The Squid log analyzer

SquidAnalyzer

Squid proxy native log analyser and reports generator with full statistics about times, hits, bytes, users, networks, top urls and top domains. Statistic reports are oriented toward user and bandwidth control; this is not a pure cache statistics generator.

SquidAnalyzer use flat files to store data and don't need any SQL, SQL Lite or Berkeley databases.

This log analyzer is incremental and should be run in a daily cron or more often on huge network trafic.

News

Multiprocess benchmark

You can find a benchmark here about performances speed improvement by using multiprocess mode.

6.0 - Sat Aug 30 21:48:14 CEST 2014

This major release adds several new features, lot of speed improvement and some major bug fixes.

  • Multiple access.log files can be processed at the same time.
  • Multiprocess mode can be activated using the -j N command line option.
  • New ExcludedMimes configuration directive to exclude from statistics a comma separated list of mime-type or using regex like text/.*.
  • New ExcludedMethods configuration directive to exclude from statistics a comma separated list of HTTP methods (GET,POST,CONNECT,...).
  • New translation available: pl_PL

Using 4 CPU cores (-j 4) to run SquidAnalyzer can divide by 4 the time used in single process mode. In single process mode building a 1.4GB access.log file takes 50 minutes on my computer, using 4 cpus take around 15 minutes.

	squid-analyzer --no-year-stat -j 4 /var/log/squid3/access.log*

Here the full list of changes:

	- Freshmeat/Freecode site is down, release announcement will be done on
	  twitter now, see https://twitter.com/SquidAnalyzer
	- Allow multiple log files to be given at command line arguments.
	- Add support to ETCDIR instead of CONFDIR during installation process,
	  Where real config files are installed on some distributions (BSD).
	- Add support to parse multiple access log file at a time in multi-
	  process mode.
	- Add documentation about multiprocess usage.
	- Add multiprocess support to SquidAnalyzer, see -j option. This can
	  improve a lot speed performances. See notes at bottom of issue #18
	  for more details. Great thanks to Francisco Rodriguez for his help.
	- Add more timing information during SquidAnalyzer execution.
	- Add some other minor speed improvement.
	- Removed call to tell,  we were spending too much time in this method
	  unnecessary.
	- Fix reports with --no-year-stat. It now reports cache stat only in year
	  and month view instead of empty page.
	- Remove intermediate build of week reports.
	- Fix Mime-Type transfer's chart title to reflect the unit used: MBytes.
	  Thanks to IMiGS for the report.
	- Little fix in a translation. Thanks to atlhon for the patch.
	- Fix case where days in calendar does not appear when DateFormat was
	  changed. Thanks to joseh-henrique for the report.
	- Update Makefile with META_MERGE and MAN3PODS informations.
	- Fix missing cleaning of pid file when early error occurs.
	- Automatically remove \r when reading configuration file.
	- Improve incremental mode by seeking directly to last position in
	  logfile and automatic detection of already parsed log.
	- Fix issue on calendar when days of a month are over 6 week. Thanks
	  to Michael Gauthier for the report.
	- Update cs_CZ language file. Thanks to Martin Kylian for the patch.
	- Fix weeks graph when a week overlaps over 2 months.
	- Add missing install of included file. Thanks to Klaus Tachtler for
	  the patch.
	- Force removing of pid file after die of the process. Thanks to Klaus
	  Tachtler for the report.
	- Fix german language de_DE.txt. Thanks to Klaus Tachtler for the patch.
	- Add ExcludedMimes configuration directive to allow exclusion from
	  statistics of a comma separated list of mime-type full name or using
	  regex like text/.*. Thanks to Ajayaks for the feature request.
	- Add ExcludedMethods configuration directive to allow exclusion from
	  statistics of a comma separated list of HTTP methods. Thanks to Ajayaks
	  for the feature request.
	- Fix error when rebuilding with old data repository that does not have
	  week view repository. Thanks to Adam Ciarcinski for the report.
	- Add pl_PL translation. Thanks to Adam Ciarcinski for the patch.
	- Fix en_US translation. Thanks to Adam Ciarcinski for the patch.

UPDATE: you must override all your installation, Perl scripts, configuration file, CSS and Javascript files. Backward compatibility with 5.x data files is preserved.

Release annoucement on twitter

Release annoucements used to be done on freshmeat.net or freecode.net, but unfortunately the site is now down. Please follow us on twitter now to receive release annoucement and latest news.

Screenshot of 5.x branch

This is a report for 2 days in January and one home user. This is just to show the interface and is not representative of the real activity in a Squid proxy.

May 04 2014 SquidAnalyzer version 5.4

This release fixes several bugs and adds more translation files (Ukrainian, Czech, Spanidh and Russian). It also allow a custom definition of the logo and title. Add new weekly statistics and reports as well as TCP DENIED report in cache statistics.

	- Change CSS to adjust calendar position. Thanks to Nathanael Bonnin
	  for the patch.
	- Add Czech language file. Thanks to Martin Kylian for the patch.
	- Add Ukrainian translation file. Thanks to Oleg A. Deordiev for the
	  patch.
	- Add CustomHeader to allow custom definition of the logo and title.
	  See squidanalyzer.conf or documentation for more explanation and
	  example. Thanks to Raktim Chatterjee.
	- Replace logrotate example that was too simplistic. Thanks to Gaetan
	  Slongo for the patch
	- Update documentation about -P option.
	- Prevents the script to run multiple instances using a pid file. This
	  patch adds a new command line option -P to change the default pid file
	  (/tmp/squid-analyzer.pid). Thanks to Gaetan Slongo for the patch.
	- Remove useless command to delete special files because those files are
	  not installed by make install. Thanks to David Walser for the report.
	- Fix spec file to build RPM. Thanks to Jonathan Gibert for the patch.
	- Fix usage to show $DEFAULT_CONFIGFILE that may not always be the
	  right place following the installation. Thanks to Jonathan Gibert for
	  the patch.
	- Remove squid requirement from RPM spec file. Thanks to Jonathan Gibert
	  for the report.
	- Add spanish language file. Thanks to BhEaN for the patch.
	- Remove ^M when reading translation files. Thanks to Filipe Teixeira
	  for the report.
	- Add new weekly reports. Thanks to Amish for the feature reports.
	- Add documentation about the new "Include" configuration directive.
	- Add build and storage of weekly statistics.
	- Fix CSS to reduce font size used in the calendar and increase size
	  of the header.
	- Add WeekDay translation string.
	- Remove link on week, the week view is not available yet.
	- Add week day and week number to the calendar.
	- Fix bug where statistics from the last parsed day was count twice
	  in months and years statisics.
	- Add file etc/included to limit entries to matching users, networks
	  or ip address. Thanks to giovanniredix for the feature request.
	- Show values of all dataset when mouse is over graph.
	- Update pt_BR.txt language file. Thanks to Joseh-Henrique for the
	  patch.
	- Update flotr2 to most recent version. Replace bars in graphs with
	  line to have mouse tracking on each dataset. Change line graphs
	  colors and reformat label shown on mouse over graph data.
	- Remove special hidden character from documentation that prevent
	  pod2man from working. Thanks to David Walser for the report.
	- Add TCP DENIED report in cache statistics. Thanks to Karl Dawkins
	  for the feature request.
	- Fix missing semi-colon after graph width definition. Add missing
	  graph on mime type.
	- Add russian translation. Thanks to Sergey Kononenko for the patch.
	- Fix german lang file about new second level domain label. Thanks
	  to Klaus Tachtler for the patch.
	- Update translation files with new Second_domain_graph_hits_title
	  and Second_domain_graph_bytes_title variables.
	- Klaus Tachtler have made some small text improvements for the
	  German language file.
	- Add ordering of pie legend following percentage. Thanks to Klaus
	  Tachtler for the report.
	- Change trackFormatter so that mouse over the pie-chart color shows
	  the domain/TLD name also with the value. Thanks to Jeetendra Poojari
	  for the patch.
	- Add advice about parsing access.log after logrotate. Thanks to James
	  Harper for the report.
	- Fix some more issue on top second level report.

Slackbuild script approved

Thanks to the great work of Oleg A. Deordiev, SquidAnalyzer can be downloaded as a slackbuild ! You can downlaod it at http://slackbuilds.org/repository/14.1/network/squidanalyzer/

Dec 01 2013 SquidAnalyzer version 5.3

This release fixes several bugs and adds two new pie charts about the most use top second level domains. It is also possible to do DNS lookup of Ip addresses inside SquidAnalyzer, see UseClientDNSName new configuration directive. This can slow down dramatically the squid-analyzer performances but you can adjust the DNS lookup timeout to prevent waiting slow DNS server, see DNSLookupTimeout new configuration directive.

        - Update and fix first and second top level domain name. Thanks to Luis
          Armando Roca Fumero for the report.
        - Add new directive DNSLookupTimeout to change the default timeout for
          DNS lookup. Add 0.0001 second timeout when SquidAnalyzer look for a DNS
          name and can't find a name server. Thanks to alex for the patch.
        - Add pie chart of top second level domains. Thanks to Jeetendra Poojari
          for the feature request.
        - Fix some HTML tag issues and table ordering on Top domain hits and Top
          url hits. Thanks to Jeetendra Poojari for the report.
        - Update INSTALL file to remove GD::Graph requirements.
        - Change underscore used to replace space in user name by the special
          string _SPC_ so that underscore will not be wrongly replaced on HTML
          output.
        - Fix pt_BR translation with charset to utf-8 and a few words with
          accentuation fix. Thanks to Joseh-Henrique for the fix.
        - Allow Ip addresses on user names to be replaced by their DNS name, this
          feature is activated by a new directive: UseClientDNSName. Thanks to
          Pfisztner Tamas for the patch.
        - Add missing description of --no-year-stat option to documentation and
          squid-analyzer usage.

UPDATE: you must override all your installation, Perl scripts, configuration file, CSS and Javascript files. Backward compatibility with data files is fully preserved. Note that if you run a Perl version < 5.8 you may need the CPAN module Time::HiRes.

Sep 01 2013 SquidAnalyzer version 5.2

This is release fixes lot of minor bugs and adds support to CIDR notation in network-alias and exclude files as well as some code rewriting and options to improve speed. New feature to show the top N users that look at an url or a domain. Adds a new configuration directive, TopUrlUser, to control the number of users to show in reports.

	- Fix pod2man error with perl 5.18 about non-ascii character. Thanks to
	  David Walser for the report.
	- Skip some unwanted code execution when --no-year-stat is enabled.
	- Fix graphs on domain statistics reports.
	- Fix badly ordered check on squid log file to parse. Thanks to wordelle
	  for the report.
	- Fix issue where squid-analyzer use default configuration file even if
	  a custom one is specified with the -c option. Thanks to Thibaud Aubert
	  for the report.
	- Add --no-year-stat to disable year statistics, reports will start from
	  month level only. This allow to save time during reports generation.
	- Allow composed top level domain statistics in Top Domain report, like
	  co.uk. Thanks to Thibaut Aubert for the feature request.
	- Add support to CIDR notation in network-alias file. Thanks to Thibaud
	  Aubert for the feature request.
	- Fix issue in Domain report where domain of top url was not found in
	  the domain report even if the download was bigger than those reported.
	  Thanks to Aubert Thibaud for the report.
	- Fix bug in last/first hour of a day data storage that mixed collected
	  data over the two days and stored false first and last visit time.
	- Fix translation for user and count strings.
	- Add -b | --build_date command line option to limit rebuilt to a given
	  date, the format of its value can be yyyy-mm-dd, yyyy-mm or yyyy. This
	  should be used to prevent rebuilding html files from all data files.
	- New feature to show the top N users that look at an url or a domain.
	  Add a new configuration directive, TopUrlUser, to control the number
	  of users to show. Thanks to Mr-Ed for the feature resquest.
	- Fix first and last visit that was displayed in random order. Thanks
	  to MangOuste for the report.
	- Allow CIDR notation in exclude file to skip line matching the given
	  xxx.xxx.xxx.xxx/n network. Thanks to the MangOuste for the patch.
	- Add WriteDelay configuration directive (default to 3600 - stores one
	  hour of statistics from log file) to be able to save resources on huge
	  log files. On small log file it is not a good idea to set a lower
	  value as SquidAnalyzer will spend his time to write and read from
	  disk. This is a first attempt to speedup statistic reports, especially
	  with year and month, but do not expect lot of performances gain.
	- Add regex checker for alias file and improve speed for network and
	  alias replacement. Thanks to jcputter for the report.
	- Fix unwanted message display in quiet mode.
	- Call localtime as little time as possible. Thanks to ammdispose for
	  the patch.
	- Remove the use of backstick to call system command in loop to gain
	  some speed. Thanks to ammdispose for the patch.
	- Revome the call to date and iconv system command each time a HTML file
	  is genrated, call it just one time if directive Locale is defined at
	  initialisation. Thanks to ammdispose for the report.
	- Fix some print statements were outputting to STDOUT instead of STDERR.
	  Thanks to ammdispose for the patch.
	- Allow peer SIBLING_HIT to be added to HIT counters. Thanks to Maher
	  Kassem forn the patch.
	- Recode initial column sorted by default to fix pending sort issues.
	- Fix bug on ordering column with sorttable. Thanks to David Walser for
	 the report.
	- Fix support to regex with coma in user and network alias file. Regex
	  like "1stFloor 192.168.1.1\d{0,2}" in user-aliases and network-aliases	  was not permitted. Thanks to Steffen Junick for the report.
	- Fix issue with log file provided at command line (option -l) and non
	  existant LogFile in squidanalyzer.conf. Thanks to Kalin Kozhuharov for
	  the report.
	- Fix test to exclude old months and days from the statistics builder.
	  Thanks to ammdispose for the patch.
	- Fix some code comment about month obsolescence. Thanks to ammdispose
	  for the patch.
	- Change code to remove obsolete statistics to more stable code with
	  less iterations. Thanks to ammdispose for the patch.
	- Fix install of configuration files. Thanks to David Wasler for the
	  patch.

Jan 30 2013 SquidAnalyzer version 5.1

This release adds more graphs, fixes several major issues and allow automatic removing of obsolete statistics. The Javascript library sorttable.js have been updated.

	- Add squid-analyzer usage information into documentation.
	- Update INSTALL file about FreeBSD issue.
	- Add -v | --version and -d | --debug command line options.
	- Add support to automatic deletion of obsolete statistics and data.
	- Add -p | --preserve option to allow automatic removing of obsolete statistic. Thanks to ammdispose for the feature request.
	- Fix documentation about OrderUrl that is limited to User detailed Urls reports, not Top Url and Top domain.
	- Set default sorttable sort in descending order first.
	- Fix default order in Mime, Network, User and User details views.  Thanks to Wesley Bresson for the report.
	- Update sorttable.js to allow sort on page load using the new function sorttable.innerSortFunction().
	- Add documentation about FreeBSD install to avoid issue with error: FreeBSD: Cannot determine short module description.
          Thanks to hackson99 for the report.
	- Fix first and last visit time in the user detail view that was not displayed at all.
	- Fix parser issue when squid logformat has additional field at end.  Thanks to nuxsmin for the patch.
	- Add pie chart on Mime type statistics.
	- Add MinPie configuration directive to set the percentage of pie graph data which under item will be stored in others label.
	- Force left alignment on largest Url column.
	- Add Locale configuration directive to set the locale when retrieving date from system. Thanks to Klaus Tachtler for the suggestion.
	- Fix missing Up link translation and some German translation. Thanks to Klaus Tachtler for the patch.
	- Fix HTML code in user and url reports. Thanks to Nathanael Bonin for the patch.
	- Fix typo in fr_FR.txt lang file. Thanks to Nathanael Bonin for the patch.
	- Replace .orig extension on configuration files by .sample. Thanks to David Walser for the report.
	- Update squidanalyzer.spec for RPM build. Thanks to David Walser for the patch.
	- Modify sorttable.js to be able to sort column with both value and percentage as numeric.
	- Remove percentage column and store the percent information directly in the column between parenthesis.
          Thanks to Nathanael Bonin for the patch.
	- Add missing class CSS style to user graphs table. Thanks to Nathanael Bonin for the report.

UPGRADE: You must reinstall all. Backward compatibility is preserved on data
files, but the HTML reports have changed, so after re-installing SquidAnalyzer
you may want to run manually squid-analyzer with the --rebuild command line
option to regenerate those files.

Dec 11 2012 SquidAnalyzer version 5.0

I'm proud to annouce the new major release of SquidAnalyzer v5.0. This release introduce a completely new design thanks to the great work of Nathanael Bonin based on CSS. I've also replaced all graphs based on libgd and GD::Graph by charts based on the Flotr2 javascript library, this mean that there's no more dependencies to install SquidAnalyzer. Below are some screenshots of the new interface.

    - Add TransfertUnit configuration directive to allow changing default Bytes transfert unit. Possible values are BYTES, KB, MB and GB.
    - Fix date encoding with call to date+iconv or fallback to strftime().
    - Fix sorttable with missing tbody tag.  Thanks to Nathanael Bonin for the patch.
    - Fix accent within report, now SquidAnalyzer will use the CharSet translation directive to set the charset following the language.
      Default to utf-8, french, german and portuges will use iso-8859-15.
    - Fix issue where per user/ip network stats was wrong and very high. Thanks to Ben Tullis for the report.
    - Fix several design issues, moved png images in a dedicated directory. Thanks to Nathanael Bonin for the patches.
    - Reorder source tree and apply tree change on Perl install.
    - Removed FooterFile that like HeaderFile is no more used.
    - Remove code for graphs generated using GD::Graph libraries, replaced by the Javascript library Flotr2.
    - Fix extra .0 in networks report.
    - Add the use %M to replace month by its 3 letters abbreviation into DateFormat configuration directive. Thanks to pk-its for the feature request.
    - Add missing german translation. Thanks to pk-its for the patch.
    - Remove space in user alias to prevent corruption of data files. Thanks to Robert Becskei for the report.
    - Fix stat_user.dat format error when username have a space in its login name. Thanks to Moise Michel for the report.
    - Force date/time used in report generation date to used 24h notation. Thanks to Klaus Tachtler for the report
    - Fix install of squidanalyzer.conf, network-aliases, user-aliases and excluded to be copied with the .orig extention so that nones of these
      file can be overriden at upgrade. Thanks to Klaus Tachtler for the report.

Aug 21 2012 SquidAnalyzer on GitHub

The development code of SquidAnalyzer is now available from GitHub.org

Aug 21 2012 v4.4

This release adds support to bzip2 compressed log file and allow exclusion of URL from the report. There's also a new German translation file.

    - Add support to bzip2 compressed log file. Feature request by Edmonds Namasenda.
    - Allow exclusion of URL from the report, the format of the exclude file has changed, see header of the excluded file.
    - Fix some RPM spec file issue. Patch by David Walser.
    - Add german translation file. Translation by Klaus Tachtler.
    - Fix translation file path into squidanalyzer.conf.

Upgrade: a full reinstallation is requiered unless you know what files to override.

Screenshot of 4.x branch

This is a report for 2 days in december and one user (me :-). This is just to show the interface and is not representative of the real activity in a Squid proxy.

Apr 27 2012 v4.3

This release fix three bugs and add sibling cache hit/miss into cache statistics.

	- Fix broken statistics when an access log file run over multiple years. Thanks to Jugurtha Massinissa for the report.
	- Prevent buildHtml to create directories on month/day without log.
	- Sort year's view per decreasing years.
	- Peer CD_SIBLING_HIT status will now be taken as local cache hit. A new
	  configuration directive (SiblingHit) to disable this feature. Default
	  is enable. Documentation and configuration file has been updated.
	  Thanks to Nathanael BONIN for the feature request.
	- Add spec file for RPM packaging. Thanks to David Walser for the patch.
	- Fix path and comment to translation file in squidanalyzer.conf. Thanks
	  to Jugurtha Massinissa for the report.

Upgrade: If you don't want to reinstall all you can just override SquidAnalyzer.pm into your perl installation.

Jan 22 2012 v4.2

This release fixes several issues related to network and user aliases. It also adds support to squidmime logformat and allow full user anonymization.

	- Fix network alias issue broken in last release, now data files will
	  store the network alias instead of the default class C network. Thank
	  to Fabrice Prigent for the report.
	- Add support to multiple regex pattern to match networks aliases. See
	  network-aliases file for more information.
	- User aliases are now directly changed into data files instead of html
	  file like before.
	- Add support to multiple regex pattern to match users aliases. See
	  users-aliases file for more inforamtion.
	- Add AnonymizeLogin configuration directives to allow full anomymization
	  of all users/ip.  Thanks to Fabrice Prigent for the request.
	- Add support to squidmime logformat, original request and reply header
	  are simply omitted. Squid and squidmime native format are still the
	  only log format supported.

UPGRADE: If you use network and/or user aliases, even if I try to preserved backward
compatibility, you may want to start with new data files as these informations are
now replaced directly into the data file instead of the HTML files. Changes only
concern file SquidAnalyzer.pm so you can just override it. There's also a new
configuration directive 'AnonymizeLogin' so you may copy/paste his definition in
default squidanalyzer.conf file.

Jan 18 2012 v4.1

This release fix some issues encountered during the install process and fixes a line format error detected on some data file.

	- Fix data file parser issue with largest_file_size or largest_file_url are empty.
	  Thanks to David Walser for the report.
	- Add line number when reporting bad format into a data file.
	- Add missing DESTDIR variable into the allowed commande line arguments. Thanks to
	  David Walser for the report.
	- Fix error when header file are missing as they are no more used now. Thanks to
	  David Walser for the report.
	- Remove salogo.png from install script, it is now replaced by file logo-squidanalyzer.png.
	  Thanks to David Walser for the report.
	- Install of README, INSTALL and Changelog is now disabled by default, you must set DOCDIR
	  parameter to install those files in the desired directory. Thanks to David Walser for
	  the request.
	
UPGRADE: Only SquidAnalyzer.pm have changed so you can just override this file. Other are related to the installation part.

Dec 28 2011 v4.0

The long pending new version of SquidAnalyzer has been released today after 4 years. Here are the complete changelog:

        - Fix regex search failure on fake domain. Thanks to Michel Reischl for the report.
        - Add support to squid log with client dnsname instead of ip address. Thanks to Matthew Jurgens for the patch.
        - Add options --logfile (-l) and --configfile (-c) to squid-analyzer command line argument. Thanks to Matthew Jurgens for the patch.
        - Add the first and last time an url is viewed by a user on day and hour views. Thanks to Franco Mazz for the feature request.
        - Allow column sorting using the Javascript sorttable library available at http://www.kryogenix.org/code/browser/sorttable/.
	  Thanks to Matthew Jurgens for the patch.
        - Renamed file Changes as ChangeLog.
        - Makefile.PL is now able to do a full install by creating directories and moving files.
        - Allow parsing of compressed log file. SquidAnalyzer will automaticaly use zcat when the logfile extention ends with .gz
        - The cost column now is not shown when the cost price is 0.
        - Change CSS style of links.
        - Complete rewrite of report menu and calendar sot that they appears on all HTML page.
	- Change License from Artistic to GPL v3.
	- Remove HeaderFile, replaced by squidanalyzer.css file.
	- Add --rebuild option to allow rebuild of all html and graph output from data files. This should be done after upgrading.
	- Add ..._REFRESH_UNMODIFIED into cache HIT statistics.

UPGRADE: Even if backward compatibility is preserved on data files, the HTML and
graph have changed, so after re-installing SquidAnalyzer you may want to run
squid-analyzer with the --rebuild command line option to regenerate those files.
So run the following commands:

        tar xzf SquidAnalyzer-4.0.tar.gz
        cd SquidAnalyzer-4.0/
        perl Makefile.PL HTMLDIR=/var/www/squidanalyzer
        make && make install
        /usr/local/bin/squid-analyzer --rebuild

should the normal procedure to upgrade. Change the path to the HTML output if required.

Dec 27 2011 Web site

SquidAnalyzer has now a dedicated web site!

Help support
SquidAnalyzer!

 

Copyright (c) 2001-2012 Gilles Darold - All rights reserved. (GPL v3).