The Squid log analyzer

SquidAnalyzer

Squid proxy native log analyser and reports generator with full statistics about times, hits, bytes, users, networks, top urls and top domains. Statistic reports are oriented toward user and bandwidth control; this is not a pure cache statistics generator.

SquidAnalyzer use flat files to store data and don't need any SQL, SQL Lite or Berkeley databases.

This log analyzer is incremental and should be run in a daily cron or more often on huge network trafic.

News

Jan 30 2013 SquidAnalyzer version 5.1

This release adds more graphs, fixes several major issues and allow automatic removing of obsolete statistics. The Javascript library sorttable.js have been updated.

	- Add squid-analyzer usage information into documentation.
	- Update INSTALL file about FreeBSD issue.
	- Add -v | --version and -d | --debug command line options.
	- Add support to automatic deletion of obsolete statistics and data.
	- Add -p | --preserve option to allow automatic removing of obsolete statistic. Thanks to ammdispose for the feature request.
	- Fix documentation about OrderUrl that is limited to User detailed Urls reports, not Top Url and Top domain.
	- Set default sorttable sort in descending order first.
	- Fix default order in Mime, Network, User and User details views.  Thanks to Wesley Bresson for the report.
	- Update sorttable.js to allow sort on page load using the new function sorttable.innerSortFunction().
	- Add documentation about FreeBSD install to avoid issue with error: FreeBSD: Cannot determine short module description.
          Thanks to hackson99 for the report.
	- Fix first and last visit time in the user detail view that was not displayed at all.
	- Fix parser issue when squid logformat has additional field at end.  Thanks to nuxsmin for the patch.
	- Add pie chart on Mime type statistics.
	- Add MinPie configuration directive to set the percentage of pie graph data which under item will be stored in others label.
	- Force left alignment on largest Url column.
	- Add Locale configuration directive to set the locale when retrieving date from system. Thanks to Klaus Tachtler for the suggestion.
	- Fix missing Up link translation and some German translation. Thanks to Klaus Tachtler for the patch.
	- Fix HTML code in user and url reports. Thanks to Nathanael Bonin for the patch.
	- Fix typo in fr_FR.txt lang file. Thanks to Nathanael Bonin for the patch.
	- Replace .orig extension on configuration files by .sample. Thanks to David Walser for the report.
	- Update squidanalyzer.spec for RPM build. Thanks to David Walser for the patch.
	- Modify sorttable.js to be able to sort column with both value and percentage as numeric.
	- Remove percentage column and store the percent information directly in the column between parenthesis.
          Thanks to Nathanael Bonin for the patch.
	- Add missing class CSS style to user graphs table. Thanks to Nathanael Bonin for the report.

UPGRADE: You must reinstall all. Backward compatibility is preserved on data
files, but the HTML reports have changed, so after re-installing SquidAnalyzer
you may want to run manually squid-analyzer with the --rebuild command line
option to regenerate those files.

Screenshot of 5.x branch

This is a report for 2 days in January and one home user (me :-). This is just to show the interface and is not representative of the real activity in a Squid proxy.

Dec 11 2012 SquidAnalyzer version 5.0

I'm proud to annouce the new major release of SquidAnalyzer v5.0. This release introduce a completely new design thanks to the great work of Nathanael Bonin based on CSS. I've also replaced all graphs based on libgd and GD::Graph by charts based on the Flotr2 javascript library, this mean that there's no more dependencies to install SquidAnalyzer. Below are some screenshots of the new interface.

    - Add TransfertUnit configuration directive to allow changing default Bytes transfert unit. Possible values are BYTES, KB, MB and GB.
    - Fix date encoding with call to date+iconv or fallback to strftime().
    - Fix sorttable with missing tbody tag.  Thanks to Nathanael Bonin for the patch.
    - Fix accent within report, now SquidAnalyzer will use the CharSet translation directive to set the charset following the language.
      Default to utf-8, french, german and portuges will use iso-8859-15.
    - Fix issue where per user/ip network stats was wrong and very high. Thanks to Ben Tullis for the report.
    - Fix several design issues, moved png images in a dedicated directory. Thanks to Nathanael Bonin for the patches.
    - Reorder source tree and apply tree change on Perl install.
    - Removed FooterFile that like HeaderFile is no more used.
    - Remove code for graphs generated using GD::Graph libraries, replaced by the Javascript library Flotr2.
    - Fix extra .0 in networks report.
    - Add the use %M to replace month by its 3 letters abbreviation into DateFormat configuration directive. Thanks to pk-its for the feature request.
    - Add missing german translation. Thanks to pk-its for the patch.
    - Remove space in user alias to prevent corruption of data files. Thanks to Robert Becskei for the report.
    - Fix stat_user.dat format error when username have a space in its login name. Thanks to Moise Michel for the report.
    - Force date/time used in report generation date to used 24h notation. Thanks to Klaus Tachtler for the report
    - Fix install of squidanalyzer.conf, network-aliases, user-aliases and excluded to be copied with the .orig extention so that nones of these
      file can be overriden at upgrade. Thanks to Klaus Tachtler for the report.

Aug 21 2012 SquidAnalyzer on GitHub

The development code of SquidAnalyzer is now available from GitHub.org

Aug 21 2012 v4.4

This release adds support to bzip2 compressed log file and allow exclusion of URL from the report. There's also a new German translation file.

    - Add support to bzip2 compressed log file. Feature request by Edmonds Namasenda.
    - Allow exclusion of URL from the report, the format of the exclude file has changed, see header of the excluded file.
    - Fix some RPM spec file issue. Patch by David Walser.
    - Add german translation file. Translation by Klaus Tachtler.
    - Fix translation file path into squidanalyzer.conf.

Upgrade: a full reinstallation is requiered unless you know what files to override.

Screenshot of 4.x branch

This is a report for 2 days in december and one user (me :-). This is just to show the interface and is not representative of the real activity in a Squid proxy.

Apr 27 2012 v4.3

This release fix three bugs and add sibling cache hit/miss into cache statistics.

	- Fix broken statistics when an access log file run over multiple years. Thanks to Jugurtha Massinissa for the report.
	- Prevent buildHtml to create directories on month/day without log.
	- Sort year's view per decreasing years.
	- Peer CD_SIBLING_HIT status will now be taken as local cache hit. A new
	  configuration directive (SiblingHit) to disable this feature. Default
	  is enable. Documentation and configuration file has been updated.
	  Thanks to Nathanael BONIN for the feature request.
	- Add spec file for RPM packaging. Thanks to David Walser for the patch.
	- Fix path and comment to translation file in squidanalyzer.conf. Thanks
	  to Jugurtha Massinissa for the report.

Upgrade: If you don't want to reinstall all you can just override SquidAnalyzer.pm into your perl installation.

Jan 22 2012 v4.2

This release fixes several issues related to network and user aliases. It also adds support to squidmime logformat and allow full user anonymization.

	- Fix network alias issue broken in last release, now data files will
	  store the network alias instead of the default class C network. Thank
	  to Fabrice Prigent for the report.
	- Add support to multiple regex pattern to match networks aliases. See
	  network-aliases file for more information.
	- User aliases are now directly changed into data files instead of html
	  file like before.
	- Add support to multiple regex pattern to match users aliases. See
	  users-aliases file for more inforamtion.
	- Add AnonymizeLogin configuration directives to allow full anomymization
	  of all users/ip.  Thanks to Fabrice Prigent for the request.
	- Add support to squidmime logformat, original request and reply header
	  are simply omitted. Squid and squidmime native format are still the
	  only log format supported.

UPGRADE: If you use network and/or user aliases, even if I try to preserved backward
compatibility, you may want to start with new data files as these informations are
now replaced directly into the data file instead of the HTML files. Changes only
concern file SquidAnalyzer.pm so you can just override it. There's also a new
configuration directive 'AnonymizeLogin' so you may copy/paste his definition in
default squidanalyzer.conf file.

Jan 18 2012 v4.1

This release fix some issues encountered during the install process and fixes a line format error detected on some data file.

	- Fix data file parser issue with largest_file_size or largest_file_url are empty.
	  Thanks to David Walser for the report.
	- Add line number when reporting bad format into a data file.
	- Add missing DESTDIR variable into the allowed commande line arguments. Thanks to
	  David Walser for the report.
	- Fix error when header file are missing as they are no more used now. Thanks to
	  David Walser for the report.
	- Remove salogo.png from install script, it is now replaced by file logo-squidanalyzer.png.
	  Thanks to David Walser for the report.
	- Install of README, INSTALL and Changelog is now disabled by default, you must set DOCDIR
	  parameter to install those files in the desired directory. Thanks to David Walser for
	  the request.
	
UPGRADE: Only SquidAnalyzer.pm have changed so you can just override this file. Other are related to the installation part.

Dec 28 2011 v4.0

The long pending new version of SquidAnalyzer has been released today after 4 years. Here are the complete changelog:

        - Fix regex search failure on fake domain. Thanks to Michel Reischl for the report.
        - Add support to squid log with client dnsname instead of ip address. Thanks to Matthew Jurgens for the patch.
        - Add options --logfile (-l) and --configfile (-c) to squid-analyzer command line argument. Thanks to Matthew Jurgens for the patch.
        - Add the first and last time an url is viewed by a user on day and hour views. Thanks to Franco Mazz for the feature request.
        - Allow column sorting using the Javascript sorttable library available at http://www.kryogenix.org/code/browser/sorttable/.
	  Thanks to Matthew Jurgens for the patch.
        - Renamed file Changes as ChangeLog.
        - Makefile.PL is now able to do a full install by creating directories and moving files.
        - Allow parsing of compressed log file. SquidAnalyzer will automaticaly use zcat when the logfile extention ends with .gz
        - The cost column now is not shown when the cost price is 0.
        - Change CSS style of links.
        - Complete rewrite of report menu and calendar sot that they appears on all HTML page.
	- Change License from Artistic to GPL v3.
	- Remove HeaderFile, replaced by squidanalyzer.css file.
	- Add --rebuild option to allow rebuild of all html and graph output from data files. This should be done after upgrading.
	- Add ..._REFRESH_UNMODIFIED into cache HIT statistics.

UPGRADE: Even if backward compatibility is preserved on data files, the HTML and
graph have changed, so after re-installing SquidAnalyzer you may want to run
squid-analyzer with the --rebuild command line option to regenerate those files.
So run the following commands:

        tar xzf SquidAnalyzer-4.0.tar.gz
        cd SquidAnalyzer-4.0/
        perl Makefile.PL HTMLDIR=/var/www/squidanalyzer
        make && make install
        /usr/local/bin/squid-analyzer --rebuild

should the normal procedure to upgrade. Change the path to the HTML output if required.

Dec 27 2011 Web site

SquidAnalyzer has now a dedicated web site!

Help support
SquidAnalyzer!

 

Copyright (c) 2001-2012 Gilles Darold - All rights reserved. (GPL v3).