AXS records visits to your web page and processes those records into meaningful graphs and database listings. The scripts tell you where visitors are coming from, charts their flow through your site, and tells you which links they follow when leaving. In addition, it records their server name, IP address, type of web browser, and time of visit.
View Demo - this demo logs visits to our mirror of the Perl 5 Documentation. To see records of your own visits, tour that directory and then demo AXS. These scripts usually require two passwords to access - our public demo doesn't use them. The script you see in the demo is a beta version which can only be downloaded from the FTP site.
Download the Code and Instructions (axs.zip)
To request help or discuss this script, please visit the local script forum.
If you're experiencing problems, please make sure you're using the latest, released May 2, 2000.
If you like AXS, please Rate it at CGI Resources.com.
It'll keep you from getting bored with your own web page
Apathy is the webmaster's worst enemy. AXS provides the essential dynamic content that keeps you coming back. Which of my links are the most popular? What search terms do people type in to get to my page? Are people from fbi.gov scanning my site? Now you'll know, with more information arriving every minute.Because it's worthy
The AXS Script Set is the most complete site tracking system available (as of this writing, to the authors' knowledge of course). You have control of eighteen graphing options and five filtering methods. But AXS doesn't overwhelm you with pure information - short browser names, abbreviated referrers, and by-file filtering can turn a one megabyte log file into exactly the page-full of information you were looking for.Do it because it's free
We at Fluid Dynamics have better things to do than charge you for code. Take advantage of this! Future scripts-in-the-works will not have this valuable feature... ;)
To run AXS you must have access to the Internet and a web page (either a personal web page or your own server, it doesn't matter). Version 2.3 will run equally well on Unix/Linux and Windows NT servers so long as they are equipped with Perl 4 or higher. Of course, your provider must allow custom CGI scripts and it helps if they allow server side includes. To view the graphs you should have a table-capable browser which supports scaled images (Netscape 1.2 or higher).If you don't have such a web page, they are available for a few dollars per month from hundreds of companies. A couple favorites are Illumanti Online and Pair Networks.
Installation of this script will take an hour or two (including the fifteen minutes needed to read this section and the trouble-shooting tips thoroughly). Script configuration takes about fifteen minutes and website configuration takes about thirty. Trouble-shooting & testing account for the rest.
- Downloading the Files
- Files and Permissions
- Configuring the Logging Module
- Configuring the Admin Module
- Configuring your Web Page
Downloading the Files
Either download the axs.zip file with everything, or download 'em one by one:If you download scripts as straight text files, change the file extension from .txt to either .cgi or .pl. Treat script files as straight ASCII text for downloading and editing.
The two scripts, ax.cgi and ax-admin.cgi, should be readable and executable by all. On Unix systems, these permissions are set with the following commands:chmod 755 ax.cgi chmod 755 ax-admin.cgiThe two data files, log.txt and axs.dat, should be readable and writable by all. On Unix systems, these permissions are set with the following commands:
chmod 777 axs.dat chmod 777 log.txtThe images, red.gif and tracker.jpg, should be readable by all. Read permissions are the default, so no special commands are needed.
On our Windows NT account, we had to request a permissions change from the system administrator. You may have to do the same.
Configuring the Logging Module
Open ax.cgi with a text editor. The first variable to set is the location of the access log, log.txt, on your system. We highly recommend that you place the access log file in the same directory as the CGI scripts. If you do, this variable should be left exactly as is. However, if you put the access log file somewhere else, enter the full path to it, like so:On Windows NT systems, enter the full path from the drive - we've found that virtual directories don't work so well. You might also have to use single quotes rather than double quotes on NT.$logfile = "/u2/home/xav/log.txt";
Next, enter the domain of your web page. Do not include a trailing slash:Even if you web page isn't at the top domain, this variable should still hold the domain URL only. For example, if your web page is http://io.com/~zoltan, you'd set $domain = "http://io.com";.$domain = "http://www.xav.com";
Now we specify the Content-type header that will be used for SSI calls. 99% of you will leave this as is. Some systems don't like Content-type in SSI, though, so you folks would have $header="";.If you scroll down through the script, there is a section that lets you ignore some domains. We usually ignore ourselves to keep the logs honest. If you don't want to ignore people, just delete that whole section.$header = "Content-type: text/html\n\n";
Configuring the Admin Module
Open ax-admin.cgi with a text editor. There are several variables. The ones toward the top must be changed. The ones toward the bottom can probably be left as is.
The first variable is $http_addr. This is a string which occurs in the URL of your web site. This is used for pattern matching - AXS will decide whether a particular URL is on your website or someone else's based on whether it includes this variable. If your site is referred to as both "http://me.com" and "http://www.me.com", you can just enter the common letters. This is the case for our web page, so we set $http_addr = "xav.com";.Next you need a password for the script. Be creative. You can turn off password locking by setting it to nothing, i.e. $password="";.$http_addr = "xav.com";There is never a username for this script. When you go to use the script, you will be asked for a username, but don't give one. It is a trick and if you give one the script will fail. The password is case sensitive.$password = "IronMAN";
The next set of variables form a hyperlink that appears at the top of the output graphs. This provides you with a quick link back to your main page. Setting these variables isn't required.Now we need the URL to the two images. You can leave these as is if you'd like.$link_url = "http://www.xav.com/scripts/"; $link_title = "Scripts Page";By default, this script has a debug option. Instructions on how to use it are in the Trouble Shooting section. The debugger prints out critical system settings. Once AXS is working, set $Debug = "OFF"; so that hackers cannot acquire this information for their subversive purposes.$red = "http://www.xav.com/images/red.gif"; $tracker = "http://www.xav.com/images/tracker.jpg";Next, we specify the location of the data files. If you leave these data files in the same directory as your scripts, you can leave this part as is. If you put the data files somewhere else, or if you use Windows NT, you must specify the full path to the files.$Debug = "ON";In any case, do not use virtual paths on NT and do not enter a URL to the data files. These things won't work.# Default (good): $logfile = "log.txt"; $prefs = "axs.dat"; # Example of full path for Unix systems: $logfile = "/u2/home/xav/log.txt"; $prefs = "/u2/home/xav/axs.dat"; # Example of full path for Windows NT: $logfile = "D:\users\www\bob\data\log.txt"; $prefs = "D:\users\www\bob\data\axs.dat";
This script calls itself, and so it needs to know its own URL. 99% of the time, this information is stored in the $ENV{'SCRIPT_NAME'} variable. When this is the case, the $cgi_url variable doesn't need to be changed. However, if you have errors getting past the first page of output, you'll need to replace this with the full URL to the ax-admin.cgi script.Now we type out some URLs to our favorite lookup services. You probably won't need to change this.$cgi_url = "$ENV{'SCRIPT_NAME'}";This script is about 50% straight HTML, so even if you don't know Perl you can still edit quite a bit. Please respect our Copyright Policy.$nslookup = "http://www.xav.com/cgi-bin/nslookup.cgi"; $whois = "http://rs.internic.net/cgi-bin/whois";
The ax-admin.cgi script should now be ready to run, but before we can count any hits we'll have to configure the web page itself.
Configuring Your Web Page
The preferred tracking method is with server side includes. To use this method, place the following SSI call in all of your HTML pages. If the logging script is in a directory other than cgi-bin, substitute directory names for the following examples:<!--#exec cgi="/cgi-bin/ax.cgi" -->For this to work, your web server must support this type of command. Most do. This command tells the web server to execute ax.cgi before sending the document to the visitor. If they view the source code from their browser, they will just see a blank space where the SSI call used to be.
Note that some webservers forbid the "exec cgi" tag as a matter of policy. If you're unable to get "exec cgi" to work, but you've been assured that SSI is allowed, try this alternate syntax:
<!--#include virtual="/cgi-bin/ax.cgi" -->
Sometimes you have to rename your .html documents to .shtml or .stm in order for server side includes to function. Other times you have to place a custom .htaccess file in your home directory with a special AddType directive. Other times you have to harass your provider into enabling SSI.
If your server doesn't allow SSI, or you have trouble getting it to work, then we use the transparent redirect method. On every document, place the following image tag:where "whatever" is any string of letters or numbers. This produces a tiny one pixel by one pixel transparent gif. It won't really show up and it will let you log everyone without SSI. The image redirects won't let you see where people are coming from, though, and they only work if the visitor has his graphics cranked up.<IMG SRC="/cgi-bin/ax.cgi?whatever.gif">
The Microsoft Internet Explorer web browser has a nasty habit of placing script-generated images into the visitor's cache. When this happens, the first visit is logged, but when MSIE gets to another page with the embedded image tag, it prefers to pull the image out of its own cache instead of off your web server. Because of this, the ax.cgi script won't be executed again, and the visit to the second (or third or fourth) documents won't be recorded.
You can trick MSIE into downloading a new image each time by making the query string random. On your main document you'll have SRC="ax.cgi?one.gif" and on another document you could have SRC="ax.cgi?xxx22.gif". MSIE will treats these as separate images and will initiate a clean download.
Whichever method is used to log visitors to internal pages, all links to off-site web pages should be re-written as redirects:The above link takes the visitor to ax.cgi without them noticing anything, and then the logging script redirects the visitor to the intended web page. This allows you to track which links are used as people leave your site.<A HREF="/cgi-bin/ax.cgi?http://www.yahoo.com/">Yahoo!</A>
If you have documents on your page which aren't HTML and therefore don't contain the SSI call, you should link to those as a redirect too:In theory you can also link to embedded objects (sounds, images, class files, and so on) as redirects:<A HREF="/cgi-bin/ax.cgi?http://xav.com/readme.txt">Read ME</A>Do not get carried away. It is best to avoid writing to the access log more than once each time a page is hit. If each page download writes to the log twice, then there is a high chance that the data will get corrupted and the whole log will be lost.<IMG SRC="/cgi-bin/ax.cgi?http://www.xav.com/pict.gif">
Similarly, if you have a frames page, include the SSI call or image redirect on only one of the documents.
If this sounds confusing, don't worry - as long as you are consistent, you shouldn't get into any trouble. Use SSI on every HTML document you have, and redirect on every non-HTML or offsite link.
As this script shows, it is possible to log the URL that a visitor is at before he arrives at your page. The AX-ADMIN script creates a list of such URL's for you. The list is hyperlinked, so as you scan which web pages are referencing yours, and you may click on the links from your admin screen and jump right to the site in question. The list of referring URL's is convenient, valuable information, but it is also a privacy issue. Some people don't want you to know which URL they were at before, or the URL might contain sensitive information and so it's location is a secret.The web address of your admin module might be considered one such URL with sensitive information, whose location should remain secret. But if you follow one of the links that this script creates, your previous URL (the URL of your admin module) can then be recorded by a webmaster at another site, who can then backtrack it and view your access log at will. It's kind of a case of HTTP_REFERER coming full circle.
Therefore, it is in some cases advisable not to follow the links from the admin page. Look but don't click. You can always copy the URL and then safely enter it in the address window. Of course, there is little danger in back-tracking search engine hits to see where you fall in the hierarchy, or in back-tracking links to your friends' sites. This danger is one of the reasons we added the script-based password protection - be sure to use it. As a general precaution you may wish to also place AX-ADMIN within a directory that has htaccess protection. This will require that those using the directory give a valid username and password - anyone backtracking to your admin module will be greeted with a login screen instead of your access stats.
To set up .htaccess protection, create the following text file named .htaccess and place it in the same directory as your ax-admin.cgi script (you might want to create a separate sub-directory specifically for this purpose):
where /home/adam is your personal login directory, and adam is replaced with the username that you would like to use. The file we just created will tell the Apache webserver (and variants thereof) to authenticate (ask for username and password) before a visitor can use the directory in which it resides. It says that the visitor must have username adam, and that his password must agree with the one in the .htpasswd file. So our next step is to create this password file.AuthType Basic AuthName Access Log Manager AuthGroupFile /dev/null AuthUserFile /home/adam/.htpasswd <LIMIT GET POST PUT> require user adam </LIMIT>At the unix prompt, type:
where /home/adam is your personal login directory, and adam is replaced with the username that you would like to use. Your server will ask you for twice for a password. If you want to change it later, you can run the same command later without the "-c" attribute.htpasswd -c /home/adam/.htpasswd adamWe may have to make the .htaccess file readable and executable by the webserver:
Setting up .htaccess authentication is sometimes difficult, and not all unix machines support both the .htaccess protection and the htpasswd program. Windows NT doesn't support it at all. If your provider does not support these things, try a different one. ;)chmod 555 .htaccessWe hope that this section does not make your overly paranoid - we just want to let you know what some of the dangers are, and to give some suggestions on how to decrease the risk. Some people might wish to let everyone view the access log (that's what we do around here).
Trouble shooting is divided into three types. The first type is when the script doesn't execute at all, and you get some kind of server error. The second type is when the script executes and shows you the menu of options, but there are problems getting it to perform as desired or problems getting the access log to display hits. The final type involves SSI-specific problems.
Server Errors, Access Forbidden Warnings, and the Like
The script is run by going to the URL of the ax-admin.cgi file - there is no HTML page to anchor it. When you go to this URL, you may get a "malformed header" or "premature end of script headers" message. If so, it may be because the script was transferred as a binary file at some point. Always transfer scripts in ASCII format.
Another possibility involves the location of the Perl executable on your system. Most servers have it at /usr/bin/perl, and that is how these scripts are set up. You may have to change the first lines to #!/usr/local/bin/perl if that's where they've put the Perl compiler, or you may have to type whereis perl at the command line to find where they have hidden it.
If neither an ASCII transfer nor a "whereis perl" fix a Malformed Headers error, you will have to run the script under a Perl debugger. This is done by entering telnet and typing perl -w ax-admin.cgi at the command line. When you do this, the script will kick out either the line with the error on it, or it will execute just fine. If it executes just fine, check the permissions on your data files, double check the location of the Perl compiler, and verify that a Content-Type header or Location header is returned in the HTML output. There will also be a lot of "Use of uninitialized value" errors, which don't matter.
Since some providers allow FTP but not telnet access, you may have to request a debug from your provider. If this step is required, note that tech support people respond better to humble pleas than to threats against their lives or allegations that their Perl compiler is broken. The author works in tech support and knows this to be true.
Finally, if you get an "Access Forbidden" or similar message, either the permissions are not set to 755 or there is an .htaccess file somewhere that is preventing your from running scripts. Some installations of the Front Page extensions for Unix involve such a restriction. If this is the case, you will have to remove the type POST from the line containing <Limit POST PUT DELETE> in your main .htaccess file. You may also use the fix below, which allows you to call AXS with the GET method.
You may also have to rename your scripts from .pl to .cgi.
501 Method Not Supported (or something like that) - this message comes up sometimes when the POST method is not allowed. We've seen several Windows NT servers where this is the case. Typical symptoms are that you can get to the authentication page, but get an error when logging in, or you can get to the first screen with the password turned off, but cannot retrieve any graphical or database data. A fix for this has been created. The fix allows you to use either the GET or POST method. The file above will be used to replace your current ax-admin.cgi file.
See also the MSA CGI FAQs by Matt Wright.
Partial Success Debugging
Partial success is when you can go to the URL of the ax-admin.cgi file and have it show a list of options or a user authentication screen. This means that the script is executing okay in Perl and will soon be working.
If you repeatedly get a "Cannot find Preferences" or "Cannot find Log File" error, check the path to make sure your data files are present. You may have to rewrite the variables to show the full system path to where these data files are located.
If you cannot get past the password screen, try setting the password to blank and worry about it later. If you can get past the password screen once but always get kicked back to it when trying to make graphs, then your axs.dat file is not writable. You will have to reset the read/write/execute permissions for it.
If you can get the options screen to appear, and can get past the password screen, then the only error remaining is to get the access log to grow as more people visit your site. Try clearing your browser cache and visiting all your pages. Then go back to ax-admin.cgi and verify that the hits have been recorded.
If they have not, try going to the URL of the ax.cgi script. This should kick out a blank screen, or maybe a "Document contains no data" error. If you get a script error or forbidden message, run the trouble shooting tips above. Once ax.cgi works, go back to the admin module and see whether the log is growing.
If it is still not growing, there are three possible reasons. The first reason is that the $logfile variable in ax.cgi is incorrect. Verify that it agrees with the $logfile variable in ax-admin.cgi. The second possibility is that the log file is not writable. Reset the permissions to 777. The third and final possibility is that your server might be listed in the @ignore array in the ax.cgi. By default, military and government servers are in this array. If this is the case, remove your server from the array and try again.
Logging works, but prints broken image to the screen - this error comes up when using the transparent image logging. The default ax.cgi script in v2.01 included the binary code for a transparent image, but this code didn't always work. A fix for this has been created. This fix uses a real GIF image that will not appear broken.
All pages are logged twice - this had come up with the transparent gif logging method. The above fix should cure it. If you are using the SSI method of logging, you might want to check that you only have one SSI tag per page. Even if you have only one, you may still want to use the above fix.
Hits to Local Documents contains duplicates due to different hostnames - this situation comes up when your website can be hit by different hostnames, such as "http://xav.com" and "http://www.xav.com". When viewing hits to local documents, there will be separate entries for "http://xav.com/index.html" and "http://www.xav.com/index.html" - though it's the same file. The solution is to use the %Maps hash in ax.pl. Open the file and scroll down to the mappings. Add an entry that replaces the inconsistent hostname with the one you want to standardize on. For example, in our example, an entry mapping "http://xav.com" to "http://www.xav.com" would correct the problem. Note that mappings are done in a case insensitive, substring fashion. As many mappings as desired can be added. [Feature added in AXS 2.3.0.0008 ]
Logging hits to CGI scripts - this isn't a bug, it's a feature. Sometimes you want to log visits to a dynamic, CGI-generated web page. You can log these with the transparent image method, or with the CGI logging fix. This fix allows you to include a logging procedure inside of any other Perl script.
See also the CGI Documentation.
When Server Side Includes Won't Work
There are two types of server side includes errors. The trivial ones are those where the SSI call doesn't even try to execute, and is simply passed as part of the HTML source code. If this happens, you will have to enable SSI by either changing your documents to file.shtml, or talking to your provider. You may also have to make your web files executable - try chmod 755 *.*htm*.
If you get a 500 error message or a "an error occurred while processing this directive message", it usually means that the path to the CGI script is wrong. Try using a straight relative path. For example, if the .shtml document and the ax.cgi script are in the same directory, use:If the .shtml document is in your main directory, and ax.cgi is in the cgi-bin subdirectory, use:<!--#exec cgi="ax.cgi" --><!--#exec cgi="cgi-bin/ax.cgi" -->See also the SSI Documentation.
"Hits From Other Sites" shows local sites as well
In ax-admin.pl, set the $My_Web_Address variable to your domain name, rather than $ENV{'HTTP_HOST'}."Hits Followed From This Site" shows links followed internally
In ax-admin.pl, set the $My_Web_Address variable to your domain name, rather than $ENV{'HTTP_HOST'}.
If you have a problem and this FAQ does not help, write to axs@nickname.net and we will try to assist. Sometimes mail takes a couple months to be read and replied to, and sometimes mail is forgotten about or accidently deleted. If it's been three or four months and you haven't gotten a response, send your letter again.
If you really need to get this script working, we'll happily install it for you for $80 US, assuming a small to medium size site with less than a hundred documents. Larger sites may run a little more.
Note that this script can work for documents on your web page and elsewhere on the net. You can post an image to USENET as:and then view the statistics on people who view the picture from their browser-based newsreaders. The people being logged will never see your site, they may never know what happened, but you'll have their access stats.<IMG SRC="http://www.xav.com/cgi-bin/ax.cgi?http://www.xav.com/pict.gif">This script could also be set up on a CGI-enhanced provider and used to log accesses on a different, non-CGI-enhanced web page (though only the redirect method will work in these offsite cases, and the full URL to the redirect script will have to be written out as in the above example).
If you go to a guestbook on your friend's page and enter your URL as:
you will be able to log each time that link is followed. The same works for mailto links:http://www.provider.com/cgi-bin/ax.cgi?http://www.provider.comand just about any other URL-based expression. You can always add a redirect to a URL, and you can chain your redirects as many times as you'd like (though, again, this may cause data corruption if you set up your redirect calls back-to-back).http://www.provider.com/cgi-bin/ax.cgi?mailto:me@provider.com
These scripts are considered strict freeware - they are distributed free, and cannot be resold in any way. You cannot charge for hosting or installing this script. If you typically charge a residual fee to your clients for hosting and maintaining their pages, you must communicate to them that the AXS script is not included in any fee and is hosted/maintained/installed free. If you need help installing the script and cannot find free help, or would like somebody to host the script and cannot find free script hosting, we will gladly assist for a reasonable fee (currently about $50 per hour for installation, with a typical install taking two hours).
Distribution of all scripts requires consent of Fluid Dynamics. Several people have written with proposed multi-lingual versions of this script set, or with various improvements. Distribution of such versions is encouraged but requires consent so that bug fixes and updated versions can be consistently made available to all users.
You are free to modify the script files so long as you leave the first ten lines (the copyright information) intact and leave lower three lines of HTML output intact on each output page of the ax-admin.cgi script (specifically, the link to us and the copyright information). Considerable work has gone into these scripts and they are distributed to build name recognition (among other things). They are not distributed so that others can make money off of them and not out of goodwill nor love of humanity, all which are highly over rated.
To receive notification via email when an updated version is released, send an email message to axs@nickname.net to that effect. To help drive new updates, please send any suggestions and bug reports to the same address.
The following people have helped out a lot with this script by reporting bugs and making suggestions. If you report a bug or make a suggestion your name will be placed here too. Sometimes we forget, so if you have already reported a bug or made a suggestion and would like your name here, let us know.
Nick of CyberNet (offline) risked his site during the alpha & beta testing. Thanks, Nick. Jeff Carnahan of Terminal Productions (offline?) provided the new nslookup module. Worth layout tips came from Jason Moore of Hypenet, Simon of Byte-Sized, Richard of AB Lagerling Bostadsrätter and Leonardo Piccioli.
Amjad Alhait (offline) of Novell (big names here, woo-hoo!) pointed out the first error in version 2.0, a literal @ sign that has been fixed.
Cindy Chinn of The Art Department (offline) constructed the "Tracked by AXS" graphics based on the larger tracker.jpg that Jeff Verhagen put together for us. The red.gif image is on loan from Darryl C. Burgdorf of Affordable Web Space Design. Darryl also contributed the "Abbreviated Browser Names" and "Operating System" modules.
Donnie Lee of Nexus Federation (offline) loaned us an NCSA server for testing. NW Nexus loaned us a Windows NT server for testing.
Thanks to TJK of NY Politics (offline) for pointing out the one-day offset error in date filtering. TJK also pointed out recent errors in the date filtering for database output and "Average Hits" module. Joe of Rainy Day Web Design pointed out the graphing errors for small hit items with large database sizes.Erik Sivertsen discovered and fixed the $MaxLogSize bug in version 2.3.0.0005.
Dieter discovered and fixed the "No referring URL's" problem in ver 2.3.0.0005, the "Hits to local docs" bug in ver 2.3.0.0006, the REQUEST_URI SSI logging bug in 2.3.0.0009, and countless other fixes.
Thone fixed the leap year problem with the "Hits by Day of the Year" graph.
Marty Irvine pointed out the problem with binary data in the log file, leading to the fixes of version 2.3.0.0010.
| [ Contact | Search | Main Page | CGI Scripts | AXS 2.3.0.0010 ] | © 1997-2000 by Fluid Dynamics |
