wiki:Software/Productionalisation/ImportingOldLogs
Last modified 7 years ago Last modified on 11/07/11 13:38:55

Importing Historical Log Files into Raptor

Raptor provides two facilities for importing log files. The first allows the user to upload small log files throught the RaptorWeb interface. The second allows much larger log files to be uploaded directly through the MUA. Importantly, the following log file formats are currently supported:

  • Shibboleth 2.x idp-audit.log (only the audit log files)
  • Shibboleth 1.3 idp-access.log (only the access log files)
  • Ezproxy's generic log file.

Uploading Through RaptorWeb

The Setup page of RaptorWeb allows the user to upload Ezproxy or Shibboleth 1.3 / 2.x Idp log files that are under 10mb in size - see the image below. Only two log files can be uploaded at anyone time due to limitions with the current in-memory storage and transfer engine.

There are three steps involved in uploading a log file. Firstly, click the add button and select up to a maximum of two log files each under 10mb from the local file system - once selected, these will show in the interface. By clicking on upload, the log files will be uploaded into RaptorWeb (stored in internal memory). Then, for each log file uploaded, the event type needs to be selected manually (currently Raptor has no automated way for detecting the type of log files). Once the event type has been selected, the 'Upload To MUA' sends that log file to the attached MUA for processing. Once the MUA has sucessfully parsed - but not at that stage stored - events in the log file, the Status message will be updated, and you can safely clear the parsed data - in so doing freeing up internal memory.

Direct MUA Upload

The MUA can import log files directly off the local file system. In this way larger sized files (up to 50mb) can be uploaded at once, and there is no limit on the number of files that can be sat waiting in the queue ready to be uploaded. The default install of the MUA includes a directory called batchimport, with sub directories shib13, ezproxy and shib2. By placing log files in the correct directory e.g. Shibboleth 1.3 in the batchimport/shib13 directory etc. the MUA will recognise and import them in sequence (please wait up to 1 minute before they are recognised).

Our Shibboleth and Ezproxy log files are set to roll over each day and hence rarely exceed 40mb. Consequently, for our test installation, we can simple copy all relevant log files into their appropriate directories and leave the MUA import them. However, if your log files exceeed 50mb, there are two options for importing them:

  1. Use the linux split (or similar) command to seperate larger log files into smaller ones e.g
    split --line-bytes=50M <filename>
    
  2. Alternatively, you can process larger file sizes by increasing the amount of Heap Memory assigned to Java. More specifically, edit the /etc/init.d/raptormuad file, and change the line:
    javaArgs="-DconfigurationFiles=/opt/raptor/mua/conf/ -jar raptor-mua.jar -server -Xmx512M"
    
    to
    javaArgs="-DconfigurationFiles=/opt/raptor/mua/conf/ -jar raptor-mua.jar -server -Xmx1024M"
    
    Where the maximum heap space is defined by the arguments -Xmx<number>M. The configuration above allocates 1024 megabytes, but a higher number can be used as required.

Please NOTE,

  1. while the MUA is processing the batch import of log files, no other operations are permitted on the MUA. For example, any attached RaptorWeb instance will not be able to produce graphs until the import is complete.
  2. The ezproxy log file parser in the default Raptor install only parses Ezproxy log files in the format specified in EzproxyConfig. If another format is used, the parsing format will need to be modified as per instructions [NOT FINISHED].

Attachments