Like any other application, X-Road is producing logs that contain information about how different components of the application are operating. Application logs can be used for multiple purposes, e.g. monitoring, debugging error conditions, verifying transactions etc. In general, application logs can be divided in three different groups: technical logs, business logs and audit logs. X-Road produces logs in all the three categories.
Technical logs
X-Road consists of three main components: Central Server, Configuration Proxy and Security Server. Each of these components is internally divided into several lower level components or modules – some of which are shared between the main components. For example, Security Server’s main components are proxy, signer and confclient. Proxy is a Security Server specific component whereas signer is used by all the main components. When it comes to technical logs each component has its own log file where it writes information regarding its operations. The level of detail can be configured using configuration files – each lower level component has its own configuration file. Technical log files are stored in /var/log/xroad directory. More information about the technical logs can be found at:
Business logs
Security Server’s business log is stored in the message log database which contains all the messages processed by the Security Server. Each message is time-stamped and signed which makes it possible to verify the message content afterwards. However, verifying a message requires that the message payload has been logged – logging message payload can also be disabled. By default, time-stamped messages are archived from the database to disk every six hours. Time-stamped and archived messages are kept in the message log database for 30 days until they are removed automatically. Different intervals regarding archiving of messages can be configured through configuration files. More information about the message log can be found at:
Audit log
Security Server’s audit log is stored on the disk and it contains information about all the actions completed by an administrator through the Security Server UI. In this way all the actions that change the configuration or state of the Security Server are logged and they can be traced afterwards. The audit log is stored in /var/log/xroad directory. More information about the audit log can be found at:
Archiving logs
It is worth mentioning that all the logs produced by the Security Server are local – also in clustered environments. This means that log records are not replicated inside Security Server cluster. In addition, log records are not automatically transferred to an external host or log storage for archiving purposes.
In most cases there are both business and legal requirements regarding the retention period of different types of logs. X-Road produces the logs, but it is the administrator’s responsibility to configure the transfer of the logs to a long-term storage. This is strongly recommended for saving hard disk space and avoiding loss of log records during Security Server crash. Therefore, it is a good idea to transfer the logs to a centralized logging system or log storage rather than storing them locally on Security Server.
Technical logs and audit log can be configured to be redirected to an external location using rsyslog. In this case a batch type of transfer is not required as rsyslog forwards the log messages nearly real time. However, is not possible to use rsyslog for transferring archived message log record files so a batch transfer must be used. This can be implemented using a transfer script shipped with the Security Server or using rsync.
Please don’t forget disk space
Another reason why it is highly recommended to transfer log records to an external storage rather than storing them locally is hard disk space that they consume. Especially message log database and message log archive files may consume a significant amount of disk space on high traffic Security Servers.
Each SOAP request contains 9.5 kB of metadata (SOAP namespace definitions, headers, signatures) and response message contains around 11.2 kB of metadata. Based on this, each successful query produces around 21 kB of metadata in the message log. In addition, once in a minute security server batch time-stamps all the messages processed since the previous time-stamp operation which generates additional 3.6 kB of metadata per minute.
Message log space requirements can be estimated using the formula below:
3.6kB + N * (21kB + R + A) = S
N = number of requests per minute
R = size of request payload in kB
A = size of response payload in kB
S = disk usage per minute (kB / min)
Example
Let's assume that a system receives 100 requests per minute. Request payload size is 4 kB and response payload size is 8 kB.
3.6 kB + 100 * (21 kB + 4 kB + 8 kB) = 3303.6 kB / min = 3.3 MB / min
This makes 4.8 GB per day and 143 GB per month.
In addition, archived message log records require around 43 % of the space required by the active message log records so the required disk space in total is:
4.8 + 2.1 GB = 6.9 GB / day
143 + 61 GB = 204 GB / month
1716 + 738 GB = 2454 GB / year
When looking at the example above it is easy to understand why handling and archiving of log records should be planned in advance. Message log may generate a huge amount of data and no one wants to end up in a situation where a full disk causes a service break in a critical business system.
What next?
Many of you may ask what’s the sense of collecting and archiving all these logs. Wouldn’t it be easier to free up disk space by just deleting them? The features related to logging are not a coincidence and they are there for a reason. In the next blog post regarding the X-Road logs I will give more insights on the background of the message log and why it’s working the way it is.