HTTP Access Logging

HTTP access logs help you monitor your application's usage with information such as the persons who access it, how many hits it received, what the errors are, etc. This information is useful for troubleshooting errors. WSO2 Identity Server can enable access logs for the HTTP servlet transport. This servlet transport works on 9443/9763 ports, and it receives admin/operation requests. Therefore, access logs for the servlet transport are useful for analyzing operational/admin-level access details.

Configure access logs for the HTTP servlet transport

To configure access logs for HTTP servlet transport:

1. Open the <IS_HOME>/repository/conf/deployment.toml file.

2. Add the following configuration.

   [http_access_log]
   pattern = "%h %l %u %t %r %s %b %{Referer}i %{User-Agent}i %T"
   

   The attributes that are used by default are explained below. See the descriptions of the Tomcat-supported Access Log Valve attributes.

   directory	The path to the directory that will store the access log file. By default, this location is set to ${carbon.home}/repository/logs in all WSO2 products.
   prefix	The prefix added to the log file's name.
   suffix	The suffix added to the log file's name. By default, this is .log for all WSO2 products.
   pattern	The attribute defines the format for the log pattern, which consists of the information fields from the requests and responses that should be logged. The pattern format is created using the following attributes: A standard value to represent a particular string. For example, "%h" represents the remote hostname in the request. See the list of string replacement values supported by the Tomcat valve . %{xxx}i is used to represent the header in the incoming request (xxx=header value). %{xxx}o is used to represent the header in the outgoing request (xxx=header value). While you can use the above attributes to define a custom pattern, the standard patterns shown below can be used. common (Apache common log pattern): pattern=%h %l %u %t "%r" %s %b combined (Apache combined log pattern): pattern=%h %l %u %t "%r" %s %b "%{Referer}i" "%{User-Agent}i"

3. Restart the server. According to the configurations, a log file named http_access.{DATE}.log is created by default inside the <IS_HOME>/repository/logs directory. The log is rotated daily.

Routing access logs to the Log4j2 logger

By default, HTTP access logs write to a separate http_access.log file using the Tomcat Access Log Valve. WSO2 Identity Server also supports routing HTTP access logs through the Log4j2 logger, which gives you full control over where those logs go.

To enable Log4j2-based access logging, set useLogger = true in <IS_HOME>/repository/conf/deployment.toml:

[http_access_log]
useLogger = true

After enabling this, update <IS_HOME>/repository/conf/log4j2.properties to add the HTTP access log appender and logger.

Route to a dedicated rolling log file

If you want to keep HTTP access logs in their own file, separate from other server logs, you can route them to a rolling log file that rotates on a schedule and rolls over when it reaches a configured size. To do so,

1. Add HTTP_ACCESS to the appenders list:

   appenders = CARBON_CONSOLE, CARBON_LOGFILE, AUDIT_LOGFILE, ATOMIKOS_LOGFILE, CARBON_TRACE_LOGFILE, DELETE_EVENT_LOGFILE, TRANSACTION_LOGFILE, HTTP_ACCESS
   

2. Add HTTP_ACCESS to the loggers list:

   loggers = HTTP_ACCESS, AUDIT_LOG, trace-messages, ...
   

3. Add the appender and logger configuration:

   logger.HTTP_ACCESS.name = HTTP_ACCESS
   logger.HTTP_ACCESS.level = INFO
   logger.HTTP_ACCESS.appenderRef.HTTP_ACCESS.ref = HTTP_ACCESS
   logger.HTTP_ACCESS.additivity = false
   
   appender.HTTP_ACCESS.type = RollingFile
   appender.HTTP_ACCESS.name = HTTP_ACCESS
   appender.HTTP_ACCESS.fileName = ${sys:carbon.home}/repository/logs/http_access.log
   appender.HTTP_ACCESS.filePattern = ${sys:carbon.home}/repository/logs/http_access-%d{MM-dd-yyyy}.log
   appender.HTTP_ACCESS.layout.type = PatternLayout
   appender.HTTP_ACCESS.layout.pattern = [%X{Correlation-ID}] %mm%n
   appender.HTTP_ACCESS.policies.type = Policies
   appender.HTTP_ACCESS.policies.time.type = TimeBasedTriggeringPolicy
   appender.HTTP_ACCESS.policies.time.interval = 1
   appender.HTTP_ACCESS.policies.time.modulate = true
   appender.HTTP_ACCESS.policies.size.type = SizeBasedTriggeringPolicy
   appender.HTTP_ACCESS.policies.size.size = 10MB
   appender.HTTP_ACCESS.strategy.type = DefaultRolloverStrategy
   appender.HTTP_ACCESS.strategy.max = 20
   appender.HTTP_ACCESS.filter.threshold.type = ThresholdFilter
   appender.HTTP_ACCESS.filter.threshold.level = INFO
   

Route to standard output

In Kubernetes environments, logs are typically collected from standard output rather than files. To make HTTP access logs part of that flow, route them to the console alongside all other server logs. To do so,

1. Set appenders to CARBON_CONSOLE only:

   appenders = CARBON_CONSOLE
   

2. Add HTTP_ACCESS to the loggers list:

   loggers = HTTP_ACCESS, AUDIT_LOG, trace-messages, ...
   

3. Add the logger configuration:

   logger.HTTP_ACCESS.name = HTTP_ACCESS
   logger.HTTP_ACCESS.level = INFO
   logger.HTTP_ACCESS.appenderRef.HTTP_ACCESS.ref = CARBON_CONSOLE
   logger.HTTP_ACCESS.additivity = false
   

You can define other logging patterns and targets for the HTTP_ACCESS logger using standard Log4j2 configuration. See the Log4j2 documentation for available options.

Customizing access logs by pattern

Given below are a few sample configurations for customizing the pattern attribute:

Example 1: Logging request headers

The configuration is as follows:

[http_access_log]
pattern = "%{Content-Type}i %{Accept}i %{Accept-Encoding}i"

This sample configuration logs the Content-type, Accept, and Accept-encoding headers of every request coming to the server. For example, in the following example, we use the RequestInfoExample to send the HTTP request:

GET http://<IP>:<PORT>/example/servlets/servlet/RequestInfoExample?abc=xyz

The following log entry is recorded in the http_access.{DATE}.log file.

text/plain; charset=utf-8        */*        gzip,deflate,sdch

Example 2: Logging response headers

The configuration is as follows:

[http_access_log]
pattern = "%{Content-Type}o %{Content-Length}o %{Date}o %{Server}o"

The above configuration sample logs the Content-type, Content-Length, Date, and Server headers of every response coming from the server as follows:

text/html;charset=ISO-8859-1       662       Tue, 09 Jul 2013 11:21:50 GMT        WSO2 Carbon

Example 3: Logging other variable values

The configuration is as follows:

[http_access_log]
pattern = "%r %q %h"

The above sample configuration logs the first line of the request (method and request URI), query string (prepended with a '?' if it exists), and a remote hostname (or IP) of every request coming to the server as follows:

“GET http://<IP>:<PORT>//example/servlets/servlet/RequestInfoExample?abc=xyz HTTP/1.1”      ?abc=xyz     10.100.0.67

Example 4: Logging URL encoded parameters

You cannot use the AccessLogValve to log URL-encoded parameters. However, you can use the ExtendedAccessLogValve attribute for this purpose. In this example, only two values (namely, className, and pattern ) are modified from the previous configuration. Hence this will be added as a new valve.

The configuration is as follows:

[catalina.valves.valve.properties]
className = "org.apache.catalina.valves.ExtendedAccessLogValve"
directory="${carbon.home}/repository/logs"
prefix="localhost_access_log_extended."
suffix=".log"
pattern="x-P(param1) x-P(param2)"

Send the POST request together with the URL encoded values such as param1 = value1 and param2 = value2 as follows:

POST http://<IP>:<PORT>/example/servlets/servlet/RequestInfoExample

The above sample configuration logs the following:

'value1'     'value2'