Working with Product Observability¶

Product observability enables rapid debugging of product issues. WSO2 Identity Server (WSO2 IS) facilitates product observability by logging the time taken for LDAP and JDBC database calls. This helps to track down any latencies caused by database calls in an instance. The request calls and response calls are correlated via a correlation ID that is sent in the request call.

Let's explore the following topics to learn more.

Configuring product observability¶

log4j configs¶

Follow the steps below to set up the correlation logs related to the database calls.

1. Open the log4j2.properties file in the <IS_HOME>/repository/conf directory.

2. Append the appender CORRELATION to the list of all appenders as follows.

   appenders = CARBON_CONSOLE, CARBON_LOGFILE, AUDIT_LOGFILE, ATOMIKOS_LOGFILE, CARBON_TRACE_LOGFILE, 
   DELETE_EVENT_LOGFILE, TRANSACTION_LOGFILE, osgi, CORRELATION

   3. Append the loggger correlation for list of all loggers as follows.

   loggers = AUDIT_LOG, trace-messages, org-apache-coyote, com-hazelcast, Owasp-CsrfGuard, 
   org-apache-axis2-wsdl-codegen-writer-PrettyPrinter, org-apache-axis2-clustering, org-apache-catalina, 
   org-apache-tomcat, org-wso2-carbon-apacheds, org-apache-directory-server-ldap, org-apache-directory-server-core-event, com-atomikos, org-quartz, org-apache-jackrabbit-webdav, org-apache-juddi, org-apache-commons-digester-Digester, org-apache-jasper-compiler-TldLocationsCache, org-apache-qpid, org-apache-qpid-server-Main, qpid-message, qpid-message-broker-listening, org-apache-tiles, org-apache-commons-httpclient, org-apache-solr, me-prettyprint-cassandra-hector-TimingLogger, org-apache-axis-enterprise, org-apache-directory-shared-ldap, org-apache-directory-server-ldap-handlers, org-apache-directory-shared-ldap-entry-DefaultServerAttribute, org-apache-directory-server-core-DefaultDirectoryService, org-apache-directory-shared-ldap-ldif-LdifReader, org-apache-directory-server-ldap-LdapProtocolHandler, org-apache-directory-server-core, org-apache-directory-server-ldap-LdapSession, DataNucleus, Datastore, Datastore-Schema, JPOX-Datastore, JPOX-Plugin, JPOX-MetaData, JPOX-Query, JPOX-General, JPOX-Enhancer, org-apache-hadoop-hive, hive, ExecMapper, ExecReducer, net-sf-ehcache, axis2Deployment, equinox, tomcat2, StAXDialectDetector, org-apache-directory-api, org-apache-directory-api-ldap-model-entry, TRANSACTION_LOGGER, DELETE_EVENT_LOGGER, org-springframework, org-opensaml-xml-security-credential-criteria, org-wso2-carbon-user-core, org-wso2-carbon-identity, org-wso2-carbon-identity-sso-saml, correlation

   4. Following are the default correlation appender configuration. You can change any of these values using the log4j2 .properties.

   appender.CORRELATION.type = RollingFile
   appender.CORRELATION.name = CORRELATION
   appender.CORRELATION.fileName =${sys:carbon.home}/repository/logs/correlation.log
   appender.CORRELATION.filePattern =${sys:carbon.home}/repository/logs/correlation-%d{MM-dd-yyyy}.%i.log
   appender.CORRELATION.layout.type = PatternLayout
   appender.CORRELATION.layout.pattern = %d{yyyy-MM-dd HH:mm:ss,SSS}|%X{Correlation-ID}|%t|%mm%n
   appender.CORRELATION.policies.type = Policies
   appender.CORRELATION.policies.time.type = TimeBasedTriggeringPolicy
   appender.CORRELATION.policies.time.interval = 1
   appender.CORRELATION.policies.time.modulate = true
   appender.CORRELATION.policies.size.type = SizeBasedTriggeringPolicy
   appender.CORRELATION.policies.size.size=10MB
   appender.CORRELATION.strategy.type = DefaultRolloverStrategy
   appender.CORRELATION.strategy.max = 20
   appender.CORRELATION.filter.threshold.type = ThresholdFilter
   appender.CORRELATION.filter.threshold.level = INFO

Enabling observability¶

Follow the steps below to enable product observability.

1. Navigate to the <IS_HOME>/bin directory on the command prompt.

       cd <IS_HOME>/bin

2. To set the -DenableCorrelationLogs property to true, execute the following command.

       For macOS/Linux   --> sh wso2server.sh -DenableCorrelationLogs=true start
       For Windows     --> wso2server.bat -DenableCorrelationLogs=true start

   Note

   By default, this property is set to false .

3. Navigate to the <IS_HOME>/repository/logs directory.

       cd <IS_HOME>/repository/logs

   Notice that a separate log file called correlation.log is created.

   

Now you are ready to test the product observability of WSO2 IS.

Log patterns¶

Following are the log patterns that support product observability.

JDBC database call logging¶

Format¶

timestamp | correlationID | threadID | duration | callType | startTime | methodName | query | connectionUrl 

Example¶

2018-10-22 17:54:46,869|cf57a4a6-3ba7-46aa-8a2b-f02089d0172c|http-nio-9443-exec-2|4|jdbc|1540211086865|executeQuery|SELECT ID, TENANT_ID, IDP_ID, PROVISIONING_CONNECTOR_TYPE, IS_ENABLED, IS_BLOCKING  FROM IDP_PROVISIONING_CONFIG WHERE IDP_ID=?|jdbc:mysql://localhost:13306/apimgtdb?autoReconnect=true&useSSL=false

LDAP database call logging¶

Format¶

timestamp | correlationID | threadID | duration | callType | startTime | methodName | providerUrl | principal | argsLengeth | args

Example¶

2018-10-2310:55:02,279|c4eaede8-914d-4712-b630-73f6534b8def|http-nio-9443-exec-18|19|ldap|1540272302260|search|ldap://localhost:10392|uid=admin,ou=system| ou=Users,dc=wso2,dc=org,(&(objectClass=person)(uid=admin)),javax.naming.directory.SearchControls@6359ae3a

Beginning of the request call¶

Format¶

timestamp | correlationID | threadID | duration | HTTP-In-Request | startTime | methodName | requestQuery | requestPath

Example¶

2018-11-0514:57:06,757|f884a93d-e3a3-431f-a1ea-f6973e125cb6|http-nio-9443-exec-28|0|HTTP-In-Request|1541410026757|GET|null|/carbon/admin/images/favicon.ico

Ending of the request call¶

Format¶

timestamp | correlationID | threadID | totalDurationForRequest | HTTP-In-Response | startTime | methodName | requestQuery | requestPath

Example¶

2018-11-05 14:57:06,764|f884a93d-e3a3-431f-a1ea-f6973e125cb6|http-nio-9443-exec-28|7|HTTP-In-Response|1541410026764|GET|null|/carbon/admin/images/favicon.ico

Reading the logs¶

Let's analyze the following sample log lines to find if there are any timing delays for the JDBC or LDAP calls.

Advanced scenarios¶

Following are a few advance scenarios that are related to product observability in WSO2 IS.

Defining denylists for the threads¶

Certain threads continuously print unnecessary logs. Defining denylists prevents the unwanted threads from printing logs thereby improving the readability of the logs.

Follow the steps below to configure denylists for threads.

1. Open either of the following files in the <IS_HOME>/bin directory on a command prompt.
   1. For macOS/Linux: wso2server.sh file
   2. For Windows: wso2server.bat file

2. Add the following configuration as a system property.

   -Dorg.wso2.CorrelationLogInterceptor.blacklistedThreads=threadName1,threadName2 \

   Tip

   Make sure to add it before the org.wso2.carbon.bootstrap.Bootstrap $* line.

   Note

   This configuration is not required by default, as all unnecessary threads are already a part of the denylist in the MessageDeliveryTaskThreadPool thread. If the above configuration is added, the default value will be overridden.

3. Restart the WSO2 IS server.

       sh wso2server.sh -DenableCorrelationLogs=true stop
       sh wso2server.sh -DenableCorrelationLogs=true start

4. To send the authentication request, execute the following cURL command.

       curl -v -k -X POST --basic -u <CLIENT_KEY>:<CLIENT_SECRET> -H "Content-Type: application/x-www-form-urlencoded;charset=UTF-8" -H "customHeader1:correlationvalue1" -H "customHeader2:correlationvalue2" -d "grant_type=client_credentials" https://localhost:9443/oauth2/token

   Tip

   Use the client key and client secret of the service provider you created after enabling product observability

5. Open the correlation.log on a command prompt and notice the related logs.

   tail -f ../repository/logs/correlation.log

   

OpenTelemetry-based tracing in WSO2 Identity Server¶

Attaching a Java observability agent, such as the Datadog Java Agent, enables WSO2 Identity Server to automatically trace common operations, including API calls and database interactions. The agent’s built-in auto instrumentation generates these traces.

To provide even more comprehensive observability, WSO2 Identity Server offers explicit OpenTelemetry-based instrumentation for Lightweight Directory Protocol (LDAP) operations, such as search, bind, and lookup. Enabling this feature extends visibility to LDAP calls alongside other automatically traced operations.

The following sections explain how to configure tracing with the Datadog Java Agent. Once you attach and configure the agent, you can enable instrumentation for LDAP through the WSO2 Identity Server configuration.

Configure the Datadog Java Agent¶

To set up tracing with the Datadog Java Agent you need the following components:

• The Datadog Java Agent (dd-java-agent.jar) attached to the WSO2 Identity Server Java Virtual Machine.
• A running Datadog Agent that receives spans from the Java Agent and forwards them to the Datadog platform.

To configure the Datadog Java Agent with WSO2 Identity Server, follow these steps:

1. Install and run the Datadog Agent following the Datadog Agent installation guide.

2. Download the Datadog Java Agent (dd-java-agent.jar) from the Datadog releases page.

3. To enable tracing, add the agent and related configurations as Java options when starting WSO2 Identity Server. To do so,

   • Open the corresponding to your operating system.

     macOS/Linux:
     <IS_HOME>/bin/wso2server.sh
     
     Windows:
     <IS_HOME>/bin/wso2server.bat

   • Extend the JAVA_OPTS variable to include the Java agent option.

     -javaagent:/path/to/dd-java-agent.jar

4. In the same file, add the required system properties as Java options to define service metadata and agent settings specific to the environment. For example:

   -Ddd.service="wso2is" \
   -Ddd.env="dev" \
   -Ddd.version=1.0 \
   -Ddd.logs.injection=true \
   -Ddd.trace.agent.host=localhost \
   -Ddd.trace.otel.enabled=true \

   Refer to the Datadog Java Agent configuration documentation for the full set of options.

5. Restart WSO2 Identity Server and confirm that traces appear in the Datadog APM dashboard.

Enable LDAP tracing in WSO2 Identity Server¶

WSO2 Identity Server extends the Datadog Java Agent with OpenTelemetry-based LDAP instrumentation, capturing operations such as search, bind, and lookup for full tracing alongside API and database calls.

You can enter the following configuration in the <IS_HOME>/repository/conf/deployment.toml file to enable LDAP tracing.

[tracing.opentelemetry]
ldap.enabled = true
ldap.scope_name = "wso2isldap"

When enabled, LDAP operations produce spans that include metadata such as operation type, base or target DN, search filters, execution duration, and error status where applicable. The configured Java agent exports these spans to the observability platform. The following example shows a sample span for the ldap.search operation (from Datadog).

{
  "ldap": {
    "filter": "(&(objectClass=***)(member=uid=***,ou=***,dc=***,dc=***))",
    "search": {
      "controls": "{scope=2, countLimit=0, timeLimit=0, returningAttributes=[cn], derefLinkFlag=false, returningObjFlag=false}"
    },
    "success": "true",
    "dn": "ou=***,dc=***,dc=***"
  },
  "peer": {
    "hostname": "localhost",
    "port": 10390,
    "service": "ldap"
  },
  "correlation_id": "8787f3ba-6149-4aa7-a620-77889664bae3",
  "language": "java",
  "thread": {
    "name": "https-jsse-nio-9443-exec-9",
    "id": 244
  },
  "env": "dev",
  "version": "1.0",
  "span": {
    "kind": "client"
  },
  "duration": 1211208
}