ELK-based Analytics Installation Guide¶
This guide shows you how to configure ELK-based Analytics for WSO2 Identity Server. ELK-based Analytics solution supports ELK version 8.X.X.
Enable Analytics in WSO2 Identity Server¶
Follow the steps below to enable ELK-based analytics in WSO2 Identity Server.
-
Download and install WSO2 Identity Server. For detailed information on how to install WSO2 IS, see Installing the Product.
-
Navigate to the
<IS_HOME>/repository/conf/
directory and open thedeployment.toml
file. -
Add the following configuration to the
deployment.toml
file.[analytics.elk] enable=true
Enable Logs in WSO2 Identity Server¶
-
Navigate to the
<IS_HOME>/repository/conf
directory and open thelog4j2.properties
file. -
Add the following configurations to the
log4j2.properties
file.-
Add
ANALYTICS_EVENT_LOGFILE
to the list of all appenders as follows:
appenders = {other appenders} , ANALYTICS_EVENT_LOGFILE
-
Add the following appender configs:
appender.ANALYTICS_EVENT_LOGFILE.type = RollingFile appender.ANALYTICS_EVENT_LOGFILE.name = ANALYTICS_EVENT_LOGFILE appender.ANALYTICS_EVENT_LOGFILE.fileName =${sys:carbon.home}/repository/logs/analytics_events.log appender.ANALYTICS_EVENT_LOGFILE.filePattern = ${sys:carbon.home}/repository/logs/analytics_events-%d{MM-dd-yyyy}.%i.log appender.ANALYTICS_EVENT_LOGFILE.layout.type = PatternLayout appender.ANALYTICS_EVENT_LOGFILE.layout.pattern=TID: [%tenantId] [%appName] [%d] [%X{Correlation-ID}] %5p {%c} — %mm%ex%n appender.ANALYTICS_EVENT_LOGFILE.policies.type = Policies appender.ANALYTICS_EVENT_LOGFILE.policies.time.type = TimeBasedTriggeringPolicy appender.ANALYTICS_EVENT_LOGFILE.policies.time.interval = 1 appender.ANALYTICS_EVENT_LOGFILE.policies.time.modulate = true appender.ANALYTICS_EVENT_LOGFILE.policies.size.type = SizeBasedTriggeringPolicy appender.ANALYTICS_EVENT_LOGFILE.policies.size.size=10MB appender.ANALYTICS_EVENT_LOGFILE.strategy.type = DefaultRolloverStrategy appender.ANALYTICS_EVENT_LOGFILE.strategy.max = 20 appender.ANALYTICS_EVENT_LOGFILE.filter.threshold.type = ThresholdFilter appender.ANALYTICS_EVENT_LOGFILE.filter.threshold.level = INFO
Note
To change the name of the log file and its location, change the fileName and filePattern parameters accordingly.
-
Add
org.wso2.carbon.event.output.adapter.logger.LoggerEventAdapter
to the list of all loggers as follows:loggers = {existing loggers}, org-wso2-carbon-event.output-adapter-logger-LoggerEventAdapter
-
Add the following logger configs
logger.org-wso2-carbon-event.output-adapter-logger-LoggerEventAdapter.name=org.wso2.carbon.event.output.adapter.logger.LoggerEventAdapter logger.org-wso2-carbon-event.output-adapter-logger-LoggerEventAdapter.level=INFO logger.org-wso2-carbon-event.output-adapter-logger-LoggerEventAdapter.appenderRef.CUSTOM_LOGFILE.ref=ANALYTICS_EVENT_LOGFILE
Note
The
analytics_events.log
file will be rolled each day or when the log size reaches the limit of 1000 MB by default. Furthermore, only 10 revisions will be kept and older revisions will be deleted automatically. You can change these configurations by updating the configurations provided in step 2 given above in this. section.
-
Configure ELK¶
Install Elasticsearch¶
-
Install Elasticsearch according to your operating system.
-
Make sure Elasticsearch is up and running.
Install Filebeat¶
-
Install Filebeat according to your operating system.
-
Open the filebeat.yml file in the root directory and enter these configurations.
Info
To configure Filebeat to mask sensitive information in logs, follow this guide.
Install Logstash¶
- Install Logstash according to your operating system.
-
In the Logstash directory, create a file with the .conf extension and add these configurations.
-
Start the logstash server with the
-f
flag set to the configuration file you created.
Installing Kibana¶
- Install Kibana according to your operating system and do this one time configuration. .
- Once Kibana is running, enter its web interface using the following address:
http://127.0.0.1:5601
Configure ELK Analytics Dashboards¶
-
Navigate to Kibana installation folder and run the following command to install the Kibana Enhanced Table plugin.
./bin/kibana-plugin install https://github.com/fbaligand/kibana-enhanced-table/releases/download/vA.B.C/enhanced-table-A.B.C_X.Y.Z.zip
Info
Replace A,B,C with the plugin version and X,Y,Z with the Kibana version you have installed.
-
Restart Kibana service and log in to Kibana.
- Navigate to Stack Management > Index Management and select the Index Templates tab.
-
Under the Index patterns column, if you have any index patterns created under the following names, delete them before moving to the next step.
- wso2-iam-alert-auth*
- wso2-iam-alert-session*
- wso2-iam-auth-raw*
- wso2-iam-session-raw*
- wso2-iam-session-time-series*
-
Download the artifact file here.
- Navigate to Stack Management > Saved Objects.
- Click Import, add the downloaded artifact file as an import object, and import.
- Login to MyAccount of Identity Server using default admin credentials.
- Navigate to the Dashboard section of Kibana to view the created Auth and Session dashboards.