

Performance Tuning Recommendations¶

This section describes some recommended performance tuning configurations to optimize WSO2 Identity Server.

Important

Performance tuning requires you to modify important system files, which affect all programs running on the server. We recommend you to familiarize yourself with these files using Unix/Linux documentation before editing them.
The parameter values we discuss below are just examples. They might not be the optimal values for the specific hardware configurations in your environment. We recommend you to carry out load tests on your environment to tune the Identity Server accordingly.

OS-level settings¶

When it comes to performance, the OS that the server runs plays an important role.

Info

If you are running on MacOS Sierra or High Sierra, and experiencing long start-up times for WSO2 Identity Server, try mapping your Mac hostname to 127.0.0.1 and ::1 in the /etc/hosts file.

127.0.0.1   localhost <my_computer_hostname>
::1         localhost <my_computer_hostname>

Example:

127.0.0.1   localhost Alex-MacBook-Pro.local
::1         localhost Alex-MacBook-Pro.local

To optimize network and OS performance, configure the following settings in /etc/sysctl.conf file of Linux. These settings specify a larger port range, a more effective TCP connection time-out value, and a number of other important parameters at the OS-level.

Info

It is not recommended to use net.ipv4.tcp_tw_recycle = 1 when working with network address translation (NAT), such as if you are deploying products in EC2 or any other environment configured with NAT.
```
net.ipv4.tcp_fin_timeout = 30
fs.file-max = 2097152
net.ipv4.tcp_tw_recycle = 1
net.ipv4.tcp_tw_reuse = 1
net.core.rmem_default = 524288
net.core.wmem_default = 524288
net.core.rmem_max = 67108864
net.core.wmem_max = 67108864
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.ip_local_port_range = 1024 65535      
```
To alter the number of allowed open files for system users, configure the following settings in /etc/security/limits.conf file of Linux (be sure to include the leading * character).
```
* soft nofile 4096
* hard nofile 65535
```
Optimal values for these parameters depend on the environment.
To alter the maximum number of processes your user is allowed to run at a given time, configure the following settings in /etc/security/limits.conf file of Linux (be sure to include the leading * character). Each carbon server instance you run would require up to 1024 threads (with default thread pool configuration). Therefore, you need to increase the nproc value by 1024 per each carbon server (both hard and soft).
```
* soft nproc 20000
* hard nproc 20000
```

Set the thread execution limit for multitenant mode¶

In multi-tenant mode, the Carbon runtime limits the thread execution time. That is, if a thread is stuck or taking a long time to process, Carbon detects such threads, interrupts and stops them. Note that Carbon prints the current stack trace before interrupting the thread. This mechanism is implemented as an Apache Tomcat valve. Therefore, it should be configured in the <IS_HOME>/repository/conf/deployment.toml file as shown below.

[catalina.valves.valve.properties]
className = "org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve"
threshold="600"

The className is the Java class name used for the implementation. This must be set to org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve .
The threshold gives the minimum duration in seconds after which a thread is considered stuck. The default value is 600 seconds.

JVM settings¶

JVM heap size (Xmx) depends on your load. Given below are the general settings but if you are on a production environment, this might not be sufficient . In such situations, you can increase the heap size accordingly .

For instance, if you want to increase the JVM heap size to 4GB, open wso2server.sh located in <IS_HOME>/bin/ and do the following changes.
```
JVM_MEM_OPTS="-Xms4096m -Xmx4096m"
if [ "$java_version" \< "1.8" ]; then
    JVM_MEM_OPTS="$JVM_MEM_OPTS -XX:MaxPermSize=512m"
fi
```
When an XML element has a large number of sub-elements and the system tries to process all the sub-elements, the system can become unstable due to a memory overhead. This is a security risk.

To avoid this issue, you can define a maximum level of entity substitutions that the XML parser allows in the system. You do this by adding the entity expansion limit attribute to the <IS_HOME>/bin/wso2server.bat file (for Windows) or the <IS_HOME>/bin/wso2server.sh file (for Linux/Solaris). The default entity expansion limit is 64000.
```
set CMD_LINE_ARGS=-Xbootclasspath/a:%CARBON_XBOOTCLASSPATH% -Xms256m -Xmx1024m -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="%CARBON_HOME%\repository\logs\heap-dump.hprof"  -Dcom.sun.management.jmxremote -classpath %CARBON_CLASSPATH% %JAVA_OPTS% 
```
In a clustered environment, the entity expansion limit has no dependency on the number of worker nodes.

Database level settings¶

Set up the THRIFT_SESSION database index in the Identity Server database to improve performance:

create index IDX_ITS_LMT on IDN_THRIFT_SESSION (LAST_MODIFIED_TIME);

Info

If you want to remove unused tokens from the database, see Removing Unused Tokens from the Database.

JDBC pool configuration¶

Within WSO2 Identity Server, we use Tomcat JDBC pooling as the default pooling framework due to its production-ready stability and high performance. The goal of tuning the pool properties is to maintain a pool that is large enough to handle peak load without unnecessarily utilizing resources. These pooling configurations can be tuned for your production server in general in the <IS_HOME>/repository/conf/deployment.toml file.

Info

For more information on configuring pooling configurations for other databases, see Work with Databases.

The following parameters should be considered when tuning the connection pool:

The application's concurrency requirement.
The average time used for running a database query.
The maximum number of connections the database server can support.

The table below indicates some recommendations on how to configure the JDBC pool. For more details about recommended JDBC configurations, see Tomcat JDBC Connection Pool.

Property	Description	Tuning Recommendations
maxActive	This denotes the maximum number of active connections that can be allocated from the connection pool at the same time. The default value is `100.`	The maximum latency (approximately) = (P / M) * T, where, M = maxActive value P = Peak concurrency value T = Time (average) taken to process a query. Therefore, by increasing the maxActive value (up to the expected highest number of concurrency), the time that requests wait in the queue for a connection to be released will decrease. But before increasing the Max. Active value, consult the database administrator, as it will create up to maxActive connections from a single node during peak times, and it may not be possible for the DBMS to handle the accumulated count of these active connections. Note that this value should not exceed the maximum number of requests allowed for your database.
maxWait	The maximum time that requests are expected to wait in the queue for a connection to be released. This property comes into effect when the maximum number of active connections allowed in the connection pool (see maxActive property) is used up.	Adjust this to a value slightly higher than the maximum latency for a request, so that a buffer time is added to the maximum latency. That is, If the maximum latency (approximately) = (P / M) * T, where, M = maxActive value, P = Peak concurrency value, T = Time (average) taken to process a query, then, the maxWait = (P / M) * T + buffer time.
minIdle	The minimum number of connections that can remain idle in the pool, without extra ones being created. The connection pool can shrink below this number if validation queries fail. Default value is 0.	This value should be similar or near to the average number of requests that will be received by the server at the same time. With this setting, you can avoid having to open and close new connections every time a request is received by the server.
maxIdle	The maximum number of connections that can remain idle in the pool.	The value should be less than the maxActive value. For high performance, tune maxIdle to match the number of average, concurrent requests to the pool. If this value is set to a large value, the pool will contain unnecessary idle connections.
testOnBorrow	The indication of whether connection objects will be validated before they are borrowed from the pool. If the object validation fails, the connection is dropped from the pool, and there will be an attempt to borrow another connection.	When the connection to the database is broken, the connection pool does not know that the connection has been lost. As a result, the connection pool will continue to distribute connections to the application until the application actually tries to use the connection. To resolve this problem, set "Test On Borrow" to "true" and make sure that the "ValidationQuery" property is set. To increase the efficiency of connection validation and to improve performance, `validationInterval` property should also be used.
validationInterval	This parameter controls how frequently a given validation query is executed (time in milliseconds). The default value is `30000` (30 seconds). That is, if a connection is due for validation, but has been validated previously within this interval, it will not be validated again.	Deciding the value for the "validationInterval" depends on the target application's behavior. Therefore, selecting a value for this property is a trade-off and ultimately depends on what is acceptable for the application. If a larger value is set, the frequency of executing the Validation Query is low, which results in better performance. Note that this value can be as high as the time it takes for your DBMS to declare a connection as stale. For example, MySQL will keep a connection open for as long as 8 hours, which requires the validation interval to be within that range. However, note that the validation query execution is usually fast. Therefore, even if this value is only large by a few seconds, there will not be a big penalty on performance. Also, specially when the database requests have a high throughput, the negative impact on performance is negligible. For example, a single extra validation query run every 30 seconds is usually negligible. If a smaller value is set, a stale connection will be identified quickly when it is presented. This maybe important if you need connections repaired instantly, e.g. during a database server restart.
validationQuery	The SQL query used to validate connections from this pool before returning them to the caller. If specified, this query does not have to return any data, it just can't throw an SQLException. The default value is null. Example values are SELECT 1(mysql), select 1 from dual(oracle), SELECT 1(MS Sql Server).	Specify an SQL query, which will validate the availability of a connection in the pool. This query is necessary when `testOnBorrow` property is true.
MaxPermSize	The memory size allocated for WSO2 Identity Server.	The default memory allocated for the product via this parameter is as follows: `-Xms256m -Xmx512m -XX:MaxPermSize=256m` You can increase the performance by increasing this value in the `<IS_HOME>/bin/wso2server.sh` file as follows: `-Xms2048m -Xmx2048m -XX:MaxPermSize=1024m`

Note

When it comes to web applications, users are free to experiment and package their own pooling framework such BoneCP.
If you are using an Oracle database, you may sometimes come across an error ( ORA-04031 ) indicating that you have not allocated enough memory for the shared pool of connections. To overcome this, you can allocate more memory to the shared pool by adjusting the following parameters in the < ORACLE_HOME>/dbs/init<SID>.ora file of your Oracle database: SHARED_POOL_RESERVED_SIZE, SHARED_POOL_SIZE and LARGE_POOL_SIZE.

Cache configuration¶

The <Cache> element configured in the deployment.toml file (stored in the <IS_HOME>/repository/conf directory) sets the global caching timeout in minutes for your server. This value specifies the time period after which, the cache will refresh. If the components in your product do not have specific caching timeout values configured, the global caching timeout will be applicable by default. Be sure to restart the server if you change the default caching timeout in the deployment.toml file shown below.

[server]
default_cache_timeout= "15"

Pool LDAPS connections¶

Connection pooling does not apply to LDAPS connections (SSL-enabled LDAP connections) by default. Therefore, you need to enable connection pooling for LDAPS connections at the time of starting your server:

Stop the server.
Open a command terminal, navigate to the <IS_HOME>/bin directory, and execute the relevant startup script:
- On Linux: wso2server.sh
- On Windows: wso2server.bat
Add the following system property to the script:
```
-Dcom.sun.jndi.ldap.connect.pool.protocol=ssl
```
Start the server.

Top