Deployment Patterns¶
The following sections provide high level information on the recommended deployment pattern.
Introduction¶
You can run multiple nodes of WSO2 Identity Server in a cluster mode to achieve two requirements.
- Handle requests seamlessly: If one node becomes unavailable or is experiencing high traffic, another node will seamlessly handle the requests.
- Balancing traffic handling: Multiple nodes can handle the traffic together so that cluster throughput is higher than the throughput of a single node.
For complete information on clustering concepts, see Clustering Overview. The following sections guide you through setting up the deployment pattern, which is an HA Clustered Deployment of 2 WSO2 Identity Server nodes.
Deployment prerequisites¶
As a first step in planning your deployment, ensure that you have the necessary system requirements and a compatible environment.
System requirements¶
The above recommendations can change based on the expected concurrency & performance. |
Note
For more information on prerequisites, see Installation Prerequisites
Environment compatibility¶
Operating systems | For information on tested operating systems, see Tested Operating Systems and JDKs . |
---|---|
RDBMS | For information on tested DBMSs, see Tested DBMSs . |
Directory services | Supports Directory Services implementing following LDAP Protocols:
For information on tested LDAPs, see Tested LDAPs . |
Java | Oracle JDK 1.8 |
Web browsers | For more information on tested web browsers, see Tested Web Browsers . |
Load balancers | For more information about load balancers, see Configuring the load balancer . |
Configuring Databases¶
In a clustered deployment, all WSO2 Identity Server nodes are pointed to the same databases to ensure the integrity of the data. Also, you can configure multiple logical databases if you require to keep your data logically separated in the environment. Following tutorial demonstrates deployment with an identity database (IDENTITY_DB) and a user database (UM_DB).
Note
Alternatively, you can create more databases for each type of data to separate the data logically.
This will NOT make a difference in performance and is not mandatory. Separating databases logically may
sometimes help to have a different backup and scaling strategy when the deployment is large and complex.
If you do wish to separate the data logically into separate databases,
see Setting Up Separate Databases for Clustering.
Tip
If you have configured the shared database correctly, the deployment.toml
in
<IS_HOME>/repository/conf
directory, should have the following configurations.
NOTE : Following is a sample configuration. Therefore parameter values might be different.
[database.identity_db]
type = "mysql"
hostname = "localhost"
name = "regdb"
username = "regadmin"
password = "regadmin"
port = "3306"
[database.shared_db]
type = "mysql"
hostname = "localhost"
name = "regdb"
username = "regadmin"
password = "regadmin"
port = "3306"
The following diagram is a high-level component diagram showing how the system would look like when two databases are used.
Note
For instructions on how to configure the data sources for other databases and more information related to databases, see Working with Databases
Mounting the shared registry¶
WSO2 Identity Server comprises of three different registry repositories.
-
Local Repository: Store configuration and runtime data that is local to the server.
-
Configuration Repository: Store product-specific configurations.
-
Governance Repository: Store configuration and data that are shared across the whole platform. This typically includes services, service descriptions, endpoints or data sources.
Info
For more information about the registry, see Working with the Registry.
In this cluster setup, we use the default h2 database as the local registry in each node individually and the governance and configuration registries should be mounted to share across all nodes. In WSO2 Identity Server, config and governance registries are mounted by default.
Note
The production recommendation is to set the <versionResourcesOnChange>
property in the registry.xml
file to false. This is because the automatic versioning of resources can be an extremely expensive operation.
<versionResourcesOnChange>false</versionResourcesOnChange>
To make sure the configurations were applied correctly:
- Log in to the management console.
-
Navigate to Home > Registry > Browse.
Note that the governance collection is shown with the symlink icon.
Clustering-related configurations¶
Following configurations need to be done to both WSO2 Identity Server nodes in order to enable clustering between them.
WSO2 supports the following membership schemes for clustering - well-known address (WKA) - Multicast membership - AWS membership - AWS ECS membership - Kubernetes membership
-
Enable clustering on node 1 and node 2 by setting the membership scheme that fits your deployment by editing the
<IS_HOME>/repository/conf/deployment.toml
file.Info
The simplest is the well-known address (WKA) based clustering method. It only suites where all the nodes are deployed on machines having static IP addresses. For more information, see About Membership Schemes. Configurations for each membership scheme are listed below.
Click to see the instructions for WKA scheme
Edit the
<IS_HOME>/repository/conf/deployment.toml
file to add following configurations. Configure thelocalMemberHost
andlocalMemberPort
entries. Add the IP of the editing node itself.
Under the[clustering] membership_scheme = "wka" local_member_host = "192.168.2.1" local_member_port = "4000" members = ["192.168.2.1:4000", "192.168.2.2:4001"]
members
section, add thehostName
andport
for each WKA member. As we have only two nodes in our sample cluster configuration, we will configure both nodes as WKA nodes.You can also use IP address ranges for the hostName. For example,
192.168.1.2-10
. This should ensure that the cluster eventually recovers after failures. One shortcoming of doing this is that you can define a range only for the last portion of the IP address. You should also keep in mind that the smaller the range, the faster the time it takes to discover members since each node has to scan a lesser number of potential members.Click to see the instructions for AWS ECS membership scheme
-
Create a working AWS ECS Cluster. Note the following when creating a cluster.
- Note the
name
andVPC CIDR block
of the cluster as you will require them later for configurations. - Ensure that the
Container instance IAM role
that you assign to the ECS cluster has the following permission policy attached.{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "ec2:DescribeAvailabilityZones", "ec2:DescribeInstances" ], "Resource": [ "*" ] } ] }
- Make sure that the security group of the cluster instances has an inbound rule to allow incoming
traffic on the Hazelcast default port range
(5701 - 5708)
. It is advised to restrict the access to instances in the same security group for this inbound rule.
- Note the
-
Create a
deployment.toml
file in a preferred directory to add following configurations. Configure the following entries.
Under the[clustering] membership_scheme = "aws-ecs" [clustering.properties] region = "us-east-1" clusterName = "ECS-IS-CLUSTER" vpcCidrBlock = "10.0.*.*"
clustering.properties
section, set theregion
,clusterName
, andvpcCidrBlock
based on the AWS ECS cluster that you created in the previous step.
Note
Once all the configurations are complete, build a docker image including the configurations. You can consume this docker image to create a
Task Definition
and run a newService
or aTask
on theAWS ECS cluster
that you created. -
-
Configure caching.
Note
From WSO2 Identity Server 5.2.0 onwards, distributed caching is disabled and it is not recommended to use this due to many practical issues that are related to configuring and running distributed caching properly. WSO2 Identity Server employs Hazelcast as the primary method of implementing cluster messages while using distributed caching in a simple setup.
About Caching
- Why cachingCaching is an additional layer on top of the databases. It enables to keep the recently used data that are fetched from the database in local memory, so that for subsequent data requests instead of fetching from the database the data can be served from the local memory. Caching has certain advantages and disadvantages that you need to evaluate when deciding on your caching strategy.
- Advantages
- The load on the underlying database or LDAP is reduced as data is served from already fetched data in memory.
- Improved performance due to the reduced number of database calls for repetitive data fetching.
- Disadvantages
- Coherency problems may occur when the data change is not immediately reflected on cached data if one node or an external system updates the database.
- Data in memory can become stale yet be served, e.g., serving data from memory while its corresponding record in the database is deleted.
Caching in WSO2 Identity Server
Historically WSO2 Identity Server used distributed caching to utilize the above-mentioned advantages as well as to minimize the coherence problem. However, in newer deployment patterns where the network is not tightly controlled, distributed caching fails in unexpected ways. Hence, we no longer recommend using distributed caching. Instead, it is recommended to have local caches (if required) and cache invalidation messages (if required) by considering the information given below.
- The ForceLocalCache property
When Hazelcast clustering is enabled certain caches act as distributed caches. The
force_local_cache
property in the<IS_HOME>/repository/conf/deployment.toml
directory is there to mark that all the caches should act like local caches even in a clustered setup. (This is by default set totrue
).
Cache invalidation uses Hazelcast messaging to distribute the invalidation message over the cluster and invalidate the caches properly. This is used to minimize the coherence problem in a multi-node setup.[server] force_local_cache = true
- Typical clustered deployment cache scenarios
Scenario Local Caching Distributed Caching Hazelcast Clustering Distributed Invalidation Description 1. All caches are local with distributed cache invalidation Enabled Not Applicable Enabled Enabled - This is the recommended approach.
- Hazelcast messaging invalidates the caches.
2. All caches are local without distributed cache invalidation Enabled Not Applicable Disabled Disabled - Invalidation clears only the caches in specific nodes. Other caches are cleared at cache expiration.
- Hazelcast communication is not used.
- As the decisions take time to propagate over nodes (default cache timeout is 15 minutes), there is a security risk in this method. To reduce the risk, reduce the default cache timeout period. To learn how to reduce the default cache timeout period, see Configuring Cache Layers - timeout.
3. No caching Disabled Disabled Disabled Disabled - The data are directly acquired from the database.
- Eliminates the security risks caused due to not having cache invalidation.
- This method will create a performance degradation due to the lack of caching.
4. Certain caches are disabled while the remaining are local Enabled for the available local caches Not Applicable Enabled Enabled - To reduce the security risk created in the second scenario and to improve performance in comparison with the third scenario, disable the security-related caches and sustain the performance-related caches as local caches.
- This requires identification of these caches depending on the use case.
5. Distributed caching enabled Disabled—the force_local_cache
is set tofalse
.Enabled Enabled Not Applicable - This scenario is only recommended if the network has tight tolerance where the network infrastructure is capable of handling high bandwidth with very low latency.
- Typically this applies only when you deploy all the nodes in a single server rack having fiber-optic cables. In any other environments, this implementation will cause cache losses. Thus, this implementation is not recommended for general use.
-
Go to the
<IS_HOME>/repository/conf/deployment.toml
file and add the proxy port as443
. The port 443 is the Load Balancer frontend port.Example
[transport.http.properties] proxyPort = 80 [transport.https.properties] proxyPort = 443
-
You may change the
<IS_HOME>/repository/conf/deployment.toml
file to access the servers using a hostname instead of the raw IP. This hostname is the one the external applications try to look up WSO2 Identity Server endpoints. ThehostName
should be resolved to the Load Balancer front-end IP address.[server] hostname = "wso2.is"
Note
This hostname is used by the WSO2 Identity Server cluster and therefore it must be specified in the
/etc/hosts
file in each node so that internal calls will not be routed through the Load Balancer.Example:
192.168.2.1 wso2.is
Enabling artifact synchronization¶
To enable synchronization for runtime artifacts you must have a shared file system. You can use one of the following depending on your environment.
- Network File System (NFS): This is one of the most commonly known shared file systems and can be used in a linux environment.
- Server Message Block (SMB) file system: This can be used in a Windows environment.
- Amazon EFS: This can be used in an AWS environment.
Once you have chosen a file system,
- Mount it in the nodes that are participating in the cluster.
- If the userstores need to be updated at runtime, create a directory called
Userstores
in the shared file system and create a symlink from the<IS_HOME>/repository/deployment/server/userstores
path to theUserstores
directory. - If multi-tenancy is required, create a directory called
Tenants
in the shared file system and create a symlink from the<IS_HOME>/repository/tenants
path to theTenants
directory.
Note
Instead of mounting the file system directly to the <IS_HOME>/repository/deployment/server/userstores
and
<IS_HOME>/repository/tenants
paths, a symlink is created to avoid issues that may occur
if you delete the product to redeploy it, the file system would get mounted to a non-existing path.
Fronting with a load balancer¶
In order to access the two WSO2 Identity Server nodes, you need to front the system with a load balancer. You can use any load balancer that is available to your system.
Info
To learn how to front the 2 node cluster with an Nginx load balance, see Fronting with the Nginx load balancer. If you’ve changed the hostnames of nodes, make sure to keep that in mind when you’re configuring your load balancer.
Running the cluster¶
- Start the load balancer and WSO2 Identity Server nodes.
- Access the management console at
https://wso2.is/carbon/
. Traffic will be served by one of the nodes in the cluster, depending on your load balancer. - Shut down the cluster node 1 and observe that the traffic is served by node 2.
- Start node 1 and shut down node 2. Note that traffic will be served by node 1.