Products » Connect2id server » Documentation » Configuration »

Database, caching and clustering configuration

The Connect2id server utilises Infinispan to:

Store its data, such as client registrations, supporting a variety of database technologies, including low-latency in-memory storage for short-lived and cached objects on the JVM heap or in Redis.
Cluster two or more Connect2id server nodes together.

1. Supported databases

Choose a database for persisting Connect2id server data:

Database	Version	Type	Deployment
MySQL	5.7.8+	SQL	On premise Cloud
PostgreSQL	9.5+	SQL	On premise Cloud
Microsoft SQL Server	2016+	SQL	On premise Cloud
Oracle Database	12c r1+	SQL	On premise Cloud
DynamoDB	n/a	NoSQL	AWS
H2	1.4+	SQL	Embedded

Notes:

MySQL: You can also use a compatible database, such as MariaDB 10.2.7+ or AWS Aurora.
PostgreSQL: You can also use a compatible database, such CockroachDB.
The lightweight relational H2 database is intended for testing and development purposes. Out of the box the Connect2id server is configured with an embedded H2 database instance.
To ensure the high-availability and durability of your Connect2id server data, use a database cluster set up for replication and fail-over.

2. Supported cluster modes

Choose a cluster mode for your scaling needs:

	Replication Mode	Stateless Mode
Server nodes	1 - 2	1 - ∞
Storage of short-lived and cached data	Local: JVM heap	External: Redis or the main database
Ideal for	small to medium loads predictable loads	small to large loads quick up/down scaling

To make your OpenID and OAuth 2.0 service highly-available deploy a Connect2id server cluster with at least two nodes. A single node handles in the order of 100 to 300 OpenID Connect requests (in the code flow) per CPU core per second. Server nodes can be added and removed dynamically to / from the cluster.

Replication mode -- Ideal for medium, predictable loads. For improved performance sessions and other short-lived objects are stored in Infinispan on the JVM heap. Persisted server data that is frequently used is also cached there. The in-memory data is continuously replicated between the nodes. The JVM on each node should be provisioned with sufficient heap capacity for the expected maximum number of sessions and other objects stored in memory.

Stateless mode -- Ideal for large loads, allows quick up / down cluster scaling. The Connect2id server state is stored entirely in the underlying database(s).

Basic -- Using the configured SQL or DynamoDB database to store all short and long-lived server data.
Redis -- Improves performance and latency by using Redis as an additional in-memory database to cache frequently used server data and store sessions and other short-lived objects. The Redis instance(s) should be provisioned with sufficient memory to handle peak usage.

Standalone -- Intended primarily for Connect2id server testing and development. Clustering is disabled, data is persisted to an embedded relational H2 database, caching is on the JVM heap. This is the out-of-the-box configuration.

One way to estimate your memory requirements is to load the Connect2id server with a number of simulated sessions containing typical data via the session store API.

3. Configuration

When you've chosen a backend database and cluster mode the actual set up is as simple as:

Link to the appropriate Infinispan XML configuration file.
Configure the database connection.
If the replication cluster mode is selected, allow inbound port access for the sync messages.
Configure a health check URL.

The four steps are explained below.

3.1 Link to the Infinispan XML configuration file

The Connect2id server comes with a set of Infinispan configuration files for each cluster mode and database combination in the WEB-INF directory of its c2id.war package.

Cluster in replication mode:

/WEB-INF/infinispan-replication-mysql.xml
/WEB-INF/infinispan-replication-postgres95.xml
/WEB-INF/infinispan-replication-sqlserver.xml
/WEB-INF/infinispan-replication-oracle.xml
/WEB-INF/infinispan-replication-dynamodb.xml

Cluster in stateless mode:

/WEB-INF/infinispan-stateless-mysql.xml
/WEB-INF/infinispan-stateless-postgres95.xml
/WEB-INF/infinispan-stateless-sqlserver.xml
/WEB-INF/infinispan-stateless-oracle.xml
/WEB-INF/infinispan-stateless-dynamodb.xml

Cluster in stateless mode, Redis used to store the short-lived objects and cache long-lived objects from the database:

/WEB-INF/infinispan-stateless-redis-mysql.xml
/WEB-INF/infinispan-stateless-redis-postgres95.xml
/WEB-INF/infinispan-stateless-redis-sqlserver.xml
/WEB-INF/infinispan-stateless-redis-oracle.xml

Cluster in stateless mode, Redis used to store the short-lived objects and DynamoDB to store the long-lived (note, transparent caching of the long-lived objects in DynamoDB can be enabled with DAX):
```
/WEB-INF/infinispan-stateless-redis-dynamodb.xml
```
Local (no clustering) mode:
```
/WEB-INF/infinispan-local-h2.xml
```

Out of the box the Connect2id server is configured in local mode with an embedded H2 database:

/WEB-INF/infinispan-local-h2.xml

For the multi-tenant Connect2id server edition:

Cluster in stateless mode:

/WEB-INF/infinispan-multitenant-stateless-mysql.xml
/WEB-INF/infinispan-multitenant-stateless-postgres95.xml
/WEB-INF/infinispan-multitenant-stateless-sqlserver.xml
/WEB-INF/infinispan-multitenant-stateless-oracle.xml
/WEB-INF/infinispan-multitenant-stateless-dynamodb.xml

Cluster in stateless mode, Redis used to store the short-lived objects and cache long-lived objects from the database:

/WEB-INF/infinispan-multitenant-stateless-redis-mysql.xml
/WEB-INF/infinispan-multitenant-stateless-redis-postgres95.xml
/WEB-INF/infinispan-multitenant-stateless-redis-sqlserver.xml
/WEB-INF/infinispan-multitenant-stateless-redis-oracle.xml

Local (no clustering) mode:

/WEB-INF/infinispan-multitenant-local-h2.xml

To switch to the configuration of your choice, e.g. stateless cluster mode with a PostgreSQL database, you have two possibilities:

Set the infinispan.configurationFile Java system property:

-Dinfinispan.configurationFile=/WEB-INF/infinispan-stateless-postgres95.xml

Or, edit the infinispan.configurationFile context parameter in /WEB-INF/web.xml:

<context-param>
    <description>
        The location of the Infinispan configuration file. Must be
        relative to the web application root directory.
    </description>
    <param-name>infinispan.configurationFile</param-name>
    <param-value>/WEB-INF/infinispan-stateless-postgres95.xml</param-value>
</context-param>

3.2 Configure the database connection

3.2.1 MySQL

To persist data to an MySQL 5.7.8+ database:

Create а new database, the character set must be UTF8:
```
CREATE DATABASE c2id CHARACTER SET utf8
```
Set the JDBC properties in the Infinispan XML configuration, or better still, override them with system properties:
- dataSource.url -- the JDBC URL. The provided MariaDB JDBC driver supports MySQL, MariaDB and AWS Aurora specific configurations, with a set of fail-over and load-balancing modes. The default value is jdbc:mysql://localhost/c2id?useUnicode=yes&characterEncoding=UTF-8.
- dataSource.user -- the database user, the default value is c2id
- dataSource.password -- the database password
- dataSource.createTableIfMissing -- controls automatic table creation and table schema updating (when upgrading to a new major server version) on Connect2id server startup, the default value is true. Since v14.0.
- dataSource.maxPoolSize -- controls the maximum size the SQL connection pool is allowed to reach, including both idle and in-use connections. The default size is 5. Since v14.0.
The Connect2id server uses HikariCP to manage its SQL connection pool. The default pool settings are suitable for most deployments. To tune it add an appropriate property to the SQL store parameters in the Infinispan XML configuration.
If automatic table creation is disabled manually apply the SQL statements provided in WEB-INF/sql/create-table-statements-mysql.sql (available since v14.0) before starting the Connect2id server.

3.2.2 PostgreSQL

To persist data to a PostgreSQL 9.5+ database:

Create а new database, the character set must be UTF8:
```
CREATE DATABASE c2id ENCODING UTF8
```
Set the JDBC properties in the Infinispan XML configuration, or better still, override them with system properties:
- dataSource.host -- the database host name / IP address, the default value is localhost
- dataSource.user -- the database user, the default value is c2id
- dataSource.password -- the database password
- dataSource.databaseName -- the database name, the default value is c2id
- dataSource.createTableIfMissing -- controls automatic table creation and table schema updating (when upgrading to a new major server version) on Connect2id server startup, the default value is true. Since v14.0.
- dataSource.maxPoolSize -- controls the maximum size the SQL connection pool is allowed to reach, including both idle and in-use connections. The default size is 5. Since v14.0.
The Connect2id server uses HikariCP to manage its SQL connection pool. The default pool settings are suitable for most deployments. To tune it add an appropriate property to the SQL store parameters in the Infinispan XML configuration.
If automatic table creation is disabled manually apply the SQL statements provided in WEB-INF/sql/create-table-statements-postgres.sql (available since v14.0) before starting the Connect2id server.

3.2.3 Microsoft SQL Server

To persist data to an SQL Server 2016+ database:

Create а new database, the collation is irrelevant, the character should be the default UCS-2 / UTF-16 supporting i18n or may be set to UTF8 (supported from SQL Server version 2019 on):
```
CREATE DATABASE c2id
```
Set the JDBC properties in the Infinispan XML configuration, or better still, override them with system properties:
- dataSource.url -- the JDBC URL, the default value is jdbc:sqlserver://localhost:1433;databaseName=c2id;applicationName=c2id;
- dataSource.user -- the database user, the default value is c2id
- dataSource.password -- the database password
- dataSource.createTableIfMissing -- controls automatic table creation and table schema updating (when upgrading to a new major server version) on Connect2id server startup, the default value is true. Since v14.0.
- dataSource.maxPoolSize -- controls the maximum size the SQL connection pool is allowed to reach, including both idle and in-use connections. The default size is 5. Since v14.0.
The Connect2id server uses HikariCP to manage its SQL connection pool. The default pool settings are suitable for most deployments. To tune it add an appropriate property to the SQL store parameters in the Infinispan XML configuration.
If automatic table creation is disabled manually apply the SQL statements provided in WEB-INF/sql/create-table-statements-sqlserver.sql (available since v14.0) before starting the Connect2id server.

3.2.4 Oracle Database

To persist data to an Oracle 12c r1+ database:

Create а new database. The configured national database character set must be UTF8 (NLS_NCHAR_CHARACTERSET=AL32UTF8).
```
CREATE DATABASE c2id
```
Set the JDBC properties in the Infinispan XML configuration, or better still, override them with system properties:
- jdbc.url -- the JDBC URL, the default value is jdbc:oracle:thin:@//localhost:1521/XE
- dataSource.user -- the database user, the default value is c2id
- dataSource.password -- the database password
- dataSource.databaseName -- the database name, the default value is c2id
- dataSource.createTableIfMissing -- controls automatic table creation and table schema updating (when upgrading to a new major server version) on Connect2id server startup, the default value is true. Since v14.0.
- dataSource.maxPoolSize -- controls the maximum size the SQL connection pool is allowed to reach, including both idle and in-use connections. The default size is 5. Since v14.0.
The Connect2id server uses HikariCP to manage its SQL connection pool. The default pool settings are suitable for most deployments. To tune it add an appropriate property to the SQL store parameters in the Infinispan XML configuration.
If automatic table creation is disabled manually apply the SQL statements provided in WEB-INF/sql/create-table-statements-oracle.sql (available since v14.0) before starting the Connect2id server.

3.2.5 DynamoDB

To persist data to DynamoDB in the AWS cloud:

Set the connection properties in the Infinispan XML configuration, or better still, override them with system properties:
- dynamodb.region -- the preferred AWS region, e.g. us-east-1.
  
  If the system property is left undefined the default AWS region provider chain will take over to determine the region, using alternative methods like reading the AWS_REGION environment variable.
  
  The default AWS region provider chain is helpful in cases like deployments in AWS EKS or where the AWS access credential is a web identity token obtained from the AWS STS.
  
  Note, prior to v12.10 the system property had a default value of us-east-1, preventing the default AWS region provider chain from kicking in.
- dynamodb.httpProxyHost, dynamodb.httpProxyPort -- optional proxy for accessing the DynamoDB endpoint.
- dynamodb.encryptionAtRest -- set to true to enable server-side encryption. The default value is false.
- dynamoDB.hmacSHA256Key -- optional 256-bit secret key (BASE64-encoded) for applying client-side SHA-256 HMAC protection (integrity and authenticity check) to the stored DynamoDB items. The HMAC will be appended to each item in a _hmac#256 binary attribute.
  
  To generate a key with OpenSSL:
```
openssl rand -base64 32
```
  If a retrieved DynamoDB item is missing the _hmac#256 attribute or its verification fails the Connect2id server will produce an HTTP 500 Internal Server Error, log the error with a "DS0131" code and increment the <cache-name>.dynamoDB.invalidItemHmacCounter metric.
  
  The HMAC protection must be enabled for newly provisioned or empty DynamoDB tables. Enabling the protection for DynamoDB tables with prior existing items will cause HMAC validation errors on their retrieval.
- dynamodb.enableStream -- set to true to enable streaming for global tables. The default value is false.
- dynamodb.enableContBackups -- set to true to enable continuous backups / point-in-time recovery for all tables where crucial or long-lived data is persisted: id_access_tokens, long_lived_authorizations, revocation_journal, clients, federation_clients and tenants (in the multi-tenant Connect2id server edition). Applied at Connect2id server startup on new table creation as well as for existing tables. The default value is false. Since v12.15.
- dynamodb.tablePrefix -- optional prefix for all tables created by the Connect2id server, defaults to none.
- dynamodb.applyRangeKey -- optional range (sort) key, enables sharing of the Connect2id server DynamoDB tables between multiple server deployments. If set the value should be tid. Not available in the multi-tenant Connect2id server edition.
- dynamodb.rangeKeyValue -- the value of the optional range key, should uniquely identify the deployment. Not available in the multi-tenant edition of the Connect2id server.
- dynamodb.enableTTL.sessionStore.sessionMap -- set to true to enable automatic expiration of subject sessions by DynamoDB (via a "ttl" attribute). Also set the sessionStore.sessionMap.expirationInterval Java system property to zero (0) to disable the Infinispan expiration thread. Expiration may be moved to DynamoDB if there are no relying parties registered for OpenID Connect back-channel logout notifications, as session expiration notifications can no longer be delivered to them.
Make sure the AWS credentials for accessing the DynamoDB tables are configured in way that the default AWS credentials provider chain can look them up, ideally via IAM instance profile roles.

The Connect2id server requires access to the following DynamoDB actions:
- "dynamodb:BatchGetItem"
- "dynamodb:BatchPutItem"
- "dynamodb:CreateTable"
- "dynamodb:DeleteItem"
- "dynamodb:DescribeTable"
- "dynamodb:GetItem"
- "dynamodb:PutItem"
- "dynamodb:Query"
- "dynamodb:Scan"
- "dynamodb:UpdateItem"
Note, when the AWS access credential is a web identity token obtained from the AWS STS the dynamodb.region system property must be left undefined in order to let the default AWS region provider chain determine the region, typically by reading the AWS_REGION environment variable. The region of the AWS STS endpoint is set by the AWS_STS_REGIONAL_ENDPOINT environment variable.

Example to set the DynamoDB and STS endpoints to the same region:
```
AWS_REGION=eu-west-2
AWS_STS_REGIONAL_ENDPOINTS=regional
```
The Connect2id server will automatically create the tables it requires when it accesses DynamoDB for the first time.
After the DynamoDB tables are created enable auto-scaling for each table.
If you want to run Connect2id server instances in two or more AWS regions, replicating data between them asynchronously, set up global DynamoDB tables.
Note, data in DynamoDB tables can be transparently cached for quicker access via DAX.

3.2.6 H2

To persist data to an H2 2.2+ database:

Set the JDBC properties in the Infinispan XML configuration, or better still, override them with system properties:
- dataSource.user -- the database user, the default value is c2id
- dataSource.password -- the database password
- dataSource.createTableIfMissing -- controls automatic table creation and table schema updating (when upgrading to a new major server version) on Connect2id server startup, the default value is true
The Connect2id server uses the default settings of its Hikari JDBC connection pool. They can be tuned by adding an appropriate property to the SQL store parameters in the Infinispan XML configuration.
If automatic table creation is disabled manually apply the SQL statements provided in WEB-INF/sql/create-table-statements-h2.sql (available since v14.0) before starting the Connect2id server.

3.2.7 Redis

If the Connect2id server is deployed in stateless cluster mode where short-lived and cached data will be stored in Redis:

We recommend that you set up two separate Redis servers:
- One Redis server for transient data, such as sessions. This Redis server should be provisioned with enough memory to hold all current sessions and other objects in it. Make sure the eviction policy of this Redis server is set to volatile-lru:
```
maxmemory-policy volatile-lru
```
  An practical way to estimate the Redis memory requirements is to load the Connect2id server with N sample sessions via the session POST API and then check the resulting memory usage.
  
  The Redis store for transient data may be additionally configured to persist the data to disk using two strategies - periodic snapshots (RDB) and / or write log (AOF).
- Another Redis server for caching data only. Configure it with an allkeys-lru policy, to let Redis evict the least recently keys when the maxmemory limit is reached. There is no particular recommendation how much memory to allocate to the caching Redis server. See what works best for you in terms of service latency and budget.
```
maxmemory-policy allkeys-lru
```
  Persisting the cached data is not necessary and brings no benefits.
- The two Redis servers must be configured with 24 logical databases each. Every Redis database will be dedicated to store a particular Connect2id server object type.
```
databases 24
```
  Database zero (0) will never be used, to prevent a conflict in AWS ElastiCache which inserts a special ElastiCacheMasterReplicationTimestamp key in the default database (at zero index) to aid replication.
- Note that the Redis cluster topology is not supported because it cannot handle logical databases.
Set the Redis connection properties in the Infinispan XML configuration, or better still, override them with system properties:
- redisMapHost, redisMapPort -- to point to the host and port of the Redis server for storing transient data, such as sessions.
- redisCacheHost, redisCachePort -- to point to the host and port of the Redis server for cached data.
- redisMapPassword, redisCachePassword -- optional password to access Redis. Since v13.0.

3.3 Open inbound ports (replication cluster mode)

TCP

If you're going to run the Connect2id server in a replication cluster mode with an SQL database or DynamoDB allow inbound TCP traffic to the following ports:

port 7800
port 7900 through 7905

These ports are required by the clustering layer (JGroups) to receive sync messages (7800) and detect node failures (7900-7905).

The Connect2id server will pick a "site local" (non-routable) IP address on the host machine to attach the TCP listening sockets, e.g. from the 192.168.0.0 or 10.0.0.0 address range. To specify the address otherwise set the jgroups.tcp.address Java system property, e.g. like this:

-Djgroups.tcp.address=match-interface:eth0

Check out the bind_addr section in the JGroups docs for information.

If for some reason IPv6 networking gets used by mistake, you can force the JVM to use IPv4:

-Djava.net.preferIPv4Stack=true

3.4 Health check URL

Configure your load balancer / reverse proxy to use the health-check endpoint for determining a node's status. The health-check guide has helpful information.

4. How to override an Infinispan XML configuration setting with a Java system property

Any ${property-name:default-value} placeholder found in the Infinispan XML configuration file can be overridden with a Java system property:

In this example, if a Java system property named dataSource.password is found, it will be used to set the database password, otherwise the default value ("secret") will apply:

<property name="dataSource.password">${dataSource.password:secret}</property>

The external configuration guide has tips for setting system properties from environment variables, local files and other locations.

5. FAQ

5.1 How to add a new node to a Connect2id server cluster?

Simply start the new node and let the load balancer direct HTTP requests to it once the health check endpoint becomes available and returns status 200.

If your cluster is in replication mode, the joining node will automatically discover the coordinator (via the configured discovery protocol, e.g. JDBC_PING) and start communicating with it and the other nodes.

If your cluster is in stateless mode, the nodes don't sync data between them, and hence there is no need for them to discover one another.

5.2 How to remove a node from a Connect2id server cluster?

Make the load balancer stop directing HTTP requests to it, then shut down the node graciously. On Tomcat you can do this by calling its bin/shutdown.sh script.

5.3 How to track the cluster size?

For a cluster in replication mode check out the infinispan.numClusterMembers metric.

The gauge will get incremented when a new node joins, decremented when a node leaves.

5.4 Where are the cluster events and errors logged?

The Infinispan and JGroups messages are appended to the log of the Connect2id server.

Check out our guide how to interpret Infinispan and JGroups messages written to the log.

5.5 What JGroups protocols are used?

In a replication mode the JGroups clustering layer is configured to use the following protocols:

For message transport:
- TCP -- for Connect2id server setups with an SQL database or DynamoDB.
For initial node discovery:
- JDBC_PING -- for setups with an SQL database.
- DYNAMODB_PING -- for setups with DynamoDB backend database.

5.7 Is WAN / cross-datacentre replication supported?

Yes. Full cross-datacentre replication can be implemented in the "stateless" mode when configured with a DynamoDB backend database and global tables.

Partial replication is possible in the "stateless" mode with an SQL backend, for the persisted data, such as client registrations and long-lived authorisations. An OAuth 2.0 or OpenID Connect client application would not be able to switch seamlessly between regions, e.g. in the middle of an OAuth 2.0 flow. Its credentials and authorisations (refresh tokens), however, will work. The subject sessions established with the IdP can be replicated, but the Connect2id server must be configured to persist them to the SQL store. Normally the subject sessions are kept in memory (Redis) only, for quick access.

Contact support to receive further assistance.

5.8 How to size the SQL connection pool?

The default maximal size of the SQL connection pool to the configured database is 5. This value is sufficient for a Connect2id server using 2 CPU cores.

Check the HikariCP guide for how the connection pool is optimally sized.

The Connect2id server exposes metrics to monitor the SQL connection pool. Note that all *.sqlStore.pool.* metrics are sourced from the same connection pool.

6. Description of the Infinispan maps and caches

Here is a description of the individual Infinispan key / value maps and caches used by the Connect2id server. This information can be useful in order to fine tune the storage parameters.

Key / value maps that must be persisted:

clients.registrationsMap -- The registered OAuth 2.0 / OpenID Connect clients. For optimal performance aim to have all client entries cached in memory.
federation.registrationsMap -- The registered OpenID Connect Federation 1.0 clients. For optimal performance aim to have all client entries cached in memory.
authzStore.longLivedAuthzMap -- The long-lived (sticky) OAuth 2.0 authorisations (consent), including the associated refresh tokens. Caching of these entries speeds up processing of login requests as well as refresh token lookup.
authzStore.revocationJournalMap -- The authorisation revocation journal.
tenantRegistry.tenants -- The Connect2id server tenants. Applies to the multitenant edition of the server only.

Key / value maps where persistence is recommended, but not critical:

authzStore.accessTokenMap -- The active identifier-based OAuth 2.0 access tokens. If only self-contained (JWT-encoded) access tokens are issued, this map will not be utilised. If full persistence is not configured, the map should use passivation to move entries evicted from memory to the backend database.
sessionStore.sessionMap -- The end-user sessions with the Connect2id server. If full persistence is not configured, the map should use passivation to the backend database.
sessionStore.subjectMap -- An index of the sessions for each logged in end-user. If full persistence is not configured, the map should use passivation to the backend database.

Key / value maps and caches where persistence generally isn't needed:

authzStore.codeMap -- The pending OAuth 2.0 authorisation codes awaiting exchange for a token. If stored in memory passivation is recommended.
authzStore.expendedTokens -- The keys of expended rotated self-contained (JWT-encoded) refresh tokens and expended initial client registration tokens.
op.authSessionMap -- The authentication session state for each end-user who has a login page currently opened. If stored in memory enough capacity should be allocated to accommodate the expected number of open login pages at any one time, else passivation must be configured.
op.consentSessionMap -- The consent session state for each end-user who has an login page currently opened. If stored in memory enough capacity should be allocated to accommodate the expected number of open login pages at any one time, else passivation must be configured.
clients.remoteJWKSetCache -- Caches remote client JWK sets referenced by URL.
clients.remoteRequestJWTClaimsCache -- Caches the claims of validated OAuth 2.0 request objects referenced by URL.