Clustering: User Provided TLS Credentials

Clustering

Eclipse Amlen allows multiple instances of the software to be put into a cluster to provide scalability. The connection between the two instances can optionally have TLS enabled. This uses credentials in the binary to secure the connection for initial pairing. If a cluster has been created using the default credentials it is recommended to switch to user provided credentials as the default credentials can be read from the open source repository.

Before creating a cluster place the certificate, key and optionally the certificate authority on each server. For Eclipse Amlen the default location is: /var/lib/amlen-server/data/certificates/keystore The key must be called Cluster_key.pem and must be in pem format. The certificate must be called Cluster_cert.pem and must be in pem format.The keystore location can be changed in the server.cfg.

For details on how to create credentials see TLS Authentication Methods.

Certificate credentials can be disabled to allow servers using different credentials to form a cluster. This is not recommended as a permanent solution, however can be used as a temporary solution during upgrade scenarios. The primary use-case is during upgrade from a version of IBM® Watson IoT Platform Message Gateway that does not support user provided credentials. This is done by setting RequireCertificates to false in the ClusterMembership configuration of the Eclipse Amlen Instance. An example of this is:


        curl -X POST    -d  "{
              \"ClusterMembership\": {
                    \"EnableClusterMembership\": true,
                    \"ControlAddress\": \"10.0.1.5\",
                    \"MessagingAddress\": \"10.0.1.5\",
                    \"DiscoveryServerList\":\"10.0.1.5:9104,10.0.1.4:9104\",
                    \"UseMulticastDiscovery\": false,
                    \"ClusterName\": \"MyCluster\",
                    \"MessagingUseTLS\":true,
                    \"CertificatesRequired\":false}
              }"  http://127.0.0.1:9089/ima/v1/configuration

If user provided credentials are supplied then certificates are always required and the RequireCertificates setting will be ignored

Migrating A Cluster To New User Provided Credentials

This procedure can be used for migrating a Cluster from either the default credentials or from previous user provided credentials.

For Eclipse Amlen the default location for the credentials is: /var/lib/amlen-server/data/certificates/keystore. The key must be called Cluster_key.pem and must be in pem format. The certificate must be called Cluster_cert.pem and must be in pem format. The keystore location can be changed in the server.cfg.

Support for clustering between IBM Watson IoT Platform Message Gateway and Eclipse Amlen requires the latest iFix from IBM Watson IoT Platform Message Gateway 5.0.0.2 only. On IBM Watson IoT Platform Message Gateway the default location for the key and certificate is:/var/messagesight/data/certificates/keystore

For details on how to create credentials see TLS Authentication Methods.

During this procedure, instance will in turn be removed form the cluster, they will not be able to re-join the cluster until all instances have been restarted.

The procedure is:

  1. Place the credentials (key, certificate and cafile if appropriate) on all instances.
  2. If using High Availability pairing restart all standby instances wait for them to resynchronize.
  3. Select one instance as the last instance to restart.
  4. Restart each instance in turn checking that the cluster status on the last instance has the disconnected server count incremented: 
    
               curl -m 10 -u "${IMA_USER}" -k -X GET "${IMA_ADMIN}/service/status/Cluster"
                    {  "Version":"v1",
                       "Cluster": {
                           "Status": "Active",
                           "Name": "MyCluster",
                           "Enabled": true,
                           "ConnectedServers": 2,
                           "DisconnectedServers": 1
                       }
                    }
  5. Restart the last instance
  6. Wait for the cluster status to be available on the last instance and check that the connected server count is back to what it started as: 
    
                curl -m 10 -u "${IMA_USER}" -k -X GET "${IMA_ADMIN}/service/status/Cluster"
                    {  "Version":"v1",
                       "Cluster": {
                           "Status": "Active",
                           "Name": "MyCluster",
                           "Enabled": true,
                           "ConnectedServers": 3,
                           "DisconnectedServers": 0 
                        }
                    }