High Availability Pairing: User Provided TLS Credentials
High Availability Pairing
Eclipse Amlen allows two instances of the software to be put into an Active-Passive relationship to enable a fast failover mechanism to provide a Highly Available system. The connection between the two instances can optionally have TLS enabled. This uses credentials in the binary to secure the connection for initial pairing. If a pair has been created it is highly recommended to switch to user provided credentials as the default credentials can be read from the open source repository.
For details on how to create credentials see TLS Authentication Methods.
Before creating a High Availability Pair place the certificate, key and optionally the certificate authority on both servers. For Eclipse Amlen the default location is: /var/lib/amlen-server/data/certificates/keystore The key must be called ha_key.pem and must be in pem format. The certificate must be called ha_cert.pem and must be in pem format. The keystore location can be changed in the server.cfg.
Certificate credentials can be disabled to allow servers using different credentials to form a pair. This is not recommended as a permanent solution, however can be used as a temporary solution during upgrade scenarios. The primary use-case is during upgrade from a version of IBM® Watson IoT Platform Message Gateway that does not support user provided credentials. This is done by setting RequireCertificates to false in the HighAvailability configuration of the Eclipse Amlen Instance. An example of this is:
curl -m 10 -u "${IMA_USER}" -k -X POST -d
'{ "HighAvailability":
{ "Group": "haPair1",
"EnableHA": true,
"StartupMode": "AutoDetect",
"RemoteDiscoveryNIC": "10.0.1.24",
"LocalReplicationNIC": "10.0.1.25",
"LocalDiscoveryNIC": "10.0.1.26",
"PreferredPrimary": true,
"UseSecuredConnections":true,
"RequireCertificates": false,
}
}' "${IMA_ADMIN}/configuration"
If user provided credentials are supplied then certificates are always required and the RequireCertificates setting will be ignored
Migrating HA Pairs to New User Provided Credentials
This procedure can be used for migrating a High Availability Pair from either the default credentials or from previous user provided credentials.
For Eclipse Amlen the default location for the credentials is: /var/lib/amlen-server/data/certificates/keystore. The key must be called HA_key.pem and must be in pem format. The certificate must be called HA_cert.pem and must be in pem format. The keystore location can be changed in the server.cfg.
Support for pairing between IBM Watson IoT Platform Message Gateway and Eclipse Amlen requires the latest iFix from IBM Watson IoT Platform Message Gateway 5.0.0.2 only. On IBM Watson IoT Platform Message Gateway the default location for the key and certificate is:/var/messagesight/data/certificates/keystore
For details on how to create credentials see TLS Authentication Methods.
The procedure is:
- Place the key and certificate on the standby instance and define the truststore if using Certificate Authorities
- Restart the standby instance. The standby instance will not become active in the pair. Example results from HighAvailability will be:
curl -m 10 -u "${IMA_USER}" -k -X GET "${IMA_ADMIN}/service/status/HighAvailability" Response from the Primary: { "Version":"v1", "HighAvailability": { "Status": "Active", "Enabled": true, "Group": "haPair1", "NewRole": "PRIMARY", "OldRole": "PRIMARY", "ActiveNodes": 1, "SyncNodes": 1, "PrimaryLastTime": "2021-05-05T16:48:22Z", "PctSyncCompletion": -1, "ReasonCode": 0, "RemoteServerName":"3ac46d703cde:9089" } } Running the command again against the rcently restarted secondary: curl -u "${IMA_USER}" -k -X GET "${IMA_ADMIN}/service/status/HighAvailability" Response: { "Version":"v1", "HighAvailability": { "Status": "Active", "Enabled": true, "Group": "haPair1", "NewRole": "UNSYNC", "OldRole": "UNSYNC", "ActiveNodes": 0, "SyncNodes": 0, "PrimaryLastTime": "", "PctSyncCompletion": -1, "ReasonCode": 0, "RemoteServerName":"" } } - Place the key and certificate on the primary instance and define the truststore if using Certificate Authorities.
-
Restart the primary instance
The initial HighAvailability status will show:
The primary instance will restart and remain the primary instance
The unsynchronized instance will start to synchronize and once completed will become a standby.
Once the procedure has completed, they should return to:#Running the command against the primary curl -m 10 -u "${IMA_USER}" -k -X GET "${IMA_ADMIN}/service/status/HighAvailability #Response: "{ "Version":"v1", "HighAvailability": { "Status": "Active", "Enabled": true, "Group": "haPair1", "NewRole": "PRIMARY", "OldRole": "UNSYNC", "ActiveNodes": 2, "SyncNodes": 1, "PrimaryLastTime": "2021-05-05T16:48:39Z", "PctSyncCompletion": 1, "ReasonCode": 0, "RemoteServerName":"" } } #Running the command against the secondary curl -m 10 -u "${IMA_USER}" -k -X GET "${IMA_ADMIN}/service/status/HighAvailability" #Response: { "Version":"v1", "HighAvailability": { "Status": "Active", "Enabled": true, "Group": "haPair1", "NewRole": "UNSYNC", "OldRole": "UNSYNC", "ActiveNodes": 2, "SyncNodes": 0, "PrimaryLastTime": "", "PctSyncCompletion": -1, "ReasonCode": 0, "RemoteServerName":"" } }curl -m 10 -u "${IMA_USER}" -k -X GET "${IMA_ADMIN}/service/status/HighAvailability" { "Version":"v1", "HighAvailability": { "Status": "Active", "Enabled": true, "Group": "haPair1", "NewRole": "PRIMARY", "OldRole": "PRIMARY", "ActiveNodes": 2, "SyncNodes": 2, "PrimaryLastTime": "2021-05-05T17:16:07Z", "PctSyncCompletion": -1, "ReasonCode": 0, "RemoteServerName":"3ac46d703cde:9089" } } curl -m 10 -u "${IMA_USER}" -k -X GET "${IMA_ADMIN}/service/status/HighAvailability" { "Version":"v1", "HighAvailability": { "Status": "Active", "Enabled": true, "Group": "haPair1", "NewRole": "STANDBY", "OldRole": "UNSYNC", "ActiveNodes": 2, "SyncNodes": 2, "PrimaryLastTime": "", "PctSyncCompletion": -1, "ReasonCode": 0, "RemoteServerName":"3704cacfea12:9089" } }