Table of Contents

Configuring multiple Elasticsearch clusters

Important

Elasticsearch is only supported up to version 6.8, which is no longer supported by Elastic. We therefore recommend using Storage as a Service instead, or if you do want to continue using self-hosted storage, using OpenSearch.

Note

This procedure can be followed both on Linux and Windows setups. However, we highly recommend using Linux.

From DataMiner 10.2.0/10.1.3 onwards, you can have data offloaded to multiple Elasticsearch clusters, i.e. one main cluster and several replicated clusters. Data is always read from the main cluster, but data updates are sent to all clusters.

Configuring multiple Elasticsearch clusters becomes crucial when on-premises setups are considered, as Elastic strongly advises against setting up clusters across high-latency nodes. Rather than spreading Elasticsearch nodes across locations with high latency, it is preferable to offload data to multiple Elasticsearch clusters. This approach allows for geo-redundancy, ensuring that data remains accessible even if one cluster becomes temporarily unavailable. With this configuration, if inconsistencies arise between the Elasticsearch clusters, synchronization issues can be avoided.

Important
  • A setup with multiple Elasticsearch clusters is only recommended for high latency scenarios. For low or medium latency, we strongly advise using the standard Elasticsearch setup.
  • Contact Skyline if you are unsure about the setup that best fits your circumstances.
Tip

For more information, see Failover setups (with geo-redundancy).

To configure this setup:

  1. In the C:\Skyline DataMiner\ folder, open DB.xml.

  2. For each Elasticsearch cluster, add a DataBase tag with the following configuration:

    • active attribute: Set this to "true" if you want your Elasticsearch cluster to be added to the list of clusters to offload data to.

    • search attribute: Always set this to "true".

    • ID attribute: This should be a unique number, preferably starting at 0 and increasing by 1 for each additional cluster.

    • priorityOrder attribute: Indicates the priority of the different clusters. The lower the value, the greater the priority. The cluster with the lowest value is the main cluster, from which data will be read.

    • type attribute: Always set this to "ElasticSearch".

    • DBServer: The hosts of the cluster, separated by commas (see Indexing database settings)

    • UID: The username to connect to Elasticsearch (see Specifying custom credentials for Elasticsearch).

    • PWD: The password to connect to Elasticsearch (see Specifying custom credentials for Elasticsearch).

    • DB: The prefix for the Elasticsearch indexes (see Specifying a custom prefix for the Elasticsearch indexes).

    • FileOffloadIdentifier: String used to identify this connection. Each connection should have a different identifier, which will be used for file offloads.

    Example:

    <DataBases>
     <!-- Reads will be handled by the ElasticCluster with the lowest priorityOrder -->
     <DataBase active="true" search="true" ID="0" priorityOrder="0" type="ElasticSearch">
         <DBServer>localhost</DBServer>
         <UID />
         <PWD>root</PWD>
         <DB>dms</DB>
         <!--
         Used by file offload (i.e. when the connection with the Elastic cluster is not
         available) for tagging the offloaded data with.
        Should be different for each defined Elastic cluster connection.
         -->
         <FileOffloadIdentifier>cluster1</FileOffloadIdentifier>
     </DataBase>
     <DataBase active="true" search="true" ID="1" priorityOrder="1" type="ElasticSearch">
         <DBServer>10.11.1.44,10.11.2.44,10.11.3.44</DBServer>
         <UID />
         <PWD>root</PWD>
         <DB>dms</DB>
         <FileOffloadIdentifier>cluster2</FileOffloadIdentifier>        
     </DataBase>
</DataBases>

  3. Remove or disable any previous Elasticsearch configuration from DB.xml.

  4. Restart DataMiner.

Note

If an exception occurs for one of the replicated clusters, an alarm will be generated in the Alarm Console, indicating that not all data might be replicated. If further errors occur, no new alarms are created until the DMA is restarted.

Tip

See also: DB.xml.

Troubleshooting

During normal operation, DataMiner will offload the data written to both Elasticsearch clusters simultaneously.

If one of the clusters goes down, an error will be displayed in the Alarm Console to indicate that the indexing cluster is down: All nodes in the indexing cluster are down and/or All nodes in the replicated indexing cluster [IP1, IP2,...] are down.

If this happens, you need to alter the priorityOrder (see above) so that the backup indexing cluster becomes active. Specifically, this means setting priorityOrder="0" where the priorityOrder used to be 1 and the other way around. After you have done so, restart DataMiner. This should restore DataMiner to a functional state.

Important

If the main Elasticsearch cluster goes down, it will need to be restored afterwards. To ensure that the data is in sync, first follow the procedure on Verifying Elasticsearch synchronization, then follow the procedure to restore on Taking a snapshot of one Elasticsearch cluster and restoring it to another.

Propose changes