Table of Contents

Storage as a Service (STaaS)

With DataMiner Storage as a Service, you can connect your DataMiner System to a scalable, easy-to-use cloud-native storage platform, without the effort of setting up and managing your own storage databases.

STaaS

Advantages of DataMiner Storage as a Service (STaaS) include:

  • Fast, secure, and reliable storage

  • Easy and flexible configuration

  • Outstanding performance, scalability, and redundancy

  • Cost-effectiveness

  • Safe and secure data

Tip

If you have any questions or need support with the STaaS feature, contact staas@dataminer.services.

Setting up STaaS

For a self-hosted DataMiner System, follow the steps below to set up STaaS.

Note

  1. Upgrade your DataMiner System to DataMiner 10.4.0 [CU0]/10.4.1 or higher.

    Note

    To be able to use non-indexed logger tables, upgrade to DataMiner 10.4.0 [CU5]/10.4.8 or higher.

    Important

    We recommend always upgrading DataMiner to the latest available version to get the latest features and performance updates.

  2. Make sure your DataMiner System is connected to dataminer.services.

  3. Make sure that all Agents in your DataMiner System have internet access and are able to reach the following endpoints, depending on the data location that will be used:

    • STaaS West Europe: 20.76.71.123
    • STaaS UK South: 20.162.131.128
    Note

    All communication for STaaS happens through HTTPS. The DataMiner System initiates all outbound connections.

  4. Make sure you have at least DataMiner CloudGateway 2.8.0 installed on the system. See Upgrading nodes to the latest DxM versions.

  5. Register your system to use STaaS:

    1. Go to the Admin app.

    2. In the sidebar on the left, go to Organization > Overview.

    3. In the DataMiner Systems section, click the system you want to register.

    4. At the top of the page, in the Storage as a Service box, click the button Get Started with STaaS.

    5. Fill in your preferred region and click Initialize.

    Important

    Only owners of a DataMiner System can register their system.

  6. Optionally, migrate your existing data to STaaS.

    If you do so, wait until the migration has been completed and verified before continuing with this procedure.

  7. On each DataMiner Agent in the cluster, in the C:\Skyline DataMiner folder, open DB.xml and edit it corresponding to your setup:

    • For setups without proxy, use the following configuration:

      <?xml version="1.0"?>
<DataBases xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://www.skyline.be/config/db">
    <DataBase active="true" local="true" search="true" cloud="true" type="CloudStorage"/>
</DataBases>

    • For setups with proxy (this requires DataMiner 10.4.5 or higher), use the following configuration, filling in the fields as required:

      <?xml version="1.0"?>
<DataBases xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://www.skyline.be/config/db">
    <DataBase active="true" local="true" search="true" cloud="true" type="CloudStorage">
       <Proxy>
          <Address>[Enter Address Here]</Address>
          <Port>[Enter Port Here]</Port>
          <UserName>[Enter UserName Here]</UserName>
          <Password>[Enter Password Here]</Password>
       </Proxy>
    </DataBase>
</DataBases>
      Note

      If the proxy does not require authentication, you can leave the UserName and Password fields blank or remove them.

  8. Restart DataMiner to begin using STaaS.

Data location and redundancy

DataMiner STaaS relies on Azure Storage, which stores multiple copies of your data to make sure it is always available even in case outages or disasters occur. Different storage redundancy setups are possible. STaaS supports zone-redundant storage and geo-redundant storage. When you contact Skyline to register your system to use STaaS, you can include your preferences as to the region(s) where your data should be stored and the type of storage redundancy that should be used.

Note

DataMiner STaaS's standard supported regions are West Europe (The Netherlands), UK South, North Central US, UAE North, Southeast Asia (Singapore), and Australia East. Choosing regions outside this standard list will incur additional charges.

  • Zone-redundant storage (ZRS) copies your data synchronously across three Azure availability zones in one region. Each availability zone is a separate physical location with independent power, cooling, and networking. By default, DataMiner STaaS uses ZRS.

  • Geo-redundant storage (GRS) copies your data synchronously three times within a single physical location in the primary region and then also copies your data asynchronously to a single physical location in the secondary region. Only specific regions can be combined in such a setup, e.g. if the primary region is Switzerland North, the secondary region can only be Switzerland West. For an overview of the supported regions, see Azure paired regions. GRS is available upon request, but will result in additional charges. If you wish to use DataMiner STaaS with GRS, contact staas@dataminer.services.

Tip

For detailed information, see Azure Storage redundancy on learn.microsoft.com

Data resilience and backups

To ensure data resilience for potential recovery scenarios, protecting against user errors and accidental changes, your data is backed up with a granularity of 1 day. Backups are stored for 30 days.

  • Daily backups: STaaS performs backups with a granularity of 1 day and maintains a 30-day rolling snapshot of your data.

  • Data restoration and support: In the event a rollback is necessary, our support team will assist you. To submit a rollback request, contact the support team by sending an email to staas@dataminer.services. They will guide you through the necessary steps to ensure a successful data restoration.

Data security and availability

With STaaS, the data for a specific DMS is isolated in a logical partition. You can only ever access the logical partition dedicated to your own DMS, and all partitions are strictly isolated from each other.

To access your data, you use a connection authenticated with a Service Principal. With this connection, you can only access the logical partition dedicated to a specific DMS, which means that all data of a DMS is strictly isolated.

The data is encrypted both at rest and in transit.

If ZRS is used, STaaS has an expected availability of 99.90%. With GRS, it has an expected availability of 99.95%. For more information, please contact sales@skyline.be.

TTL

It is not yet possible to configure time-to-live (TTL) values for STaaS. In the table below, you can find the default TTL values for each data type.

Data type TTL
Real-time trending 7 days
Average trending (short) 3 months
Average trending (medium) 2 years
Average trending (long) 10 years
State changes 5 years
Spectrum traces 1 year
Alarm events 1 year

Cost estimation

To request a cost estimation, follow the procedure below:

  1. Follow the setup procedure until you come to the step where you need to wait to receive confirmation of your registration.

  2. Wait until you have received confirmation of your registration by email.

  3. Deploy the STaaS Migration Script package from the DataMiner Catalog.

  4. In the Automation module in DataMiner Cube, locate the CloudStorageMigration script and execute the script.

    Note

    When you run the Automation script on a Failover pair, make sure the currently active Agent is the main Failover Agent (i.e. the first Agent in the Failover configuration). Otherwise, the Automation script will not function correctly.

  5. Initialize the migration:

    1. Optionally, configure a proxy for the migration if necessary. This is supported from DataMiner 10.4.6 onwards.

    2. Click Init migration to initialize the migration.

  6. Start the migration:

    1. Make sure Replication only is selected.

    2. Select all data types.

      This way you will have the most accurate estimation of your system when running the script.

    3. Click Start migration.

  7. Let the script run for 24 hours without restarting the DataMiner System (DMS).

  8. After the 24-hour period, restart the DMS to stop the estimation process.

  9. After this, at approximately 2 AM UTC, you will be able to view your cost estimation in the Admin app, under Overview > Usage for the relevant organization.

    Note

    You will only be able to see the Usage module if you are an Owner or Admin of the organization.

If you have any questions regarding this cost estimation, please contact staas@dataminer.services.

Important

Cost estimations can currently only be performed for the West Europe and UK South regions.

Tip

To optimize the cost efficiency of a STaaS solution, adhere to the best practices to prevent storing unnecessary data with Automation scripts or with connectors.

Migrating existing data to STaaS

Before migrating your data over to STaaS, make sure you are aware of the limitations for migration. Then follow the procedure below:

  1. Follow the setup procedure until you come to the step where you have received confirmation that the registration is completed.

  2. Deploy the STaaS Migration Script package from the DataMiner Catalog.

  3. In the Automation module in DataMiner Cube, locate the CloudStorageMigration script and execute the script.

    Note

    When you run the Automation script on a Failover pair, make sure the currently active Agent is the main Failover Agent (i.e. the first Agent in the Failover configuration). Otherwise, the Automation script will not function correctly.

  4. Initialize the migration:

    1. Optionally, configure a proxy for the migration if necessary. This is supported from DataMiner 10.4.6 onwards.

    2. Click Init migration to initialize the migration.

  5. Start the migration:

    • Make sure Replication only is not selected.

    • Select the desired storage types for migration.

      Note

      For systems with a lot of real-time trending, we urge you to consider if you really need this data to be migrated. This data is typically only stored for 1 day, so when there is a lot of data, this gives an overhead on the rest of the types that need to be migrated, and this can cause the migration to take longer.

    • Click Start migration to start the migration.

    The script will initiate the migration process, but the migration itself will not be completed immediately.

  6. To monitor the migration progress, run the CloudStorageMigrationProgress script.

    This will log the progress of the migration for each storage type as information events.

  7. Keep an eye on the progress until the status for all storage types that were triggered shows State=Completed.

    This indicates that the migration has successfully finished.

Note

In case of issues during or after the migration, revert the DB.xml file to its previous state and re-trigger the migration process. If you want to be certain no data inconsistencies are possible, contact STaaS support.

Limitations

To migrate existing data to STaaS, the following limitations apply:

  • Migration is supported in 10.4.0 and the latest available 10.4.x feature release.

  • Migration of a setup with multiple OpenSearch/Elasticsearch clusters is not yet supported.

  • Migration from a MySQL setup is not yet supported.

  • Migration using a proxy is supported from DataMiner 10.4.6 onwards.

  • If you start the migration while an element with a logger table is stopped, the data of that element will not be migrated.

In addition, the following other limitations currently apply:

  • Jobs, Ticketing, and API Deployment data are not supported.

  • The following indexing engine functionality is not supported: Alarm Console search tab, search suggestions in the Alarm Console, aliases, and aggregation.

  • Custom configuration of TTL values is not yet supported.

  • Direct queries from DataMiner Cube to the database are not supported.

  • The SLReset tool is not supported.

  • Exporting trend data to a .dmimport file is not supported.

  • DMZ setups are currently not supported.

  • Adding a DataMiner Agent to a DMS using STaaS requires additional manual configuration steps.

  • Regarding logger tables:

Troubleshooting

For troubleshooting information related to STaaS, see Troubleshooting – STaaS.

Note

If you experience any issues during setup or while using Storage as a Service, and you cannot resolve these using the available troubleshooting information, contact staas@dataminer.services.

Propose changes