Creating an anomaly overview dashboard

This tutorial illustrates DataMiner's behavioral anomaly detection feature in the context of Dashboards and Low-Code Apps. While creating an anomaly overview dashboard, you will learn how to access the behavioral change event data in a dashboard and gain insight in how to build great dashboards and low-code apps.

Estimated duration: 35 minutes.

Note

The content and screenshots for this tutorial were created in DataMiner 10.4.9. To follow the complete tutorial, you will need at least DataMiner 10.4.7. If you only create the statistical overview but do not follow the last two steps of the tutorial, DataMiner 10.3.12 is sufficient.

Tip

• For more information, such as technical limitations of anomaly detection, see Working with behavioral anomaly detection.
• See also: Anomaly Detection Tutorial, Kata #12: Automatically detect anomalies with DataMiner on DataMiner Dojo, and Kata #39: Building an anomaly overview dashboard on DataMiner Dojo.

Prerequisites

• A DataMiner System connected to dataminer.services.

• Storage as a Service (STaaS) (recommended) or a self-managed Cassandra database with indexing.

• Behavioral anomaly detection is enabled (in DataMiner Cube: System Center > System settings > analytics config).

• The time to live for trending is at least 1 day for real-time trending and 1 month for minute records.

Overview

The tutorial consists of the following steps:

• Step 1: Install the example package from the Catalog
• Step 2: Compile a list of anomalies
• Step 3: Include statistical overview with filter options
• Step 4: Enhance the table view by replacing IDs with names
• Step 5: Add a trending component and timeline view to visualize the anomalies

Step 1: Install the example package from the Catalog

1. Go to https://catalog.dataminer.services/details/c270f156-3d8d-4b40-a6d9-0d0cc599062a.

2. Deploy the Catalog item to your DataMiner Agent by clicking the Deploy button.

   This will deploy both the custom operators that you will need in later in this tutorial and an example of the end result of the tutorial.

Step 2: Compile a list of anomalies

1. Create a new dashboard.

2. Drag and drop a Time range visualization from the pane on the left onto the dashboard to add a time range component.

   

   This component will allow users to select a specific time range, which will be used in a query to retrieve behavioral change points later on.

3. While the component is selected, go to the Settings tab on the right and change the default range to This week.

4. Drag and drop the Table visualization from the pane on the left to the dashboard.

   

   This will add a table component. In this component, you will show the output of a GQI query.

5. In the data pane, open the QUERIES section and click the "+" icon.

   

6. Name your query Anomalies filtered on time.

7. Select the data source Get behavioral change events.

8. Add a Filter operator, and configure it as follows:

   • Column: Anomalous.
   • Filter method: equals.
   • Select Value, which will set the value to true.

   

9. Add another Filter operator for the start time:

   • Column: Start time.

   • Filter method: greater than.

   • Value: Click the link icon in the value box, then select your time range feed and the property From, and click Apply.

     

10. Add a final Filter operator for the end time:

    • Column: End time.
    • Filter method: less than or equals.
    • Value: Click the link icon in the value box, then select your time range feed and the property To, and click Apply.

    The resulting query should look like this:

    

11. Drag the query from the data pane onto the table component.

The table component should now display the raw anomalies from the system within the time range that is selected. If you cannot see any items in your table, select a wider time range with the time range component.

Step 3: Include statistical overview with filter options

1. Duplicate the query you made in the previous step, by clicking the ... icon to the right of the query and selecting Duplicate.

   

2. Name the new query Anomalies by Type.

3. Add an Aggregate operator, and configure it as follows:

   • Column: Change Point ID.
   • Method: Count.

   

4. Add a Group by operator, and set it to the column Change type.

   

5. Drag and drop the State visualization from the pane on the left to the top of the dashboard.

6. Drag the query Anomalies by Type onto the State component.

   The state component should now show the number of anomalies for each type.

   

7. Fine-tune the layout of the state component:

   1. While the component is selected, go to the Layout tab on the right.
   2. Clear the checkbox Display query name
   3. Under Design, select Large.

   

8. Click one of the State items in the component to make a selection.

9. Modify the Anomalies filtered on time query so it also takes that selection into account:

   1. Click the pencil icon next to the query to start editing it.

   2. Rename the query to Anomalies filtered on time and type.

   3. Add a Filter operator, and configure it as follows:

   • Column: Change type.
   • Filter method: equals.
   • Value: Click the link icon in the value box, then select your state component, and click Apply.

Now the statistical overview at the top of the dashboard will give a good indication of the anomalies that have been detected in the system in the specified time range, and clicking one of the change types in the overview will filter the table so it only shows the selected type.

Step 4: Enhance the table view by replacing IDs with names

The Element ID and Parameter ID columns in the table contain the IDs of the elements and parameters where the anomalies occur. To enhance the overview, you can show the names of the elements and parameters instead.

1. To resolve the element names, modify the Anomalies filtered on time and type query:

2. Add a Join operator of type Left.

3. Add the data source Get elements, and configure it as follows:

   • Operator: Select
   • Columns: Element ID, Protocol, ProtocolVersion, and Name.

4. Below this, select which column should be used to join the table by selecting Element ID both on the left and on the right.

   

5. Optionally, add the ParameterKeyToSeparateColumns custom operator to also include the table key for table parameters:

   1. At the bottom of your query, select Apply custom operator, and then select ParameterKeyToSeparateColumns.

   2. In the Parameter Key Column box, select Parameter ID.

   

6. Add the Parameter KeyToName custom operator:

   1. At the bottom of your query, select Apply custom operator, and then select Parameter KeyToName.

   2. In the Parameter Key Column box, select Parameter ID.

   3. In the Protocol Name and Protocol Version boxes, select Protocol and ProtocolVersion, respectively.

   

7. Add the Rename column custom operator:

   1. Select Apply custom operator, and then select Rename column.

   2. In the Column box, select Name.

   3. In the New name box, enter Element.

   

8. Add an operator to concatenate the parameter name with the parameter key into one single parameter column:

   1. At the bottom of your query, select Column manipulations.

   2. Select the manipulation method Concatenate.

   3. Select the columns Parameter ID and Parameter Name.

   4. Using the placeholders {0} and {1}, in the Format box, indicate how the concatenated data should be displayed, e.g. {1} ({0}).

   5. In the New column name box, specify Parameter.

   

9. Add a Select operator, and make sure only the columns Element, Parameter, Change type, Start time and End time are selected.

   

10. Optionally, drag and drop the column headers of the table to rearrange the columns, so that the Element and Parameter columns are shown first.

Note

The custom operators mentioned above are included in the package you installed in step 1. You can also find them on GitHub: Parameter KeyToName, ParameterKeyToSeparateColumns, and RenameColumn. Note that the Parameter KeyToName operator can make a dashboard load significantly more slowly.
Key in the Parameter KeyToName operator are IGQIOnInit and its GQIDMS interface, which provides access to protocol and parameter information via the element or protocol. The operator has a very basic protocol cache in place. Specifying the Protocol and ProtocolVersion in the Parameter KeyToName operator is optional, but will result in better performance because the operator will be able to reuse protocol information for the elements of the same protocol and version. Overridden parameter names (e.g. information template) will then go back to the name from the protocol.
Key in the ParameterKeyToSeparateColumns operator is the ParamID Object (slnettypes.dll).

Step 5: Add a trending component and timeline view to visualize the anomalies

In this step, you will add a line chart to show the trending of the parameter linked to the selected anomaly. Below the chart, you will add a timeline that will show a template for the selected anomaly.

1. Drag and drop the Line & area chart from the pane on the left onto the dashboard, below the table component.

   

2. Drag and drop the Timeline from the pane on the left onto the dashboard, below the line & area chart.

   

3. Select a row in the table so that it becomes available as a feed.

4. In the data pane on the right, go to Feeds > Table x > Selected rows, and drag the Parameters row to the line & area chart component.

5. In the data pane on the right, go to Feeds > Table x > Selected rows, and drag the Timespans row to the line & area chart component as a filter.

6. In the data pane on the right, go to Feeds > Table x > Selected rows, and drag the Query rows row to the timeline component.

7. Edit the template of the timeline so it shows the change type of the anomaly and appears in a custom color:

   1. While the timeline component is select, go to the Layout pane on the right, scroll down to the Item templates section, and click Edit.

      

      This will open the template editor.

   2. In the pane on the right, while the text layer is selected, instead of {Element}, specify {Change type}, and modify the layout according to your preferences.

      

   3. Select the rectangle layer instead, and tweak the layout according to your preferences.

      

   4. Click Save.

   Tip

   See also: Template editor.

8. Link the time ranges of the timeline and the line chart:

   1. Select the timeline component and go to the Settings pane on the right.

   2. Next to Link time range to feed, click the link button.

      

   3. Select Line & area chart x and click Apply.

9. Finally, add another custom operator to your query in order to apply a time range that adds a bit of time before and after the anomaly start and end time, so that displayed time range is more user-friendly:

   1. In the data pane, click the pencil icon to edit the Anomalies filtered on time and type.

   2. At the bottom of the query, click Apply custom operator.

   3. Select the operator Time Range Feed Extending.

      Note

      This operator was also included in the package you installed in step 1 and is also available separately on Github. This is an operator with relatively simple logic, which illustrates two nice capabilities of custom operators:

      • Adding extra columns with new information, in this case, the columns TimeRange Feed Start and TimeRange Feed End, which show the result of the applied offsets on the linked Start and End columns.
      • Overriding the Time range feed by setting new values for the StartTime and EndTime in the TimeRangeMetadata in the rows.

   4. In the Start and End boxes, select Start time and End time, respectively.

   5. In the Amount of hours before box, specify 8.

   6. In the Amount of hours after box, specify 4.

   

The resulting dashboard could for instance look like this: