When defining alert conditions (see Step 4 in How to create an alert) you can define a number of additional (optional) conditions that enable you to fine-tune your alert.
Important: An alert is triggered only when the conditioning metrics meet the additional specified conditions. For example, where more than one Influencing Metrics condition is defined in the alert, all the Influencing Metrics conditions must be met for the alert to trigger.
These additional conditions include the following:
- Auto-Tuned Settings (including minimum delta settings, when to override the delta duration period, and when to ignore low volume anomalies)
- Add multiple event conditions
- Number of metrics in an anomaly
- Metrics values in an anomaly
- Add influencing metrics
Defining Auto-Tuned Settings
The Auto-Tuned Settings section includes three main options that enable you to automatically tune your alert conditions.
Setting a Minimum Delta Value
Set the Minimum Delta of the metric value as either an absolute value, percentage, or both; this sets a minimum delta of the metric value from the normal range.
Note: If the Automatic option switch is selected, Anodot sets a minimum delta of 10%. This calculation is based on the 90th percentile of the average values of the metrics in the alert. For a single metric, the minimum delta is 10% of its average value. The minimum delta is recalculated and updated periodically.
However, be aware of a monitoring phenomenon known as "Single Dip". In such a situation, there are data points (mostly because of a data streaming issue) below the delta defined. Note that for illustration purposes, we added the blue arrows below to demonstrate the delta.
As can be seen in the image above, a single data point exceeded the delta and therefore would generate an alert. If you would like to filter such alerts, use the Override Delta Duration option, as described below.
Defining when to Override the Delta Duration
Select the Override Delta Duration checkbox to set the threshold above which an alert will be triggered. Note that the value for this is bound by the time duration and minimum duration you defined initially for the alert. For example, if you have set a minimum duration of 30 minutes, you can define an override period which is up to 30 minutes.
Ignoring Low Volume Anomalies
The Low Volume Condition alert is designed to help reduce alert false positives. This condition will automatically ignore high/low values, according to the alert's metrics.
IMPORTANT: An alert is triggered only when the alert's metrics meet the specified condition value (the condition is applied on metrics with the “counter” aggregation type). If the low volume condition is off, (i.e., Ignore Low Volume Anomalies is not selected), the condition will be disregarded.
- Select the Ignore Low Volume Anomalies checkbox. Note that this will not trigger an alert if the value is in the past.
- Turn off the Automatic option switch.
Note: If Automatic is selected, the low volume condition will automatically apply to each new alert created from counter metrics. The metric value is recalculated and updated periodically. You can also set a different value. - Set the metric value. The Low Volume Condition alert is not triggered if the metric value is lower/higher than the set value in the defined time interval (based on what you defined in the basic alert conditions). If the specified duration is more than “1”, the value is compared to the sum of the data points in the given period.
Adding multiple event conditions
You can define a number of event conditions for an alert, including different groups of influencing of events in alerts, and optional matching properties between events and metrics. This option is also relevant to all alert types: Anomaly, Static and No data.
For example, if you are collecting error counts (and other KPIs) on certain servers, you would expect an increase in the error count/rate of those servers when adding new deployments. The ability to add multiple event conditions enables you to send each deployment event with the server property and match the server to the server in the alert metrics.
- In the Events section, click the Add button and then select from one of Display, Influence, Suppressing, or Office hours.
- If you select Display, select the relevant events on triggers using their title, description, source, category or other properties. You can also map event properties in each user event group to dimensions in the alert metrics.
- If you select Influence, select the relevant influencing events over the metrics using their title, description, source, category or other properties (see also Influencing Events for further information). You can also map event properties in each user event group to dimensions in the alert metrics; these specific events will influence the specific dimensions selected.
- If you select Suppressing, select the relevant events to suppress certain metrics according to the event start and end time (the events need to include some kind of “end time”, either in the same event, or by using a closing event). Note that if an event suppresses a subset of the alert metrics, it will be shown in the trigger; if it suppresses all metrics, it cannot be shown as there is no trigger to show it in. You can also map up to four properties in each user event group to dimensions in the alert metrics.
Note: If a metric is already in an anomalous state when the suppression event start time begins, it will not be suppressed by that event. An event that took place after an anomaly started is “ignored” for that anomaly; the anomaly might send a trigger even if the event exists. - If you select Office hours, select the relevant events that will not issue any alert triggers in the designated period. Note that the events need to include some kind of “end time”, either in the same event, or by using a closing event. Once the event takes place, no triggers are sent (as in a "do not disturb" scenario for the defined hours).
- If you select Display, select the relevant events on triggers using their title, description, source, category or other properties. You can also map event properties in each user event group to dimensions in the alert metrics.
- Repeat as required to add multiple event conditions for the alert. For example, you can add a number of Display events together with a number of Suppressing events
Defining the number of Metrics in an Anomaly
- Click the Number of metrics in an anomaly option switch.
- Define the Minimum number of metrics (the minimum number of metrics in an anomaly to trigger the alert). You can set the minimum as absolute values or percentages, or both.
Defining Metrics Values
- Click the Metrics values in an anomaly option switch.
- Define the metric value that will trigger the alert if it drops Below or is Below & Equals to the entered value.
Adding Influencing Metrics
You can select relevant influencing metrics to which the conditions will be applied.
Note that influencing metrics influence both the alert and the missing data (no-data).
- Click the Add button to display the Influencing Metrics dialog box.
- Click Select a Measure to define a metric, the same way you would in any Metrics expression.
Note: There is no limit to the number of metrics you can add, but the 'what' of the metric must be a single 'what'; for example anomalies_simulation_hit_rate. - Set a condition on the metric values. The condition can be one of the following:
- Greater than a value you set
- Greater than or equal to a value you set
- Less than a value you set
- Less than or equal to a value you set
- In the range of a value you set (inclusive)
- Not in the range of a value you set - Set the value or value range, depending on the condition you selected in Step 3.
- Set the time properties by which the metrics will be aggregated [in the last Hour / Day / Week], and if to use the values Before or During the anomaly.
- Select the Send alerts even if the influencing metric is missing checkbox to send alerts if the influencing metrics are not found.
