Managing delays in Composite Metrics
The Challenge: Delays in Data Streams
Data is streamed to Anodot via numerous collectors and APIs, using a variety of schedulers, structures, and frequencies. Anodot aggregates the data samples to partitions referred to as buckets - as data samples arrive, they are added into an open bucket, until that bucket is closed.
Closing a bucket provides a signal to calculate composite metrics/alerts, as defined by the users. Since data samples can arrive at different delays, and even out of order, the indication to close a bucket and move to the next one becomes complex; the indication to calculate composite metrics based on the data samples in the bucket is even more complex.
Anodot provides a delay mechanism to overcome this challenge, as described in the following section.
The Solution: Delay Mechanisms
There are three delay mechanisms available:
Watermark time-based delay
Using Anodot Data Collectors or the Metric 3.0 REST API, you can send an explicit watermark timestamp. This watermark timestamp commits that no data samples with timestamps less than or equal to it will be sent. In other words, all data samples for “this bucket” have arrived and do not need to wait anymore. The next steps, such as composite calculations, can start on all buckets closed by the timestamp watermark.
Use this delay mechanism when all the raw metrics included in the composite are from the same stream, and the search expression includes @stream with the stream name.
Alternatively, use this mechanism when the stream is an Anodot DC stream, or Metrics 3.0 based.
-
Send the data using Anodot DC streams or the Metric 3.0 protocol.
-
Make sure relevant composite metrics have the Auto calculated delay enabled checkbox selected (see image below).
- You should now be good to go - the mechanism is enabled in the account and is applied on composite metrics according to their relevancy.
Automatic Composite Metric Delay
Anodot calculates a delay that enables 85% of the metric samples to be included in the composite calculation. Anodot keeps monitoring and updating the required delay to maintain this rate of 85% over time.
Use this delay mechanism when the timestamp watermark solution cannot be applied.
-
Create a composite metric according to your needs.
-
In the Composite Delay dialog (as shown in the Watermark time-based delay section). select the Auto calculated delay enabled checkbox.
Manual Composite Metric Delay
The user sets the required delay to be used by Anodot.
Use this delay mechanism when the automatic composite delay solution cannot be applied.
-
Create a composite metric according to your needs.
-
In the Composite Delay dialog (as shown in the Watermark time-based delay section). clear the Auto calculated delay enabled checkbox.
-
Select the relevant value in the scale below the Auto calculated delay enabled checkbox; this scale enables you to choose the preferred combination of data points received and time delay, according to your business requirements.
Choosing the right Delay Mechanism
To optimize for accurate insights:
- Ensure samples for each timestamp bucket are sent to Anodot in chronological order to avoid buckets closing prematurely.
- Be aware that waiting an extra amount of time ensures all samples for the current bucket are received.
To optimize for faster insights:
- In order to gain faster insights, be aware that you may have to compromise on accuracy and the false positive rate. Note that the indefinite time delay capabilities Anodot provides are more flexible than the company’s current tools.
Updated over 1 year ago