Managing Delays in Composite Metrics

The Challenge: Delays in Composite Metrics
The Solution: Delays Mechanisms
    1. Watermark Timestamp-based Delay
       When to Use it
       How to Use It
    2. Automatic Composite Metric Delay
        When to Use It
        How to Use It
    3. Manual Composite Metric Delay
        When to Use It
        How to Use it
Choosing the Suitable Mechanism for Your Business
        Optimize for Accuracy of Insights
        Optimize for Faster Insights

 THE CHALLENGE: DELAYS IN DATA STREAMS

  • Data is streamed to Anodot via numerous collectors and APIs, using a variety of schedulers, structures, and frequencies.
  • Anodot aggregates the data samples to partitions referred to as buckets.
    • As data samples arrive, they are added into an open bucket, until that bucket is closed.
  • Closing a bucket provides a signal to calculate composite metrics/alerts as defined by the users.
  • Since data samples may arrive at different delays, and even out of order, the indication to close a bucket and move to the next one becomes complex.
    • Hence the indication to calculate composite metrics based on the data samples in the bucket is complex.
  • Anodot provides a delay mechanism to provide control over this challenge.

THE SOLUTION: DELAY MECHANISMS

1.  Watermark time-based delay 

When to Use It

i.  All raw metrics in the composite are from the same stream
ii. The stream is an Anodot DC stream, or Metrics 3.0 based

How to Use It

 i. Send the data using Anodot DC streams or Metric 3.0 protocol.

- Using Anodot Data Collectors, or REST Metric 3.0 allows you send an explicit watermark timestamp.
- A watermark timestamp commits that no data samples with timestamps less than or equal to it will be sent. In other words, all data samples for “this bucket” have arrived and no need to wait anymore.

ii. All set

- The mechanism is turned on in the account and is applied on composite metrics according to their relevancy.

2.  Automatic Composite Metric Delay

When to Use It

i. The timestamp watermark solution cannot be applied.

How to Use It

i.  Create a composite metric according to your needs.
ii.  Go to the Composite delay option and check the Auto calculated delay enabled.
iii. Anodot will consider 85% of the metrics’ samples as the required delay and will keep monitoring and updating the required delay to enable these 85%.


Composite_metrics_01.png

3.  Manual Composite Metric Delay

When to Use it

i. The automatic composite delay solution cannot be applied.

How to Use It

i.   Create a composite metric according to your needs
ii.  Go to the Composite Delay option and uncheck Auto calculated delay enabled.
iii. Anodot provides a scale to choose the preferred combination according to your business requirements.

Composite_metrics_02.png

CHOOSING THE SUITABLE MECHANISM FOR YOUR BUSINESS

Optimize for Accuracy of Insights

  • Ensure samples for each timestamp bucket come to Anodot in chronological order to avoid buckets closing prematurely.
  • Willing to wait some extra amount of time to ensure all samples for the current bucket are received.

Optimize for Faster Insights

  • Willing to compromise some accuracy and false positive rate in order to gain speedier insights, noting that the indefinite time delay capabilities Anodot provides are more flexible than the company’s current tools. 
Was this article helpful?
0 out of 0 found this helpful
-->