Collecting and Streaming Data

This article describes how to get started with data sources and data streams, and includes:

Overview

To stream data into Anodot, either from a website source or to upload a file from your local drive, select INTEGRATIONS > Data Management from the Navigation Panel. By default, all current data streams are displayed. 

DataManagement-2.jpg

To filter the displayed list to display streams associated with a specific source, click in the Source Type option box and select the relevant source(s) from the displayed list.

SourceType.jpg

To expand/collapse the displayed list of streams, click the More options button in the top right corner of the screen and select Expand All / Collapse All.

Streams-Collapse-Expand.jpg

Note: The Metrics Usage bar shows the current usage of metrics for your account. 

Data Sources

As described in the previous section, you can filter the displayed data sources to show only the streams relevant for a specific data source.

You can also perform the following:

  • Add a new data source, as described in Adding New Data Sources. 
  • View and/or rename a data source by selecting the data source in the Source Type option box, hovering over the relevant source in the displayed list, and clicking the Edit (pencil) icon, as shown below.
    DataStream-ViewSource.jpg

  • Delete a source by clicking the More icon and selecting Delete.
    DataStream-Delete.jpg

Adding New Data Sources

  1. In the Data Management screen, click + New Stream in the top right corner.
  2. In the displayed dialog, click Add a New Source.

    DataStream-Add.jpg

  3. In the displayed list of data sources, hover over the relevant data source.
    Then click Start and follow the onscreen instructions.
    OR
    Click Instructions to open the relevant Knowledge Base page.

    DataSources.jpg

    For further details on defining each of the available data sources, see **GUIDELINES BY DATA SOURCE TYPE**.

Note: If you do not see your data source, use the Search bar in the top right corner.

Data Streams

Stream status

The current status of stream queries is shown in the list of streams. To view, change the status or edit the query, highlight the stream query you want to change, and click the More icon.

DataStream-More.jpg

Queries can be changed as follows: 

Status

Description 

Click to: 
Live Historical data gathered or live data being gathered 
In Progress Gathering historical data, that has not reached the present date
Paused A Live or Initialized data stream was paused by a user

 
Incomplete The Stream definition has not been completed
  • Edit
  • Duplicate
  • Delete
Done For Uploaded files only - streaming completed
Error The stream is in an error state
  • Hover over the icon to
    display a description of
    the error
  • More>Resume, to
    resume streaming

[1] Renaming a Stream 

  • Changes the stream reference to the metrics created from that stream.
  • In Alerts/Composites/Anoboards/Dashboards, if the stream name is used without wildcards in an alert expression, it will be replaced in the expression automatically. If it is used with wildcards, the expression will not be changed automatically and you will need to change it manually.

Creating Stream Queries

Each source can hold several stream queries. The following steps describe how to create new queries:

  1. Define a stream query per the requirements of the data source:

    - Data to collect: measures, dimensions, and timestamp
    - Collection scheduling
    See Creating a Stream Query from a Database or the guidelines for the specific data source.

  2. Data to collect: measures, dimensions and timestamp
  3. Review the data in the Stream Table before processing it to Anodot.
    Note: To create more meaningful metrics, you can transform and filter the measures and dimensions from the data collectors before streaming the data.
    See Stream Tables, Measure Columns, and Dimension Columns