Amazon S3

This article contains information on the steps required to create an AWS S3 data source and stream query:

Creating an Amazon S3 Source

  1. In the Navigation Panel, click Catalog.
  2. Use the Search box OR click the Storage filter to locate the data source.
  3. Click Start on the Amazon S3 tile. The Amazon S3 dialog box is displayed.

    s3-stream.png

  4. From the dropdown menu select the Amazon regional bucket location.
  5. Enter the bucket name you created in your Amazon account.
    [Optional] To restrict access to a specific folder in the bucket, enter the name of the folder in the Folder Name field.
  6. Setup the AWS cross account role (as described here) to enable anodot.com to access your S3 bucket
  7. Click CONTINUE to display the Stream Query window.

Creating an Amazon S3 Stream Query

If you have just created an Amazon S3 data source, skip to step 3.

  1. In the Sources page (accessed by clicking INTEGRATIONS > Sources in the Navigation Panel), filter the list of streams to find the source for which you want to create a stream query.
    Note: If the Streams panel is empty, no stream queries exist for that source.
  2. Hover over the Amazon S3 data source, and click + New Stream. The Stream Query page is displayed.
  3. In the Files Path field enter a path relative to the data source.
    [Optional] You can further restrict access to the source by continuing the path to a specific folder within the bucket.
  4. Choose a File Name Date Pattern.
    Note: The file name indicates the start of the collection interval. See the examples below.

    Filename Timestamp Pattern

    Example

    Supported Intervals
    yyyyMMdd

    test_20180715_daily.csv.gz

    Daily

    yyyyMMddHH 

    test_2018071503_hourly.csv.gz 
    test_2018071504_hourly.csv.gz 
    test_2018071505_hourly.csv.gz

    Daily, hourly [1]

    yyyyMMddHHmm 

    test_201807150305_5min.csv.gz
    test_201807150310_5min.csv.gz
    test_201807150300_15min.csv.gz
    test_201807150315_15min.csv.gz
    test_201807150330_15min.csv.gz
    test_201807150345_15min.csv.gz

    Minute, Daily, hourly,
    15 minutes, 5 minutes, 1 minute [2]

    [1] If daily - the most recent file will be used
    [2] For daily, hourly, 15 minutes, 5 minutes, 1 minute - the most recent file will be used

    File Name Examples
    File Name  Holds data from...to
    test_2018071503_hourly.csv.gz 
    test_2018071504_hourly.csv.gz 
    test_2018071505_hourly.csv.gz    		 hourly data from 03:00 to 03:59
    hourly data from 04:00 to 04:59
    hourly data from 05:00 to 05:59
    test_201807150305_5min.csv.gz
    test_201807150310_5min.csv.gz
    test_201807150300_15min.csv.gz
    test_201807150345_15min.csv.gz    		 minute-based data from 03:05 to 03:09
    minute-based data from 03:10 to 03:14
    15-minutes of data from 03:00 to 03:14
    15-minutes of data from 03:45 to 03:59
  5. In the File name prefix field, enter the prefix (e.g. test_ ).
  6. In the File name suffix field, enter suffix and extension (e.g. demo.csv.gz).

    AWS_S3_stream_query_prefix_suffix_australia.png

    Notes:

    • In each of the Timestamp/Prefix/Suffix fields enter the specified data only.
    • Files may be compressed [.gz].
    • If you get an error message, fix the error in the bucket or files, and click RETRY to re-fetch the files.
  7. To preview the file, click View-icon.png. [Optional]
  8. To change the Parsing and Import Settings, click Settings-icon.png.
    See CSV - Parsing and Importing Settings.
  9. Click GO! The imported Measures & Dimensions are displayed.
    AWS_S3_no_measures_wCrumbs.png

Note: If there are no Measures in the Stream Query, an editable Events Count column is automatically inserted in the Stream Table.

Editing a Stream Query

  1. To edit the Stream Query, click the edit icon in the Measures & Dimensions panel.

S3_MeasuresDimensionsPanel.png

  • To add an item, drag from the repository on the left to the relevant Measure or Dimension  panel. At least one dimension and one measure [ a numerical value] are required.
  • To clear an item, click the X icon on the item.
  • Choose a time format from the dropdown menu.
  • Choose a time zone. 

2. Click X to accept the edits and return to the Stream Query window.

3. To edit the Schedule File Collection, click the edit icon in the Schedule panel.

S3_Schedule_File_Collection.png

In Schedule File Collection, define the following:

  • From the Collect files every menu, set the intraday or daily collection interval.
    Note: The timestamp at the start of the source file name indicates the period of the minute, hourly or daily data. Each row in a file will increase incrementally, either by the intraday or daily interval chosen.
  • Set the Ignore files older than time span.
    Note: "Older than" refers to back filing of historical data and does not affect ongoing collection.
  • Set the Files Time Zone scheduling time zone according to the data records time zone.
  • Set the Lagging Files Policy number of intervals to wait for lagging files.

Note: The system behavior, in general, is to keep the data ordered, and create as minimal gaps as possible. Therefore, the heuristics are the following:

- While there is no new data, meaning no new files, the system will retry to collect from the last file collected. In this case, the lag policy parameter does not affect anything.
- When there is a gap followed by new files, the Lagging files parameter is used to state "how many intervals the system should wait before processing the new files". So that the file hold processing is not too long, due to missing files that may never or are too-long in arriving.

Example
File 2018032806_test.csv was collected OK - Anodot has collected the data up to 2018/03/28 at 06:59.
The files for 07,08, 09 have not arrived to the relevant folder.
The file for 10, called 2018032810_test.csv arrived at 11:15 - therefore there is a gap.

The Lagging files parameter determines the processing of the 10 o'clock file:

Lag policy = 0 --> The 10 file will be processed upon arrival, and a gap in the previous hours is created.

Lag policy = 1 --> The 10 file will not be processed yet, it will wait to 12:15 and then be processed. [If during the waiting time, older files arrive, they will be processed before.]

Lag policy = 2 --> The 10 file will not be processed yet, it will wait to 13:15 and then be processed. [If during the waiting time, older files arrive, they will be processed before.]

4. Click X to accept the scheduling settings.

5. Click NEXT. The Stream Table is displayed. 

Recommended Practices

Before uploading a data set file to an Amazon S3 source type:

  1. Prepare the file in a test folder; check that the file format meets the S3 data source file parameters.
  2. Records within files must be sorted chronologically.
    Note: Files smaller than 16MB are automatically sorted by timestamp by Anodot.
  3. Move the file to the production folder only after you have verified that the file is complete.

Notes:

  • If the file timestamp is epoch seconds or epoch milliseconds, don't define a timezone - it will be treated as an error.
  • The file must be formatted using UTF-8 or ASCII character sets.
  • An S3 bucket can be accessed from one Anodot account.
  • To ensure there is no impact on performance, the S3 folder should be configured to contain up to a maximum of 5000 files.

See Also:
Using Data Collectors
Collecting and Streaming Data
Stream Tables
Stream Summaries