AWS S3 Parquet Collector

This article describes the two main steps required to create an S3 Parquet Collector:

• Creating an S3 Parquet Collector data source  
• Creating an S3 Parquet Collector stream query

Creating an S3 Parquet Collector Data Source

1. In the Streams page, click Sources +.
2. In Source Types, click START on the S3 Parquet tile.
3. In the displayed window, select a bucket region, enter the bucket name, and enter a folder name (optional).
   
   
   
4. Update the bucket policy (as described in AWS S3) to enable anodot.com to access your S3 Parquet account.
5. Click Continue to display the Stream Query window, as shown in the following section. 

Creating an S3 Parquet Collector Stream Query

1. In the Streams page, choose the S3 Parquet source for which you want to create a stream query. 
   Note: The streams associated with that source are displayed. If the Streams panel is empty, no stream queries exist for that source.
2. Click Streams + to display the Stream Query window. 
   
   
3. At the top of the Stream Query window, set a name for the stream.
4. In the Files to Collect section, define an (Optional) Root Path (such as example-month-il) and the Daily Partition Pattern (such as {{DD}}-{{MM}}-{{YYYY}}). 
5. In the File Properties section, define a Format (by default, Parquet is selected), and Compression (choose from None or Snappy).
   Note: Non-Parquet files cannot be uploaded to the Parquet files directory.
   
6. Click the Measures & Dimensions Edit icon to define measures for the stream.
   
   
   - To add an item, drag from the Available fields repository on the left to the relevant Measures or Dimensions panel. At least one dimension and one measure [a numerical value] are required.
   - Select a time format from the dropdown menu, and then select a time zone.
   - When finished, click X to accept the edits and return to the Stream Query window.
   
   
7. Click the Schedule File Collection Edit icon to define file collection settings for the stream.
   
   
   Define the following:
   - Set the collection interval in the Collect Files Every field.
   - Set the scheduling time zone of the data records time zone in the Files Time Zone field.
   - Set the delay according to the time it takes the data to be available in the Delay (Minutes) field.
   - Set a time span in the Ignore Files Older Than field. 
   - Set the number of intervals to waits for lagging files in the Lagging Files Policy field.
   
   For further details on the above fields and their settings, see Editing a Stream Query.
   
   
8. Click X to confirm the scheduling settings.
9. Click NEXT. The Stream Table is displayed, as described in Stream Tables. 

SEE ALSO:
Using Data Collectors
Collecting and Streaming Data
Stream Tables
Stream Summaries