Kinesis to S3 Application

Summary

Ingest messages from kinesis and write to S3 bucket. The source code is available at: https://github.com/DataTorrent/app-templates/tree/master/kinesis-to-s3.

Please send feedback or feature requests to: feedback@datatorrent.com

This document has a step-by-step guide to configure, customize, and launch this application.

Steps to launch application

  1. Click on the AppHub tab from the top navigation bar. AppHub link from top navigation bar

  2. Page listing the applications available on AppHub is displayed. Search for Kinesis to see all applications related to Kinesis. AppHub search for Kinesis_or_s3 Click on import button for Kinesis to S3 App.

  3. Notification is displayed on the top right corner after application package is successfully imported. App import Notification

  4. Click on the link in the notification which navigates to the page for this application package. App details page Detailed information about the application package like version, last modified time, and short description is available on this page. Click on launch button for kinesis-to-S3 application.

  5. Launch Kinesis-to-S3 dialogue is displayed. One can configure name of this instance of the application from this dialogue. Launch dialogue

  6. Select Specify Launch properties option. This expands a key-value editor with mandatory properties for this application. Specify Launch properties

  7. Specify the mandatory properties under Specify Launch Properties Properties editor For example, suppose we wish to process all messages from Kinesis stream transactions and write them to output.txt under /user/appuser/output on S3. Properties should be set as follows:

    name value
    Stream Name For Kinesis Input transactions
    Access Key For Kinesis Input KINESIS_ACCESS_KEY
    Secret Key For Kinesis Input KINESIS_SECRET_KEY
    End Point For Kinesis Input KINESIS_END_POINT
    Access Key For S3Output S3_ACCESS_KEY
    Secret Access Key For S3Output S3_SECRET_KEY
    S3Output Bucket Name S3_BUCKET_NAME
    S3Output Directory Path /user/appuser/output

    Details about configuration options are available in Configuration options section.

  8. Click on the Launch button on lower right corner of the dialog to launch the application. A notification is displayed on the top right corner after application is launched successfully and includes the Application ID which can be used to monitor this instance and find its logs. Application launch no tification

  9. Click on the Monitor tab from the top navigation bar. Monitor tab

  10. A page listing all running applications is displayed. Search for current application based on name or application id or any other relevant field. Click on the application name or id to navigate to application instance details page. Apps monitor listing

  11. Application instance details page shows key metrics for monitoring the application status. logical tab shows application DAG, Stram events, operator status based on logical operators, stream status, and a chart with key metrics. Logical tab

  12. Click on the physical tab to look at the status of physical instances of the operator, containers etc. Physical tab

Configuration options

Mandatory properties

End user must specify the values for these properties.

Property Description Type Example
Stream Name For Kinesis Input Name of the stream from where the records to be fetched String transactions
Access Key For Kinesis Input Indicates the accessKey which have read access to the Kinesis stream String KINESIS_ACCESS_KEY
Secret Key For Kinesis Input Indicates the secret AccessKey which have read access to the Kinesis stream String KINESIS_SECRET_KEY
End Point For Kinesis Input Indicates the endpoint of the kinesis stream String KINESIS_END_POINT
Access Key For S3Output AWS access key which have write access to the S3 bucket String S3_ACCESS_KEY
Secret Access Key For S3Output AWS Secret key which have write access to the S3 bucket String S3_SECRET_KEY
S3Output Bucket Name Name of S3 bucket Sting S3_BUCKET_NAME
S3Output Directory Path Output path for S3 files String /user/appuser/output

Steps to customize the application

  1. Make sure you have following utilities installed on your machine and available on PATH in environment variables

  2. Use following command to clone the examples repository:

    git clone git@github.com:DataTorrent/app-templates.git

  3. Change directory to 'examples/tutorials/kinesis-to-s3':

    cd examples/tutorials/kinesis-to-s3

  4. Import this maven project in your favorite IDE (e.g. eclipse).

  5. Change the source code as per your requirements. Some tips are given as commented blocks in the Application.java for this project

  6. Make respective changes in the test case and properties.xml based on your environment.

  7. Compile this project using maven:

    mvn clean package

    This will generate the application package with .apa extension in the target directory.

  8. Go to DataTorrent UI Management console on web browser. Click on the Develop tab from the top navigation bar. Develop tab

  9. Click on upload package button and upload the generated .apa file. Upload

  10. Application package page is shown with the listing of all packages. Click on the Launch button for the uploaded application package.
    Follow the steps for launching an application.