AWS Kinesis 1.1
Collect, process, and analyze real-time, streaming data so you can get timely insights and react quickly to new information
OverviewCopy
The AWS Kinesis connector currently allows you to load data onto a Kinesis firehose.
AuthenticationCopy
To use the Kinesis firehose operations you need to create a user with the following permissions to put records onto the firehose:
1"firehose:ListDeliveryStreams"2"firehose:PutRecord"3"firehose:PutRecordBatch"
Adding records to a Kinesis firehoseCopy
An AWS Kinesis firehose allows you to send data into other AWS services, such as S3, Lambda and Redshift, at high scale. The Kinesis connector includes 2 operations that allow you to either send a single item of data (Put Record) or send multiple items (Put Batch Record) to a Kinesis firehose.
To get started with a Kinesis firehose, its recommended that you follow the AWS guide to setting one up.
Putting RecordsCopy
Using the put record operations is as simple as setting an AWS region, choosing a delivery stream and then sending some raw text content.
The raw text content can be of any type, but usually depends on how you have setup your firehose. For example if you want to load your data directly into redshift the content can be a comma seperated list of column values. Alternatively you may want to copy raw JSON into redshift.
PLEASE NOTE: If you are planning putting a JSON object onto your firehose, you need to useInterpolated Mode to send the JSON as raw string. If you try and map the output of a previous stepdirectly into the Data field, it will not stringify the JSON object.