Kinesis stream – EC2 producer using “Agent”

Configure EC2 instance for AWS-Kinesis Agent to Stream logs to Kinesis Stream :

Step:1 Create an IAM role with following default AWS policies.

Step:2 Launch an EC2 instance attaching the above IAM role.

Step:3 Install “aws-kinesis-agent” in ec2 instance:

  • yum install aws-kinesis-agent -y
  • service aws-kinesis-agent start
  • chkconfig aws-kinesis-agent on

Step:4 Configure “TCPDUMP” to create continuous log file

  • yum install -y tcpdump
  • nohup tcpdump >> /tmp/tcpdump.log
  • tail -f /tmp/tcpdump.out

Step:5 Configure the aws-kinesis-agent to forward log records to kinesis stream “tcpdump”  *** I have created stream “tcpdump”

cat /etc/aws-kinesis/agent.json
{
  “cloudwatch.emitMetrics”: true,
  “kinesis.endpoint”: “”,
  “firehose.endpoint”: “”,
 
  “flows”: [
    {
      “filePattern”: “/tmp/tcpdump.out“,
      “kinesisStream”: “tcpdump“,
      “partitionKeyOption”: “RANDOM”
    }
  ]
}

 

Step:6 Restart the kinesis-agent and verify the agent logs:

# service aws-kinesis-agent restart
aws-kinesis-agent shutdown                                 [  OK  ]
aws-kinesis-agent startup                                  [  OK  ]
# tail -f /var/log/aws-kinesis-agent/aws-kinesis-agent.log

2016-08-02 21:32:51.816+0000 ip-172-31-4-2 (Agent.MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.Agent [INFO] Agent: Progress: 179 records parsed (143508 bytes), and 94 records sent successfully to destinations. Uptime: 90057ms

 

Step:7 Read the data from Kinesis stream (get-record)

$ SHARD_ITERATOR=$(aws kinesis get-shard-iterator --shard-id shardId-000000000002 --shard-iterator-type TRIM_HORIZON --stream-name tcpdump)

*** to read it , decode the Base64 format output.
https://www.base64decode.org/ --> copy the "bmFtZTpTZW50aGlsLCBBZ2U6MzgsIFBsYWNlOiBDaGVubmFp"
(or)
command line decode:
 $ aws kinesis get-records --shard-iterator $SHARD_ITERATOR | awk '{print $3}' | base64 --decode

Leave a Comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>