AWS Kinesis-Firehose delivery stream to S3 bucket (stream TCPdump logs from EC2 )

Configure EC2 instance for AWS-Kinesis Agent to Stream logs to Kinesis Firehose “and push to S3 bucket “tcpdump2525

Prerequisite:

AWS console –> Create a Kinesis-Firehose stream & with S3 bucket “tcpdump2525″ and attache IAM role “firehose_delivery_role” with all other default configuration.

Step:1 Create an IAM role with following default AWS policies.

  • AmazonKinesisFirehoseFullAccess
  • AmazonCloudwatchFullAccess

Step:2 Launch an EC2 instance attaching the above IAM role.

Step:3 Install “aws-kinesis-agent” in ec2 instance:

  • yum install aws-kinesis-agent -y
  • service aws-kinesis-agent start
  • chkconfig aws-kinesis-agent on

Step:4 Configure “TCPDUMP” to create continuous log file

  • yum install -y tcpdump
  • nohup tcpdump >> /tmp/tcpdump.log
  • tail -f /tmp/tcpdump.out

Step:5 Configure the aws-kinesis-agent to forward log records to kinesis stream “tcpdump”  *** I have created stream “tcpdump”

cat /etc/aws-kinesis/agent.json
{
  “cloudwatch.emitMetrics”: true,
  “kinesis.endpoint”: “”,
  “firehose.endpoint”: “”,
 
  “flows”: [
    {
      “filePattern”: “/tmp/tcpdump.out“,
      “kinesisStream”: “tcpdump“,
      “partitionKeyOption”: “RANDOM”
    },
    {
      “filePattern”: “/tmp/tcpdump.log*”,
      “deliveryStream”: “FHtoS3-TCPDUMP”
    }
  ]
}

 

Step:6 Restart the kinesis-agent and verify the agent logs:

# service aws-kinesis-agent restart
aws-kinesis-agent shutdown                                 [  OK  ]
aws-kinesis-agent startup                                  [  OK  ]
# tail -f /var/log/aws-kinesis-agent/aws-kinesis-agent.log

2016-08-03 18:36:38.906+0000 ip-172-31-4-2 (FileTailer[fh:FHtoS3-TCPDUMP:/tmp/tcpdump.out].MetricsEmitter RUNNING) com.amazon.kinesis.streaming.agent.tailing.FileTailer [INFO] FileTailer[fh:FHtoS3-TCPDUMP:/tmp/tcpdump.out]: Tailer Progress: Tailer has parsed 1298 records (169164 bytes), and has successfully sent 1166 records to destination

 

Step:7 Verify the data in S3 bucket

$ aws s3 ls s3://tcpdump2525
**Note: check for bucket sub-folders for your data.

AWS-CLI Examples:

Delete steam:
$ aws firehose delete-delivery-stream --delivery-stream-name firehose-demo001-s3

List stream:

 $ aws firehose list-delivery-streams
{
“DeliveryStreamNames”: [
“FHtoS3-TCPDUMP”,
“firehost-redshift-s3-demo0001″
],
“HasMoreDeliveryStreams”: false
}

 

Describe stream:

$ aws firehose describe-delivery-stream –delivery-stream-name FHtoS3-TCPDUMP
{
“DeliveryStreamDescription”: {
“HasMoreDestinations”: false,
“LastUpdateTimestamp”: 1470249607.553,
“VersionId”: “2”,
“CreateTimestamp”: 1470244230.444,
“DeliveryStreamARN”: “arn:aws:firehose:us-east-1:086043250494:deliverystream/FHtoS3-TCPDUMP”,
“DeliveryStreamStatus”: “ACTIVE”,
“DeliveryStreamName”: “FHtoS3-TCPDUMP”,
“Destinations”: [
{
“DestinationId”: “destinationId-000000000001″,
“S3DestinationDescription”: {
“RoleARN”: “arn:aws:iam::086043250494:role/firehose_delivery_role”,
“Prefix”: “”,
“BufferingHints”: {
“IntervalInSeconds”: 300,
“SizeInMBs”: 5
},
“EncryptionConfiguration”: {
“NoEncryptionConfig”: “NoEncryption”
},
“CompressionFormat”: “UNCOMPRESSED”,
“BucketARN”: “arn:aws:s3:::tcpdump2525″
}
}
]
}
}

 Put record:

$  aws firehose put-record –delivery-stream-name FHtoS3-TCPDUMP –record Data=test
{
“RecordId”: “2allgu0ndNyfq0L/oCdckqCPiojGbnQUZD/bDRnc8PDkkY5d2D0stJGoFKzxs1mrgHZClc+M0G2jo5xoQ/QZ0pqNABmohbTrzN+yk5xKWeGoy7H4q6A1GKN91VepLJU3feaAwmgIdIjm0wvoHteZmPaJIxYGVb1ASgCLrxnwO7VjpvDmiS0FHUMmAqY7wrWKKvw2+qwbZA2UvtPsrUvmPgrmeezZVlhI”
}

Leave a Comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>