ELK (Elasticsearch + Logstash + Kibana) – To analysis TCPDUMP output from EC2 instance

What is Logstash?

 

Logstash is an open source data collection engine with real-time pipelining capabilities. 
Logstash can dynamically unify data from disparate sources and normalize the data into destinations of your choice.
Cleanse and democratize all your data for diverse advanced downstream analytics and visualization use cases.
 Process Any Data, From Any Source
  • Centralize data processing of all types
  • Normalize varying schema and formats
  • Quickly extend to custom log formats
  • Easily add plugins for custom data sources
  • The ingestion workhorse for Elasticsearch and kibana and more
  • Community-extensible and developer-friendly plugin ecosystem

 

1. Launch an ec2 instance.

2. Install <TCP dump>  and run in background to collect logs.

$ yum install tcpdump -y

$ nohup tcpdump >> /tmp/tcpdump.out &

3. Install logstash and configure

Download package:

a. wget https://download.elastic.co/logstash/logstash/packages/centos/logstash-all-plugins-2.3.4-1.noarch.rpm

b. yum install logstash-all-plugins-2.3.4-1.noarch.rpm

c. vi  /etc/logstash/conf.d/logstash.conf
input {
file {
path => “/tmp/tcpdump.out”
}
}
output {
elasticsearch {
hosts => “http://myexampledomain0000000001.us-east-1.es.amazonaws.com”
index => “hari”
}
}

d. service logstash start

e. chkconfig logstash on

f. check tail -f /var/log/logstash/logstash.log

{:timestamp=>”2016-07-29T23:40:58.595000+0000″, :message=>”Pipeline main started”}

4. Check the aws elasticseach to ensure new index “tcpdump” created.

 

 

 

Leave a Comment

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>