We will start with Introduction to Logstash and end post with installing logstash from binaries and docker.
Logstash is data processing pipeline which ingests the data simultaneously from multiple data sources, transform it and send it to different
stash i.e. Elasticsearch, Redis, database, rest endpoint etc. For example; Ingesting logs files; cleaning and transforming it to machine and human readable formats.
There are three components in Logstash i.e. Inputs, Filters and Outputs.
It ingests data of any kind, shape and size. For examples: Logs, AWS metrics, Instance health metrics etc.
Logstash filters parse each event, build a structure, enrich the data in event and also transform it to desired form. For example: Enriching geo-location from IP using GEO-IP filter, Anonymize PII information from events, transforming unstructured data to structural data using GROK filters etc.
This is the sink layer. There are many output plugins i.e. Elasticsearch, Email, Slack, Datadog, Database persistence etc.
As of writing Logstash(6.2.3) requires Java 8 to run. To check the java version run the following command.
The output on my system is
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
If Java 8 is not installed then please download it from Oracle website and follows instruction for installation. Also, set the
You can directly download the binaries from its official site.
Installation with APT
//ADD PUBLIC SIGNING KEY
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
sudo apt-get install apt-transport-https
//save the repository definition
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
sudo apt-get update && sudo apt-get install logstash
Installation with YUM
// Download and install the public signing key
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearch
Add the following in a new .repo file in your
name=Elastic repository for 6.x packages
// Installation command
sudo yum install logstash
docker pull docker.elastic.co/logstash/logstash:6.2.3