We will start with Introduction to Logstash and end post with installing logstash from binaries and docker.
Logstash is data processing pipeline which ingests the data simultaneously from multiple data sources, transform it and send it to different stash i.e. Elasticsearch, Redis, database, rest endpoint etc. For example; Ingesting logs files; cleaning and transforming it to machine and human readable formats.
There are three components in Logstash i.e. Inputs, Filters and Outputs.
It ingests data of any kind, shape and size. For examples: Logs, AWS metrics, Instance health metrics etc.
Logstash filters parse each event, build a structure, enrich the data in event and also transform it to desired form. For example: Enriching geo-location from IP using GEO-IP filter, Anonymize PII information from events, transforming unstructured data to structural data using GROK filters etc.
This is the sink layer. There are many output plugins i.e. Elasticsearch, Email, Slack, Datadog, Database persistence etc.
As of writing Logstash(6.2.3) requires Java 8 to run. To check the java version run the following command.
java -versionThe output on my system is
java version "1.8.0_161"
Java(TM) SE Runtime Environment (build 1.8.0_161-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.161-b12, mixed mode)
If Java 8 is not installed then please download it from Oracle website and follows instruction for installation. Also, set the
JAVA_HOMEvariable.
You can directly download the binaries from its official site.
Installation with APT
//ADD PUBLIC SIGNING KEY
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
//add https-transports
sudo apt-get install apt-transport-https
//save the repository definition
echo "deb https://artifacts.elastic.co/packages/6.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-6.x.list
//installation command
sudo apt-get update && sudo apt-get install logstashInstallation with YUM
// Download and install the public signing key
rpm --import https://artifacts.elastic.co/GPG-KEY-elasticsearchAdd the following in a new .repo file in your /etc/yum.repos.d/ directory.
[logstash-6.x]
name=Elastic repository for 6.x packages
baseurl=https://artifacts.elastic.co/packages/6.x/yum
gpgcheck=1
gpgkey=https://artifacts.elastic.co/GPG-KEY-elasticsearch
enabled=1
autorefresh=1
type=rpm-md
// Installation command
sudo yum install logstashdocker pull docker.elastic.co/logstash/logstash:6.2.3