As of the time of writing this, the logs are stored on the server where the application runs. The process of debugging, searching and garnering insights on the logs is very difficult.

Hence, Filebeat along with Elastic Search and Kibana has been used to make this process simpler.

The architecture of this setup is as follows:

As we can see in the diagram, Filebeat is a light weight local daemon that runs on every server where our applications run.

The only job of a Filebeat daemon is to collect logs from multiple log files that are generated within a server and publish them over to Elastic Search.

The path to one or more log files can be configured through a “filebeat.yml” file. This file also contains the necessary information to connect to the Elastic Search instance.

The role of the Elastic search instance is to store and index the logs as per the format sent by Filebeat and to allow for future searching and retrieval.

Kibana acts as an UI and allows for visualizing, analyzing, and exploring logs stored within Elastic search.

The Filebeat binary is usually deployed to /var/filebeat with the filebeat.yml configuration file being present in the same directory.

The deployment is done through a Filebeat Ansible Role, for more information, refer https://bitbucket.org/errigal/filebeat/