What’s up!

The software/Web applications we build often rely on more than one web servers and several database servers. In these circumstances, we usually don’t have centralized tools for analyzing and storing logs. This makes locating or identifying different types of events and mapping them with other types of events an almost complicated mess.

What’s the Problem?

A simple exception or error condition, somewhere in the middle of the application system, can be miserable for both the development team and the end user. A significant packet-loss can occur within your network due to some mistake, causing a straightforward and simple job turn into an enormous headache.

The sensible solution would be to know precisely what train of events occurred and created the problem to rise originally and how to avoide it from recurring.

What’s the Solution?

This however, means gathering logs from individual machines, compiling and joining them through some kind of id before you can even start identifying the problem.

This is all within the context that the logs you should be verifying stand in the computer’s working memory or when cat or grep or xargs is inadequate for investigation or analysis.

At present, Applications have moved to cloud platforms such as Digital Ocean, Google Cloud, or Amazon Web Services (AWS).

Cloud Platform’s security tools and logging platforms are crucial tools to these services. Despite the fact these are individual tools, they were designed and built to collaborate.

Tools to facilitate the Solution…

There are numerous tools and their implementations available to handle this issue. Some of them are paid, while many are open source. The main competitors are as follows:

  1. The Elastic Stack ELK (Elastic Search, Logstash and Kibana)
  2. Graylog
  3. Sensu Stack
  4. FluentD

These are some frameworks that can be acquired as hosted and can also be implemented by devs on any free instance as open source and free solution.

Some simple loggers are also available, that only provide logging solution and not monitoring or alerts of any type, which are :

  1. Octopussy.
  2. LOGalyze.
  3. LogPacker.
  4. Logwatch

Comparison between ELK and Graylog (Open Source)

The two better solutions among all the other solutions are ELK and Graylog, Both have some pros and cons.

ELK:

the ELK stack uses Filebeat and many other different Beats , solution to forward and centralize logs. After they’ve been forwarded, they get refined into desired fields with Logstash and made into an ElasticSearch cluster.further, they start the display process using the Kibana dashboard of the stack.

Usability:

Being an advanced suite of products, ELK has a reasonably sheer training period. Moreover, it is complicated to sustain. Against this background, it permits you to perform nearly anything you need from an individual tool. As soon as you overcome the learning curve, it’s a great solution. Logs, metrics, and visualizations are fairly good, and if extra utility is required, you can explore the extensive library of available plugins.

Pricing:

The ELK stack is a free and open-source, However it can get quite expensive. The total cost fluctuates substantially from company to company but relies on features like the quantity of data that your system or systems produce, data retention period, and data accessibility.

Pros:

  • Stable and powerful solution.
  • Logstash enables you to create personalize log reprocessing.
  • Diversity of plugins.
  • Kibana visualizations.
  • Check on how to index data in ElasticSearch.

Cons:

  • Learning curve is quite steep
  • No default logging dashboards are provided by kibana.
  • Management is quite Time consuming and exhaustive.
  • Authentication and Alerting are paid features
  • Cost , not “free” as advertised (system, server, storage and maintenance cost)

All in all, ELK is a strong answer for log management, after you overcome it’s steep learning curve. utilized as a independent and self-contained solution, or if more functionality is required, can be combined with other applications.

GrayLog:

Graylog is a powerful log management solution and uses MongoDB and Elasticsearch to funcion. It has limited in functionality when you step out of the scope of what it is good at. In other words if we try to achieve something aside from its general scope, it’ll turns into something pretty complicated and time-consuming.

For elaborate or complex graphs you will likely have to add other tools like Grafana. an InfluxDB or Graphite data store regarding complex parsing, Graylog provides a great but all or none kind of solution.

Usability:

Graylog’s learning curve is simple to overcome, that provides an almost entirely functional layout in a comparably less time. Additionally all important items are easy to find in the GUI which makes usability leisurely.

Pricing:

Graylog is an open-source tool, which means you can use it for free. There is also enterprise licensing available. Contact sales for more details.

Pros:

  • Authentication and Authorization free.
  • Alerting,Parsing and some elemental graphing.
  • Easy to learn /  learning curve not steep.
  • Quick setup.
  • Mostly GUI-based.

Cons:

  • Restricted scope of what it does well. i.e central logging.
  • Strong parsing ability
  • Graphing is basic. needs to use Grafana or Kibana.
  • Not many plugins available compared to ELK.

Graylog is quick and simple to set up and easy to learn. It provides useful features, but for anything outside of scope, other tools might be incorporated.

Have any questions? Do ask in comments 🙂

Quick read about PM2, Process manager for Node JS applications. Read this article.