What & Why: Log Management in Cloud

Logging is often not the most exciting part of an application, but definitely an important one. Logging and managing logs becomes even more important in a multi node clustered environments where logs are spread across nodes. It is not a very productive proposition to log on to each node to check what is happening on a node or to trace the transaction which might be spread across multiple nodes. Moreover ability to filter and parse logs on a node is limited by native tools such as grep on *nix systems. Is there a better solution? Absolutely! Welcome to log management or centralized logging, demonstrated at a high level in following image

This might sound complicated to setup, but let’s look at the benefits and then see how easy it is to setup log management.

Production Troubleshooting & Application Debugging

If you are troubleshooting a live application, logs will be the lifeline which will save your day. On development environments, there are enough tools at hand which make debugging application much easier. But for an application which is live in production, logs provide key information about what is going on in the application. On the other hand too verbose logging might hamper performance of an application,but that is a separate discussion and there are tools available to control level of logging based on environments.

“debugging statements stay with the program; debugging sessions are transient”

If you have applications deployed in private/public cloud, or even deployed within enterprise in a clustered fashion, it will be much easier to troubleshoot and debug application if all your logs can be searched and parsed all in one place. Inefficiencies of scrolling over thousands of lines of raw data through multiple nodes and trying to connect dots would be solved by a log management system which provides a unified view of all logs and a much richer query interface.

Data Analysis

A running application has multiple user interactions and transactions running and being able to gather real time insights from these would be extremely valuable.  Applications can be designed to log desired information about transactions and this information from logs can be analyzed and monitored in real time. Just imagine if CEO of a retail company could see in real time statistics of how many shopping carts are converted in actual buys. While analytics is a huge field in itself, you got the idea of possibilities.

Monitoring and alerting

Being able to report on errors, exceptions and crashes in real time can lead to actions which reduce downtime and much proactive response to issues in a live application. Alerts can be setup based on severity level and actions can be taken accordingly.

Features of a Log management system

While every product and service offers different flavours of features, we consolidate here some features you should look out for.

Drill Down Capability

Time based drill down is a very standard feature of any log management system and offers ability to look at patterns plotted vs time, which can be drilled down further. For example look at a screenshot of LogStash:

Regular expressions and filtering

Being able to filter on random text, or being able to click on strings from the result shown on UI makes navigating and finding relevant logs much easier and quicker. Also a regular expression library like grok for example shipped with LogStash helps using regular expressions a easier job.

Alerts and notifications

A good log management system should be able to hook to multiple systems like email, websockets for delivering information. That increases reach of the system and it’s usefulness.

API

Last but not the least, having a API of the log management system exposed can greatly increase adoption and extensibility of platform in some cases.

Factors to Consider

Here is a definitive questionnaire you should analyze for choosing your log management system:

• Which type of applications you have and do you have sufficient hooks available to plug into log management system? For example if you would like Android apps to log data to log management server, do you have a lightweight library supporting the same?
• Does the language in which applications are developed has easier ways to communicate to a log management server? For example log4j has SysLogAppender which can talk to Syslog which in turn can talk to log management system.
• Would you like to setup your own log management systems or you would like to use log management provider’s services? (Refer to following sections for both approaches)
• If your application is hosted on a PaaS platform, can you integrate the platform with your own or hosted log management servers?

Log Management: Set Up Yourself

If you are planning to set up your own log management system, either within your enterprise or on hosted environments you have quite a few choices. If your organization has budgets to buy support and want enterprise grade system without a team to maintain the system inhouse, then Splunk is probably a good fit. If you have a team inhouse which is ready to look after the system and prefers a DIY approach, then you can choose from a set of tools for each specific purpose. Log storage and indexing can be done in MongoDB, ElasticSearch to name a few. Tools like LogStash, Kibana, Graphite, Greylog2 offer different features and you can mix and match to suit your needs.

Log Management as a Service

If you want to add Log Management as a service to your existing apps in cloud, you have choices like Loggly, papertrail, logentries among the main players. While we won’t be discussing them in detail in this post, each one offers it’s unique flavors of log management strategy.

Conclusion

You can decide not to give enough importance to logging in your applications but the benefits of logging and a good log management system are far reaching. In a world where horizontally scaled environments are the norm and cloud is being adopted day by day, a good log management system is a necessity but can prove to be a differentiator.  Organizations have already started reaping benefits of log management system from production debugging to analytics. Has your company started the journey yet? Let us know your thoughts.