Grafana Loki - Like Prometheus but for Logs

Grafana is building a new logging solution to treat logs as a first class citizen. inspired by Prometheus and using labels instead of full indexing.

Grafana is building a new logging solution to treat logs as a first-class citizen. inspired by Prometheus and using labels instead of full indexing.

Episode Transcription

Welcome back to The Byte. In this episode, we're going to discuss Loki, the Grafana-based logging solution, so, following the previous episode, we talked about Grafana a little bit, now we're going to talk about some of the features within Grafana, and Loki is one of the new features available. It's still in beta, so we're really looking into this, and I've already installed it. I'm using it to really understand how it works, but the idea behind Loki is to treat logs like Prometheus treats metrics, and they want to make it easy to use, highly scalable, multi-tenant, and they want to take the same concept that Prometheus uses with metrics and labels and apply it to logs.

What does that mean? That means they will not be doing full indexing of logs like, for example, the ELK Stack. They'll only take advantage of labels within the logs, and then you can reuse the same labels between your metrics and your logs, so, if you're using Grafana, you can actually combine some of these data sources. It's going to be easier to use and operate because, obviously, you don't have to do all this full indexing. It's just basically raw logs and it's querying the labels.

Now, as I said, it's still in early days, but I've already played around with it. Now, to install it, it comes with three different components, so Promtrail is an agent that you need to install, and it gathers and ships the logs back to Loki. Loki can either be run in Grafana Cloud or as an open-source. It can run on your own stack and use Grafana as the front end to visualize this information.

Now, I have it running, and I must say it's quite good. It's still rough around the edges. There's still a lot to be desired, but you can see logs, you can dive into details, and it's the same user interface that we have with Grafana, so I see the real value here is having one user interface to see logs in metrics and can combine these things into a single pane of glass as they say.

Now, Grafana has been contributing a lot back to Prometheus, and they say they want to treat Loki as a Prometheus solution and logs should be a first-class citizen, and they say they've asked lots of developers, and you know they're really creating so much logging now that it's just not cost-effective, so they want to make a cost-effective solution and send everything to Loki, and it's just an easier situation, so it's not going to be as detailed as like an Elasticsearch Stack, but they'll be more ... They'll be easier to use and have more efficiency, so I think that's quite key in this aspect.

Now, what's also interesting is it's going to be a separate data source, so, within Grafana, you have different data sources, so you can use Prometheus and [FlockDB 00:03:10], and Loki will be its own data source. Now, you cannot at the moment use the data source to build graphs or different visualizations, but it's coming. I mean, when you try to do one, it actually pops up and tells you, "Hey, it's not ready yet, but check back soon."

I really see the value because they're basing it on the same principle as the cloud native foundation of keeping it super simple, flexible, scalable, et cetera. I mean, if I just look through the interface, I can really dive down quickly into the metrics and I have ... On the left-hand side, I have an explore button, so I click on explore and then I have my Loki data source and I can see all the log files that I'm collecting, and I can just quickly jump into var logs and then it starts parsing var logs.

Now, it gives you information. It's just raw format, but you can then quickly query based on the different labels within here, so, if I click on one of the labels, it pops out and gives me all the metadata for this particular label. It then tells me flags for the labels or errors, unknowns, infos, warnings.

Like I said, if the concept continues the way it's going, they want to make logs a first-class citizen and make it easy to consume and you don't need a giant cluster to store your logs anymore, I see a huge benefit. I have it running on my test system, and it's not much overhead, so I'm shipping basically all the container logs, all the system logs to it, and I don't notice much of a performance hit. I mean, I'm not generating a lot of traffic, but, at the same time, it's not consuming a lot of resources just to run either, so there is some positives there.

I'll continue looking into it. I'll provide the links. Grafana Loki is the project, so [grafana.com/loki](https://grafana.com/loki) and [GitHub](https://github.com/grafana/loki). That's all for this episode. Thank you and we'll see you next time.


Brian Christner