Comparing ELK and GrayLog
-
In doing projects in the NTG Lab, I've been working with log management. I've always focused on ELK as the leading open source log management system but GrayLog comes up regularly as well. I've now installed both and have both running and am interested in comparing the two to understand the strengths and weaknesses.
First thing is, it is not the straightforward comparison that you would think. In both cases the solution is a stack, not a single product. In both cases the base of the stack, the core database, is ElasticSearch - a powerful, scalable, NoSQL database that handles easy clustering.
ELK is ElasticSearch, LogStash and Kibana as a stack. GrayLog is ElasticSearch and GrayLog or optionally, ElasticSearch, LogStash and GrayLog. If you choose to use LogStash, it really should be thought of as the ELK vs. ELG stacks with only the user interface being unique.
Some key differences thus far:
ELK is more up to date and runs on ElasticSearch 2. GrayLog is still limited to the older, but rather mature, ElasticSearch 1 products (we are testing on the latest ElasticSearch 1.7 system.)
Kibana is extremely difficult to use and is not intuitive at all. GrayLog seems to be easier to get initial reports out of.
Kibana does not have user management and relies on selling an additional, non-free, component to handle that. GrayLog includes user management (via a local MongoDB database) or will attach to LDAP or Active Directory for user management for free as part of the open source solution. There are no "paid add ons" with GrayLog.
First look seems like ELK is far easier to get logs into than GrayLog. But moving to ELG might fix this.
-
Does anyone have an updated comparison to share?
-
Currently I'm leaning towards GrayLog with Logstash, and then adding Kibana or Grafana in the future if needed
-
Gods, I still need to do something with this so bad.
-
@flaxking said in Comparing ELK and GrayLog:
Currently I'm leaning towards GrayLog with Logstash, and then adding Kibana or Grafana in the future if needed
Overall, Graylog has been better for most shops because it is more complete without needing to do a lot of plumbing on your own.
-
@scottalanmiller said in Comparing ELK and GrayLog:
@flaxking said in Comparing ELK and GrayLog:
Currently I'm leaning towards GrayLog with Logstash, and then adding Kibana or Grafana in the future if needed
Overall, Graylog has been better for most shops because it is more complete without needing to do a lot of plumbing on your own.
The Elastic stack looks like it would be fun, but Kibana login authentication would be a problem. Offloading login to nginx + LDAP would be ok, but not being able to define access roles would be a problem for us.
It sounds like there's a decent open source alerts plugin for the Elastic Stack, but built in authentication + alerting does make GrayLog the more attractive solution.
-
I'm getting the feeling that if you want a real Enterprise setup with the Elastic Stack, eventually you will end up having to pay $$
-
We've been planning for Prometheus + Grafana for metrics, so if we open up Elastic Stack as on option, we will have to see if we want to use it for metrics instead.
It doesn't help in the initial decision making that these products can all be combined in all kinds of ways, but I suppose it will help in the future if we need something different without having redo the entire setup to meet a new business need.
-
And what does everyone think of packetbeat? My gut feeling is that it would be a bad idea.
-
@flaxking said in Comparing ELK and GrayLog:
I'm getting the feeling that if you want a real Enterprise setup with the Elastic Stack, eventually you will end up having to pay $$
Above 5GB/day, yes.
-
@flaxking said in Comparing ELK and GrayLog:
I'm getting the feeling that if you want a real Enterprise setup with the Elastic Stack, eventually you will end up having to pay $$
I think the open source version pretty much does what you need.
-
@scottalanmiller said in Comparing ELK and GrayLog:
@flaxking said in Comparing ELK and GrayLog:
I'm getting the feeling that if you want a real Enterprise setup with the Elastic Stack, eventually you will end up having to pay $$
I think the open source version pretty much does what you need.
We need Ops to have access and then Devs to have access only to data from the project they are on the team for.
I suppose we could try to put additional authentication in front of elasticsearch, and then just have multiple Kibana instances all with different access to elasticsearch. Failing that, we would be looking at separate ELK deployments per project - which could be an option, but might kind of suck for Ops
-
@flaxking said in Comparing ELK and GrayLog:
@scottalanmiller said in Comparing ELK and GrayLog:
@flaxking said in Comparing ELK and GrayLog:
I'm getting the feeling that if you want a real Enterprise setup with the Elastic Stack, eventually you will end up having to pay $$
I think the open source version pretty much does what you need.
We need Ops to have access and then Devs to have access only to data from the project they are on the team for.
Doesn't the free open source GrayLog do that for you as it is?
-
@flaxking said in Comparing ELK and GrayLog:
I suppose we could try to put additional authentication in front of elasticsearch, and then just have multiple Kibana instances all with different access to elasticsearch. Failing that, we would be looking at separate ELK deployments per project - which could be an option, but might kind of suck for Ops
Right, this is why ELK doesn't do what you want, but Graylog does. That's exact why Graylog is the general recommendation here, ELK requires a lot of add ons or the enterprise version that you pay for to get basic functionality. But Graylog does it all for free.
-
@scottalanmiller said in Comparing ELK and GrayLog:
@flaxking said in Comparing ELK and GrayLog:
@scottalanmiller said in Comparing ELK and GrayLog:
@flaxking said in Comparing ELK and GrayLog:
I'm getting the feeling that if you want a real Enterprise setup with the Elastic Stack, eventually you will end up having to pay $$
I think the open source version pretty much does what you need.
We need Ops to have access and then Devs to have access only to data from the project they are on the team for.
Doesn't the free open source GrayLog do that for you as it is?
Yeah, which is why I'm leaning towards GrayLog
-
@flaxking said in Comparing ELK and GrayLog:
@scottalanmiller said in Comparing ELK and GrayLog:
@flaxking said in Comparing ELK and GrayLog:
@scottalanmiller said in Comparing ELK and GrayLog:
@flaxking said in Comparing ELK and GrayLog:
I'm getting the feeling that if you want a real Enterprise setup with the Elastic Stack, eventually you will end up having to pay $$
I think the open source version pretty much does what you need.
We need Ops to have access and then Devs to have access only to data from the project they are on the team for.
Doesn't the free open source GrayLog do that for you as it is?
Yeah, which is why I'm leaning towards GrayLog
Oh, I misunderstood your comment about needing to pay for the Elastic stack. Because Graylog is an Elastic stack as wel. ELK and Graylog are competing Elastic stacks.
-
Yes, the ELK stack you must pay to get it working in an enterprise way, that's for certain.
-
@scottalanmiller said in Comparing ELK and GrayLog:
@flaxking said in Comparing ELK and GrayLog:
@scottalanmiller said in Comparing ELK and GrayLog:
@flaxking said in Comparing ELK and GrayLog:
@scottalanmiller said in Comparing ELK and GrayLog:
@flaxking said in Comparing ELK and GrayLog:
I'm getting the feeling that if you want a real Enterprise setup with the Elastic Stack, eventually you will end up having to pay $$
I think the open source version pretty much does what you need.
We need Ops to have access and then Devs to have access only to data from the project they are on the team for.
Doesn't the free open source GrayLog do that for you as it is?
Yeah, which is why I'm leaning towards GrayLog
Oh, I misunderstood your comment about needing to pay for the Elastic stack. Because Graylog is an Elastic stack as wel. ELK and Graylog are competing Elastic stacks.
ELK + Beats is now rebranded as "The Elastic Stack"
Strategic marketing decision -
@flaxking said in Comparing ELK and GrayLog:
@scottalanmiller said in Comparing ELK and GrayLog:
@flaxking said in Comparing ELK and GrayLog:
@scottalanmiller said in Comparing ELK and GrayLog:
@flaxking said in Comparing ELK and GrayLog:
@scottalanmiller said in Comparing ELK and GrayLog:
@flaxking said in Comparing ELK and GrayLog:
I'm getting the feeling that if you want a real Enterprise setup with the Elastic Stack, eventually you will end up having to pay $$
I think the open source version pretty much does what you need.
We need Ops to have access and then Devs to have access only to data from the project they are on the team for.
Doesn't the free open source GrayLog do that for you as it is?
Yeah, which is why I'm leaning towards GrayLog
Oh, I misunderstood your comment about needing to pay for the Elastic stack. Because Graylog is an Elastic stack as wel. ELK and Graylog are competing Elastic stacks.
ELK + Beats is now rebranded as "The Elastic Stack"
Strategic marketing decisionOh man, that's confusing.
-
Because Graylog has been an Elastic Stack for a really long time now, as has ELK. Rebranding something new as something that already exists is a mess.
