Starting Clean - Kibana
-
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
If the logs are going to be forwarded by syslog (rsyslog, in this case)
Does syslog have to be replaced by rsyslog on the XS box?
.
.
.
.then that needs to follow the Digital Ocean guide that I linked, not my guide as mine is for Filebeat which uses local files, not syslogging daemons. The two cannot be mixed together, it will make a mess at best and won't work at all at worst.
syslog is a protocol, rsyslog is an implementation. rsyslog is a syslog server.
Without looking at the DO install instructions - what does rsyslog do in this case?
It writes the local logs by default. You can configure it to send elsewhere if you want.
-
@Dashrender said in Starting Clean - Kibana:
Why is the step 4 needed at all, at least in regards to rsyslog? Why can't you just use the native syslog to forward the logs? Is it because the DO instructions assume you want to leave a copy of the logs local as well?
Which step four?
-
OK In reading the DO instructions - I see that rsyslog converts the log data into the JSON format. Maybe syslog can do that, maybe not, but in this case, the DO instructions are definitely having rsyslog do this.
Does anyone know if syslog can be set to output the log data as JSON compliant (based on a provided template) so the rsyslog portion can be skipped altogether?
-
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
Why is the step 4 needed at all, at least in regards to rsyslog? Why can't you just use the native syslog to forward the logs? Is it because the DO instructions assume you want to leave a copy of the logs local as well?
Which step four?
Let's skip that and just move on to my next question
-
@Dashrender said in Starting Clean - Kibana:
OK In reading the DO instructions - I see that rsyslog converts the log data into the JSON format. Maybe syslog can do that, maybe not, but in this case, the DO instructions are definitely having rsyslog do this.
Does anyone know if syslog can be set to output the log data as JSON compliant (based on a provided template) so the rsyslog portion can be skipped altogether?
What is this syslog server that you are talking about?
-
@Dashrender you do realize that when you say "the syslog server" and when you say "rsyslog" that in both cases you are discussing the same process?
-
Oh, I see the confusion, hold on....
-
Okay, in the DO example, they are using a separate rsyslog aggregator for some reason, probably because they are doing a site to site trunking of the logs, rather than having every device at the remote site send the logs individually with its own connection. Okay, so @Dashrender is referring to the central rsyslog server as rsyslog and the local one as syslog. All of the syslogs, everywhere, are rsyslog. syslog is a generic name for any syslog server, rsyslog is the specific implementation used on nearly any Linux system today.
That's that confusion.
-
@scottalanmiller so the question is, how do we setup Kibana with Elk and Logstash and redirect all of the XS logs to it?
That's the end all of this conversation..
-
@DustinB3403 said in Starting Clean - Kibana:
@scottalanmiller so the question is, how do we setup Kibana with Elk and Logstash and redirect all of the XS logs to it?
That's the end all of this conversation..
- Kibana isn't part of that conversation, that's how we look at the logs after this is all done.
- It wouldn't be with Filebeat, it has to be directly with rsyslog, so this thread is the wrong one as this whole thread is about filebeat and Kibana, rather than Logstash and rsyslog.
-
@scottalanmiller said in Starting Clean - Kibana:
@DustinB3403 said in Starting Clean - Kibana:
@scottalanmiller so the question is, how do we setup Kibana with Elk and Logstash and redirect all of the XS logs to it?
That's the end all of this conversation..
- Kibana isn't part of that conversation, that's how we look at the logs after this is all done.
- It wouldn't be with Filebeat, it has to be directly with rsyslog, so this thread is the wrong one as this whole thread is about filebeat and Kibana, rather than Logstash and rsyslog.
Well lets divert our attention away from this fiasco and start a new topic....
-
-
Looking through the old threads on this that I can find, the first mention of Filebeat was by @DustinB3403 and that's what sent us down this path, not someone suggesting it (as far as I can tell.) Then he posted on the Filebeat article, which firmed up this path even more. Then in this thread, there was no talk of anything else.
So that Filebeat wasn't the right tool was never really considered because Filebeat was injected from the beginning. That's what led to the crazy confusion.
So a new thread all about using rsyslog to send to Logstash in ELK is what is needed. And the issue appears to be that ELK was never configured to accept syslog files because it's not open by default to listen for them.
And mentioning Kibana doesn't help. KIbana is the K in ELK, but it's not a part that processes logs. You can use Kibana for other things too, like just showing system graphs.
-
Thanks to @scottalanmiller for driving me to drink (now I've got an excuse to give the SO)
-
@scottalanmiller said in Starting Clean - Kibana:
Okay, in the DO example, they are using a separate rsyslog aggregator for some reason, probably because they are doing a site to site trunking of the logs, rather than having every device at the remote site send the logs individually with its own connection. Okay, so @Dashrender is referring to the central rsyslog server as rsyslog and the local one as syslog. All of the syslogs, everywhere, are rsyslog. syslog is a generic name for any syslog server, rsyslog is the specific implementation used on nearly any Linux system today.
That's that confusion.
wow - OK yep - had NO CLUE that was the case. I realize you can replace almost any process with some other process on a Linux system.. but damn - talk about confusion!!!!
-
@scottalanmiller said in Starting Clean - Kibana:
Looking through the old threads on this that I can find, the first mention of Filebeat was by @DustinB3403 and that's what sent us down this path, not someone suggesting it (as far as I can tell.) Then he posted on the Filebeat article, which firmed up this path even more. Then in this thread, there was no talk of anything else.
So that Filebeat wasn't the right tool was never really considered because Filebeat was injected from the beginning. That's what led to the crazy confusion.
So a new thread all about using rsyslog to send to Logstash in ELK is what is needed. And the issue appears to be that ELK was never configured to accept syslog files because it's not open by default to listen for them.
And mentioning Kibana doesn't help. KIbana is the K in ELK, but it's not a part that processes logs. You can use Kibana for other things too, like just showing system graphs.
Yeah I get the whole Kibana wasn't where the problem was - but this was because Dustin (and I) didn't understand where the error was.
The simple setup Dustin did - install Kiwi syslog server, change the XS log config file to send all logs to the Kiwi syslog server (took less than 10 mins) was so brain dead simple, neither of us knew what was failing in ELK. So Dustin started troubleshooting at the point he had direct contact with, Kibana.
