Starting Clean - Kibana
-
The goal, very simply is to have no logs local to the XS boot media at all. (In most of our cases USB drives).
Just to clarify.
Now what digitalocean guide @scottalanmiller ?
-
-
@Dashrender said in Starting Clean - Kibana:
Not sure why you guys wanted him to install filebeat in the first place. Filebeat only seems useful as long as you are keeping log files on the local server in addition to forwarding them to something like an ELK server.
Correct. That's all that my guide is built for.
-
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
If the logs are going to be forwarded by syslog (rsyslog, in this case)
Does syslog have to be replaced by rsyslog on the XS box?
.
.
.
.then that needs to follow the Digital Ocean guide that I linked, not my guide as mine is for Filebeat which uses local files, not syslogging daemons. The two cannot be mixed together, it will make a mess at best and won't work at all at worst.
syslog is a protocol, rsyslog is an implementation. rsyslog is a syslog server.
-
-
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
Didn't I link that one above?
yes I just brought it down for him
-
OK so lets make a new topic, one where everything is very clearly explained. . . .
FML spent like 2 days pulling my hair out and @scottalanmiller @Danp and everyone else here is telling me "is this installed?".. . .
gah
-
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
If the logs are going to be forwarded by syslog (rsyslog, in this case)
Does syslog have to be replaced by rsyslog on the XS box?
.
.
.
.then that needs to follow the Digital Ocean guide that I linked, not my guide as mine is for Filebeat which uses local files, not syslogging daemons. The two cannot be mixed together, it will make a mess at best and won't work at all at worst.
syslog is a protocol, rsyslog is an implementation. rsyslog is a syslog server.
Without looking at the DO install instructions - what does rsyslog do in this case?
-
Why is the step 4 needed at all, at least in regards to rsyslog? Why can't you just use the native syslog to forward the logs? Is it because the DO instructions assume you want to leave a copy of the logs local as well?
-
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
If the logs are going to be forwarded by syslog (rsyslog, in this case)
Does syslog have to be replaced by rsyslog on the XS box?
.
.
.
.then that needs to follow the Digital Ocean guide that I linked, not my guide as mine is for Filebeat which uses local files, not syslogging daemons. The two cannot be mixed together, it will make a mess at best and won't work at all at worst.
syslog is a protocol, rsyslog is an implementation. rsyslog is a syslog server.
Without looking at the DO install instructions - what does rsyslog do in this case?
It writes the local logs by default. You can configure it to send elsewhere if you want.
-
@Dashrender said in Starting Clean - Kibana:
Why is the step 4 needed at all, at least in regards to rsyslog? Why can't you just use the native syslog to forward the logs? Is it because the DO instructions assume you want to leave a copy of the logs local as well?
Which step four?
-
OK In reading the DO instructions - I see that rsyslog converts the log data into the JSON format. Maybe syslog can do that, maybe not, but in this case, the DO instructions are definitely having rsyslog do this.
Does anyone know if syslog can be set to output the log data as JSON compliant (based on a provided template) so the rsyslog portion can be skipped altogether?
-
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
Why is the step 4 needed at all, at least in regards to rsyslog? Why can't you just use the native syslog to forward the logs? Is it because the DO instructions assume you want to leave a copy of the logs local as well?
Which step four?
Let's skip that and just move on to my next question
-
@Dashrender said in Starting Clean - Kibana:
OK In reading the DO instructions - I see that rsyslog converts the log data into the JSON format. Maybe syslog can do that, maybe not, but in this case, the DO instructions are definitely having rsyslog do this.
Does anyone know if syslog can be set to output the log data as JSON compliant (based on a provided template) so the rsyslog portion can be skipped altogether?
What is this syslog server that you are talking about?
-
@Dashrender you do realize that when you say "the syslog server" and when you say "rsyslog" that in both cases you are discussing the same process?
-
Oh, I see the confusion, hold on....
-
Okay, in the DO example, they are using a separate rsyslog aggregator for some reason, probably because they are doing a site to site trunking of the logs, rather than having every device at the remote site send the logs individually with its own connection. Okay, so @Dashrender is referring to the central rsyslog server as rsyslog and the local one as syslog. All of the syslogs, everywhere, are rsyslog. syslog is a generic name for any syslog server, rsyslog is the specific implementation used on nearly any Linux system today.
That's that confusion.
-
@scottalanmiller so the question is, how do we setup Kibana with Elk and Logstash and redirect all of the XS logs to it?
That's the end all of this conversation..
-
@DustinB3403 said in Starting Clean - Kibana:
@scottalanmiller so the question is, how do we setup Kibana with Elk and Logstash and redirect all of the XS logs to it?
That's the end all of this conversation..
- Kibana isn't part of that conversation, that's how we look at the logs after this is all done.
- It wouldn't be with Filebeat, it has to be directly with rsyslog, so this thread is the wrong one as this whole thread is about filebeat and Kibana, rather than Logstash and rsyslog.
-
@scottalanmiller said in Starting Clean - Kibana:
@DustinB3403 said in Starting Clean - Kibana:
@scottalanmiller so the question is, how do we setup Kibana with Elk and Logstash and redirect all of the XS logs to it?
That's the end all of this conversation..
- Kibana isn't part of that conversation, that's how we look at the logs after this is all done.
- It wouldn't be with Filebeat, it has to be directly with rsyslog, so this thread is the wrong one as this whole thread is about filebeat and Kibana, rather than Logstash and rsyslog.
Well lets divert our attention away from this fiasco and start a new topic....
