Starting Clean - Kibana
-
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
XS1 - configs say to send all logs to ELK server)
ELK - accepting logsDoes it say that? Which config are you talking about? Filebeat says that, but it sounds like Filebeat also sees no logs to send.
Of course there is nothing for filebeat to see, the syslog inside the XS1 server is forwarding all logs to the ELK server directly.
Not sure why you guys wanted him to install filebeat in the first place. Filebeat only seems useful as long as you are keeping log files on the local server in addition to forwarding them to something like an ELK server.
-
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
Syslog is the process that writes the logs. Without it, logs don't exist. They don't get written locally, they don't get sent anywhere, Filebeat has nothing to read....
OK here is the discussion issue.
You're saying syslog is the process on each server generating the logs.
I'm saying that syslog is a deamon that accepts log data forwarded to it.is it both?
Of course. Individual processes on the server send to syslog so that it can process them. That's how they all end up aggregated together in /var/log/messages. It's like SMTP in that way... mail in, mail out. Only here it is logs in, logs out. Syslog servers talk to syslog servers. ELK is a syslog server, too.
So, Dustin turned off the external syslog daemon on the syslog1 server in my example, AND set the syslog server on XS1 to forward logs to the ELK server.
But then what is Filebeat for? Filebeat only works if the logs are not fowarded elsewhere. So there we have a disconnect. If the logs are going to be forwarded by syslog (rsyslog, in this case) then that needs to follow the Digital Ocean guide that I linked, not my guide as mine is for Filebeat which uses local files, not syslogging daemons. The two cannot be mixed together, it will make a mess at best and won't work at all at worst.
-
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
Syslog is the process that writes the logs. Without it, logs don't exist. They don't get written locally, they don't get sent anywhere, Filebeat has nothing to read....
OK here is the discussion issue.
You're saying syslog is the process on each server generating the logs.
I'm saying that syslog is a deamon that accepts log data forwarded to it.is it both?
Of course. Individual processes on the server send to syslog so that it can process them. That's how they all end up aggregated together in /var/log/messages. It's like SMTP in that way... mail in, mail out. Only here it is logs in, logs out. Syslog servers talk to syslog servers. ELK is a syslog server, too.
So, Dustin turned off the external syslog daemon on the syslog1 server in my example, AND set the syslog server on XS1 to forward logs to the ELK server.
But then what is Filebeat for? Filebeat only works if the logs are not fowarded elsewhere. So there we have a disconnect. If the logs are going to be forwarded by syslog (rsyslog, in this case) then that needs to follow the Digital Ocean guide that I linked, not my guide as mine is for Filebeat which uses local files, not syslogging daemons. The two cannot be mixed together, it will make a mess at best and won't work at all at worst.
You guys told him to install it when he said there were no logs on the ELK server - not realizing he used syslog's own ability to forward the logs to ELK directly.
-
@scottalanmiller said in Starting Clean - Kibana:
The two cannot be mixed together, it will make a mess at best and won't work at all at worst.
now we're having a conversation!
Sweet, progress!
-
@scottalanmiller said in Starting Clean - Kibana:
If the logs are going to be forwarded by syslog (rsyslog, in this case)
Does syslog have to be replaced by rsyslog on the XS box?
.
.
.
.then that needs to follow the Digital Ocean guide that I linked, not my guide as mine is for Filebeat which uses local files, not syslogging daemons. The two cannot be mixed together, it will make a mess at best and won't work at all at worst.
-
The goal, very simply is to have no logs local to the XS boot media at all. (In most of our cases USB drives).
Just to clarify.
Now what digitalocean guide @scottalanmiller ?
-
-
@Dashrender said in Starting Clean - Kibana:
Not sure why you guys wanted him to install filebeat in the first place. Filebeat only seems useful as long as you are keeping log files on the local server in addition to forwarding them to something like an ELK server.
Correct. That's all that my guide is built for.
-
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
If the logs are going to be forwarded by syslog (rsyslog, in this case)
Does syslog have to be replaced by rsyslog on the XS box?
.
.
.
.then that needs to follow the Digital Ocean guide that I linked, not my guide as mine is for Filebeat which uses local files, not syslogging daemons. The two cannot be mixed together, it will make a mess at best and won't work at all at worst.
syslog is a protocol, rsyslog is an implementation. rsyslog is a syslog server.
-
-
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
Didn't I link that one above?
yes I just brought it down for him
-
OK so lets make a new topic, one where everything is very clearly explained. . . .
FML spent like 2 days pulling my hair out and @scottalanmiller @Danp and everyone else here is telling me "is this installed?".. . .
gah
-
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
If the logs are going to be forwarded by syslog (rsyslog, in this case)
Does syslog have to be replaced by rsyslog on the XS box?
.
.
.
.then that needs to follow the Digital Ocean guide that I linked, not my guide as mine is for Filebeat which uses local files, not syslogging daemons. The two cannot be mixed together, it will make a mess at best and won't work at all at worst.
syslog is a protocol, rsyslog is an implementation. rsyslog is a syslog server.
Without looking at the DO install instructions - what does rsyslog do in this case?
-
Why is the step 4 needed at all, at least in regards to rsyslog? Why can't you just use the native syslog to forward the logs? Is it because the DO instructions assume you want to leave a copy of the logs local as well?
-
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
If the logs are going to be forwarded by syslog (rsyslog, in this case)
Does syslog have to be replaced by rsyslog on the XS box?
.
.
.
.then that needs to follow the Digital Ocean guide that I linked, not my guide as mine is for Filebeat which uses local files, not syslogging daemons. The two cannot be mixed together, it will make a mess at best and won't work at all at worst.
syslog is a protocol, rsyslog is an implementation. rsyslog is a syslog server.
Without looking at the DO install instructions - what does rsyslog do in this case?
It writes the local logs by default. You can configure it to send elsewhere if you want.
-
@Dashrender said in Starting Clean - Kibana:
Why is the step 4 needed at all, at least in regards to rsyslog? Why can't you just use the native syslog to forward the logs? Is it because the DO instructions assume you want to leave a copy of the logs local as well?
Which step four?
-
OK In reading the DO instructions - I see that rsyslog converts the log data into the JSON format. Maybe syslog can do that, maybe not, but in this case, the DO instructions are definitely having rsyslog do this.
Does anyone know if syslog can be set to output the log data as JSON compliant (based on a provided template) so the rsyslog portion can be skipped altogether?
-
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
Why is the step 4 needed at all, at least in regards to rsyslog? Why can't you just use the native syslog to forward the logs? Is it because the DO instructions assume you want to leave a copy of the logs local as well?
Which step four?
Let's skip that and just move on to my next question
-
@Dashrender said in Starting Clean - Kibana:
OK In reading the DO instructions - I see that rsyslog converts the log data into the JSON format. Maybe syslog can do that, maybe not, but in this case, the DO instructions are definitely having rsyslog do this.
Does anyone know if syslog can be set to output the log data as JSON compliant (based on a provided template) so the rsyslog portion can be skipped altogether?
What is this syslog server that you are talking about?
-
@Dashrender you do realize that when you say "the syslog server" and when you say "rsyslog" that in both cases you are discussing the same process?
