Starting Clean - Kibana
-
@scottalanmiller migth I ask you try doing this very same thing on an XS installation and outline exactly what you do so that others can replicate it.
As it is now, I'm beyond upset with just trying to get this going.
-
@Dashrender said
Dustin (and I) want NO local logging on our XS boxes.
The only way I ever got this to work was the "dirty little trick" that is mentioned in the comments of that article.
Because on mine, it kept writing locally REGARDLESS of what I did.
@scottalanmiller determined the best way was to move /var/logs
-
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
Hopefully there is a way to send the logs to Logstash directly, instead of sending them via Filebeat.
Of course, syslog, which you are already running. But Filebeat makes Logstash ingest easier.
You've lost me. you say of course, syslog, but Dustin took down the syslog server when he stood the ELK server up, because the assumption was/is that the ELK server replaced the syslog server.
-
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
Hopefully there is a way to send the logs to Logstash directly, instead of sending them via Filebeat.
Of course, syslog, which you are already running. But Filebeat makes Logstash ingest easier.
You've lost me. you say of course, syslog, but Dustin took down the syslog server when he stood the ELK server up, because the assumption was/is that the ELK server replaced the syslog server.
Huh? If he turned off the syslogging server, then there ARE no logs to send on. So that explains everything. Syslog is the logging service. Without it, there are no logs to send anywhere.
-
Syslog is the process that writes the logs. Without it, logs don't exist. They don't get written locally, they don't get sent anywhere, Filebeat has nothing to read....
-
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
Hopefully there is a way to send the logs to Logstash directly, instead of sending them via Filebeat.
Of course, syslog, which you are already running. But Filebeat makes Logstash ingest easier.
You've lost me. you say of course, syslog, but Dustin took down the syslog server when he stood the ELK server up, because the assumption was/is that the ELK server replaced the syslog server.
Huh? If he turned off the syslogging server, then there ARE no logs to send on. So that explains everything. Syslog is the logging service. Without it, there are no logs to send anywhere.
I think I'm even more confused now.
Syslog is a service that accepts logs from other sources (sure one of those sources could the the server syslog is intalled on - but has nothing to do with the logging system of the syslog host server - I hope you follow that) - or at least that's how I understand it.
So, if he has the following:
XS1 - Xenserver generating logs (configs say to send all logs to syslog1 server)
syslog1 - accepting logsThis was working for him. But when he moved to the following:
XS1 - configs say to send all logs to ELK server)
ELK - accepting logsThis does not work. why not? Does he need to install a syslog daemon on this box, then install Filebeat on that syslog server.
- many posts have been writen before I posted this, but I'm leaving it for learning.
-
@scottalanmiller said in Starting Clean - Kibana:
Syslog is the process that writes the logs. Without it, logs don't exist. They don't get written locally, they don't get sent anywhere, Filebeat has nothing to read....
OK here is the discussion issue.
You're saying syslog is the process on each server generating the logs.
I'm saying that syslog is a deamon that accepts log data forwarded to it.is it both?
-
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
Syslog is the process that writes the logs. Without it, logs don't exist. They don't get written locally, they don't get sent anywhere, Filebeat has nothing to read....
OK here is the discussion issue.
You're saying syslog is the process on each server generating the logs.
I'm saying that syslog is a deamon that accepts log data forwarded to it.is it both?
Of course. Individual processes on the server send to syslog so that it can process them. That's how they all end up aggregated together in /var/log/messages. It's like SMTP in that way... mail in, mail out. Only here it is logs in, logs out. Syslog servers talk to syslog servers. ELK is a syslog server, too.
-
@Dashrender said in Starting Clean - Kibana:
XS1 - configs say to send all logs to ELK server)
ELK - accepting logsDoes it say that? Which config are you talking about? Filebeat says that, but it sounds like Filebeat also sees no logs to send.
-
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
Syslog is the process that writes the logs. Without it, logs don't exist. They don't get written locally, they don't get sent anywhere, Filebeat has nothing to read....
OK here is the discussion issue.
You're saying syslog is the process on each server generating the logs.
I'm saying that syslog is a deamon that accepts log data forwarded to it.is it both?
Of course. Individual processes on the server send to syslog so that it can process them. That's how they all end up aggregated together in /var/log/messages. It's like SMTP in that way... mail in, mail out. Only here it is logs in, logs out. Syslog servers talk to syslog servers. ELK is a syslog server, too.
So, Dustin turned off the external syslog daemon on the syslog1 server in my example, AND set the syslog server on XS1 to forward logs to the ELK server.
-
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
XS1 - configs say to send all logs to ELK server)
ELK - accepting logsDoes it say that? Which config are you talking about? Filebeat says that, but it sounds like Filebeat also sees no logs to send.
Of course there is nothing for filebeat to see, the syslog inside the XS1 server is forwarding all logs to the ELK server directly.
Not sure why you guys wanted him to install filebeat in the first place. Filebeat only seems useful as long as you are keeping log files on the local server in addition to forwarding them to something like an ELK server.
-
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
Syslog is the process that writes the logs. Without it, logs don't exist. They don't get written locally, they don't get sent anywhere, Filebeat has nothing to read....
OK here is the discussion issue.
You're saying syslog is the process on each server generating the logs.
I'm saying that syslog is a deamon that accepts log data forwarded to it.is it both?
Of course. Individual processes on the server send to syslog so that it can process them. That's how they all end up aggregated together in /var/log/messages. It's like SMTP in that way... mail in, mail out. Only here it is logs in, logs out. Syslog servers talk to syslog servers. ELK is a syslog server, too.
So, Dustin turned off the external syslog daemon on the syslog1 server in my example, AND set the syslog server on XS1 to forward logs to the ELK server.
But then what is Filebeat for? Filebeat only works if the logs are not fowarded elsewhere. So there we have a disconnect. If the logs are going to be forwarded by syslog (rsyslog, in this case) then that needs to follow the Digital Ocean guide that I linked, not my guide as mine is for Filebeat which uses local files, not syslogging daemons. The two cannot be mixed together, it will make a mess at best and won't work at all at worst.
-
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
Syslog is the process that writes the logs. Without it, logs don't exist. They don't get written locally, they don't get sent anywhere, Filebeat has nothing to read....
OK here is the discussion issue.
You're saying syslog is the process on each server generating the logs.
I'm saying that syslog is a deamon that accepts log data forwarded to it.is it both?
Of course. Individual processes on the server send to syslog so that it can process them. That's how they all end up aggregated together in /var/log/messages. It's like SMTP in that way... mail in, mail out. Only here it is logs in, logs out. Syslog servers talk to syslog servers. ELK is a syslog server, too.
So, Dustin turned off the external syslog daemon on the syslog1 server in my example, AND set the syslog server on XS1 to forward logs to the ELK server.
But then what is Filebeat for? Filebeat only works if the logs are not fowarded elsewhere. So there we have a disconnect. If the logs are going to be forwarded by syslog (rsyslog, in this case) then that needs to follow the Digital Ocean guide that I linked, not my guide as mine is for Filebeat which uses local files, not syslogging daemons. The two cannot be mixed together, it will make a mess at best and won't work at all at worst.
You guys told him to install it when he said there were no logs on the ELK server - not realizing he used syslog's own ability to forward the logs to ELK directly.
-
@scottalanmiller said in Starting Clean - Kibana:
The two cannot be mixed together, it will make a mess at best and won't work at all at worst.
now we're having a conversation!
Sweet, progress!
-
@scottalanmiller said in Starting Clean - Kibana:
If the logs are going to be forwarded by syslog (rsyslog, in this case)
Does syslog have to be replaced by rsyslog on the XS box?
.
.
.
.then that needs to follow the Digital Ocean guide that I linked, not my guide as mine is for Filebeat which uses local files, not syslogging daemons. The two cannot be mixed together, it will make a mess at best and won't work at all at worst.
-
The goal, very simply is to have no logs local to the XS boot media at all. (In most of our cases USB drives).
Just to clarify.
Now what digitalocean guide @scottalanmiller ?
-
-
@Dashrender said in Starting Clean - Kibana:
Not sure why you guys wanted him to install filebeat in the first place. Filebeat only seems useful as long as you are keeping log files on the local server in addition to forwarding them to something like an ELK server.
Correct. That's all that my guide is built for.
-
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
If the logs are going to be forwarded by syslog (rsyslog, in this case)
Does syslog have to be replaced by rsyslog on the XS box?
.
.
.
.then that needs to follow the Digital Ocean guide that I linked, not my guide as mine is for Filebeat which uses local files, not syslogging daemons. The two cannot be mixed together, it will make a mess at best and won't work at all at worst.
syslog is a protocol, rsyslog is an implementation. rsyslog is a syslog server.
-
