Starting Clean - Kibana
-
OK I'm done with this trial......
I'm just going in circles and before I break something I need a breather...
-
Damn - I'm glad Dustin ran through this first I think I would have been pulling my hair out LONG before he did.
Dustin (and I) want NO local logging on our XS boxes. We would love to have this log information inside something like ELK where we can do easy searches and graphs on it.
If Filebeat forwards the local logs to the ELK server how do we get a situation where no local logs are stored on the XS boxes?
Do we seriously have to setup a syslog server in the middle that does nothing but collect logs and run Filebeat, which then forwards the syslog's logs to the ELK server?
-
Hopefully there is a way to send the logs to Logstash directly, instead of sending them via Filebeat.
-
@DustinB3403 said in Starting Clean - Kibana:
@Danp said in Starting Clean - Kibana:
@DustinB3403 Did you use XC to change the logging to a remote location?
Yes.
Check the logs there, then.
-
@Dashrender said in Starting Clean - Kibana:
Hopefully there is a way to send the logs to Logstash directly, instead of sending them via Filebeat.
Of course, syslog, which you are already running. But Filebeat makes Logstash ingest easier.
-
@scottalanmiller said in Starting Clean - Kibana:
@DustinB3403 said in Starting Clean - Kibana:
@Danp said in Starting Clean - Kibana:
@DustinB3403 Did you use XC to change the logging to a remote location?
Yes.
Check the logs there, then.
I'm guessing there is not there - because the 'there' that he is forwarding them to is the ELK server.
-
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
@DustinB3403 said in Starting Clean - Kibana:
@Danp said in Starting Clean - Kibana:
@DustinB3403 Did you use XC to change the logging to a remote location?
Yes.
Check the logs there, then.
I'm guessing there is not there - because the 'there' that he is forwarding them to is the ELK server.
You can't both forward to ELK and use Filebeat!!
Of course this isn't working. -
@Dashrender said in Starting Clean - Kibana:
Hopefully there is a way to send the logs to Logstash directly, instead of sending them via Filebeat.
-
@scottalanmiller migth I ask you try doing this very same thing on an XS installation and outline exactly what you do so that others can replicate it.
As it is now, I'm beyond upset with just trying to get this going.
-
@Dashrender said
Dustin (and I) want NO local logging on our XS boxes.
The only way I ever got this to work was the "dirty little trick" that is mentioned in the comments of that article.
Because on mine, it kept writing locally REGARDLESS of what I did.
@scottalanmiller determined the best way was to move /var/logs
-
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
Hopefully there is a way to send the logs to Logstash directly, instead of sending them via Filebeat.
Of course, syslog, which you are already running. But Filebeat makes Logstash ingest easier.
You've lost me. you say of course, syslog, but Dustin took down the syslog server when he stood the ELK server up, because the assumption was/is that the ELK server replaced the syslog server.
-
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
Hopefully there is a way to send the logs to Logstash directly, instead of sending them via Filebeat.
Of course, syslog, which you are already running. But Filebeat makes Logstash ingest easier.
You've lost me. you say of course, syslog, but Dustin took down the syslog server when he stood the ELK server up, because the assumption was/is that the ELK server replaced the syslog server.
Huh? If he turned off the syslogging server, then there ARE no logs to send on. So that explains everything. Syslog is the logging service. Without it, there are no logs to send anywhere.
-
Syslog is the process that writes the logs. Without it, logs don't exist. They don't get written locally, they don't get sent anywhere, Filebeat has nothing to read....
-
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
Hopefully there is a way to send the logs to Logstash directly, instead of sending them via Filebeat.
Of course, syslog, which you are already running. But Filebeat makes Logstash ingest easier.
You've lost me. you say of course, syslog, but Dustin took down the syslog server when he stood the ELK server up, because the assumption was/is that the ELK server replaced the syslog server.
Huh? If he turned off the syslogging server, then there ARE no logs to send on. So that explains everything. Syslog is the logging service. Without it, there are no logs to send anywhere.
I think I'm even more confused now.
Syslog is a service that accepts logs from other sources (sure one of those sources could the the server syslog is intalled on - but has nothing to do with the logging system of the syslog host server - I hope you follow that) - or at least that's how I understand it.
So, if he has the following:
XS1 - Xenserver generating logs (configs say to send all logs to syslog1 server)
syslog1 - accepting logsThis was working for him. But when he moved to the following:
XS1 - configs say to send all logs to ELK server)
ELK - accepting logsThis does not work. why not? Does he need to install a syslog daemon on this box, then install Filebeat on that syslog server.
- many posts have been writen before I posted this, but I'm leaving it for learning.
-
@scottalanmiller said in Starting Clean - Kibana:
Syslog is the process that writes the logs. Without it, logs don't exist. They don't get written locally, they don't get sent anywhere, Filebeat has nothing to read....
OK here is the discussion issue.
You're saying syslog is the process on each server generating the logs.
I'm saying that syslog is a deamon that accepts log data forwarded to it.is it both?
-
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
Syslog is the process that writes the logs. Without it, logs don't exist. They don't get written locally, they don't get sent anywhere, Filebeat has nothing to read....
OK here is the discussion issue.
You're saying syslog is the process on each server generating the logs.
I'm saying that syslog is a deamon that accepts log data forwarded to it.is it both?
Of course. Individual processes on the server send to syslog so that it can process them. That's how they all end up aggregated together in /var/log/messages. It's like SMTP in that way... mail in, mail out. Only here it is logs in, logs out. Syslog servers talk to syslog servers. ELK is a syslog server, too.
-
@Dashrender said in Starting Clean - Kibana:
XS1 - configs say to send all logs to ELK server)
ELK - accepting logsDoes it say that? Which config are you talking about? Filebeat says that, but it sounds like Filebeat also sees no logs to send.
-
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
Syslog is the process that writes the logs. Without it, logs don't exist. They don't get written locally, they don't get sent anywhere, Filebeat has nothing to read....
OK here is the discussion issue.
You're saying syslog is the process on each server generating the logs.
I'm saying that syslog is a deamon that accepts log data forwarded to it.is it both?
Of course. Individual processes on the server send to syslog so that it can process them. That's how they all end up aggregated together in /var/log/messages. It's like SMTP in that way... mail in, mail out. Only here it is logs in, logs out. Syslog servers talk to syslog servers. ELK is a syslog server, too.
So, Dustin turned off the external syslog daemon on the syslog1 server in my example, AND set the syslog server on XS1 to forward logs to the ELK server.
-
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
XS1 - configs say to send all logs to ELK server)
ELK - accepting logsDoes it say that? Which config are you talking about? Filebeat says that, but it sounds like Filebeat also sees no logs to send.
Of course there is nothing for filebeat to see, the syslog inside the XS1 server is forwarding all logs to the ELK server directly.
Not sure why you guys wanted him to install filebeat in the first place. Filebeat only seems useful as long as you are keeping log files on the local server in addition to forwarding them to something like an ELK server.
-
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
Syslog is the process that writes the logs. Without it, logs don't exist. They don't get written locally, they don't get sent anywhere, Filebeat has nothing to read....
OK here is the discussion issue.
You're saying syslog is the process on each server generating the logs.
I'm saying that syslog is a deamon that accepts log data forwarded to it.is it both?
Of course. Individual processes on the server send to syslog so that it can process them. That's how they all end up aggregated together in /var/log/messages. It's like SMTP in that way... mail in, mail out. Only here it is logs in, logs out. Syslog servers talk to syslog servers. ELK is a syslog server, too.
So, Dustin turned off the external syslog daemon on the syslog1 server in my example, AND set the syslog server on XS1 to forward logs to the ELK server.
But then what is Filebeat for? Filebeat only works if the logs are not fowarded elsewhere. So there we have a disconnect. If the logs are going to be forwarded by syslog (rsyslog, in this case) then that needs to follow the Digital Ocean guide that I linked, not my guide as mine is for Filebeat which uses local files, not syslogging daemons. The two cannot be mixed together, it will make a mess at best and won't work at all at worst.
