Starting Clean - Kibana
-
@Danp said in Starting Clean - Kibana:
@DustinB3403 Did you use XC to change the logging to a remote location?
Yes.
-
@DustinB3403 Then it stops writing to the local logs in some cases as described at the bottom of this article.
-
OK I'm done with this trial......
I'm just going in circles and before I break something I need a breather...
-
Damn - I'm glad Dustin ran through this first I think I would have been pulling my hair out LONG before he did.
Dustin (and I) want NO local logging on our XS boxes. We would love to have this log information inside something like ELK where we can do easy searches and graphs on it.
If Filebeat forwards the local logs to the ELK server how do we get a situation where no local logs are stored on the XS boxes?
Do we seriously have to setup a syslog server in the middle that does nothing but collect logs and run Filebeat, which then forwards the syslog's logs to the ELK server?
-
Hopefully there is a way to send the logs to Logstash directly, instead of sending them via Filebeat.
-
@DustinB3403 said in Starting Clean - Kibana:
@Danp said in Starting Clean - Kibana:
@DustinB3403 Did you use XC to change the logging to a remote location?
Yes.
Check the logs there, then.
-
@Dashrender said in Starting Clean - Kibana:
Hopefully there is a way to send the logs to Logstash directly, instead of sending them via Filebeat.
Of course, syslog, which you are already running. But Filebeat makes Logstash ingest easier.
-
@scottalanmiller said in Starting Clean - Kibana:
@DustinB3403 said in Starting Clean - Kibana:
@Danp said in Starting Clean - Kibana:
@DustinB3403 Did you use XC to change the logging to a remote location?
Yes.
Check the logs there, then.
I'm guessing there is not there - because the 'there' that he is forwarding them to is the ELK server.
-
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
@DustinB3403 said in Starting Clean - Kibana:
@Danp said in Starting Clean - Kibana:
@DustinB3403 Did you use XC to change the logging to a remote location?
Yes.
Check the logs there, then.
I'm guessing there is not there - because the 'there' that he is forwarding them to is the ELK server.
You can't both forward to ELK and use Filebeat!!
Of course this isn't working. -
@Dashrender said in Starting Clean - Kibana:
Hopefully there is a way to send the logs to Logstash directly, instead of sending them via Filebeat.
-
@scottalanmiller migth I ask you try doing this very same thing on an XS installation and outline exactly what you do so that others can replicate it.
As it is now, I'm beyond upset with just trying to get this going.
-
@Dashrender said
Dustin (and I) want NO local logging on our XS boxes.
The only way I ever got this to work was the "dirty little trick" that is mentioned in the comments of that article.
Because on mine, it kept writing locally REGARDLESS of what I did.
@scottalanmiller determined the best way was to move /var/logs
-
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
Hopefully there is a way to send the logs to Logstash directly, instead of sending them via Filebeat.
Of course, syslog, which you are already running. But Filebeat makes Logstash ingest easier.
You've lost me. you say of course, syslog, but Dustin took down the syslog server when he stood the ELK server up, because the assumption was/is that the ELK server replaced the syslog server.
-
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
Hopefully there is a way to send the logs to Logstash directly, instead of sending them via Filebeat.
Of course, syslog, which you are already running. But Filebeat makes Logstash ingest easier.
You've lost me. you say of course, syslog, but Dustin took down the syslog server when he stood the ELK server up, because the assumption was/is that the ELK server replaced the syslog server.
Huh? If he turned off the syslogging server, then there ARE no logs to send on. So that explains everything. Syslog is the logging service. Without it, there are no logs to send anywhere.
-
Syslog is the process that writes the logs. Without it, logs don't exist. They don't get written locally, they don't get sent anywhere, Filebeat has nothing to read....
-
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
Hopefully there is a way to send the logs to Logstash directly, instead of sending them via Filebeat.
Of course, syslog, which you are already running. But Filebeat makes Logstash ingest easier.
You've lost me. you say of course, syslog, but Dustin took down the syslog server when he stood the ELK server up, because the assumption was/is that the ELK server replaced the syslog server.
Huh? If he turned off the syslogging server, then there ARE no logs to send on. So that explains everything. Syslog is the logging service. Without it, there are no logs to send anywhere.
I think I'm even more confused now.
Syslog is a service that accepts logs from other sources (sure one of those sources could the the server syslog is intalled on - but has nothing to do with the logging system of the syslog host server - I hope you follow that) - or at least that's how I understand it.
So, if he has the following:
XS1 - Xenserver generating logs (configs say to send all logs to syslog1 server)
syslog1 - accepting logsThis was working for him. But when he moved to the following:
XS1 - configs say to send all logs to ELK server)
ELK - accepting logsThis does not work. why not? Does he need to install a syslog daemon on this box, then install Filebeat on that syslog server.
- many posts have been writen before I posted this, but I'm leaving it for learning.
-
@scottalanmiller said in Starting Clean - Kibana:
Syslog is the process that writes the logs. Without it, logs don't exist. They don't get written locally, they don't get sent anywhere, Filebeat has nothing to read....
OK here is the discussion issue.
You're saying syslog is the process on each server generating the logs.
I'm saying that syslog is a deamon that accepts log data forwarded to it.is it both?
-
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
Syslog is the process that writes the logs. Without it, logs don't exist. They don't get written locally, they don't get sent anywhere, Filebeat has nothing to read....
OK here is the discussion issue.
You're saying syslog is the process on each server generating the logs.
I'm saying that syslog is a deamon that accepts log data forwarded to it.is it both?
Of course. Individual processes on the server send to syslog so that it can process them. That's how they all end up aggregated together in /var/log/messages. It's like SMTP in that way... mail in, mail out. Only here it is logs in, logs out. Syslog servers talk to syslog servers. ELK is a syslog server, too.
-
@Dashrender said in Starting Clean - Kibana:
XS1 - configs say to send all logs to ELK server)
ELK - accepting logsDoes it say that? Which config are you talking about? Filebeat says that, but it sounds like Filebeat also sees no logs to send.
-
@scottalanmiller said in Starting Clean - Kibana:
@Dashrender said in Starting Clean - Kibana:
@scottalanmiller said in Starting Clean - Kibana:
Syslog is the process that writes the logs. Without it, logs don't exist. They don't get written locally, they don't get sent anywhere, Filebeat has nothing to read....
OK here is the discussion issue.
You're saying syslog is the process on each server generating the logs.
I'm saying that syslog is a deamon that accepts log data forwarded to it.is it both?
Of course. Individual processes on the server send to syslog so that it can process them. That's how they all end up aggregated together in /var/log/messages. It's like SMTP in that way... mail in, mail out. Only here it is logs in, logs out. Syslog servers talk to syslog servers. ELK is a syslog server, too.
So, Dustin turned off the external syslog daemon on the syslog1 server in my example, AND set the syslog server on XS1 to forward logs to the ELK server.
