SysLog Forwarding for XenServer
-
I still have a few compressed logs (things that aren't marked to be forward to Elk/Kibana)
-
Obviously I'll need to change the syslog file to make sure those are only sent off host.
But why aren't they appearing in Elk/Kibana...
-
Everything here seems happy.
-
I still don't know why the logging isn't showing up in Kibana. . .
-
@DustinB3403 said in SysLog Forwarding for XenServer:
I still don't know why the logging isn't showing up in Kibana. . .
What do the local logs say? On both ends. There should be Logstash logs saying what has happened.
-
@scottalanmiller said in SysLog Forwarding for XenServer:
@DustinB3403 said in SysLog Forwarding for XenServer:
I still don't know why the logging isn't showing up in Kibana. . .
What do the local logs say? On both ends. There should be Logstash logs saying what has happened.
I'm still new to syslog, so what should I be looking at to answer this question?
-
You could just use Graylog. It uses rsyslog instead of file-beat (which doesn't work with journalctl anyway).
-
Since I'm having a hell of time getting this going, I setup a KiwI Syslog on a VM from one of my host, and it just works.
Just enabling the logging to the IP address, and let it go.
-
@DustinB3403 said in SysLog Forwarding for XenServer:
Since I'm having a hell of time getting this going, I setup a KiwI Syslog on a VM from one of my host, and it just works.
Just enabling the logging to the IP address, and let it go.
Instead of posting the logs to diagnose?
-
@scottalanmiller Again, where do I look for them....
-
@DustinB3403 said in SysLog Forwarding for XenServer:
@scottalanmiller Again, where do I look for them....
Check the logs.
(Boy I am glad I am not withing physical reach of you!
) -
My point with the Kiwi server is that I must've misconfigured something on the SysLog installation.
Since I've made no direct change other than the logging address in the XC settings. (which does update the settings in the server) but it doesn't change the default port etc.
-
@DustinB3403 said in SysLog Forwarding for XenServer:
@scottalanmiller Again, where do I look for them....
/var/log/messages same as always
-
@scottalanmiller said in SysLog Forwarding for XenServer:
@DustinB3403 said in SysLog Forwarding for XenServer:
@scottalanmiller Again, where do I look for them....
/var/log/messages same as always
... and what would be a decent way to view this as it breezes by at 100 lines a second?
Is there a specific event you're looking for?
-
@DustinB3403 said in SysLog Forwarding for XenServer:
@scottalanmiller said in SysLog Forwarding for XenServer:
@DustinB3403 said in SysLog Forwarding for XenServer:
@scottalanmiller Again, where do I look for them....
/var/log/messages same as always
... and what would be a decent way to view this as it breezes by at 100 lines a second?
Is there a specific event you're looking for?
It only breezes by if you tail it. Try just looking at it statically.
What is generating so many messages?
-
Looking for errors from the forwarder.
-
Have you tried? If you're seeing logs coming in from XenServer, then you should be on the right track.
tail -f|grep nameofsourceserver -
So this is what I have currently with the Kibana system running.
@dafyre tail -f|grep servername results in "tail: warning: following standard input indefinitely is ineffective"
-
Here it is with me connected to the system, and my server supposedly sending logs to it.
[root@syslog-cent ~]# tail /var/log/messages Aug 16 08:32:24 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:24+00:00","tags":[],"pid":609,"method":"post","sta tusCode":200,"req":{"url":"/elasticsearch/packetbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","ho st":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version": "4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","co ntent-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-lan guage":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"stat usCode":200,"responseTime":31,"contentLength":9},"message":"POST /elasticsearch/packetbeat-*/_field_stats?level=indices 200 31ms - 9.0B" } Aug 16 08:32:24 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:24+00:00","tags":[],"pid":609,"method":"post","sta tusCode":200,"req":{"url":"/elasticsearch/packetbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","ho st":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version": "4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","co ntent-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-lan guage":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"stat usCode":200,"responseTime":29,"contentLength":9},"message":"POST /elasticsearch/packetbeat-*/_field_stats?level=indices 200 29ms - 9.0B" } Aug 16 08:32:24 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:24+00:00","tags":[],"pid":609,"method":"post","sta tusCode":200,"req":{"url":"/elasticsearch/topbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host" :"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4. 4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","conte nt-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-langua ge":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusC ode":200,"responseTime":23,"contentLength":9},"message":"POST /elasticsearch/topbeat-*/_field_stats?level=indices 200 23ms - 9.0B"} Aug 16 08:32:24 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:24+00:00","tags":[],"pid":609,"method":"post","sta tusCode":200,"req":{"url":"/elasticsearch/topbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host" :"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4. 4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","conte nt-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-langua ge":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusC ode":200,"responseTime":32,"contentLength":9},"message":"POST /elasticsearch/topbeat-*/_field_stats?level=indices 200 32ms - 9.0B"} Aug 16 08:32:24 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:24+00:00","tags":[],"pid":609,"method":"post","sta tusCode":200,"req":{"url":"/elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1471347138543","method":"post","headers" :{"connection":"upgrade","host":"192.168.100.83","content-length":"3146","accept":"application/json, text/plain, */*","origin":"http://1 92.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52. 0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding ":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100. 83/app/kibana?"},"res":{"statusCode":200,"responseTime":8,"contentLength":9},"message":"POST /elasticsearch/_msearch?timeout=0&ignore_un available=true&preference=1471347138543 200 8ms - 9.0B"} Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","sta tusCode":200,"req":{"url":"/elasticsearch/packetbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","ho st":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version": "4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","co ntent-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-lan guage":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"stat usCode":200,"responseTime":38,"contentLength":9},"message":"POST /elasticsearch/packetbeat-*/_field_stats?level=indices 200 38ms - 9.0B" } Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","sta tusCode":200,"req":{"url":"/elasticsearch/topbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host" :"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4. 4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","conte nt-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-langua ge":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusC ode":200,"responseTime":23,"contentLength":9},"message":"POST /elasticsearch/topbeat-*/_field_stats?level=indices 200 23ms - 9.0B"} Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","sta tusCode":200,"req":{"url":"/elasticsearch/packetbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","ho st":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version": "4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","co ntent-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-lan guage":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"stat usCode":200,"responseTime":31,"contentLength":9},"message":"POST /elasticsearch/packetbeat-*/_field_stats?level=indices 200 31ms - 9.0B" } Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","sta tusCode":200,"req":{"url":"/elasticsearch/topbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host" :"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4. 4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","conte nt-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-langua ge":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusC ode":200,"responseTime":24,"contentLength":9},"message":"POST /elasticsearch/topbeat-*/_field_stats?level=indices 200 24ms - 9.0B"} Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","sta tusCode":200,"req":{"url":"/elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1471347138543","method":"post","headers" :{"connection":"upgrade","host":"192.168.100.83","content-length":"3146","accept":"application/json, text/plain, */*","origin":"http://1 92.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52. 0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding ":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100. 83/app/kibana?"},"res":{"statusCode":200,"responseTime":15,"contentLength":9},"message":"POST /elasticsearch/_msearch?timeout=0&ignore_u navailable=true&preference=1471347138543 200 15ms - 9.0B"} -
Aug 16 08:32:24 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:24+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/topbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":23,"contentLength":9},"message":"POST /elasticsearch/topbeat-*/_field_stats?level=indices 200 23ms - 9.0B"} Aug 16 08:32:24 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:24+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/topbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":32,"contentLength":9},"message":"POST /elasticsearch/topbeat-*/_field_stats?level=indices 200 32ms - 9.0B"} Aug 16 08:32:24 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:24+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1471347138543","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"3146","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":8,"contentLength":9},"message":"POST /elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1471347138543 200 8ms - 9.0B"} Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/packetbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":38,"contentLength":9},"message":"POST /elasticsearch/packetbeat-*/_field_stats?level=indices 200 38ms - 9.0B"} Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/topbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":23,"contentLength":9},"message":"POST /elasticsearch/topbeat-*/_field_stats?level=indices 200 23ms - 9.0B"} Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/packetbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":31,"contentLength":9},"message":"POST /elasticsearch/packetbeat-*/_field_stats?level=indices 200 31ms - 9.0B"} Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/topbeat-*/_field_stats?level=indices","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"178","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":24,"contentLength":9},"message":"POST /elasticsearch/topbeat-*/_field_stats?level=indices 200 24ms - 9.0B"} Aug 16 08:32:26 syslog-cent kibana: {"type":"response","@timestamp":"2016-08-16T12:32:26+00:00","tags":[],"pid":609,"method":"post","statusCode":200,"req":{"url":"/elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1471347138543","method":"post","headers":{"connection":"upgrade","host":"192.168.100.83","content-length":"3146","accept":"application/json, text/plain, */*","origin":"http://192.168.100.83","kbn-version":"4.4.2","user-agent":"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36","content-type":"application/json;charset=UTF-8","referer":"http://192.168.100.83/app/kibana?","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.8"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"http://192.168.100.83/app/kibana?"},"res":{"statusCode":200,"responseTime":15,"contentLength":9},"message":"POST /elasticsearch/_msearch?timeout=0&ignore_unavailable=true&preference=1471347138543 200 15ms - 9.0B"} Aug 16 08:41:16 syslog-cent systemd: Starting Cleanup of Temporary Directories... Aug 16 08:41:16 syslog-cent systemd: Started Cleanup of Temporary Directories.
