NGINX Just Stop Working
-
Here is the only entry in the NGINX error log for the last time NGINX stopped.
2021/01/08 22:34:03 [error] 847#847: *195 access forbidden by rule, client: 195.154.63.222, server: plextrack.jpslconsulting.ca, request: "GET / HTTP/1.1", host: "plextrack.jpslconsulting.ca"The Let's Encrypt log shows no activity immediately before the outage.
Syslog also shows no errors. It has entries from 3AM to 3:155AM and 9:59PM to 10:02PM on the day of the last incident however the outage occurred between 7:06PM and 10:00PM so the only related entries in this log are at the time the outage was discovered and Ubuntu restarted.
-
I also ran the NGINX test and all looks good.
-
certbot.timer failing?
https://stackoverflow.com/a/52967898
-
@black3dynamite I'm not seeing any evidence of this failing in the letsencrypt.log file syslog or nginx logs (both access and error). Would those logs be elsewhere? Obviously I don't want to have to manually renew certs.
-
@NashBrydges letsencrypt.log is the only one I'm aware of. Actually are using systemd to renew your certs or cronjob?
-
@black3dynamite systemd...
-
@NashBrydges said in NGINX Just Stop Working:
I also ran the NGINX test and all looks good.
If they weren't it wouldn't even start up.
-
@NashBrydges said in NGINX Just Stop Working:
Here is the only entry in the NGINX error log for the last time NGINX stopped.
The error log is where it records HTTP errors, not Nginx software errors.
-
@scottalanmiller Well at this point I'm looking at any log that has "error" in the name. Lol
-
@NashBrydges said in NGINX Just Stop Working:
@scottalanmiller Well at this point I'm looking at any log that has "error" in the name. Lol
This should show you what there is for Nginx itself....
grep nginx /var/log/messages -
@scottalanmiller said in NGINX Just Stop Working:
grep nginx /var/log/messages
/var/log/messagesDoes not exist.
-
@NashBrydges said in NGINX Just Stop Working:
@scottalanmiller said in NGINX Just Stop Working:
grep nginx /var/log/messages
/var/log/messagesDoes not exist.
Oh sorry, use Ubuntu's log. That's RHELs.
-
@scottalanmiller said in NGINX Just Stop Working:
@NashBrydges said in NGINX Just Stop Working:
Here is the only entry in the NGINX error log for the last time NGINX stopped.The error log is where it records HTTP errors, not Nginx software errors.
Which is useful for in a case I've seen where the service was started by other means, and showed all addresses were already in use.
-
@scottalanmiller said in NGINX Just Stop Working:
@NashBrydges said in NGINX Just Stop Working:
@scottalanmiller said in NGINX Just Stop Working:
grep nginx /var/log/messages
/var/log/messagesDoes not exist.
Oh sorry, use Ubuntu's log. That's RHELs.
I thought the modern distros these days were
journalctl -forjournalctl|grep blah|less -
@dafyre said in NGINX Just Stop Working:
I thought the modern distros these days were journalctl -f or journalctl|grep blah|less
Just another way to see the same thing.
-
@scottalanmiller Yeah, those log messages are in the syslog file in Ubuntu. The only log entries in syslog are related to when I rebooted the server. GREP output is too large for those related nginx entries in the log for me to post here but just reviewed every line and no errors that I could find.
-
@NashBrydges said in NGINX Just Stop Working:
@scottalanmiller Yeah, those log messages are in the syslog file in Ubuntu. The only log entries in syslog are related to when I rebooted the server. GREP output is too large for those related nginx entries in the log for me to post here but just reviewed every line and no errors that I could find.
Not necessarily looking for errors, just information as to what might have happened.
-
@scottalanmiller It's too large to upload the output here so uploaded a text file to Box and sharing link here. This is a simple txt file.
-
@NashBrydges said in NGINX Just Stop Working:
@scottalanmiller It's too large to upload the output here so uploaded a text file to Box and sharing link here. This is a simple txt file.
Well I see a major issue is that the hostname of the box appears to be nginx. So that essentially makes it unsearchable because you are trying to diagnose the nginx service. Start by giving the system a more appropriate name and that alone will clear up the logs so that you can search for services.
Rule of thumb - hostnames should be meaningful with information that you can't put elsewhere, but not overlap with things like service names.
Name the systems based on things like purpose, not code. So our standard here, if nginx is a proxy, the system would be named something that ends in rproxy to denote its role. If it was an Nginx based web server, it would be named lemp and so forth.
That's why you have a huge log that you can't sort through instead of a tiny one that's super easy to sort through.
-
@NashBrydges are you sure that you are on the right box? The nginx process has literally no logs, at all. The service has never logged that it started, stopped, ran or anything else.
Also, LXD is running on this system. It appears that this is a hypervisor host rather than a web server.
