502 Bad Gateway; nginx/1.16.1

gjacobse

Nothing in the /var/log/nginx/error.log so I checked /var/log/nginx/nextcloud_https_error.log

cat nextcloud_https_error.log
2019/12/31 00:09:16 [error] 760#760: *281 connect() failed (111: Connection refused) while connecting to upstream, client: MY-IP, server: _, request: "GET /csrftoken HTTP/1.1", upstream: "fastcgi://127.0.0.1:9000", host: "ServerIP"

gjacobse

Testing the nginx config:

root@TuDallasNC:/var/log/nginx# sudo nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

scottalanmiller

Get us a...

netstat -tulpn

There are a lot of unknowns here. My guess is that whatever is running NextCloud, isn't.

gjacobse

@scottalanmiller said in 502 Bad Gateway; nginx/1.16.1:

netstat -tulpn

root@TuDallasNC:/var/log/nginx# netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:443             0.0.0.0:*               LISTEN      759/nginx: master p
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      766/redis-server 12
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      759/nginx: master p
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      681/systemd-resolve
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      760/sshd
tcp        0      0 0.0.0.0:61209           0.0.0.0:*               LISTEN      733/python3
tcp6       0      0 :::443                  :::*                    LISTEN      759/nginx: master p
tcp6       0      0 ::1:6379                :::*                    LISTEN      766/redis-server 12
tcp6       0      0 :::80                   :::*                    LISTEN      759/nginx: master p
tcp6       0      0 :::22                   :::*                    LISTEN      760/sshd
udp        0      0 127.0.0.53:53           0.0.0.0:*                           681/systemd-resolve
udp        0      0 ServerIP:68        0.0.0.0:*                           656/systemd-network

gjacobse

Checking to see if nginx is running:

root@TuDallasNC:/var/log/nginx# systemctl status nginx
● nginx.service - nginx - high performance web server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-12-31 01:58:03 UTC; 40min ago
     Docs: http://nginx.org/en/docs/
  Process: 741 ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf (code=exited, status=0/SUCCESS)
 Main PID: 759 (nginx)
    Tasks: 2 (limit: 2287)
   Memory: 5.9M
   CGroup: /system.slice/nginx.service
           ├─759 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
           └─761 nginx: worker process

Dec 31 01:58:03 TuDallasNC systemd[1]: Starting nginx - high performance web server...
Dec 31 01:58:03 TuDallasNC systemd[1]: nginx.service: Can't open PID file /run/nginx.pid (yet?) after start: No such file or directory
Dec 31 01:58:03 TuDallasNC systemd[1]: Started nginx - high performance web server.

black3dynamite

@gjacobse said in 502 Bad Gateway; nginx/1.16.1:

Checking to see if nginx is running:

root@TuDallasNC:/var/log/nginx# systemctl status nginx
● nginx.service - nginx - high performance web server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-12-31 01:58:03 UTC; 40min ago
     Docs: http://nginx.org/en/docs/
  Process: 741 ExecStart=/usr/sbin/nginx -c /etc/nginx/nginx.conf (code=exited, status=0/SUCCESS)
 Main PID: 759 (nginx)
    Tasks: 2 (limit: 2287)
   Memory: 5.9M
   CGroup: /system.slice/nginx.service
           ├─759 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.conf
           └─761 nginx: worker process

Dec 31 01:58:03 TuDallasNC systemd[1]: Starting nginx - high performance web server...
Dec 31 01:58:03 TuDallasNC systemd[1]: nginx.service: Can't open PID file /run/nginx.pid (yet?) after start: No such file or directory
Dec 31 01:58:03 TuDallasNC systemd[1]: Started nginx - high performance web server.

You can fix the nginx.service: Can't open PID file /run/nginx.pid (yet?) after start: No such file or directory by doing following:

sudo mkdir /etc/systemd/system/nginx.service.d
printf "[Service]\nExecStartPost=/bin/sleep 0.1\n" | \
sudo tee /etc/systemd/system/nginx.service.d/override.conf
sudo systemctl daemon-reload
sudo systemctl restart nginx
sudo systemctl stop nginx

https://bugs.launchpad.net/ubuntu/+source/nginx/+bug/1581864

https://medium.com/@parsher/nginx-service-failed-to-parse-pid-from-file-var-run-nginx-pid-invalid-argument-bugs-fda95eeeda73

gjacobse

I was reading that a few minutes ago, and search to see if the file was there,.. and it is.

However, Let me run through it.

gjacobse

Issue continues

JaredBusch

Nextcloud runs on Apache by default, or at least used to. So I have no idea what script you followed to get Nginx.

Emad R

@gjacobse said in 502 Bad Gateway; nginx/1.16.1:

@JaredBusch said in 502 Bad Gateway; nginx/1.16.1:

Did you install Nextcloud with Nginx instead of Apche?

I do not recall - as that was months ago. I used the Vultr scripted base install. I've kept it update/upgraded but took the time to upgrade the distro from 18.04 to 19.10

thats why i dont like the scripted installs, they are good and gets you going fast, but when times come to troubleshoot... cause you didnt do it, it will take you x100 time to solve it.

IRJ

@JaredBusch said in 502 Bad Gateway; nginx/1.16.1:

Nextcloud runs on Apache by default, or at least used to. So I have no idea what script you followed to get Nginx.

Hmm... Is apache running perhaps?

I've had the reverse of that issue before where apache was started and brought a site down

gjacobse

It was the Vultr stock server build for Nextcloud.

In trying to fix things, I think I have made matters worse, so at this point, I've downloaded the few files that where there, and will re-install the server nuking the current data.

Of note; the data was found at:

/var/www/html/data/user17049/files/SHARENAME

marcinozga

You have the answer in the logs you posted. Nginx can't connect to php handler daemon on port 9000, most likely php-fpm.