Proxmo upgrades hung

JaredBusch

I have 2 PVE 7 systems running at home. Mostly just to have a non-client, non-production, setup handy.

I could not log in to the web interface of the cluster by FQDN or the direct IP of either unit in the cluster. FQDN did through a cert expired error. So, something is going wrong, as the certificate manager is supposed to handle thing.

SSH works just. I know the password, confirmed by using it for SSH.

Things I saw on a google search did not help. So a reboot. Still did not help. So I ran updates from the CLI, not it has been stuck at 98% for 10 minutes on both systems.

JaredBusch

It finally finished..

While I was waiting, I looked at pvecm status and found out it thought I had 4 nodes (see expected votes), when I only have 2 nodes.

I used pvecm expected 2 on both and suddenly the updates moved on and I was able to log in to the web interface immediately.

In the web interface, it showed a pve3 and pve4. These were some test setups I did months ago.
I deleted the nodes from the CLI and everything looks clean again.

JaredBusch

When I tried to manually renew the certificate from the command line, it said I have been rate limited.

root@pve:~# pvenode acme cert renew
Loading ACME account details
Placing ACME order
Error: POST to https://acme-v02.api.letsencrypt.org/acme/new-order {   "type": "urn:ietf:params:acme:error:rateLimited",   "detail": "Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: pm.daerma.com: see https://letsencrypt.org/docs/rate-limits/",   "status": 429 }
Task Error: POST to https://acme-v02.api.letsencrypt.org/acme/new-order {   "type": "urn:ietf:params:acme:error:rateLimited",   "detail": "Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: pm.daerma.com: see https://letsencrypt.org/docs/rate-limits/",   "status": 429 }

JaredBusch

Looks like the cert expired in April, so I bet PVE has been hammering ACME.....

JaredBusch

It finally finished..

While I was waiting, I looked at pvecm status and found out it thought I had 4 nodes (see expected votes), when I only have 2 nodes.

I used pvecm expected 2 on both and suddenly the updates moved on and I was able to log in to the web interface immediately.

In the web interface, it showed a pve3 and pve4. These were some test setups I did months ago.
I deleted the nodes from the CLI and everything looks clean again.

Solved Proxmo upgrades hung