HAProxy fails to start on reboot

JaredBusch

Can someone explain why my system refuses to start the HAProxy service on reboot?

But if I subsequently log in and issue systemctl start haproxy it works?

Looking at journalctl seems to point to a problem attaching to some ports initially.

-- Reboot --
Feb 25 21:36:42 hap.domain.local systemd[1]: Starting HAProxy Load Balancer...
Feb 25 21:36:43 hap.domain.local systemd[1]: Started HAProxy Load Balancer.
Feb 25 21:36:43 hap.domain.local haproxy-systemd-wrapper[670]: [ALERT] 055/213643 (678) : Starting frontend ft_stats: cannot bind socket [10.202.0.20:8080]
Feb 25 21:36:43 hap.domain.local haproxy-systemd-wrapper[670]: [ALERT] 055/213643 (678) : Starting frontend ft_exchange_https: cannot bind socket [10.202.0.19:80]
Feb 25 21:36:43 hap.domain.local haproxy-systemd-wrapper[670]: [ALERT] 055/213643 (678) : Starting frontend ft_exchange_https: cannot bind socket [10.202.0.19:443]
Feb 25 21:36:43 hap.domain.local haproxy-systemd-wrapper[670]: haproxy-systemd-wrapper: exit, haproxy RC=1
Feb 25 21:36:43 hap.domain.local systemd[1]: haproxy.service: Main process exited, code=exited, status=1/FAILURE
Feb 25 21:36:43 hap.domain.local systemd[1]: haproxy.service: Unit entered failed state.
Feb 25 21:36:43 hap.domain.local systemd[1]: haproxy.service: Failed with result 'exit-code'.
Feb 25 21:41:24 hap.domain.local systemd[1]: Starting HAProxy Load Balancer...
Feb 25 21:41:24 hap.domain.local systemd[1]: Started HAProxy Load Balancer.

scottalanmiller

My guesses would be that for some reason it is trying to fire up as a non-root user, or SELinux is messing with it, but then changing later so as not to be a problem.

JaredBusch

@scottalanmiller said in HAProxy fails to start on reboot:

My guesses would be that for some reason it is trying to fire up as a non-root user,

As is typical for my client work, everything is done as the root user to enable simpler hand off in case we part ways.

I manage things with my own user account and sudo, but the setup is all originally done under root.

So because of that, I cannot see what would be causing this from that point of view.

@scottalanmiller said in HAProxy fails to start on reboot:

or SELinux is messing with it, but then changing later so as not to be a problem.

A possibility I guess. I can disable it permanently, reboot, and see what happens as a test.

JaredBusch

set it to permissive in /etc/selinux/config and rebooted. same result.

this be annoying.

scottalanmiller

@jaredbusch said in HAProxy fails to start on reboot:

set it to permissive in /etc/selinux/config and rebooted. same result.

this be annoying.

Well at least that is ruled out.

scottalanmiller

This will be odd but... any chance that something else is running and bound to those ports at the time that the system is starting up?

JaredBusch

@scottalanmiller said in HAProxy fails to start on reboot:

This will be odd but... any chance that something else is running and bound to those ports at the time that the system is starting up?

Single purpose VM, so should not be. Especially on 8080.

Obsolesce

@jaredbusch said in HAProxy fails to start on reboot:

@scottalanmiller said in HAProxy fails to start on reboot:

This will be odd but... any chance that something else is running and bound to those ports at the time that the system is starting up?

Single purpose VM, so should not be. Especially on 8080.

netstat -apn after reboot?

scottalanmiller

I saw that someone needed this. But should not affect you with SELinux off.

setsebool -P haproxy_connect_any=1

JaredBusch

@scottalanmiller said in HAProxy fails to start on reboot:

I saw that someone needed this. But should not affect you with SELinux off.
setsebool -P haproxy_connect_any=1

And it is running with it enforcing

scottalanmiller

What do you get with ip addr | grep 10

JaredBusch

Llooks like Nginx is there, but there is no conf for it and no standard /etc/nginx folder.

WTF, wonder if HAProxy uses it.

scottalanmiller

@jaredbusch said in HAProxy fails to start on reboot:

Llooks like Nginx is there, but there is no conf for it and no standard /etc/nginx folder.

WTF, wonder if HAProxy uses it.

That's really odd. I didn't think that it used that. I think HA-Proxy is older than Nginx, actually.

JaredBusch

@scottalanmiller said in HAProxy fails to start on reboot:

What do you get with ip addr | grep 10

shows the vif that the instructions had me setup.

# ip addr | grep 10
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 10.202.0.20/23 brd 10.202.1.255 scope global dynamic eth0
    inet 10.202.0.19/23 brd 10.202.1.255 scope global secondary eth0:0

scottalanmiller

That looks like it should.

JaredBusch

@tim_g said in HAProxy fails to start on reboot:

@jaredbusch said in HAProxy fails to start on reboot:

@scottalanmiller said in HAProxy fails to start on reboot:

This will be odd but... any chance that something else is running and bound to those ports at the time that the system is starting up?

Single purpose VM, so should not be. Especially on 8080.

netstat -apn after reboot?

$ sudo netstat -apn
[sudo] password for jbusch: 
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      660/sshd            
tcp        0    356 10.202.0.20:22          10.254.103.22:36038     ESTABLISHED 855/sshd: jbusch [p 
tcp6       0      0 :::22                   :::*                    LISTEN      660/sshd            
udp        0      0 127.0.0.1:323           0.0.0.0:*                           617/chronyd         
udp        0      0 0.0.0.0:68              0.0.0.0:*                           800/dhclient        
udp6       0      0 ::1:323                 :::*                                617/chronyd         
raw6       0      0 :::58                   :::*                    7           638/NetworkManager  
Active UNIX domain sockets (servers and established)
Proto RefCnt Flags       Type       State         I-Node   PID/Program name     Path
unix  2      [ ACC ]     STREAM     LISTENING     17962    1/systemd            /run/lvm/lvmpolld.socket
unix  2      [ ACC ]     SEQPACKET  LISTENING     17965    1/systemd            /run/udev/control
unix  2      [ ]         DGRAM                    20289    617/chronyd          /var/run/chrony/chronyd.sock
unix  2      [ ACC ]     SEQPACKET  LISTENING     18047    1/systemd            /run/systemd/coredump
unix  3      [ ]         DGRAM                    11927    1/systemd            /run/systemd/notify
unix  2      [ ]         DGRAM                    11929    1/systemd            /run/systemd/cgroups-agent
unix  2      [ ACC ]     STREAM     LISTENING     18087    1/systemd            /run/lvm/lvmetad.socket
unix  14     [ ]         DGRAM                    11944    1/systemd            /run/systemd/journal/dev-log
unix  2      [ ACC ]     STREAM     LISTENING     11948    1/systemd            /run/systemd/journal/stdout
unix  6      [ ]         DGRAM                    11950    1/systemd            /run/systemd/journal/socket
unix  2      [ ACC ]     STREAM     LISTENING     22988    638/NetworkManager   /var/run/NetworkManager/private-dhcp
unix  2      [ ACC ]     STREAM     LISTENING     20173    1/systemd            /run/dbus/system_bus_socket
unix  2      [ ]         DGRAM                    23506    858/systemd          /run/user/1000/systemd/notify
unix  2      [ ACC ]     STREAM     LISTENING     23508    858/systemd          /run/user/1000/systemd/private
unix  2      [ ACC ]     STREAM     LISTENING     23514    858/systemd          /run/user/1000/bus
unix  2      [ ACC ]     STREAM     LISTENING     17885    1/systemd            /run/systemd/private
unix  3      [ ]         STREAM     CONNECTED     23561    855/sshd: jbusch [p  
unix  3      [ ]         STREAM     CONNECTED     20320    615/dbus-daemon      
unix  2      [ ]         DGRAM                    20583    625/hypervkvpd       
unix  2      [ ]         DGRAM                    18222    472/systemd-journal  
unix  2      [ ]         DGRAM                    23020    800/dhclient         
unix  3      [ ]         STREAM     CONNECTED     20386    472/systemd-journal  /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     20006    591/auditd           
unix  2      [ ]         DGRAM                    23552    855/sshd: jbusch [p  
unix  2      [ ]         DGRAM                    23652    889/sudo             
unix  3      [ ]         STREAM     CONNECTED     23560    864/sshd: jbusch@pt  
unix  3      [ ]         STREAM     CONNECTED     20432    624/systemd-logind   
unix  2      [ ]         DGRAM                    20598    624/systemd-logind   
unix  3      [ ]         STREAM     CONNECTED     20319    615/dbus-daemon      
unix  3      [ ]         STREAM     CONNECTED     20524    472/systemd-journal  /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     20433    472/systemd-journal  /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     20385    623/hypervvssd       
unix  2      [ ]         DGRAM                    20575    623/hypervvssd       
unix  3      [ ]         STREAM     CONNECTED     20005    591/auditd           
unix  3      [ ]         STREAM     CONNECTED     20609    624/systemd-logind   
unix  3      [ ]         STREAM     CONNECTED     20523    625/hypervkvpd       
unix  2      [ ]         DGRAM                    20004    591/auditd           
unix  2      [ ]         DGRAM                    22822    622/python3          
unix  3      [ ]         STREAM     CONNECTED     18662    500/systemd-udevd    
unix  3      [ ]         STREAM     CONNECTED     23470    472/systemd-journal  /run/systemd/journal/stdout
unix  2      [ ]         DGRAM                    20951    638/NetworkManager   
unix  3      [ ]         STREAM     CONNECTED     22279    707/polkitd          
unix  3      [ ]         STREAM     CONNECTED     21448    638/NetworkManager   
unix  3      [ ]         STREAM     CONNECTED     20737    615/dbus-daemon      /run/dbus/system_bus_socket
unix  2      [ ]         DGRAM                    18676    500/systemd-udevd    
unix  2      [ ]         DGRAM                    21424    675/crond            
unix  3      [ ]         STREAM     CONNECTED     18431    490/lvmetad          
unix  3      [ ]         STREAM     CONNECTED     20787    638/NetworkManager   
unix  3      [ ]         STREAM     CONNECTED     21129    660/sshd             
unix  3      [ ]         STREAM     CONNECTED     21449    615/dbus-daemon      /run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     20269    615/dbus-daemon      
unix  2      [ ]         DGRAM                    22282    707/polkitd          
unix  3      [ ]         STREAM     CONNECTED     20965    638/NetworkManager   
unix  3      [ ]         STREAM     CONNECTED     20321    615/dbus-daemon      /run/dbus/system_bus_socket
unix  2      [ ]         DGRAM                    23495    858/systemd          
unix  3      [ ]         STREAM     CONNECTED     20271    472/systemd-journal  /run/systemd/journal/stdout
unix  2      [ ]         DGRAM                    20274    617/chronyd          
unix  2      [ ]         DGRAM                    21028    615/dbus-daemon      
unix  3      [ ]         STREAM     CONNECTED     20736    622/python3          
unix  2      [ ]         DGRAM                    23487    860/(sd-pam)         
unix  3      [ ]         DGRAM                    18750    500/systemd-udevd    
unix  3      [ ]         STREAM     CONNECTED     20610    615/dbus-daemon      /run/dbus/system_bus_socket
unix  2      [ ]         DGRAM                    18432    1/systemd            
unix  3      [ ]         STREAM     CONNECTED     20966    615/dbus-daemon      /run/dbus/system_bus_socket
unix  3      [ ]         DGRAM                    18749    500/systemd-udevd    
unix  3      [ ]         STREAM     CONNECTED     21398    675/crond            
unix  3      [ ]         STREAM     CONNECTED     20225    1/systemd            
unix  3      [ ]         STREAM     CONNECTED     18499    472/systemd-journal  /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     20789    472/systemd-journal  /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     18664    472/systemd-journal  /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     23466    858/systemd          
unix  3      [ ]         STREAM     CONNECTED     21130    472/systemd-journal  /run/systemd/journal/stdout
unix  3      [ ]         STREAM     CONNECTED     22280    615/dbus-daemon      /run/dbus/system_bus_socket
unix  3      [ ]         STREAM     CONNECTED     21399    472/systemd-journal  /run/systemd/journal/stdout
Active Bluetooth connections (servers and established)
Proto  Destination       Source            State         PSM DCID   SCID      IMTU    OMTU Security
Proto  Destination       Source            State     Channel

But again by the time I log in, I can immediately start the service.

scottalanmiller

In /etc/sysctl.conf can you add this line?

net.ipv4.ip_nonlocal_bind = 1

Or just do...

echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf

JaredBusch

@scottalanmiller said in HAProxy fails to start on reboot:

In /etc/sysctl.conf can you add this line?
net.ipv4.ip_nonlocal_bind = 1
Or just do...
echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf

That did it.
WTF did I just do? I can read, so I get the general idea.

JaredBusch

@jaredbusch said in HAProxy fails to start on reboot:

net.ipv4.ip_nonlocal_bind

Google tells me this.
https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7-beta/html/load_balancer_administration/s1-initial-setup-forwarding-vsa

scottalanmiller

@jaredbusch said in HAProxy fails to start on reboot:

@scottalanmiller said in HAProxy fails to start on reboot:
In /etc/sysctl.conf can you add this line?
net.ipv4.ip_nonlocal_bind = 1
Or just do...
echo "net.ipv4.ip_nonlocal_bind = 1" >> /etc/sysctl.conf
That did it.
WTF did I just do? I can read, so I get the general idea.

For load balancing it needs to be able to bind to non-local ports. For some reason the install did not set this which it seems like it should have done.