CentOS 7 VM on Hyper-V losing DHCP assigned address

scottalanmiller

Are we sure that ntp.org was the issue? Is this repeatable?

JaredBusch

@scottalanmiller said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

Are we sure that ntp.org was the issue? Is this repeatable?

Second time it has caught me. Completely different server. Completely different client.

coliver

@JaredBusch said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

@scottalanmiller said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

Are we sure that ntp.org was the issue? Is this repeatable?

Second time it has caught me. Completely different server. Completely different client.

Was it the same centos pool? I haven't had issues with the us pool just the default centos ones.

JaredBusch

The end of the log is me changing it to 3 local NTP sources and it fixing itself. Prior to that is was 2 local sources and 2 ntp.org sources.

10.202.1.11
10.202.1.1
3.us.pool.ntp.org
4.us.pool.ntp.org

changed to

10.202.1.11
10.202.1.1
10.202.0.21

No it has not always been a problem, but it picked bad time more than once.

# grep chrony /var/log/messages* > chrony.logs
# cat chrony.logs
/var/log/messages:Sep 19 14:14:44 owncloud chronyd[765]: Can't synchronise: no selectable sources
/var/log/messages:Sep 21 17:02:28 owncloud chronyd[758]: chronyd version 2.1.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +DEBUG +ASYNCDNS +IPV6 +SECHASH)
/var/log/messages:Sep 21 17:02:28 owncloud chronyd[758]: Frequency -25.319 +/- 0.011 ppm read from /var/lib/chrony/drift
/var/log/messages:Sep 20 14:12:20 owncloud chronyd[758]: Selected source 209.208.79.69
/var/log/messages:Sep 20 14:12:20 owncloud chronyd[758]: System clock wrong by -96617.583787 seconds, adjustment started
/var/log/messages:Sep 20 14:12:20 owncloud chronyd[758]: System clock was stepped by -96617.583787 seconds
/var/log/messages:Sep 20 14:12:21 owncloud chronyd[758]: Selected source 104.238.179.130
/var/log/messages:Sep 20 19:02:11 owncloud chronyd[758]: Selected source 209.208.79.69
/var/log/messages:Sep 22 08:35:19 owncloud chronyd[758]: chronyd version 2.1.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +DEBUG +ASYNCDNS +IPV6 +SECHASH)
/var/log/messages:Sep 22 08:35:19 owncloud chronyd[758]: Frequency -25.359 +/- 0.039 ppm read from /var/lib/chrony/drift
/var/log/messages:Sep 21 11:26:54 owncloud chronyd[758]: Selected source 10.202.1.1
/var/log/messages:Sep 21 11:26:54 owncloud chronyd[758]: System clock wrong by -76114.165928 seconds, adjustment started
/var/log/messages:Sep 21 11:26:54 owncloud chronyd[758]: System clock was stepped by -76114.165928 seconds
/var/log/messages:Sep 21 11:43:40 owncloud chronyd[758]: chronyd exiting
/var/log/messages:Sep 21 11:43:40 owncloud chronyd[5238]: chronyd version 2.1.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +DEBUG +ASYNCDNS +IPV6 +SECHASH)
/var/log/messages:Sep 21 11:43:40 owncloud chronyd[5238]: Frequency -24.924 +/- 0.508 ppm read from /var/lib/chrony/drift
/var/log/messages:Sep 21 11:43:45 owncloud chronyd[5238]: Selected source 10.202.1.1
/var/log/messages-20160828:Aug 22 07:58:01 owncloud chronyd[755]: Selected source 64.6.144.6
/var/log/messages-20160828:Aug 22 08:10:35 owncloud chronyd[755]: Selected source 129.250.35.250
/var/log/messages-20160919:Oct 19 03:22:07 owncloud chronyd[762]: chronyd version 2.1.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +DEBUG +ASYNCDNS +IPV6 +SECHASH)
/var/log/messages-20160919:Oct 19 03:22:08 owncloud chronyd[762]: Frequency -25.194 +/- 0.096 ppm read from /var/lib/chrony/drift
/var/log/messages-20160919:Sep 15 21:22:30 owncloud chronyd[762]: Selected source 129.6.15.28
/var/log/messages-20160919:Sep 15 21:22:30 owncloud chronyd[762]: System clock wrong by -2872803.736327 seconds, adjustment started
/var/log/messages-20160919:Sep 15 21:22:30 owncloud chronyd[762]: System clock was stepped by -2872803.736327 seconds
/var/log/messages-20160919:Sep 16 07:55:53 owncloud chronyd[762]: chronyd exiting
/var/log/messages-20160919:Sep 16 18:26:18 owncloud chronyd[759]: chronyd version 2.1.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +DEBUG +ASYNCDNS +IPV6 +SECHASH)
/var/log/messages-20160919:Sep 16 18:26:18 owncloud chronyd[759]: Frequency -25.324 +/- 0.011 ppm read from /var/lib/chrony/drift
/var/log/messages-20160919:Sep 16 07:56:36 owncloud chronyd[759]: Selected source 10.202.1.1
/var/log/messages-20160919:Sep 16 07:56:36 owncloud chronyd[759]: System clock wrong by -37792.972761 seconds, adjustment started
/var/log/messages-20160919:Sep 16 07:56:36 owncloud chronyd[759]: System clock was stepped by -37792.972761 seconds
/var/log/messages-20160919:Sep 16 08:00:04 owncloud chronyd[759]: chronyd exiting
/var/log/messages-20160919:Sep 16 17:26:05 owncloud chronyd[764]: chronyd version 2.1.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +DEBUG +ASYNCDNS +IPV6 +SECHASH)
/var/log/messages-20160919:Sep 16 17:26:05 owncloud chronyd[764]: Frequency -25.324 +/- 0.014 ppm read from /var/lib/chrony/drift
/var/log/messages-20160919:Sep 16 17:25:42 owncloud chronyd[764]: Selected source 10.202.1.1
/var/log/messages-20160919:Sep 16 17:25:42 owncloud chronyd[764]: System clock wrong by -31.644643 seconds, adjustment started
/var/log/messages-20160919:Sep 16 17:25:42 owncloud chronyd[764]: System clock was stepped by -31.644643 seconds
/var/log/messages-20160919:Sep 16 17:30:01 owncloud chronyd[764]: Selected source 66.228.59.187
/var/log/messages-20160919:Sep 18 15:35:10 owncloud chronyd[764]: chronyd exiting
/var/log/messages-20160919:Sep 20 13:43:45 owncloud chronyd[765]: chronyd version 2.1.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP +DEBUG +ASYNCDNS +IPV6 +SECHASH)
/var/log/messages-20160919:Sep 20 13:43:45 owncloud chronyd[765]: Frequency -25.310 +/- 0.020 ppm read from /var/lib/chrony/drift
/var/log/messages-20160919:Sep 20 13:43:54 owncloud chronyd[765]: Selected source 152.2.133.52
/var/log/messages-20160919:Sep 18 15:35:51 owncloud chronyd[765]: System clock wrong by -166083.522656 seconds, adjustment started
/var/log/messages-20160919:Sep 18 15:35:51 owncloud chronyd[765]: System clock was stepped by -166083.522656 seconds
#

JaredBusch

Well this morning this system was offline again.

[root@owncloud ~]# ip a sh
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 02:50:56:17:18:7f brd ff:ff:ff:ff:ff:ff

No IP again. not new.
It says it is there and has link.

[root@owncloud ~]# nmcli d
DEVICE  TYPE      STATE      CONNECTION
eth0    ethernet  connected  Wired connection 1
lo      loopback  unmanaged  --

Let's look at the config file maybe there is something stupid..

Wait what? WTF? Where is eth0?

[root@owncloud ~]# ls /etc/sysconfig/network-scripts/ifcfg*
/etc/sysconfig/network-scripts/ifcfg-ens32 /etc/sysconfig/network-scripts/ifcfg-lo

Could not find it to save my life, but nmtui listed both Wired connection 1 and ens32. Fine. Deleted ens32 as it was the wrong MAC and edited Wired connection 1 since it was the correct MAC. Renamed it to eth0 and exited from nmtui.

Now I have ifcfg-eth0

[root@owncloud ~]# ls /etc/sysconfig/network-scripts/ifcfg*
/etc/sysconfig/network-scripts/ifcfg-eth0  /etc/sysconfig/network-scripts/ifcfg-lo

Let's see if this gets more stable now.

I have no idea WTF happened here. Nothing on this system has changed for 2 years except yum updates.

JaredBusch

Lost the IP again this morning.

But now that I have the eth0 configured i was able to systemctl restart network and it came up.

nothing in /var/log/messages for DHCP. Date it correct still.

So where do I look to resolve this?

JaredBusch

status looked like this again..

[root@owncloud ~]# ip a sh
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 02:50:56:17:18:7f brd ff:ff:ff:ff:ff:ff

JaredBusch

obviously something was making the network work prior to my change yesterday because a reboot would fix it.

So WTF else makes networking work on CentOS 7 when there is no eth0 config file?

At this point, I have to assume that process is causing the problem.

dafyre

Do you have DHCP server set to give this server a specific IP?

JaredBusch

@dafyre said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

Do you have DHCP server set to give this server a specific IP?

It is a DHCP reservation yes.

I could obviously make this static, but i hate static, because what if I change the network scope or DNS or something. Too much to manually update in that instance.

Also this is NOT the first CentOS 7 system I have seen this on. This is the second one used for ownCloud, and I have another that is simply a FTP server for some old backups that has done this in the past.

All three of the above systems are at different clients sites. with different servers and networks. All three are on Hyper-V though.
Those two systems have not done this recently, but this system is doing it daily now.

dafyre

@JaredBusch said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

@dafyre said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

Do you have DHCP server set to give this server a specific IP?

It is a DHCP reservation yes.

I could obviously make this static, but i hate static, because what if I change the network scope or DNS or something. Too much to manually update in that instance.

Also this is NOT the first CentOS 7 system I have seen this on. This is the second one used for ownCloud, and I have another that is simply a FTP server for some old backups that has done this in the past.

All three of the above systems are at different clients sites. with different servers and networks. All three are on Hyper-V though.
Those two systems have not done this recently, but this system is doing it daily now.

Is your DHCP server on the same Hyper-V server?

Also, what kind of network switch is it connected to?

JaredBusch

@dafyre said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

@JaredBusch said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

@dafyre said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

Do you have DHCP server set to give this server a specific IP?

It is a DHCP reservation yes.

I could obviously make this static, but i hate static, because what if I change the network scope or DNS or something. Too much to manually update in that instance.

Also this is NOT the first CentOS 7 system I have seen this on. This is the second one used for ownCloud, and I have another that is simply a FTP server for some old backups that has done this in the past.

All three of the above systems are at different clients sites. with different servers and networks. All three are on Hyper-V though.
Those two systems have not done this recently, but this system is doing it daily now.

Is your DHCP server on the same Hyper-V server?

Also, what kind of network switch is it connected to?

What part of happens on three different networks did you not grasp?

dafyre

@JaredBusch said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

@dafyre said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

@JaredBusch said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

@dafyre said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

Do you have DHCP server set to give this server a specific IP?

It is a DHCP reservation yes.

I could obviously make this static, but i hate static, because what if I change the network scope or DNS or something. Too much to manually update in that instance.

Also this is NOT the first CentOS 7 system I have seen this on. This is the second one used for ownCloud, and I have another that is simply a FTP server for some old backups that has done this in the past.

All three of the above systems are at different clients sites. with different servers and networks. All three are on Hyper-V though.
Those two systems have not done this recently, but this system is doing it daily now.

Is your DHCP server on the same Hyper-V server?

Also, what kind of network switch is it connected to?

What part of happens on three different networks did you not grasp?

In one eye, out the other, I reckon.

What does your DHCP server show when you restart the network services on the ownCloud box?

JaredBusch

@dafyre haven't looked. when i restart i get the address. so i would assume it is all good. I guess the server log can be checked for the time of failure though.

momurda

Is this maybe something that has happened over time to these servers as a result of updates? Nowadays the network interfaces are using a different naming scheme, starting in systemd v197. The naming of the interfaces are usually enox ensx, others unless disabled by various methods, ethx names are last in line to be used.

https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/

dafyre

@momurda said in CentOS 7 VM on Hyper-V losing DHCP assigned address:

Is this maybe something that has happened over time to these servers as a result of updates? Nowadays the network interfaces are using a different naming scheme, starting in systemd v197. The naming of the interfaces are usually enox ensx, others unless disabled by various methods, ethx names are last in line to be used.

https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames/

@JaredBusch fixed his interface name to always use eth0 on this server.

JaredBusch

This is still a problem everywhere that I have Hyper-V Server 2012 R2 or Server 2012 R2 + Hyper-V.

It is very intermittent and It has been easier to just pop in and kick the service.
But now I have the time to handle this and I need to get this figured out.

Today I have a specific VM down that also went down yesterday. There are 2 other CentOS 7 VMs on this same server that are not down right now.

This VM went down yesterday also.

JaredBusch

0_1485362583958_upload-109b49db-aa05-47e4-a71e-19e5b0efdedb

The ownCloud instance is the one in quesiton today. It was rebooted yesterday because after getting alerted to it being down, I also updated ownCloud from 9.0.x to 9.1.x and then gave it a reboot for good measure.

As you can see everything else was rebooted 2 days ago. Because I rebooted the entire hypervisor (Hyper-V Server 2012 R2) on Sunday for updates.

0_1485362842946_upload-f3743a3d-9a58-4124-8a28-925b43eed3b1

This is the entire DHCP lease list. You can see it is not a large site.
0_1485363143422_upload-8ca29cad-2f05-44e0-8ffe-9847973265e1

JaredBusch

And the same problem that I had before. Bad NTP.

0_1485363690487_upload-84fba01a-de24-4f1c-b510-2caceca2b190

But this time I am only using two internal sources. There is not a third internal source available at this site to have a proper witness.

JaredBusch

Edited chrony to use 3 external servers and rebooted..

# These servers were defined in the installation:
# server 10.201.1.7 iburst
# server 10.201.1.1 iburst
server 0.us.pool.ntp.org iburst
server 1.us.pool.ntp.org iburst
server 2.us.pool.ntp.org iburst
# Use public servers from the pool.ntp.org project.
# Please consider joining the pool (http://www.pool.ntp.org/join.html).

/var/log/messages:Jan 26 11:55:07 owncloud chronyd[617]: chronyd version 2.1.1 starting (+CMDMON +NTP +REFCLOCK +RTC +PRIVDROP $
/var/log/messages:Jan 26 11:55:07 owncloud chronyd[617]: Frequency -22.354 +/- 0.114 ppm read from /var/lib/chrony/drift
/var/log/messages:Jan 25 11:06:29 owncloud chronyd[617]: Selected source 108.61.73.244
/var/log/messages:Jan 25 11:06:29 owncloud chronyd[617]: System clock wrong by -89327.780923 seconds, adjustment started
/var/log/messages:Jan 25 11:06:29 owncloud chronyd[617]: System clock was stepped by -89327.780923 seconds