Salt Minion Problems

scottalanmiller

Here is his firewall...

# firewall-cmd --list-all
public (active)
  target: default
  icmp-block-inversion: no
  interfaces: eth0
  sources: 
  services: dhcpv6-client ssh
  ports: 4505-4506/tcp
  protocols: 
  masquerade: no
  forward-ports: 
  sourceports: 
  icmp-blocks: 
  rich rules:

scottalanmiller

Here is the error message on the Salt Master's minion process:

Jan 31 20:32:37 salt salt-minion[901]: [ERROR   ] The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate

Romo

@scottalanmiller said in Salt Minion Problems:

The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate

That happens to all minions when they are waiting for the master to approve their key.

scottalanmiller

@Romo said in Salt Minion Problems:

@scottalanmiller said in Salt Minion Problems:

The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate

That happens to all minions when they are waiting for the master to approve their key.

Yeah, the master didn't have an /etc/hosts entry so it was pointing itself to a different Salt Master. So that was awfully confusing.

scottalanmiller

Ok, so after fixing the hostname on the master, fixing the /etc/hosts file and then checking the errors, we find that that weird ip local address there is NOT from the Minion, but from the Master itself. So the Minion has never checked in at all.

# salt-key --list-all
Accepted Keys:
Denied Keys:
ip-65-75-137-152.local
Unaccepted Keys:
ip-65-75-137-152.local
Rejected Keys:
[root@lab-lnx-william-salt ~]# systemctl status salt-minion
● salt-minion.service - The Salt Minion
   Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; enabled; vendor preset: disabled)
   Active: inactive (dead) since Tue 2017-01-31 20:39:37 UTC; 2min 43s ago
  Process: 902 ExecStart=/usr/bin/salt-minion (code=exited, status=0/SUCCESS)
 Main PID: 902 (code=exited, status=0/SUCCESS)

Jan 31 20:39:22 lab-lnx-william-salt systemd[1]: Starting The Salt Minion...
Jan 31 20:39:23 lab-lnx-william-salt systemd[1]: Started The Salt Minion.
Jan 31 20:39:37 lab-lnx-william-salt salt-minion[902]: [CRITICAL] The Salt Master has rejected this minion's public key!
Jan 31 20:39:37 lab-lnx-william-salt salt-minion[902]: To repair this issue, delete the public key for this minion on the Salt Master and restart this minion.
Jan 31 20:39:37 lab-lnx-william-salt salt-minion[902]: Or restart the Salt Master in open mode to clean out the keys. The Salt Minion will now exit.

scottalanmiller

Here is the current error on the minion. I'm guessing we have the same issue:

[root@minion1 ~]# systemctl status salt-minion
● salt-minion.service - The Salt Minion
   Loaded: loaded (/usr/lib/systemd/system/salt-minion.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2017-01-31 20:21:59 UTC; 23min ago
 Main PID: 1228 (salt-minion)
   CGroup: /system.slice/salt-minion.service
           ├─1228 /usr/bin/python /usr/bin/salt-minion
           ├─1237 /usr/bin/python /usr/bin/salt-minion
           └─1241 /usr/bin/python /usr/bin/salt-minion

Jan 31 20:44:27 minion1 salt-minion[1228]: [ERROR   ] The Salt Master has cached the public key for this node, this salt minion will wait for 10 seconds before attempting to re-authenticate

scottalanmiller

Here are some issues in the /etc/hosts file that will definitely cause things to fail:

192.168.1.97



192.168.1.94 salt
192.128.1.94 salt

Dashrender

All this because you use another DNS that's not part of the project and it has overlapping details for other testers/production. lol

scottalanmiller

@Dashrender said in Salt Minion Problems:

All this because you use another DNS that's not part of the project and it has overlapping details for other testers/production. lol

No, you would need this to be correct either way.

Romo

@scottalanmiller check what does salt have as the minion_id as well, it is in /etc/salt/minion_id

scottalanmiller

Also found the /etc/hosts file had a full double copy of itself in the same file. No idea how that got there.

scottalanmiller

Salt Master will not allow me to delete the keys that are in there. I did a delete all and it said that it was, but it failed.

WrCombs

okay. what do i need to do to fix it? sorry for the headache guys

WrCombs

well it looks like Scott fixed most of the problems. ( ish ) I just logged on to the server and all the things he pointed out as wrong, arent there anymore

scottalanmiller

@WrCombs said in Salt Minion Problems:

okay. what do i need to do to fix it? sorry for the headache guys

No idea.

Dashrender

@scottalanmiller said in Salt Minion Problems:

@Dashrender said in Salt Minion Problems:

All this because you use another DNS that's not part of the project and it has overlapping details for other testers/production. lol

No, you would need this to be correct either way.

I'm sorry - correct what either way?

If he had no DNS entry for salt, sure it would fail and he would be told to put entries in /etc/hosts but since there was a working salt server in DNS, things appear to work, but really aren't going where expected...

So having an over lapping lab just made the situation harder - ultimately learning more I guess.

Dashrender

@WrCombs said in Salt Minion Problems:

well it looks like Scott fixed most of the problems. ( ish ) I just logged on to the server and all the things he pointed out as wrong, arent there anymore

Don't be - this is learning, sometimes you break things. Sometimes you don't. Scott digging into what you were doing exactly can be helpful in making guides that help future people learn with a little less pain.

@scottalanmiller can you pull a log of all of the commands he ran and perhaps find things like how the hosts file was completely doubled?

scottalanmiller

@Dashrender said in Salt Minion Problems:

@scottalanmiller said in Salt Minion Problems:

@Dashrender said in Salt Minion Problems:

All this because you use another DNS that's not part of the project and it has overlapping details for other testers/production. lol

No, you would need this to be correct either way.

I'm sorry - correct what either way?

If he had no DNS entry for salt, sure it would fail and he would be told to put entries in /etc/hosts but since there was a working salt server in DNS, things appear to work, but really aren't going where expected...

So having an over lapping lab just made the situation harder - ultimately learning more I guess.

No really, it doesn't appear to work any more than if there is no entry at all.

scottalanmiller

@Dashrender said in Salt Minion Problems:

@scottalanmiller can you pull a log of all of the commands he ran and perhaps find things like how the hosts file was completely doubled?

My guess there is "vi mishap".

WrCombs

@scottalanmiller id agree with that, but inly because i thought i was doing one thing. While doing something completely different..

Unsolved Salt Minion Problems