@scottalanmiller said in Port from SW - Salt master rsa key issue:

@dgingerich said in Port from SW - Salt master rsa key issue:

@scottalanmiller said in Port from SW - Salt master rsa key issue:

@dgingerich said in Port from SW - Salt master rsa key issue:

@scottalanmiller said in Port from SW - Salt master rsa key issue:

@dgingerich said in Port from SW - Salt master rsa key issue:

There's a lot of customization in the iptables config and other areas that are scripted so that future systems could be deployed quickly.

If that was handled in Salt, it would be all automated so that it would solve this problem, rather than create it. That's actually a reason to use Salt in that case.

The production stack has Salt working. It's the QA stack that doesn't. The production stack is 62 servers, and needs Salt. The QA stack is just 8 servers plus 2 management servers, and the salt config is supposed to be the same, but adjusted for other system names. I don't have the Salt skills to adjust the Salt scripts to work for the QA stack.

Hmmm... might want to pressure your Salt guy to automate that. In theory, you could have something like QA in the name that designates that for the future.

Yeah, the reason I'm in the middle of all this is because our Salt guy is on his honeymoon in Italy this week, and the UK next week. So, he's not going to be of much help for a while.

That was not good timing!

Fun story: He's planned this for a year. He married his wife a year ago in the US to allow her to stay in the country, to then have a real ceremony on their 1 year anniversary with her family and go on their honeymoon. In January, our company got bought out, specifically because our technology was far better than the other company's, yet they had better sales contacts and more business, and the new CTO has a very special and fast timeline for converting some of our apps to cloud for use by our new coworkers. That plan just happened to step on our Systems Engineer's wedding plans that were not able to be changed. So, we got stuck in this mess. Sometimes, various things just make the perfect storm.

@scottalanmiller said in Port from SW - Salt master rsa key issue:

@dgingerich said in Port from SW - Salt master rsa key issue:

@scottalanmiller said in Port from SW - Salt master rsa key issue:

@dgingerich said in Port from SW - Salt master rsa key issue:

There's a lot of customization in the iptables config and other areas that are scripted so that future systems could be deployed quickly.

If that was handled in Salt, it would be all automated so that it would solve this problem, rather than create it. That's actually a reason to use Salt in that case.

The production stack has Salt working. It's the QA stack that doesn't. The production stack is 62 servers, and needs Salt. The QA stack is just 8 servers plus 2 management servers, and the salt config is supposed to be the same, but adjusted for other system names. I don't have the Salt skills to adjust the Salt scripts to work for the QA stack.

Hmmm... might want to pressure your Salt guy to automate that. In theory, you could have something like QA in the name that designates that for the future.

Yeah, the reason I'm in the middle of all this is because our Salt guy is on his honeymoon in Italy this week, and the UK next week. So, he's not going to be of much help for a while.

@scottalanmiller said in Port from SW - Salt master rsa key issue:

@dgingerich said in Port from SW - Salt master rsa key issue:

There's a lot of customization in the iptables config and other areas that are scripted so that future systems could be deployed quickly.

If that was handled in Salt, it would be all automated so that it would solve this problem, rather than create it. That's actually a reason to use Salt in that case.

The production stack has Salt working. It's the QA stack that doesn't. The production stack is 62 servers, and needs Salt. The QA stack is just 8 servers plus 2 management servers, and the salt config is supposed to be the same, but adjusted for other system names. I don't have the Salt skills to adjust the Salt scripts to work for the QA stack.

@scottalanmiller said in Port from SW - Salt master rsa key issue:

@dgingerich said in Port from SW - Salt master rsa key issue:

@dgingerich

@scottalanmiller said in Port from SW - Salt master rsa key issue:

What if you create new masters with totally new names. Start fresh, including the names. A huge pain, I know, but I think that that key regeneration something horrible. Just avoid that next time, use the keys that are generated automatically only. This is a bad break that I'd love to get figured out, but I know that you have a time crunch and we want this working, too.

Tried this. It works, but the salt scripts have to be adjusted for the new names in order to work, and I have not had success with that. The guy who made the scripts is in Italy this whole week, and then UK for the week after that.

That should just be a matter of putting them into the top.sls file.

There's a lot of customization in the iptables config and other areas that are scripted so that future systems could be deployed quickly.

We have been ordered by management to go ahead with a manual build on the QA stack. So, the idea of using salt in this case is out.

@dgingerich

@scottalanmiller said in Port from SW - Salt master rsa key issue:

What if you create new masters with totally new names. Start fresh, including the names. A huge pain, I know, but I think that that key regeneration something horrible. Just avoid that next time, use the keys that are generated automatically only. This is a bad break that I'd love to get figured out, but I know that you have a time crunch and we want this working, too.

Tried this. It works, but the salt scripts have to be adjusted for the new names in order to work, and I have not had success with that. The guy who made the scripts is in Italy this whole week, and then UK for the week after that.

@msff-amman-Itofficer said in Port from SW - Salt master rsa key issue:

I too, not an expert, more like playing with salt and you seem like you know more about it than me, but this one liner helps me when I feel something is cached in the setting, or command fails cause its already running:

salt '*' saltutil.kill_all_jobs && salt-run cache.clear_all && salt '*' saltutil.clear_cache && salt '*' saltutil.sync_all

Yeah, that wouldn't work because the masters simply aren't talking to the minions.

Over the weekend, I tried to delete the DNS and recreate the masters after 12 hours, and that did sort of work, for a bit. The masters would talk to themselves, but as soon as two minions were connected, the communication just stopped again. I don't have any idea what is causing this.

Thank you very much for your time on this.

OK. I've deleted the DNS entries and the systems from Packet.net. I'm going to try again tomorrow morning after all the DNS caching should have expired.

@scottalanmiller said in Port from SW - Salt master rsa key issue:

@dgingerich said in Port from SW - Salt master rsa key issue:

@scottalanmiller said in Port from SW - Salt master rsa key issue:

@dgingerich said in Port from SW - Salt master rsa key issue:

[WARNING ] Key 'file_ignore_glob' with value None has an invalid type of NoneType, a list is required for this value

maybe you have duplicate DNS entries and round robin is getting you?

The DNS is just one address for each. I changed the to the new IPs while the systems were building at the authoritative servers. (Systems are housed at Packet.net, DNS handled through AWS) So, there should not be any DNS caching issues.

Oh okay, this is all hosted. Still, best to be sure and rule out possibilities while testing. This is weird, we use Vultr for Salt Masters and have never seen anything like this. But we avoid Ubuntu, so if there is any bug there, we'd not have seen it.

I was able to build another system, named QAICS-mastertest, that worked perfectly using the exact same methods. It's really weird.

@scottalanmiller said in Port from SW - Salt master rsa key issue:

@dgingerich said in Port from SW - Salt master rsa key issue:

[WARNING ] Key 'file_ignore_glob' with value None has an invalid type of NoneType, a list is required for this value

maybe you have duplicate DNS entries and round robin is getting you?

The DNS is just one address for each. I changed the to the new IPs while the systems were building at the authoritative servers. (Systems are housed at Packet.net, DNS handled through AWS) So, there should not be any DNS caching issues.

root@QAICS-MAN-01:~# apt-get install salt-master salt-minion
...
root@QAICS-MAN-01:/etc/salt# vi minion
root@QAICS-MAN-01:/etc/salt# vi master
root@QAICS-MAN-01:/etc/salt# service salt-master start
root@QAICS-MAN-01:/etc/salt# service salt-minion start
root@QAICS-MAN-01:/etc/salt# salt-key -L
Accepted Keys:
Denied Keys:
Unaccepted Keys:
QAICS-MAN-01
QAICS-MAN-02
Rejected Keys:
root@QAICS-MAN-01:/etc/salt# salt-key -L
Accepted Keys:
Denied Keys:
Unaccepted Keys:
QAICS-MAN-01
QAICS-MAN-02
Rejected Keys:
root@QAICS-MAN-01:/etc/salt# salt-key -A
The following keys are going to be accepted:
Unaccepted Keys:
QAICS-MAN-01
QAICS-MAN-02
Proceed? [n/Y] y
Key for minion QAICS-MAN-01 accepted.
Key for minion QAICS-MAN-02 accepted.
Key for minion QAICS-Proxy-01 accepted.
root@QAICS-MAN-01:/etc/salt# salt-key -L
Accepted Keys:
QAICS-MAN-01
QAICS-MAN-02
Denied Keys:
Unaccepted Keys:
Rejected Keys:
root@QAICS-MAN-01:/etc/salt# salt '*' test.ping
[WARNING ] Key 'file_ignore_glob' with value None has an invalid type of NoneType, a list is required for this value
[WARNING ] Key 'file_ignore_glob' with value None has an invalid type of NoneType, a list is required for this value
[WARNING ] Key 'file_ignore_glob' with value None has an invalid type of NoneType, a list is required for this value
[WARNING ] Key 'file_ignore_glob' with value None has an invalid type of NoneType, a list is required for this value
[WARNING ] jid does not exist
[WARNING ] Returner unavailable:
QAICS-MAN-02:
Minion did not return. [No response]
QAICS-MAN-01:
Minion did not return. [No response]
root@QAICS-MAN-01:/etc/salt#

@scottalanmiller The other minions were not included in this. The masters would not even log into themselves after the keys were accepted. Total blank slate machines, install salt-master and salt-minion, configure them to point to themselves, (other minions were turned off) and they still failed the test.ping and salt-minion -l debug gave the exact same result. Could it be something with the DNS info?

rebuilding the systems did not work. getting the same issue with brand new master under the same name.

Of course, trying the same sequence, I cannot reproduce the results. Looks like I'm going to have to rebuild the masters.

Maybe copying the rsa key files to the new systems will be possible.

@scottalanmiller said in Port from SW - Salt master rsa key issue:

What are the date times for the first two?

update previous post with that info

@scottalanmiller said in Port from SW - Salt master rsa key issue:

What is the contents of your PKI folder, like this...

# ll /etc/salt/pki/master/
total 28
-r-------- 1 root root 1674 Dec 16  2016 master.pem
-rw-r--r-- 1 root root  450 Dec 16  2016 master.pub
drwxr-xr-x 2 root root 4096 Jun 14 21:00 minions
drwxr-xr-x 2 root root 4096 Dec 16  2016 minions_autosign
drwxr-xr-x 2 root root 4096 Mar 19 16:26 minions_denied
drwxr-xr-x 2 root root 4096 Jun 14 21:00 minions_pre
drwxr-xr-x 2 root root 4096 Dec 16  2016 minions_rejected

Yes, the contents of my pki folder look just like that, except with different dates.

root@QAICS-MAN-01:/etc/salt/pki/master# ls -l
total 28
-r-------- 1 root root 1674 Jun 23 18:17 master.pem
-rw-r--r-- 1 root root 450 Jun 23 18:17 master.pub
drwxr-xr-x 2 root root 4096 Jun 23 18:35 minions
drwxr-xr-x 2 root root 4096 Jun 23 18:17 minions_autosign
drwxr-xr-x 2 root root 4096 Jun 23 18:17 minions_denied
drwxr-xr-x 2 root root 4096 Jun 23 18:35 minions_pre
drwxr-xr-x 2 root root 4096 Jun 23 18:17 minions_rejected
root@QAICS-MAN-01:/etc/salt/pki/master#

I am spinning up an additional system to try the "install salt, connect them, confirm communication, generate rsa keys, confirm disconnect" method. After that, I'll try generating the rsa keys before installing salt and see if that makes any difference. (I hate spinning up most systems, as they cost my company money to just start them up. I start up one, test on it, and delete it a day later, it still costs my company $36.50. So, this test will cost us $73.)

Thank you all. Wow, very active environment.

@DustinB3403 said in Port from SW - Salt master rsa key issue:

@dgingerich Hrm. . .

If you're just entering through the process I don't think it would be the RSA keys then . . . maybe there is a firewall enabled on your Masters/Minions?

I haven't had the opportunity to do anything with the firewall to this point. By default, it is wide open.