Port from SW - Salt master rsa key issue
-
I am majorly frustrated on this one. I have gone round and round with this issue and gotten nowhere. Maybe someone out there has an idea of what is going wrong.
I'm building a QA environment with 10 servers, and the production environment is going to be a total of 42 servers, so we're having to automate things. So, we're using Salt on Ubuntu masters to automate. Last week, I built 8 servers for the QA stack, 2 of which were the Salt masters. Everything was connecting, it was pushing the configurations, but an error in the iptables configurations resulted in the Ubuntu master systems and two Ubuntu minions unable to communicate. In addition to changes in the structure of the systems, I wound up having to rebuild the entire QA environment.
Yesterday, I build all 10 new systems, get Salt master installed on the masters, got them talking to the minions, and then generated rsa keys to get the masters to talk to the git repository to clone the configurations for the minions. As soon as the rsa keys were generated, the communications between masters and minions stopped, and they would never talk again.
I have tried deleting the salt keys for the minions and reaccepting them. Even though the keys were presented and accepted, the minions still complain upon logging into the masters that the masters' keys would not authenticate. I tried forcably removing all salt components and saved data, including the stuff in /etc/salt and /var/cache/salt, and reinstall, it still gets the same result. Even with just one master and the minion service on itself, it would not authenticate.
I'd rather not have to rebuild the masters. It would take several more hours of work.
I'm thinking it has something to do with the rsa keys, but I can't say for sure. Perhaps Salt is presenting the master keys on accepting the minion keys, but then uses the system rsa keys upon authenticating the minions? Does that sound likely?
Anyone have any ideas? I'm at my wits' end.
-
I thought you could only have a single Salt master in an environment. . .
-
@DustinB3403 said in Port from SW - Salt master rsa key issue:
I'm thinking it has something to do with the rsa keys, but I can't say for sure. Perhaps Salt is presenting the master keys on accepting the minion keys, but then uses the system rsa keys upon authenticating the minions? Does that sound likely?
Yes, it authenticates in both directions and it sounds like you regenerated the ID keys for the Salt master, so the minions see the master has having changed. Unless you have a copy of the original keys, I think you need to rejoin.
-
@DustinB3403 said in Port from SW - Salt master rsa key issue:
I thought you could only have a single Salt master in an environment. . .
No, it's multi-master. Salt is awesome.
-
@scottalanmiller said in Port from SW - Salt master rsa key issue:
@DustinB3403 said in Port from SW - Salt master rsa key issue:
I thought you could only have a single Salt master in an environment. . .
No, it's multi-master. Salt is awesome.
Good to know.
-
@scottalanmiller I have tried repeatedly to delete all keys and rejoin the minions. It doesn't work. The keys are seen and accepted, but then the minions refuse to authenticate. I have even deleted all keys, uninstalled salt from the minions, deleted all cached data, reinstalled salt minion, and resubmitted keys, and still, trying a ping to all minions right after accpeting the keys results in no connections. running salt-minion -l debug shows that the minions think the masters' keys don't authenticate. It is very frustrating.
-
@dgingerich said in Port from SW - Salt master rsa key issue:
@scottalanmiller I have tried repeatedly to delete all keys and rejoin the minions. It doesn't work. The keys are seen and accepted, but then the minions refuse to authenticate. I have even deleted all keys, uninstalled salt from the minions, deleted all cached data, reinstalled salt minion, and resubmitted keys, and still, trying a ping to all minions right after accpeting the keys results in no connections. running salt-minion -l debug shows that the minions think the masters' keys don't authenticate. It is very frustrating.
That's really weird. Do you have a new minion that has never joined to test? The question is... is something lingering on that we don't realize OR is there something broken no the master, like it is handing out one public key but using a different private once it receives the data.
-
This post is deleted! -
I'm no salt expert (I've only played with it a few times) but just wanted to ask and confirm something about your RSA keys.
Are you entering a password when you generate the pairs or no?
-
@scottalanmiller Yes, (a point where we think alike) I just spun up another ubuntu system for a minion to test the master. Same result. The minion submits the key, I accept the key on the master and immediately try test.ping, and nothing. salt-minion -l debug shows the exact same error about authentication. It has to be something on the masters. However, I don't want to have to rebuild the masters because the rsa keys I generated will have to be replaced on the git repository, resulting in a lost day.
Management is expecting this to be up by Monday, but they just finalized the service structure yesterday. I think they expect me to work over the weekend.
-
How is the GIT taking that long? What are you using for GIT? I use a normal user account for GIT on my masters and I can set it up in seconds.
-
@scottalanmiller It's a matter of the person putting the keys into the repository config.
-
@dgingerich said in Port from SW - Salt master rsa key issue:
@scottalanmiller It's a matter of the person putting the keys into the repository config.
We use GitLab, it's basically instant.
-
@DustinB3403 said in Port from SW - Salt master rsa key issue:
I'm no salt expert (I've only played with it a few times) but just wanted to ask and confirm something about your RSA keys.
Are you entering a password when you generate the pairs or no?
No, I did not. I used "ssh-keygen -t rsa -C [email protected]" (sensitive data redacted) as advised in a google search on the matter and chose to leave the password empty.
-
@dgingerich Hrm. . .
If you're just entering through the process I don't think it would be the RSA keys then . . . maybe there is a firewall enabled on your Masters/Minions?
-
@scottalanmiller said in Port from SW - Salt master rsa key issue:
@dgingerich said in Port from SW - Salt master rsa key issue:
@scottalanmiller It's a matter of the person putting the keys into the repository config.
We use GitLab, it's basically instant.
yeah, well, I'm not one of the ones making decisions on this project. I'm just setting up the QA stack. If I could, I would set it up entirely manually. It would take me less time. However, they want it exactly like prod except for the server numbers, and prod is too big to do manually.
-
@DustinB3403 said in Port from SW - Salt master rsa key issue:
@dgingerich Hrm. . .
If you're just entering through the process I don't think it would be the RSA keys then . . . maybe there is a firewall enabled on your Masters/Minions?
Given that it worked and the key regen broke it, it's safe to assume it's a key issue.
-
@dgingerich said in Port from SW - Salt master rsa key issue:
@scottalanmiller said in Port from SW - Salt master rsa key issue:
@dgingerich said in Port from SW - Salt master rsa key issue:
@scottalanmiller It's a matter of the person putting the keys into the repository config.
We use GitLab, it's basically instant.
yeah, well, I'm not one of the ones making decisions on this project. I'm just setting up the QA stack. If I could, I would set it up entirely manually. It would take me less time. However, they want it exactly like prod except for the server numbers, and prod is too big to do manually.
That's our prod
-
@DustinB3403 said in Port from SW - Salt master rsa key issue:
@dgingerich Hrm. . .
If you're just entering through the process I don't think it would be the RSA keys then . . . maybe there is a firewall enabled on your Masters/Minions?
I haven't had the opportunity to do anything with the firewall to this point. By default, it is wide open.
-
@scottalanmiller said in Port from SW - Salt master rsa key issue:
@DustinB3403 said in Port from SW - Salt master rsa key issue:
@dgingerich Hrm. . .
If you're just entering through the process I don't think it would be the RSA keys then . . . maybe there is a firewall enabled on your Masters/Minions?
Given that it worked and the key regen broke it, it's safe to assume it's a key issue.
I was under the assumption he replaced all of the keys.
