ML
    • Recent
    • Categories
    • Tags
    • Popular
    • Users
    • Groups
    • Register
    • Login

    Port from SW - Salt master rsa key issue

    IT Discussion
    salt salt master salt minion rsa
    6
    60
    9.5k
    Loading More Posts
    • Oldest to Newest
    • Newest to Oldest
    • Most Votes
    Reply
    • Reply as topic
    Log in to reply
    This topic has been deleted. Only users with topic management privileges can see it.
    • DustinB3403D
      DustinB3403
      last edited by DustinB3403

      I am majorly frustrated on this one. I have gone round and round with this issue and gotten nowhere. Maybe someone out there has an idea of what is going wrong.

      I'm building a QA environment with 10 servers, and the production environment is going to be a total of 42 servers, so we're having to automate things. So, we're using Salt on Ubuntu masters to automate. Last week, I built 8 servers for the QA stack, 2 of which were the Salt masters. Everything was connecting, it was pushing the configurations, but an error in the iptables configurations resulted in the Ubuntu master systems and two Ubuntu minions unable to communicate. In addition to changes in the structure of the systems, I wound up having to rebuild the entire QA environment.

      Yesterday, I build all 10 new systems, get Salt master installed on the masters, got them talking to the minions, and then generated rsa keys to get the masters to talk to the git repository to clone the configurations for the minions. As soon as the rsa keys were generated, the communications between masters and minions stopped, and they would never talk again.

      I have tried deleting the salt keys for the minions and reaccepting them. Even though the keys were presented and accepted, the minions still complain upon logging into the masters that the masters' keys would not authenticate. I tried forcably removing all salt components and saved data, including the stuff in /etc/salt and /var/cache/salt, and reinstall, it still gets the same result. Even with just one master and the minion service on itself, it would not authenticate.

      I'd rather not have to rebuild the masters. It would take several more hours of work.

      I'm thinking it has something to do with the rsa keys, but I can't say for sure. Perhaps Salt is presenting the master keys on accepting the minion keys, but then uses the system rsa keys upon authenticating the minions? Does that sound likely?

      Anyone have any ideas? I'm at my wits' end.

      scottalanmillerS 1 Reply Last reply Reply Quote 0
      • DustinB3403D
        DustinB3403
        last edited by

        I thought you could only have a single Salt master in an environment. . .

        scottalanmillerS 1 Reply Last reply Reply Quote 0
        • scottalanmillerS
          scottalanmiller @DustinB3403
          last edited by

          @DustinB3403 said in Port from SW - Salt master rsa key issue:

          I'm thinking it has something to do with the rsa keys, but I can't say for sure. Perhaps Salt is presenting the master keys on accepting the minion keys, but then uses the system rsa keys upon authenticating the minions? Does that sound likely?

          Yes, it authenticates in both directions and it sounds like you regenerated the ID keys for the Salt master, so the minions see the master has having changed. Unless you have a copy of the original keys, I think you need to rejoin.

          D 1 Reply Last reply Reply Quote 0
          • scottalanmillerS
            scottalanmiller @DustinB3403
            last edited by

            @DustinB3403 said in Port from SW - Salt master rsa key issue:

            I thought you could only have a single Salt master in an environment. . .

            No, it's multi-master. Salt is awesome.

            DustinB3403D 1 Reply Last reply Reply Quote 1
            • DustinB3403D
              DustinB3403 @scottalanmiller
              last edited by

              @scottalanmiller said in Port from SW - Salt master rsa key issue:

              @DustinB3403 said in Port from SW - Salt master rsa key issue:

              I thought you could only have a single Salt master in an environment. . .

              No, it's multi-master. Salt is awesome.

              Good to know.

              1 Reply Last reply Reply Quote 0
              • D
                dgingerich @scottalanmiller
                last edited by

                @scottalanmiller I have tried repeatedly to delete all keys and rejoin the minions. It doesn't work. The keys are seen and accepted, but then the minions refuse to authenticate. I have even deleted all keys, uninstalled salt from the minions, deleted all cached data, reinstalled salt minion, and resubmitted keys, and still, trying a ping to all minions right after accpeting the keys results in no connections. running salt-minion -l debug shows that the minions think the masters' keys don't authenticate. It is very frustrating.

                scottalanmillerS 1 Reply Last reply Reply Quote 2
                • scottalanmillerS
                  scottalanmiller @dgingerich
                  last edited by

                  @dgingerich said in Port from SW - Salt master rsa key issue:

                  @scottalanmiller I have tried repeatedly to delete all keys and rejoin the minions. It doesn't work. The keys are seen and accepted, but then the minions refuse to authenticate. I have even deleted all keys, uninstalled salt from the minions, deleted all cached data, reinstalled salt minion, and resubmitted keys, and still, trying a ping to all minions right after accpeting the keys results in no connections. running salt-minion -l debug shows that the minions think the masters' keys don't authenticate. It is very frustrating.

                  That's really weird. Do you have a new minion that has never joined to test? The question is... is something lingering on that we don't realize OR is there something broken no the master, like it is handing out one public key but using a different private once it receives the data.

                  D 1 Reply Last reply Reply Quote 0
                  • DustinB3403D
                    DustinB3403
                    last edited by

                    This post is deleted!
                    1 Reply Last reply Reply Quote 0
                    • DustinB3403D
                      DustinB3403
                      last edited by DustinB3403

                      I'm no salt expert (I've only played with it a few times) but just wanted to ask and confirm something about your RSA keys.

                      Are you entering a password when you generate the pairs or no?

                      D 1 Reply Last reply Reply Quote 0
                      • D
                        dgingerich @scottalanmiller
                        last edited by

                        @scottalanmiller Yes, (a point where we think alike) I just spun up another ubuntu system for a minion to test the master. Same result. The minion submits the key, I accept the key on the master and immediately try test.ping, and nothing. salt-minion -l debug shows the exact same error about authentication. It has to be something on the masters. However, I don't want to have to rebuild the masters because the rsa keys I generated will have to be replaced on the git repository, resulting in a lost day.

                        Management is expecting this to be up by Monday, but they just finalized the service structure yesterday. I think they expect me to work over the weekend.

                        1 Reply Last reply Reply Quote 0
                        • scottalanmillerS
                          scottalanmiller
                          last edited by

                          How is the GIT taking that long? What are you using for GIT? I use a normal user account for GIT on my masters and I can set it up in seconds.

                          D 1 Reply Last reply Reply Quote 0
                          • D
                            dgingerich @scottalanmiller
                            last edited by

                            @scottalanmiller It's a matter of the person putting the keys into the repository config.

                            scottalanmillerS 1 Reply Last reply Reply Quote 0
                            • scottalanmillerS
                              scottalanmiller @dgingerich
                              last edited by

                              @dgingerich said in Port from SW - Salt master rsa key issue:

                              @scottalanmiller It's a matter of the person putting the keys into the repository config.

                              We use GitLab, it's basically instant.

                              D 1 Reply Last reply Reply Quote 0
                              • D
                                dgingerich @DustinB3403
                                last edited by

                                @DustinB3403

                                @DustinB3403 said in Port from SW - Salt master rsa key issue:

                                I'm no salt expert (I've only played with it a few times) but just wanted to ask and confirm something about your RSA keys.

                                Are you entering a password when you generate the pairs or no?

                                No, I did not. I used "ssh-keygen -t rsa -C [email protected]" (sensitive data redacted) as advised in a google search on the matter and chose to leave the password empty.

                                DustinB3403D 1 Reply Last reply Reply Quote 0
                                • DustinB3403D
                                  DustinB3403 @dgingerich
                                  last edited by

                                  @dgingerich Hrm. . .

                                  If you're just entering through the process I don't think it would be the RSA keys then . . . maybe there is a firewall enabled on your Masters/Minions?

                                  scottalanmillerS D 2 Replies Last reply Reply Quote 0
                                  • D
                                    dgingerich @scottalanmiller
                                    last edited by

                                    @scottalanmiller said in Port from SW - Salt master rsa key issue:

                                    @dgingerich said in Port from SW - Salt master rsa key issue:

                                    @scottalanmiller It's a matter of the person putting the keys into the repository config.

                                    We use GitLab, it's basically instant.

                                    yeah, well, I'm not one of the ones making decisions on this project. I'm just setting up the QA stack. If I could, I would set it up entirely manually. It would take me less time. However, they want it exactly like prod except for the server numbers, and prod is too big to do manually.

                                    scottalanmillerS 1 Reply Last reply Reply Quote 0
                                    • scottalanmillerS
                                      scottalanmiller @DustinB3403
                                      last edited by

                                      @DustinB3403 said in Port from SW - Salt master rsa key issue:

                                      @dgingerich Hrm. . .

                                      If you're just entering through the process I don't think it would be the RSA keys then . . . maybe there is a firewall enabled on your Masters/Minions?

                                      Given that it worked and the key regen broke it, it's safe to assume it's a key issue.

                                      DustinB3403D 1 Reply Last reply Reply Quote 0
                                      • scottalanmillerS
                                        scottalanmiller @dgingerich
                                        last edited by

                                        @dgingerich said in Port from SW - Salt master rsa key issue:

                                        @scottalanmiller said in Port from SW - Salt master rsa key issue:

                                        @dgingerich said in Port from SW - Salt master rsa key issue:

                                        @scottalanmiller It's a matter of the person putting the keys into the repository config.

                                        We use GitLab, it's basically instant.

                                        yeah, well, I'm not one of the ones making decisions on this project. I'm just setting up the QA stack. If I could, I would set it up entirely manually. It would take me less time. However, they want it exactly like prod except for the server numbers, and prod is too big to do manually.

                                        That's our prod 🙂

                                        1 Reply Last reply Reply Quote 0
                                        • D
                                          dgingerich @DustinB3403
                                          last edited by

                                          @DustinB3403 said in Port from SW - Salt master rsa key issue:

                                          @dgingerich Hrm. . .

                                          If you're just entering through the process I don't think it would be the RSA keys then . . . maybe there is a firewall enabled on your Masters/Minions?

                                          I haven't had the opportunity to do anything with the firewall to this point. By default, it is wide open.

                                          scottalanmillerS 1 Reply Last reply Reply Quote 0
                                          • DustinB3403D
                                            DustinB3403 @scottalanmiller
                                            last edited by

                                            @scottalanmiller said in Port from SW - Salt master rsa key issue:

                                            @DustinB3403 said in Port from SW - Salt master rsa key issue:

                                            @dgingerich Hrm. . .

                                            If you're just entering through the process I don't think it would be the RSA keys then . . . maybe there is a firewall enabled on your Masters/Minions?

                                            Given that it worked and the key regen broke it, it's safe to assume it's a key issue.

                                            I was under the assumption he replaced all of the keys.

                                            scottalanmillerS 1 Reply Last reply Reply Quote 0
                                            • 1
                                            • 2
                                            • 3
                                            • 1 / 3
                                            • First post
                                              Last post