Redis Failover, No Good Slave

scottalanmiller

Just a test system. I have two Redis nodes. One master and one slave. If I run "info" on them the master sees itself as a master and that it has a slave, and the slave sees itself as a slave and can see its master. The slave priority is set to 100, which is normal. Only a priority of 0 should cause failover to it to be blocked.

Using Redis Sentinel, which is running on the same box as the master, to force a failover all I get is:

(error) NOGOODSLAVE No suitable slave to promote

But looking around for this error is not really turning anything up that is useful. I get that the slave is "no good". But why? The logs show nothing.

scottalanmiller

I tried this technique, telling the slave that I wanted it to be able to be a master. No luck there either. A Google Group had suggested this.

127.0.0.1:6379> slaveof no one
OK

No one wants to be a slave.

gjacobse

@thecreativeone91 said:

No one wants to be a slave.

No,.. but if you have children you are one by default. Apologies for the drift there.

What about this:

slaveof 192.168.1.1 6379

where you would put in your correct IP.

Do you need to authenticate the slave to the master?

Setting a slave to authenticate to a master
If your master has a password via requirepass, it's trivial to configure the slave to use that password in all sync operations.
To do it on a running instance, use redis-cli and type:

config set masterauth <password>

To set it permanently, add this to your config file:

masterauth <password>

both from:

http://redis.io/topics/replication

scottalanmiller

The slave says that it is fully authenticated and synced.

scottalanmiller

Tried to lower the quorum number to one, did nothing.

scottalanmiller

Added a second Sentinel but the new Sentinel can see the original, but the original doesn't see it. Argh.

scottalanmiller

Increased the slave timeout to 90 seconds. Still nothing.

scottalanmiller

Mostly just putting this here for future reference. Still not making any progress.

https://gennadny.wordpress.com/2015/01/06/meet-redis-masters-slaves-and-scaling-out/

MattSpeller

For anyone as clueless as I

"Redis is an open source, BSD licensed, advanced key-value cache and store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs."

Clarified that right up didn't it

scottalanmiller

@MattSpeller said:

For anyone as clueless as I

"Redis is an open source, BSD licensed, advanced key-value cache and store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs."

Clarified that right up didn't it

Or much simpler, it is one of the more popular NoSQL databases.

scottalanmiller

Redis is also the default database for NodeBB. But Redis doesn't scale like MongoDB and is nowhere near as robust. So MangoLassi runs on MongoDB, not Redis. We are practically the only NodeBB site doing that, though. We are the biggest NodeBB traffic site, as far as I know.

scottalanmiller

I should add that this is the command I am using to force a failover:

SENTINEL failover mymaster

scottalanmiller

Some docs here but they don't really help...

http://objectrocket.com/blog/how-to/introduction-to-redis-sentinel

mlnews

In cool, ML news, on Duck Duck Go, this thread is the number one search hit for this error. Not very encouraging for finding an answer, but good that ML is getting that kind of SEO now!

scottalanmiller

Other threads of people having the issue but no resolution.

http://grokbase.com/t/gg/redis-db/141x9xyjcw/sentinel-2-8-4-nogoodslave-in-manual-failover

scottalanmiller

Here is what the error in the logs looks like...

[14804] 04 May 12:12:05.076 # +new-epoch 18
[14804] 04 May 12:12:05.076 # +try-failover master mymaster 127.0.0.1 6379
[14804] 04 May 12:12:05.078 # +vote-for-leader 78e27a9a509327e6c853535957c808cf8afd50cb 18
[14804] 04 May 12:12:05.078 # +elected-leader master mymaster 127.0.0.1 6379
[14804] 04 May 12:12:05.078 # +failover-state-select-slave master mymaster 127.0.0.1 6379
[14804] 04 May 12:12:05.140 # -failover-abort-no-good-slave master mymaster 127.0.0.1 6379

scottalanmiller

Which really just tells us that it is trying, we already know that it is failing. No additional info in the logs. Pretty useless.

scottalanmiller

Redis doesn't appear to even have this log entry documented:

failover-abort-no-good-slave

scottalanmiller

Working through some ideas here...

https://groups.google.com/forum/#!topic/redis-db/M6WPJ0LnaWI