Redis Failover, No Good Slave
-
The slave says that it is fully authenticated and synced.
-
Tried to lower the quorum number to one, did nothing.
-
Added a second Sentinel but the new Sentinel can see the original, but the original doesn't see it. Argh.
-
Increased the slave timeout to 90 seconds. Still nothing.
-
Mostly just putting this here for future reference. Still not making any progress.
https://gennadny.wordpress.com/2015/01/06/meet-redis-masters-slaves-and-scaling-out/
-
For anyone as clueless as I
"Redis is an open source, BSD licensed, advanced key-value cache and store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs."
Clarified that right up didn't it
-
@MattSpeller said:
For anyone as clueless as I
"Redis is an open source, BSD licensed, advanced key-value cache and store. It is often referred to as a data structure server since keys can contain strings, hashes, lists, sets, sorted sets, bitmaps and hyperloglogs."
Clarified that right up didn't it
Or much simpler, it is one of the more popular NoSQL databases.
-
Redis is also the default database for NodeBB. But Redis doesn't scale like MongoDB and is nowhere near as robust. So MangoLassi runs on MongoDB, not Redis. We are practically the only NodeBB site doing that, though. We are the biggest NodeBB traffic site, as far as I know.
-
I should add that this is the command I am using to force a failover:
SENTINEL failover mymaster -
Some docs here but they don't really help...
http://objectrocket.com/blog/how-to/introduction-to-redis-sentinel
-
In cool, ML news, on Duck Duck Go, this thread is the number one search hit for this error. Not very encouraging for finding an answer, but good that ML is getting that kind of SEO now!
-
Other threads of people having the issue but no resolution.
http://grokbase.com/t/gg/redis-db/141x9xyjcw/sentinel-2-8-4-nogoodslave-in-manual-failover
-
Here is what the error in the logs looks like...
[14804] 04 May 12:12:05.076 # +new-epoch 18 [14804] 04 May 12:12:05.076 # +try-failover master mymaster 127.0.0.1 6379 [14804] 04 May 12:12:05.078 # +vote-for-leader 78e27a9a509327e6c853535957c808cf8afd50cb 18 [14804] 04 May 12:12:05.078 # +elected-leader master mymaster 127.0.0.1 6379 [14804] 04 May 12:12:05.078 # +failover-state-select-slave master mymaster 127.0.0.1 6379 [14804] 04 May 12:12:05.140 # -failover-abort-no-good-slave master mymaster 127.0.0.1 6379 -
Which really just tells us that it is trying, we already know that it is failing. No additional info in the logs. Pretty useless.
-
Redis doesn't appear to even have this log entry documented:
failover-abort-no-good-slave -
Working through some ideas here...
https://groups.google.com/forum/#!topic/redis-db/M6WPJ0LnaWI
-
@thecreativeone91 said:
No one wants to be a slave.
I was going to say "it's so hard to find a good slave"
-
@scottalanmiller said:
Redis is also the default database for NodeBB. But Redis doesn't scale like MongoDB and is nowhere near as robust. So MangoLassi runs on MongoDB, not Redis. We are practically the only NodeBB site doing that, though. We are the biggest NodeBB traffic site, as far as I know.
I kept reading for a long time that Redis was hot shoot in a champaign glass, and we tested replacing some memcached servers and some MongoDB servers with it and I have to say that we realised there were problems fairly quickly. I liked how setting up different parts of Redis was easier, however there were huge latency issues and scalability problems as far as clustering/slaving went that caused us to go back to MongoDB.
I should note that now though we're moving away from MongoDB to Cassandra, which works a lot better and actually scales even better than MongoDB and doesn't suffer from the strange paging issue (with millions of documents) that MongoDB does, i.e. works fairly well.
Right now many things are using both until we can transfer old data from our MongoDB servers to Cassandra, but it's a few dozen terabytes of data.
I'm here to say that Redis may work for some, but as far as I'm concerned, compared to memcached (it'd be nice if it did have clustering, though) and MongoDB or Cassandra, it's cold diarrhoea in a dixie cup.
-
Yup, we did the same move from MongoDB to Cassandra at Change. Although MongoDB 3 is supposed to be a big leap forward.
Redis appears to be a complete disaster, however.
-
@tonyshowoff said:
I'm here to say that Redis may work for some, but as far as I'm concerned, compared to memcached (it'd be nice if it did have clustering, though) and MongoDB or Cassandra, it's cold diarrhoea in a dixie cup.
That's definitely how I am seeing it at this point.
