Writable replicas: yes or no?

[itamarhaber]

TL;DR Since time immemorial, Redis has allowed stand-alone and cluster replicas to be configured so they can accept write requests. If you have an opinion about this feature, please add your voice to this discussion.

The primary use of Redis replicas is to provide a highly-available database service via failover and promotion. Replicas can also be used to offload reads from the master. However, a client can write to a replica under certain conditions (with the proper configuration and/or connection settings).

The motivations for making replicas writable, as far as we're aware, were mainly to support these use cases:

a. Allow creating (and possibly deleting) temporary keys, such as those that ZUNIONSTORE produces
b. Allow using commands that have the potential to write to keys, even if they don't necessarily do so, such as with using SORT in its read-only invocation form.

However, with Redis v6.2 and the upcoming v7, a slew of new _RO commands and complementary non-STORE variants have been added to address just that.

If that's indeed the case, namely that writable replicas are no longer really needed, the next question is whether they are needed at all. Supporting write replicas not only adds to the overall complexity of the project but also introduces several edge cases that, while rare, have damage potential during runtime. While the effort can be invested to address some/all of the issues, we're looking to assess the need and requirements from the feature.

So the questions we are looking to answer are mainly:

1. Are you using/have used Redis' writable replicas? If yes, what is/was the use case?
2. If you've answered yes to the above, are you writing to keys that also exist on the master?
3. And, if yes is your answer to the first question, are the keys written to the replica set with a TTL so they'll expire?

Thanks in advance for your participation!

[abrookins]

Writable replicas create some seemingly-useful caching possibilities, like pushing shared cache state to replicas through replication while allowing the replicas to develop location-specific caches through writes. Fly describes this pattern: https://fly.io/blog/last-mile-redis/

I'm not sure how widely used this approach is, but it's cool!

[itamarhaber]

Thanks for sharing the use case, cool indeed and serves as the perfect serve for pinging @mrkurt :)

[stopforumspam]

this is exactly how I use redis for driving my api rate counters

[mrkurt]

Please don't remove replica writes. Or, if possible, create a way to have loosely coupled replicas.

Are you using/have used Redis' writable replicas? If yes, what is/was the use case?

We use them for globally coordinated caches (see above URL).

If you've answered yes to the above, are you writing to keys that also exist on the master?

Yes, we write to the leader to propagate changes everywhere, purge keys, etc. Writes to replicas will overwrite those changes. We even do things like lpush on the leader and brpop on the replicas.

And, if yes is your answer to the first question, are the keys written to the replica set with a TTL so they'll expire?

No, not necessarily. Replicas can have long lived, replica only data.

We sometimes set TTLs on the leader and then reset them on the replicas. It's a handy way to push a possible change to all the replicas and then let them vanish if they don't get used.

[oranagra]

@mrkurt so you use EXPIRE on the master, and PERSIST on the replica? AFAIK that's not gonna have any effect, the master will still propagate DEL when the time comes.

There's currently a mechanism in redis that allows EXPIRE to be used on replicas (and the replica will delete the keys locally when the time comes), but this mechanism isn't very consistent, it can get messed up in some situations, for instance when a replica is promoted

I assume you're not interested in promoting replicas, and also have issues with a command arriving from the master, failing on the replica due to the different data set (maybe you don't use any commands / combinations that can cause that).
And also don't care that you lose local writes when the replica re-syncs...

p.s. which version of redis do you use?

[mrkurt]

Well now you made me wonder if we're actually using TTLs like I thought. We may not be. 😅 We definitely set TTLs on replica writes sometimes.

These replicas will never be promoted.

We have devs using replica writes across a bunch of versions. I think mostly less than 1 year old.

[soloestoy]

@mrkurt thanks for your report, seems you use writable-replica to build active-active, if I'm right I don't think it's a good use case, since the change on replica cannot propagate to master.

And another thing: from 4.0 we use proxy replication mechanism, in replication stream replicas work as a proxy, replica propagate only write commands come from master to sub-replica, users' write commands on writable-replica would not propagate to sub-replica. That means the writable-replica would become an isolated data island. I don't know which version you are using, and do you notice that.

[mrkurt]

No we're not doing active-active. This is what we're doing: https://fly.io/blog/last-mile-redis/

I don't think Redis-as-a-cache fits into normal database cluster semantics, really. We use replication to implement a loosely coupled caching model. We usually want the replicas to be islands, because caches across geographies need different data. We also want to propagate some changes to all caches. Redis replication + writable replicas solves that nicely.

If you are trying to build an elegant replication model you're probably building something we don't use Redis for.

[bpo]

Hi – I use writes on replicas! The big case that I don't see addressed here yet is when consistency is not important but availability and cache performance is.

Consider a distributed application that needs to migrate from one Redis instance to another. Redis is used as a flat cache where write order isn't important and a (Redis) cache miss incurs a performance penalty but no errors:

With replica writes enabled, we can update the application configuration(s) to point to the new Redis instance without any additional coordination. Cache hit ratio stays high during the transition because a meaningful percentage of writes from the old Redis flow through to the new Redis where both reads and writes are happening. Eventually the application stops writing to the old Redis altogether, the new Redis is promoted, and business continues as normal before the migration event occurred.

If replica writes did not exist, I think we would either have to take (1) an availability hit to coordinate the application to switch the destination of all writes at once, or (2) a drop in cache hit ratio by disconnecting replication before pointing any writes at the new Redis.

So to answer @itamarhaber's questions:

1. Yes, using replica writes
2. Yes, writing to keys that also exist on the master
3. Maybe - TTLs may or may not be set.

[soloestoy]

Hi @bpo thanks for your report.

The case you said is very common, many users in our platform migrate data from one instance to another everyday, but we use more elegant way to solve:

1. we offer DTS(Data Transmission Service) to help users to do it in our platform.

2. and we also offer redis-shake, a free open source project which can be used for synchronizing data between two redis instance, and people can use it everywhere they want, no license restrictions.

So I don't think redis needs handle this case internally, using a middle ware is more reasonable and flexible.

BTW, redis is not only cache.

[oranagra]

We also have a tool for that (not open source).
it connects to the source db as a replica (converting the RDB file into RESTORE commands), and sends commands to the destination db as a client.
one of the issues that came up is that it could have created data corruption since the destination db thought it's a master and did eviction and expiration, in which case later commands that arrive from the source could fail or get executed differently.

The solution we implemented is a feature called "pseudo-master", and "pseudo-replica".
i.e. we set the destination database in a "pseudo-replica" mode, which prevents it from doing eviction and expiration.
and the sync tools uses REPLCONF to set it's connections with a "pseudo-master" flag, which makes destination server handle the commands coming from them as if they arrive from a master (i.e. not do lazy expiry check).
in essence this makes the destination server a writable replica 8-)

The difference is that, if you don't solve this "pseudo-master" thing, you'll end up with data corruption even if you don't have writes on the target db.

[soloestoy]

Yes, the target master has data corruption issue due to expire and eviction, but in bop's case, it's not a problem because redis performs as a cache service.

About the "pseudo-master" and "pseudo-replica" feature, it's a good idea, and we had intended to implement the similar feature, but we didn't, we discussed with some users in our platform and got some conclusions:

1. if users use redis as cache, eviction and expire data corruption is not a problem
2. if users use redis as database, they always set eviction-policy to no-eviction
3. about expire, normally a key with ttl is static, no change after first write

regardless of above, "pseudo-master" and "pseudo-replica" is a good idea, and using a tool (like redis-shake or your tool) is the right way in the migrate case.

[oranagra]

i don't think i agree with the 3 bullets you wrote, but i don't think there's a point to argue about that (specifically if you agree that the pseudo-master would be a good idea for the OSS). maybe @YaacovHazan can find time to push a PR with these changes.

in any case, i wanted to stress here that this is essentially a case of a writable replica, and even if we add the pseudo-master, it's still a case of writable replica (just a bit less buggy).

Maybe we feel a little bit better about it because unlike replica-read-only it is external to redis so we can turn a blind eye, but it's still there....

[the-rarbg]

this dont work if their is any module added, like redis search

[srishtikeshari]

Is writable replica still supported? On performing any 'STORE' operation like 'ZRANGESTORE' or 'ZUNIONSTORE' on a replica node of a Redis Cluster, I a getting MOVED exception even when 'replica-read-only' is set to 'no'. Is there any other configuration required to make a replica writable?

My use case for Writable replica -
I am trying to perform secondary indexing in Redis as per https://redis.io/docs/manual/patterns/indexes/.
For secondary indexing, I use commands like 'ZRANGESTORE', 'ZUNIONSTORE' and 'ZINTERSTORE', store the results in temporary keys. As there a multiple fields on which the data is indexed, there are multiple set commands('ZRANGESTORE', 'ZUNIONSTORE' and 'ZINTERSTORE') done in a transaction and I delete all the temporary keys used for storing intermediate results in the process in the same transaction.
I want to use the replica nodes to perform all the set(indexing) operations and use the primary node just for writing data, and hence, require the write('STORE') commands in the replica node.

Redis version -
Redis server v=7.0.11 sha=00000000:0 malloc=jemalloc-5.2.1 bits=64 build=44169fc7b347b8ec

[melias122]

@srishtikeshari Did you find any solution?