The performance of the SET command degrades under high load

[muchnik]

Hi, we've encountered a strange problem under high load in our reactive redis client -- the SET command in service takes around 20 seconds according to JVM metrics and the CPU is heavily loaded. But Redis metrics shows that the commands are executed quickly and consistently.

After various tests, I've concluded that the issue might be due to the volume of the value we are writing, which is about 6 MB. If the size of the value is reduced, everything works stably. I added metrics to the serializer, and they show that it works for 10-30ms.

Has anyone encountered this problem before?

spring boot 2.7.15
lettuce 6.1.10
netty 4.1.97
reactor 3.4.32

@Bean
public ReactiveRedisTemplate<String, UserDto> userDtoReactiveRedisTemplate(ReactiveRedisConnectionFactory factory) {
    var keySerializer = new StringRedisSerializer();
    var valueSerializer = new Jackson2JsonRedisSerializer<>(UserDto.class);
    RedisSerializationContextBuilder<String, UserDto> builder = RedisSerializationContext.newSerializationContext(keySerializer);
    var context = builder
            .value(valueSerializer)
            .build();
    return new ReactiveRedisTemplate<>(factory, context);
}


public class CacheService {
    private final ReactiveRedisTemplate<String, UserDto> template;

    @Value("${service.cache.ttl:30s}")
    private Duration ttl;

    public Mono<Boolean> set(String key, UserDto value) {
        return template.opsForValue()
                       .set(key, value, ttl);
    }
}


private Mono<UserDto> getNextUserDataAndCache(...) {
    <...collect data, check if cached, update existing (fill with new information), then update cache>
                 // Mono
                        .flatMap(existingUser -> {
                                        updateUser(existingUser, newUser);
                                        return cacheService.set(key, existingUser)
                                                           .thenReturn(existingUser);
                                    });
                        });
}

[mp911de]

Thanks a lot. The issue you're seeing is that JSON serialization happens on the I/O thread which keeps your I/O thread busy.

Have you tried using a separate thread pool for encoding/decoding?

With a configuration tweak on the Lettuce side you could be able to address the issue:

ClientOptions options = ClientOptions.builder().publishOnScheduler(true).build();

// later, use it with LettuceClientConfiguration.
LettuceClientConfiguration.builder().clientOptions(options).build();

[muchnik]

@mp911de, thank you for your response. I've implemented your suggestion by configuring a custom Lettuce connection pool for our client and enabling publishOnScheduler(true). However, I haven't observed any improvements.

Further investigation revealed that the redisTemplate.opsForValue().set(key, value, duration) method is responsible for a significant amount of transmit bandwidth (response) usage during Redis transactions.
Confirming this, removing the SET operation resulted in a reduction of the redis response network load.

It seems that we might be experiencing network limitations, as the degradation becomes noticeable when the Redis transmission bandwidth is heavily utilized.

I'm curious as to why the SET operation in particular is causing such a heavy load on the Redis response network. The documentation indicates that the SET operation should return a boolean, but could there be another underlying issue that might explain this behavior?

For instance, there are spikes where we are calling the SET command. I am expecting to see received bandwidth, but I am also observing transmit bandwidth.

[mp911de]

Have you measured the payload size of the key and its value? SET returns OK as response.

[muchnik]

Key is few bytes, value is around 5MB
Probably transmit network is loaded because of redis replication?

[mp911de]

That can well be. I'd be caught by surprise if that issue happens with really small message sizes.

[muchnik]

Our load is about ~80+ tps, degradation appears when redis transmit network hits 520mbit