event data processing -Redisgears module: Data is not syncing to postgres db with redisgears module loaded via config map on top bitnami redis cluster

[Archanashetty7]

Deploy bitnami redis-cluster 7.2.3 image tag on kubernetes setup with 6 nodes and 1 replica each
redisgears module is loaded using configuration: loadmodule /var/opt/redislabs/lib/modules/redisgears.so Plugin /var/opt/redislabs/modules/rg/plugin/gears_python.so
Run redis gears recipe to sync data from redis to postgres. gears-cli run --host worker5 --port 30010 --password hello registration_details.py --requirements requirements-postgres.txt
perform hset on to redis.
yes we have customised bitnami docker image to load that modules.

FROM docker.io/bitnami/redis-cluster:7.2.3
RUN mkdir -p /var/opt/redislabs/lib/modules/
COPY redisgears.so /var/opt/redislabs/lib/modules/redisgears.so
RUN chmod +rwx /var/opt/redislabs/lib/modules/redisgears.so
RUN mkdir -p /var/opt/redislabs/modules/rg
COPY gear_runtime-jar-with-dependencies.jar /var/opt/redislabs/modules/rg/gear_runtime-jar-with-dependencies.jar
RUN mkdir -p /var/opt/redislabs/modules/rg/plugin
#RUN mkdir -p /var/opt/redislabs/modules/rg/OpenJDK
RUN mkdir -p /var/opt/redislabs/modules/rg/python3
RUN mkdir -p /var/opt/redislabs/modules/rg/python3_1.2.4
COPY plugin /var/opt/redislabs/modules/rg/plugin
#COPY OpenJDK /var/opt/redislabs/modules/rg/OpenJDK
COPY python3 /var/opt/redislabs/modules/rg/python3
COPY python3_1.2.4 /var/opt/redislabs/modules/rg/python3_1.2.4
COPY python3_1.2.4/bin/python3.7 /var/opt/redislabs/modules/rg/python3_1.2.4/bin/python3
RUN chmod -R 777 /var/opt/redislabs/modules/rg

#for postgres
RUN apt-get update && apt-get install -y build-essential libpq-dev
ENV PATH=$PATH:/usr/bin/pg_config

#for oracle
RUN mkdir -p /opt/oracle
RUN cd /opt/oracle
COPY instantclient_12_2 /opt/oracle/instantclient_12_2
RUN apt-get install -y libaio1

Set environment variables for Oracle Instant Client

ENV ORACLE_HOME=/opt/oracle/instantclient_12_2
ENV TNS_ADMIN=/opt/oracle/instantclient_12_2/network/admin
ENV LD_LIBRARY_PATH=/opt/oracle/instantclient_12_2
Expectation : data to be synced on to postgres database.

and also RG.PYEXECUTE "GB().run()" is not returning data from all the master nodes. It is only fetching data from particular node where redis-cli is executed

What do you see instead?
There is no data moving to postgres when hset is inserted into redis

These are load module logs from redis-cluster

COPYING FILE
redis-cluster 06:55:45.31 INFO ==>
redis-cluster 06:55:45.32 INFO ==> Welcome to the Bitnami redis-cluster container
redis-cluster 06:55:45.32 INFO ==> Subscribe to project updates by watching https://github.com/bitnami/containers
redis-cluster 06:55:45.32 INFO ==> Submit issues and feature requests at https://github.com/bitnami/containers/issues
redis-cluster 06:55:45.32 INFO ==>
redis-cluster 06:55:45.32 INFO ==> ** Starting Redis setup **
redis-cluster 06:55:45.36 INFO ==> Initializing Redis
redis-cluster 06:55:45.37 INFO ==> Setting Redis config file
Storing map with hostnames and IPs
redis-cluster 06:55:50.53 INFO ==> ** Redis setup finished! **
91:C 31 Jan 2024 06:55:50.572 # WARNING: Changing databases number from 16 to 1 since we are in cluster mode
91:C 31 Jan 2024 06:55:50.573 * oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
91:C 31 Jan 2024 06:55:50.573 * Redis version=7.2.3, bits=64, commit=00000000, modified=0, pid=91, just started
91:C 31 Jan 2024 06:55:50.573 * Configuration loaded
91:M 31 Jan 2024 06:55:50.573 * monotonic clock: POSIX clock_gettime
.
.-__ ''-._ _.- . . ''-._ Redis 7.2.3 (00000000/0) 64 bit
.- .-```. ```\/ _.,_ ''-._ ( ' , .-` | `, ) Running in cluster mode |`-._`-...-` __...-.-.|'_.-'| Port: 6379 | -. ._ / _.-' | PID: 91 -._ -._ -./ .-' .-'
|-._-. -.__.-' _.-'_.-'| | -.-._ _.-'_.-' | https://redis.io -._ -._-..-'.-' .-'
|-._-. -.__.-' _.-'_.-'| | -.-._ _.-'_.-' | -._ -._-..-'_.-' _.-'
-._ -..-' _.-'
-._ _.-' -..-'
91:M 31 Jan 2024 06:55:50.574 # WARNING: The TCP backlog setting of 511 cannot be enforced because /proc/sys/net/core/somaxconn is set to the lower value of 128.
Could not connect to Redis at rc-redis-cluster-0.rc-redis-cluster-headless:6379: Connection refused
Node rc-redis-cluster-0.rc-redis-cluster-headless not ready, waiting for all the nodes to be ready...
91:M 31 Jan 2024 06:55:50.584 * No cluster configuration found, I'm 59a3e3ba3a500606552d9ff08ec774478ad0ea15
91:M 31 Jan 2024 06:55:50.601 * RedisGears version 1.2.4, git_sha=0c8365367cd345a965d675343558585e923555a7, compiled_os=linux-bullseye-x64
91:M 31 Jan 2024 06:55:50.601 * Redis version found by RedisGears : 7.2.3 - oss
91:M 31 Jan 2024 06:55:50.601 * MaxExecutions:1000
91:M 31 Jan 2024 06:55:50.601 * MaxExecutionsPerRegistration:100
91:M 31 Jan 2024 06:55:50.602 * ProfileExecutions:0
91:M 31 Jan 2024 06:55:50.602 * ExecutionThreads:3
91:M 31 Jan 2024 06:55:50.602 * ExecutionMaxIdleTime:5000
91:M 31 Jan 2024 06:55:50.602 * SendMsgRetries:3
91:M 31 Jan 2024 06:55:50.602 * Plugin:[/var/opt/redislabs/modules/rg/plugin/gears_python.so]
91:M 31 Jan 2024 06:55:50.604 # Loading plugin GearsPythonPlugin version 10000
91:M 31 Jan 2024 06:55:50.604 * Initializing GearsPythonPlugin version 1.0.0
91:M 31 Jan 2024 06:55:50.604 * Python config:
91:M 31 Jan 2024 06:55:50.604 * CreateVenv : 0
91:M 31 Jan 2024 06:55:50.604 * DownloadDeps : 0
91:M 31 Jan 2024 06:55:50.604 * ForeceDownloadDepsOnEnterprise : 0
91:M 31 Jan 2024 06:55:50.604 * PythonAttemptTraceback : 1
91:M 31 Jan 2024 06:55:50.604 * OverridePythonAllocators : 1
91:M 31 Jan 2024 06:55:50.604 * PythonInstallReqMaxIdleTime : 30000
91:M 31 Jan 2024 06:55:50.604 * DependenciesUrl : http://redismodules.s3.amazonaws.com/redisgears/redisgears-python.Linux-bullseye-x86_64.1.2.4.tgz
91:M 31 Jan 2024 06:55:50.604 * DependenciesSha256 : a82d7a74dcdffdac54a1ca882aee64993f980380d791b4aef9244f618b43ff59
91:M 31 Jan 2024 06:55:50.604 * PythonInstallationDir : /var/opt/redislabs/modules/rg
91:M 31 Jan 2024 06:55:50.605 * PYENV_DIR: /var/opt/redislabs/modules/rg//python3_1.2.4/
91:M 31 Jan 2024 06:55:50.605 * PYENV_HOME_DIR: /var/opt/redislabs/modules/rg//python3_1.2.4//.venv/
91:M 31 Jan 2024 06:55:50.605 * PYENV_BIN_DIR: /var/opt/redislabs/modules/rg//python3_1.2.4//.venv//bin
91:M 31 Jan 2024 06:55:50.605 * PYENV_ACTIVATE: /var/opt/redislabs/modules/rg//python3_1.2.4//.venv//bin/activate_this.py
91:M 31 Jan 2024 06:55:50.605 * PYENV_ACTIVATE_SCRIPT: /var/opt/redislabs/modules/rg//python3_1.2.4//.venv//bin/activate
91:M 31 Jan 2024 06:55:50.605 * log file is
91:M 31 Jan 2024 06:55:50.605 * Found python installation under: /var/opt/redislabs/modules/rg//python3_1.2.4/
91:M 31 Jan 2024 06:55:50.967 * Module 'rg' loaded from /var/opt/redislabs/lib/modules/redisgears.so
91:M 31 Jan 2024 06:55:50.967 * Server initialized
91:M 31 Jan 2024 06:55:50.984 * Creating AOF base file appendonly.aof.1.base.rdb on server start
91:M 31 Jan 2024 06:55:50.998 * Creating AOF incr file appendonly.aof.1.incr.aof on server start
91:M 31 Jan 2024 06:55:50.998 * Ready to accept connections tcp
Waiting 0s before querying node ip addresses
91:M 31 Jan 2024 06:55:51.691 * configEpoch set to 1 via CLUSTER SET-CONFIG-EPOCH

Performing hash slots allocation on 3 nodes...
Master[0] -> Slots 0 - 5460
Master[1] -> Slots 5461 - 10922
Master[2] -> Slots 10923 - 16383
M: 59a3e3ba3a500606552d9ff08ec774478ad0ea15 10.233.70.209:6379
slots:[0-5460] (5461 slots) master
M: c274532721303bc566ca8991a77536c681af1adf 10.233.72.24:6379
slots:[5461-10922] (5462 slots) master
M: f6d3bd41e7d16bda0bd2552a1948e56ff09ab5c0 10.233.73.180:6379
slots:[10923-16383] (5461 slots) master
Nodes configuration updated
Assign a different config epoch to each node
Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join
.91:M 31 Jan 2024 06:55:53.009 * Cluster state changed: ok
Performing Cluster Check (using node 10.233.70.209:6379)
M: 59a3e3ba3a500606552d9ff08ec774478ad0ea15 10.233.70.209:6379
slots:[0-5460] (5461 slots) master
M: f6d3bd41e7d16bda0bd2552a1948e56ff09ab5c0 10.233.73.180:6379
slots:[10923-16383] (5461 slots) master
M: c274532721303bc566ca8991a77536c681af1adf 10.233.72.24:6379
slots:[5461-10922] (5462 slots) master
[OK] All nodes agree about slots configuration.
Check for open slots...
Check slots coverage...
[OK] All 16384 slots covered.
Cluster correctly created
redis-server "${ARGS[@]}"

And also when we execute
RG.INFOCLUSTER command we get below output, but expected output is cluster details
"no cluster mode"

[MeirShpilraien]

On oss cluster you need to run rg.refreshcluster on each node before register the gears function. Try to run it and see if you get the expected output from RG.INFOCLUSTER. Also I would advice to start with a single no cluster instance and see if you can sync the data, will be easier to troubleshoot and single shard instance.

[Archanashetty7]

We ran RG.INFOCLUSTER on each node, still RG.INFOCLUSTER is giving
127.0.0.1:6379> RG.INFOCLUSTER
"no cluster mode"

[MeirShpilraien]

Did you run rg.refreshcluster on each node before?

[Archanashetty7]

yes on every node

[MeirShpilraien]

Is it possible to share the full log file of each of the nodes and the output of info everything of each of the nodes? Thanks.

[Archanashetty7]

We have redis cluster with 3 master nodes.Below are the logs of each redis master
rediscluster0.txt
rediscluster1.txt
Rediscluster2.txt

kubectl exec -it rc-redis-cluster-0 -- sh
$ redis-cli
127.0.0.1:6379> RG.INFOCLUSTER
"no cluster mode"
127.0.0.1:6379>

kubectl exec -it rc-redis-cluster-1 -- sh
$ redis-cli
127.0.0.1:6379> RG.INFOCLUSTER
"no cluster mode"
127.0.0.1:6379>

kubectl exec -it rc-redis-cluster-2 -- sh
$ redis-cli
127.0.0.1:6379> RG.INFOCLUSTER
"no cluster mode"
127.0.0.1:6379>

[techwizardg]

it would be good if you could guide us in moving forward here as we are blocked here in syncing data to Postgresql