Decommission single broker node in Kubernetes with 3 node redpanda cluster

[mdfaizsiddiqui]

Hey

I'm running Redpanda on kubernetes cluster (Server Version: v1.27.2-eks-c12679a) with local-storage persistent volumes.

I am following this guide Decommission Brokers in Kubernetes to decommission a broker node/pod on a 3 nodes freshly built Redpanda cluster.

kubectl -n astradot exec -ti redpanda-0 -c redpanda -- \
  rpk cluster info
CLUSTER
=======
redpanda.c2498d12-511b-441d-9ed2-9f5f34b8edc4

BROKERS
=======
ID    HOST                                             PORT  RACK
0*    redpanda-0.redpanda.astradot.svc.cluster.local.  9093  us-east-1b
1     redpanda-1.redpanda.astradot.svc.cluster.local.  9093  us-east-1c
2     redpanda-2.redpanda.astradot.svc.cluster.local.  9093  us-east-1a

TOPICS
======
NAME      PARTITIONS  REPLICAS
_schemas  1           3

First I've set broker id 2 for decommission -

kubectl -n astradot exec -ti redpanda-0 -c redpanda -- \
  rpk redpanda admin brokers decommission 2
Success, broker 2 has been decommissioned! 

But when I see the cluster health, the node 2 is still part of the "All nodes" field here -

kubectl -n astradot exec -ti redpanda-0 -c redpanda -- \
  rpk cluster health
CLUSTER HEALTH OVERVIEW
=======================
Healthy:               true
Controller ID:               0
All nodes:                   [1 0 2]
Nodes down:                  []
Leaderless partitions:       []
Under-replicated partitions: []

kubectl -n astradot exec -ti redpanda-0 -c redpanda -- \
  rpk redpanda admin brokers decommission-status 2
Node 2 is not decommissioning

Logs from the broker pods -

INFO  2023-06-27 21:52:34,785 [shard 0] raft - [group_id:1, {kafka/_schemas/0}] consensus.cc:1647 - Received vote request with larger term from node {id: {0}, revision: {8}}, received 1, current 0
INFO  2023-06-27 21:52:34,817 [shard 0] storage - segment.cc:673 - Creating new segment /var/lib/redpanda/data/kafka/_schemas/0_8/0-1-v1.log
INFO  2023-06-27 21:54:20,956 [shard 0] cluster - members_table.cc:121 - changing node 2 membership state to: draining
INFO  2023-06-27 22:02:31,039 [shard 0] kafka - group_manager.cc:207 - Group offset retention is now enabled (prev {nullopt}). Legacy enabled false retention_sec {604800000} original version 9.

Can I know why it is stuck in draining state and not proceeding with decommissioning? Is there anything which is stopping it? Let me know if you need more information from me, I'll appreciate your help on this.

Thanks

[emaxerrno]

hey @mdfaizsiddiqui welcome to the community. i shared this internally on our slack. we'll try to have the k8s team chime in

cc: @dswang @StasiaZam

[bharathv]

Success, broker 2 has been decommissioned!

This is a misleading message. It just means that the decommission command has been submitted to the controller. This was fixed recently.

Node 2 is not decommissioning

Can you share the output of ... rpk redpanda admin brokers list
Also, can you attach node-0 (controller leader, per cluster info output) logs? That coordinates the decommission, so the log may have more information.

[mdfaizsiddiqui]

Output for rpk redpanda admin brokers list -

kubectl -n astradot exec -ti redpanda-0 -c redpanda -- \
  rpk redpanda admin brokers list
NODE-ID  NUM-CORES  MEMBERSHIP-STATUS  IS-ALIVE  BROKER-VERSION
0        1          active             true      v23.1.13 - 6b1c6f55d2a46871fafc7d003383b59215f12351
1        1          active             true      v23.1.13 - 6b1c6f55d2a46871fafc7d003383b59215f12351
2        1          draining           true      v23.1.13 - 6b1c6f55d2a46871fafc7d003383b59215f12351

Logs -

Defaulted container "redpanda" out of: redpanda, config-watcher, tuning (init), redpanda-configurator (init)
Detecting the current cloud vendor and VM
Detected vendor 'aws' and VM type 'm6gd.medium'
no iotune data found for VM 'm6gd.medium', of vendor 'aws'
System check "NTP Synced" failed with non-fatal error "couldn't check NTP with timedatectl or ntpstat"
no such file or directory
System check "Dir '/var/lib/redpanda/data' scheduler tuned" failed with non-fatal error "open : no such file or directory"
no such file or directory
System check "Dir '/var/lib/redpanda/data' nomerges tuned" failed with non-fatal error "open : no such file or directory"
System check "Dir '/var/lib/redpanda/data' IRQs affinity static" failed with non-fatal error "no such file or directory"
System check "Dir '/var/lib/redpanda/data' IRQs affinity set" failed with non-fatal error "no such file or directory"
System check "Fstrim systemd service and timer active" failed with non-fatal error "dial unix /var/run/dbus/system_bus_socket: connect: no such file or directory"
System check "RFS Table entries" failed with non-fatal error "open /proc/sys/net/core/rps_sock_flow_entries: no such file or directory"
System check 'Ballast file present' failed. Required: true, Current false
System check 'Clock Source' failed. Required: tsc, Current arch_sys_counter
System check 'Dir '/var/lib/redpanda/data' IRQs affinity set' failed. Required: true, Current
System check 'Dir '/var/lib/redpanda/data' IRQs affinity static' failed. Required: true, Current
System check 'Dir '/var/lib/redpanda/data' nomerges tuned' failed. Required: true, Current
System check 'Dir '/var/lib/redpanda/data' scheduler tuned' failed. Required: true, Current
System check 'Fstrim systemd service and timer active' failed. Required: true, Current
System check 'I/O config file present' failed. Required: true, Current false
System check 'Max syn backlog size' failed. Required: >= 4096, Current 256
System check 'NTP Synced' failed. Required: true, Current
System check 'RFS Table entries' failed. Required: >= 32768, Current
System check 'Swap enabled' failed. Required: true, Current false
System check 'Swappiness' failed. Required: 1, Current 60
System check - PASSED
We'd love to hear about your experience with Redpanda:
https://redpanda.com/feedback
Starting redpanda...
Running:
/opt/redpanda/bin/redpanda redpanda --redpanda-cfg /etc/redpanda/redpanda.yaml --lock-memory=false --default-log-level=info --memory=1638M --reserve-memory=204M --smp=1 --overprovisioned


Welcome to the Redpanda community!

Documentation: https://docs.redpanda.com - Product documentation site
GitHub Discussion: https://github.com/redpanda-data/redpanda/discussions - Longer, more involved discussions
GitHub Issues: https://github.com/redpanda-data/redpanda/issues - Report and track issues with the codebase
Support: https://support.redpanda.com - Contact the support team privately
Product Feedback: https://redpanda.com/feedback - Let us know how we can improve your experience
Slack: https://redpanda.com/slack - Chat about all things Redpanda. Join the conversation!
Twitter: https://twitter.com/redpandadata - All the latest Redpanda news!


WARN  2023-06-29 15:07:29,328 [shard 0] seastar - Creation of perf_event based stall detector failed, falling back to posix timer: std::__1::system_error (error system:13, perf_event_open() failed: Permission denied)
INFO  2023-06-29 15:07:29,332 [shard 0] seastar - Created fair group io-queue-0, capacity rate 2147483:2147483, limit 12582912, rate 16777216 (factor 1), threshold 2000
INFO  2023-06-29 15:07:29,333 [shard 0] seastar - IO queue uses 0.75ms latency goal for device 0
INFO  2023-06-29 15:07:29,333 [shard 0] seastar - Created io group dev(0), length limit 4194304:4194304, rate 2147483647:2147483647
INFO  2023-06-29 15:07:29,333 [shard 0] seastar - Created io queue dev(0) capacities: 512:2000:2000 1024:3000:3000 2048:5000:5000 4096:9000:9000 8192:17000:17000 16384:33000:33000 32768:65000:65000 65536:129000:129000 131072:257000:257000
INFO  2023-06-29 15:07:29,333 [shard 0] main - application.cc:347 - Redpanda v23.1.13 - 6b1c6f55d2a46871fafc7d003383b59215f12351
INFO  2023-06-29 15:07:29,333 [shard 0] main - application.cc:355 - kernel=5.15.0-1037-aws, nodename=redpanda-0, machine=aarch64
INFO  2023-06-29 15:07:29,333 [shard 0] main - application.cc:283 - System resources: { cpus: 1, available memory: 1.600GiB, reserved memory: 204.000MiB}
INFO  2023-06-29 15:07:29,333 [shard 0] main - application.cc:291 - File handle limit: 1048576/1048576
INFO  2023-06-29 15:07:29,343 [shard 0] cluster - config_manager.cc:454 - Can't load config cache: std::__1::__fs::filesystem::filesystem_error (error system:2, filesystem error: open failed: No such file or directory ["/var/lib/redpanda/data/config_cache.yaml"])
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:658 - Cluster configuration properties:
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:659 - (use `rpk cluster config edit` to change)
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.abort_index_segment_size:50000	- Capacity (in number of txns) of an abort index segment
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.abort_timed_out_transactions_interval_ms:10000	- How often look for the inactive transactions and abort them
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.admin_api_require_auth:0	- Whether admin API clients must provide HTTP Basic authentication headers
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.aggregate_metrics:0	- Enable aggregations of metrics returned by the prometheus '/metrics' endpoint. Metric aggregation is performed by summing the values of samples by labels. Aggregations are performed where it makes sense by the shard and/or partition labels.
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.alter_topic_cfg_timeout_ms:5000	- Time to wait for entries replication in controller log when executing alter configuration requst
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.append_chunk_size:16384	- Size of direct write operations to disk in bytes
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.auto_create_topics_enabled:0	- Allow topic auto creation
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_access_key:{nullopt}	- AWS access key
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_api_endpoint:{nullopt}	- Optional API endpoint
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_api_endpoint_port:443	- TLS port override
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_azure_container:{nullopt}	- The name of the Azure container to use with Tiered Storage. Note that the container must belong to 'cloud_storage_azure_storage_account'
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_azure_shared_key:{nullopt}	- The shared key to be used for Azure Shared Key authentication with the configured Azure storage account (see 'cloud_storage_azure_storage_account)'. Note that Redpanda expects this string to be Base64 encoded.
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_azure_storage_account:{nullopt}	- The name of the Azure storage account to use with Tiered Storage
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_backend:unknown	- Optional cloud storage backend variant used to select API capabilities. If not supplied, will be inferred from other configuration parameters.
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_bucket:{nullopt}	- AWS bucket that should be used to store data
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_cache_check_interval:30000	- Timeout to check if cache eviction should be triggered
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_cache_size:21474836480	- Max size of archival cache
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_credentials_host:{nullopt}	- The hostname to connect to for retrieving role based credentials. Derived from cloud_storage_credentials_source if not set. Only required when using IAM role based access.
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_credentials_source:config_file	- The source of credentials to connect to cloud services
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_disable_tls:0	- Disable TLS for all S3 connections
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_enable_compacted_topic_reupload:1	- Enable re-uploading data for compacted topics
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_enable_remote_read:0	- Default remote read config value for new topics
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_enable_remote_write:0	- Default remote write value for new topics
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_enable_segment_merging:1	- Enables adjacent segment merging. The segments are reuploaded if there is an opportunity for that and if it will improve the tiered-storage performance
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_enabled:0	- Enable archival storage
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_graceful_transfer_timeout:{5000ms}	- Time limit on waiting for uploads to complete before a leadership transfer.  If this is null, leadership transfers will proceed without waiting.
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_housekeeping_interval_ms:300000	- Interval for cloud storage housekeeping tasks
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_idle_threshold_rps:1	- The cloud storage request rate threshold for idle state detection. If the average request rate for the configured period is lower than this threshold the cloud storage is considered being idle.
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_idle_timeout_ms:10000	- Timeout used to detect idle state of the cloud storage API. If the average cloud storage request rate is below this threshold for a configured amount of time the cloud storage is considered idle and the housekeeping jobs are started.
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_initial_backoff_ms:100	- Initial backoff time for exponetial backoff algorithm (ms)
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_manifest_upload_timeout_ms:10000	- Manifest upload timeout (ms)
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_max_connection_idle_time_ms:5000	- Max https connection idle time (ms)
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_max_connections:20	- Max number of simultaneous connections to S3 per shard (includes connections used for both uploads and downloads)
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_max_materialized_segments_per_shard:{nullopt}	- Maximum concurrent readers of remote data per CPU core.  If unset, value of `topic_partitions_per_shard` multiplied by 2 is used.
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_max_readers_per_shard:{nullopt}	- Maximum concurrent readers of remote data per CPU core.  If unset, value of `topic_partitions_per_shard` is used, i.e. one reader per partition if the shard is at its maximum partition capacity.
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_max_segments_pending_deletion_per_partition:5000	- The per-partition limit for the number of segments pending deletion from the cloud. Segments can be deleted due to retention or compaction. If this limit is breached and deletion fails, then segments will be orphaned in the cloud and will have to be removed manually
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_metadata_sync_timeout_ms:10000	- Timeout for SI metadata synchronization
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_readreplica_manifest_sync_timeout_ms:30000	- Timeout to check if new data is available for partition in S3 for read replica
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_reconciliation_interval_ms:	-
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_recovery_temporary_retention_bytes_default:1073741824	- Retention in bytes for topics created during automated recovery
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_region:{nullopt}	- AWS region that houses the bucket used for storage
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_roles_operation_timeout_ms:30000	- Timeout for IAM role related operations (ms)
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_secret_key:{nullopt}	- AWS secret key
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_segment_max_upload_interval_sec:{nullopt}	- Time that segment can be kept locally without uploading it to the remote storage (sec)
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_segment_size_min:{nullopt}	- Smallest acceptable segment size in the cloud storage. Default: cloud_storage_segment_size_target/2
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_segment_size_target:{nullopt}	- Desired segment size in the cloud storage. Default: segment.bytes
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_segment_upload_timeout_ms:30000	- Log segment upload timeout (ms)
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_trust_file:{nullopt}	- Path to certificate that should be used to validate server certificate during TLS handshake
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_upload_ctrl_d_coeff:0	- derivative coefficient for upload PID controller.
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_upload_ctrl_max_shares:1000	- maximum number of IO and CPU shares that archival upload can use
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_upload_ctrl_min_shares:100	- minimum number of IO and CPU shares that archival upload can use
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_upload_ctrl_p_coeff:-2	- proportional coefficient for upload PID controller
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_upload_ctrl_update_interval_ms:60000	-
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_upload_loop_initial_backoff_ms:100	- Initial backoff interval when there is nothing to upload for a partition (ms)
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cloud_storage_upload_loop_max_backoff_ms:10000	- Max backoff interval when there is nothing to upload for a partition (ms)
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.cluster_id:{nullopt}	- Cluster identifier
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.compacted_log_segment_size:67108864	- How large in bytes should each compacted log segment be (default 256MiB)
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.compaction_ctrl_backlog_size:{nullopt}	- target backlog size for compaction controller. if not set compaction target compaction backlog would be equal to
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.compaction_ctrl_d_coeff:0.2	- derivative coefficient for compaction PID controller.
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.compaction_ctrl_i_coeff:0	- integral coefficient for compaction PID controller.
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.compaction_ctrl_max_shares:1000	- maximum number of IO and CPU shares that compaction process can use
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.compaction_ctrl_min_shares:10	- minimum number of IO and CPU shares that compaction process can use
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.compaction_ctrl_p_coeff:-12.5	- proportional coefficient for compaction PID controller. This has to be negative since compaction backlog should decrease when number of compaction shares increases
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.compaction_ctrl_update_interval_ms:30000	-
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.controller_backend_housekeeping_interval_ms:1000	- Interval between iterations of controller backend housekeeping loop
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.controller_log_accummulation_rps_capacity_acls_and_users_operations:{nullopt}	- Maximum capacity of rate limit accumulationin controller acls and users operations limit
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.controller_log_accummulation_rps_capacity_configuration_operations:{nullopt}	- Maximum capacity of rate limit accumulationin controller configuration operations limit
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.controller_log_accummulation_rps_capacity_move_operations:{nullopt}	- Maximum capacity of rate limit accumulationin controller move operations limit
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.controller_log_accummulation_rps_capacity_node_management_operations:{nullopt}	- Maximum capacity of rate limit accumulationin controller node management operations limit
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.controller_log_accummulation_rps_capacity_topic_operations:{nullopt}	- Maximum capacity of rate limit accumulationin controller topic operations limit
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.coproc_max_batch_size:32768	- Maximum amount of bytes to read from one topic read
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.coproc_max_inflight_bytes:10485760	- Maximum amountt of inflight bytes when sending data to wasm engine
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.coproc_max_ingest_bytes:655360	- Maximum amount of data to hold from input logs in memory
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.coproc_offset_flush_interval_ms:300000	- Interval for which all coprocessor offsets are flushed to disk
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.create_topic_timeout_ms:2000	- Timeout (ms) to wait for new topic creation
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.default_num_windows:10	- Default number of quota tracking windows
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.default_topic_partitions:1	- Default number of partitions per topic
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.default_topic_replications:1	- Default replication factor for new topics
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.default_window_sec:1000	- Default quota tracking window size in milliseconds
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.delete_retention_ms:604800000	- delete segments older than this - default 1 week
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.disable_batch_cache:0	- Disable batch cache in log manager
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.disable_metrics:0	- Disable registering metrics exposed on the internal metrics endpoint (/metrics)
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.disable_public_metrics:0	- Disable registering metrics exposed on the public metrics endpoint (/public_metrics)
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.election_timeout_ms:1500	- Election timeout expressed in milliseconds
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.enable_admin_api:	-
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.enable_auto_rebalance_on_node_add:0	- Enable automatic partition rebalancing when new nodes are added
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.enable_controller_log_rate_limiting:0	- Enables limiting of controller log write rate
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.enable_coproc:0	- Enable coprocessing mode
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.enable_idempotence:1	- Enable idempotent producer
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.enable_leader_balancer:1	- Enable automatic leadership rebalancing
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.enable_metrics_reporter:1	- Enable cluster metrics reporter
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.enable_pid_file:1	- Enable pid file. You probably don't want to change this.
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.enable_rack_awareness:1	- Enables rack-aware replica assignment
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.enable_sasl:0	- Enable SASL authentication for Kafka connections, authorization is required. see also `kafka_enable_authorization`
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.enable_transactions:1	- Enable transactions
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.enable_usage:0	- Enables the usage tracking mechanism, storing windowed history of kafka/cloud_storage metrics over time
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.features_auto_enable:1	- Whether new feature flags may auto-activate after upgrades (true) or must wait for manual activation via the admin API (false)
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.fetch_max_bytes:57671680	- Maximum number of bytes returned in fetch request
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.fetch_reads_debounce_timeout:1	- Time to wait for next read in fetch request when requested min bytes wasn't reached
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.fetch_session_eviction_timeout_ms:60000	- Minimum time before which unused session will get evicted from sessions. Maximum time after which inactive session will be deleted is two time given configuration valuecache
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.full_raft_configuration_recovery_pattern:{}	- Recover raft configuration on start for NTPs matching pattern
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.group_initial_rebalance_delay:3000	- Extra delay (ms) added to rebalance phase to wait for new members
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.group_max_session_timeout_ms:300000	- The maximum allowed session timeout for registered consumers. Longer timeouts give consumers more time to process messages in between heartbeats at the cost of a longer time to detect failures.
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.group_min_session_timeout_ms:6000	- The minimum allowed session timeout for registered consumers. Shorter timeouts result in quicker failure detection at the cost of more frequent consumer heartbeating, which can overwhelm broker resources.
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.group_new_member_join_timeout:30000	- Timeout for new member joins
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.group_offset_retention_check_ms:600000	- How often the system should check for expired group offsets.
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.group_offset_retention_sec:{604800s}	- Consumer group offset retention seconds. Offset retention can be disabled by setting this value to null.
INFO  2023-06-29 15:07:29,344 [shard 0] main - application.cc:634 - redpanda.group_topic_partitions:16	- Number of partitions in the internal group membership topic
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.health_manager_tick_interval:180000	- How often the health manager runs
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.health_monitor_max_metadata_age:10000	- Max age of metadata cached in the health monitor of non controller node
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.health_monitor_tick_interval:10000	- How often health monitor refresh cluster state
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.id_allocator_batch_size:1000	- Id allocator allocates messages in batches (each batch is a one log record) and then serves requests from memory without touching the log until the batch is exhausted.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.id_allocator_log_capacity:100	- Capacity of the id_allocator log in number of messages. Once it reached id_allocator_stm should compact the log.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.id_allocator_replication:	-
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.internal_topic_replication_factor:3	- Target replication factor for internal topics
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.join_retry_timeout_ms:5000	- Time between cluster join retries in milliseconds
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_admin_topic_api_rate:{nullopt}	- Target quota rate (partition mutations per default_window_sec)
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_batch_max_bytes:1048576	- Maximum size of a batch processed by server. If batch is compressed the limit applies to compressed batch size
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_client_group_byte_rate_quota:{}	- Per-group target produce quota byte rate (bytes per second). Client is considered part of the group if client_id contains clients_prefix
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_client_group_fetch_byte_rate_quota:{}	- Per-group target fetch quota byte rate (bytes per second). Client is considered part of the group if client_id contains clients_prefix
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_connection_rate_limit:{1000}	- Maximum connections per second for one core
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_connection_rate_limit_overrides:{}	- Overrides for specific ips for maximum connections per second for one core
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_connections_max:{nullopt}	- Maximum number of Kafka client connections per broker
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_connections_max_overrides:{}	- Per-IP overrides of kafka connection count limit, list of <ip>:<count> strings
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_connections_max_per_ip:{nullopt}	- Maximum number of Kafka client connections from each IP address, per broker
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_enable_authorization:{false}	- Enable authorization for Kafka connections. Values:- `nil`: Ignored. Authorization is enabled with `enable_sasl: true`; `true`: authorization is required; `false`: authorization is disabled. See also: `enable_sasl` and `kafka_api[].authentication_method`
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_enable_partition_reassignment:1	- Enable the Kafka partition reassignment API
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_group_recovery_timeout_ms:30000	- Kafka group recovery timeout expressed in milliseconds
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_max_bytes_per_fetch:67108864	- Limit fetch responses to this many bytes, even if total of partition bytes limits is higher
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_mtls_principal_mapping_rules:{nullopt}	- Principal Mapping Rules for mTLS Authentication on the Kafka API
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_nodelete_topics:{__audit, __consumer_offsets, _schemas}	- Prevents the topics in the list from being deleted via the kafka api
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_noproduce_topics:{__audit}	- Prevents the topics in the list from having message produced to them via the kafka api
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_qdc_depth_alpha:0.8	- Smoothing factor for kafka queue depth control depth tracking.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_qdc_depth_update_ms:7000	- Update frequency for kafka queue depth control.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_qdc_enable:0	- Enable kafka queue depth control.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_qdc_idle_depth:10	- Queue depth when idleness is detected in kafka queue depth control.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_qdc_latency_alpha:0.002	- Smoothing parameter for kafka queue depth control latency tracking.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_qdc_max_depth:100	- Maximum queue depth used in kafka queue depth control.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_qdc_max_latency_ms:80	- Max latency threshold for kafka queue depth control depth tracking.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_qdc_min_depth:1	- Minimum queue depth used in kafka queue depth control.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_qdc_window_count:12	- Number of windows used in kafka queue depth control latency tracking.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_qdc_window_size_ms:1500	- Window size for kafka queue depth control latency tracking.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_quota_balancer_min_shard_throughput_bps:256	- The lowest value of the throughput quota a shard can get in the process of quota balancing, in bytes/s. 0 means there is no minimum.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_quota_balancer_min_shard_throughput_ratio:0.01	- The lowest value of the throughput quota a shard can get in the process of quota balancing, expressed as a ratio of default shard quota. 0 means there is no minimum, 1 means no quota can be taken away by the balancer.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_quota_balancer_node_period_ms:750	- Intra-node throughput quota balancer invocation period, in milliseconds. Value of 0 disables the balancer and makes all the throughput quotas immutable.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_quota_balancer_window_ms:5000	- Time window used to average current throughput measurement for quota balancer, in milliseconds
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_request_max_bytes:104857600	- Maximum size of a single request processed via Kafka API
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_rpc_server_stream_recv_buf:{nullopt}	- Userspace receive buffer max size in bytes
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_rpc_server_tcp_recv_buf:{nullopt}	- Kafka server TCP receive buffer size in bytes.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_rpc_server_tcp_send_buf:{nullopt}	- Kafka server TCP transmit buffer size in bytes.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_throughput_control:{}	- List of throughput control groups that define exclusions from node-wide throughput limits. Each group consists of: ("name" (optional) - any unique group name, "client_id" - regex to match client_id). A connection is assigned the first matching group, then the connection is excluded from throughput control.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_throughput_controlled_api_keys:{produce, fetch}	- List of Kafka API keys that are subject to cluster-wide and node-wide throughput limit control
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_throughput_limit_node_in_bps:{nullopt}	- Node wide throughput ingress limit - maximum kafka traffic throughput allowed on the ingress side of each node, in bytes/s. Default is no limit.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_throughput_limit_node_out_bps:{nullopt}	- Node wide throughput egress limit - maximum kafka traffic throughput allowed on the egress side of each node, in bytes/s. Default is no limit.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kvstore_flush_interval:10	- Key-value store flush interval (ms)
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kvstore_max_segment_size:16777216	- Key-value maximum segment size (bytes)
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.leader_balancer_idle_timeout:120000	- Leadership rebalancing idle timeout
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.leader_balancer_mode:greedy_balanced_shards	- Leader balancer mode
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.leader_balancer_mute_timeout:300000	- Leadership rebalancing mute timeout
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.leader_balancer_transfer_limit_per_shard:512	- Per shard limit for in progress leadership transfers
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.legacy_group_offset_retention_enabled:0	- Group offset retention is enabled by default in versions of Redpanda >= 23.1. To enable offset retention after upgrading from an older version set this option to true.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.log_cleanup_policy:delete	- Default topic cleanup policy
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.log_compaction_interval_ms:10000	- How often do we trigger background compaction
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.log_compression_type:producer	- Default topic compression type
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.log_message_timestamp_type:CreateTime	- Default topic messages timestamp type
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.log_segment_ms:{1209600000ms}	- Default log segment lifetime in ms for topics which do not set segment.ms
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.log_segment_ms_max:31536000000	- Upper bound on topic segment.ms: higher values will be clamped to this value
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.log_segment_ms_min:60000	- Lower bound on topic segment.ms: lower values will be clamped to this value
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.log_segment_size:134217728	- Default log segment size in bytes for topics which do not set segment.bytes
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.log_segment_size_jitter_percent:5	- Random variation to the segment size limit used for each partition
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.log_segment_size_max:{268435456}	- Upper bound on topic segment.bytes: higher values will be clamped to this limit
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.log_segment_size_min:{16777216}	- Lower bound on topic segment.bytes: lower values will be clamped to this limit
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.max_compacted_log_segment_size:536870912	- Max compacted segment size after consolidation
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.max_concurrent_producer_ids:18446744073709551615	- Max cache size for pids which rm_stm stores inside internal state. In overflow rm_stm will delete old pids and clear their status
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.max_kafka_throttle_delay_ms:30000	- Fail-safe maximum throttle delay on kafka requests
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.max_version:	-
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.members_backend_retry_ms:5000	- Time between members backend reconciliation loop retries
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.memory_abort_on_alloc_failure:1	- If true, the redpanda process will terminate immediately when an allocation cannot be satisfied due to memory exhasution. If false, an exception is thrown instead.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.metadata_dissemination_interval_ms:3000	- Interaval for metadata dissemination batching
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.metadata_dissemination_retries:30	- Number of attempts of looking up a topic's meta data like shard before failing a request
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.metadata_dissemination_retry_delay_ms:320	- Delay before retry a topic lookup in a shard or other meta tables
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.metadata_status_wait_timeout_ms:2000	- Maximum time to wait in metadata request for cluster health to be refreshed
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.metrics_reporter_report_interval:86400000	- cluster metrics reporter report interval
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.metrics_reporter_tick_interval:60000	- Cluster metrics reporter tick interval
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.metrics_reporter_url:https://m.rp.vectorized.io/v2	- cluster metrics reporter url
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.min_version:	-
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.node_isolation_heartbeat_timeout:3000	- How long after the last heartbeat request a node will wait before considering itself to be isolated
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.node_management_operation_timeout_ms:5000	- Timeout for executing node management operations
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.node_status_interval:100	- Time interval between two node status messages. Node status messages establish liveness status outside of the Raft protocol.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.partition_autobalancing_concurrent_moves:50	- Number of partitions that can be reassigned at once
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.partition_autobalancing_max_disk_usage_percent:80	- Disk usage threshold that triggers moving partitions from the node
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.partition_autobalancing_mode:node_add	- Partition autobalancing mode
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.partition_autobalancing_movement_batch_size_bytes:5368709120	- Total size of partitions that autobalancer is going to move in one batch
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.partition_autobalancing_node_availability_timeout_sec:900000	- Node unavailability timeout that triggers moving partitions from the node
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.partition_autobalancing_tick_interval_ms:30000	- Partition autobalancer tick interval
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.quota_manager_gc_sec:30000	- Quota manager GC frequency in milliseconds
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.raft_heartbeat_disconnect_failures:3	- After how many failed heartbeats to forcibly close an unresponsive TCP connection.  Set to 0 to disable force disconnection.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.raft_heartbeat_interval_ms:150	- Milliseconds for raft leader heartbeats
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.raft_heartbeat_timeout_ms:3000	- raft heartbeat RPC timeout
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.raft_io_timeout_ms:10000	- Raft I/O timeout
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.raft_learner_recovery_rate:104857600	- Raft learner recovery rate limit in bytes per sec
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.raft_max_concurrent_append_requests_per_follower:16	- Maximum number of concurrent append entries requests sent by leader to one follower
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.raft_max_recovery_memory:{nullopt}	- Max memory that can be used for reads in raft recovery process by default 15% of total memory
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.raft_recovery_default_read_size:524288	- default size of read issued during raft follower recovery
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.raft_recovery_throttle_disable_dynamic_mode:0	- Disables dynamic rate allocation in recovery throttle (advanced).
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.raft_replicate_batch_window_size:1048576	- Max size of requests cached for replication
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.raft_smp_max_non_local_requests:{nullopt}	- Maximum number of x-core requests pending in Raft seastar::smp group. (for more details look at `seastar::smp_service_group` documentation)
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.raft_timeout_now_timeout_ms:1000	- Timeout for a timeout now request
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.raft_transfer_leader_recovery_timeout_ms:10000	- Timeout waiting for follower recovery when transferring leadership
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.readers_cache_eviction_timeout_ms:30000	- Duration after which inactive readers will be evicted from cache
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.reclaim_batch_cache_min_free:67108864	- Free memory limit that will be kept by batch cache background reclaimer
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.reclaim_growth_window:3000	- Length of time in which reclaim sizes grow
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.reclaim_max_size:4194304	- Maximum batch cache reclaim size
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.reclaim_min_size:131072	- Minimum batch cache reclaim size
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.reclaim_stable_window:10000	- Length of time above which growth is reset
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.recovery_append_timeout_ms:5000	- Timeout for append entries requests issued while updating stale follower
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.release_cache_on_segment_roll:0	- Free cache when segments roll
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.replicate_append_timeout_ms:3000	- Timeout for append entries requests issued while replicating entries
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.retention_bytes:{nullopt}	- Default max bytes per partition on disk before triggering a compaction
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.retention_local_target_bytes_default:{nullopt}	- Local retention size target for partitions of topics with cloud storage write enabled
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.retention_local_target_ms_default:86400000	- Local retention time target for partitions of topics with cloud storage write enabled
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.rm_sync_timeout_ms:10000	- Time to wait state catch up before rejecting a request
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.rm_violation_recovery_policy:	-
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.rpc_server_listen_backlog:{nullopt}	- TCP connection queue length for Kafka server and internal RPC server
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.rpc_server_tcp_recv_buf:{nullopt}	- Internal RPC TCP receive buffer size in bytes.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.rpc_server_tcp_send_buf:{nullopt}	- Internal RPC TCP transmit buffer size in bytes.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.rps_limit_acls_and_users_operations:1000	- Rate limit for controller acls and users operations
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.rps_limit_configuration_operations:1000	- Rate limit for controller configuration operations
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.rps_limit_move_operations:1000	- Rate limit for controller move operations
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.rps_limit_node_management_operations:1000	- Rate limit for controller node management operations
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.rps_limit_topic_operations:1000	- Rate limit for controller topic operations
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.sasl_kerberos_config:/etc/krb5.conf	- The location of the Kerberos krb5.conf file for Redpanda
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.sasl_kerberos_keytab:/var/lib/redpanda/redpanda.keytab	- The location of the Kerberos keytab file for Redpanda
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.sasl_kerberos_principal:redpanda	- The primary of the Kerberos Service Principal Name (SPN) for Redpanda
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.sasl_kerberos_principal_mapping:{DEFAULT}	- Rules for mapping Kerberos Principal Names to Redpanda User Principals
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.sasl_mechanisms:{SCRAM}	- A list of supported SASL mechanisms. `SCRAM` and `GSSAPI` are allowed.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.seed_server_meta_topic_partitions:	-
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.segment_appender_flush_timeout_ms:1000	- Maximum delay until buffered data is written
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.segment_fallocation_step:33554432	- Size for segments fallocation
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.seq_table_min_size:1000	- Minimum size of the seq table non affected by compaction
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.storage_compaction_index_memory:134217728	- Maximum number of bytes that may be used on each shard by compactionindex writers
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.storage_ignore_timestamps_in_future_sec:{nullopt}	- If set, timestamps more than this many seconds in the future relative tothe server's clock will be ignored for data retention purposes, and retention will act based on another timestamp in the same segment, or the mtime of the segment file if no valid timestamp is available
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.storage_max_concurrent_replay:1024	- Maximum number of partitions' logs that will be replayed concurrently at startup, or flushed concurrently on shutdown.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.storage_min_free_bytes:2684354560	- Threshold of minimum bytes free space before rejecting producers.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.storage_read_buffer_size:131072	- Size of each read buffer (one per in-flight read, per log segment)
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.storage_read_readahead_count:10	- How many additional reads to issue ahead of current read location
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.storage_space_alert_free_threshold_bytes:0	- Threshold of minimim bytes free space before setting storage space alert
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.storage_space_alert_free_threshold_percent:5	- Threshold of minimim percent free space before setting storage space alert
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.storage_strict_data_init:0	- Requires that an empty file named `.redpanda_data_dir` be present in the data directory. Redpanda will refuse to start if it is not found.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.storage_target_replay_bytes:10737418240	- Target bytes to replay from disk on startup after clean shutdown: controls frequency of snapshots and checkpoints
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.superusers:{}	- List of superuser usernames
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.target_fetch_quota_byte_rate:{nullopt}	- Target fetch size quota byte rate (bytes per second) - disabled default
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.target_quota_byte_rate:2147483648	- Target request size quota byte rate (bytes per second) - 2GB default
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.tm_sync_timeout_ms:10000	- Time to wait state catch up before rejecting a request
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.tm_violation_recovery_policy:	-
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.topic_fds_per_partition:{5}	- Required file handles per partition when creating topics
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.topic_memory_per_partition:{1048576}	- Required memory per partition when creating topics
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.topic_partitions_per_shard:1000	- Maximum number of partitions which may be allocated to one shard (CPU core)
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.topic_partitions_reserve_shard0:2	- Reserved partition slots on shard (CPU core) 0 on each node.  If this is >= topic_partitions_per_core, no data partitions will be scheduled on shard 0
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.transaction_coordinator_cleanup_policy:delete	- Cleanup policy for a transaction coordinator topic
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.transaction_coordinator_delete_retention_ms:604800000	- delete segments older than this - default 1 week
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.transaction_coordinator_log_segment_size:1073741824	- How large in bytes should each log segment be (default 1G)
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.transaction_coordinator_replication:	-
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.transactional_id_expiration_ms:604800000	- Producer ids are expired once this time has elapsed after the last write with the given producer id.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.tx_log_stats_interval_s:10000	- How often to log per partition tx stats, works only with debug logging enabled.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.tx_timeout_delay_ms:1000	- Delay before scheduling next check for timed out transactions
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.usage_disk_persistance_interval_sec:300000	- The interval in which all usage stats are written to disk
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.usage_num_windows:24	- The number of windows to persist in memory and disk
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.usage_window_width_interval_sec:3600000	- The width of a usage window, tracking cloud and kafka ingress/egress traffic each interval
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.use_fetch_scheduler_group:1	- Use a separate scheduler group for fetch processing
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.use_scheduling_groups:	-
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.wait_for_leader_timeout_ms:5000	- Timeout (ms) to wait for leadership in metadata cache
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.zstd_decompress_workspace_bytes:8388608	- Size of the zstd decompression workspace
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:662 - Node configuration properties:
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:663 - (use `rpk redpanda config set <cfg> <value>` to change)
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.admin:{{internal:{host: 0.0.0.0, port: 9644}}, {default:{host: 0.0.0.0, port: 9645}}}	- Address and port of admin server
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.admin_api_doc_dir:/usr/share/redpanda/admin-api-doc	- Admin API doc directory
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.admin_api_tls:{}	- TLS configuration for admin HTTP server
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.advertised_kafka_api:{{internal:{host: redpanda-0.redpanda.astradot.svc.cluster.local., port: 9093}}, {default:{host: redpanda-0, port: 31092}}}	- Address of Kafka API published to the clients
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.advertised_rpc_api:{{host: redpanda-0.redpanda.astradot.svc.cluster.local., port: 33145}}	- Address of RPC endpoint published to other cluster members
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.cloud_storage_cache_directory:{nullopt}	- Directory for archival cache. Should be present when `cloud_storage_enabled` is present
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.coproc_supervisor_server:{host: 127.0.0.1, port: 43189}	- IpAddress and port for supervisor service
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.crash_loop_limit:{nullopt}	- Maximum consecutive crashes (unclean shutdowns) allowed after which operator intervention is needed to startup the broker.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.dashboard_dir:	-
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.data_directory:{data_directory="/var/lib/redpanda/data"}	- Place where redpanda will keep the data
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.developer_mode:0	- Skips most of the checks performed at startup, not recomended for production use
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.empty_seed_starts_cluster:0	- If true, an empty seed_servers list will denote that this node should form a cluster. At most one node in the cluster should be configured configured with an empty seed_servers list. If no such configured node exists, or if configured to false, all nodes denoted by the seed_servers list must be identical among those nodes' configurations, and those nodes will form the initial cluster.
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.enable_central_config:	-
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_api:{{internal:{host: 0.0.0.0, port: 9093}:{nullopt}}, {default:{host: 0.0.0.0, port: 9094}:{nullopt}}}	- Address and port of an interface to listen for Kafka API requests
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.kafka_api_tls:{}	- TLS configuration for Kafka API endpoint
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.node_id:{nullopt}	- Unique id identifying a node in the cluster. If missing, a unique id will be assigned for this node when it joins the cluster
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.rack:{{us-east-1b}}	- Rack identifier
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.rpc_server:{host: 0.0.0.0, port: 33145}	- IpAddress and port for RPC server
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.rpc_server_tls:{ enabled: 0 key/cert files: {nullopt} ca file: {nullopt} client_auth_required: 0 }	- TLS configuration for RPC server
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - redpanda.seed_servers:{addr: {host: redpanda-0.redpanda.astradot.svc.cluster.local., port: 33145}, addr: {host: redpanda-1.redpanda.astradot.svc.cluster.local., port: 33145}, addr: {host: redpanda-2.redpanda.astradot.svc.cluster.local., port: 33145}}	- List of the seed servers used to join current cluster. If the seed_server list is empty the node will be a cluster root and it will form a new cluster
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy.advertised_pandaproxy_api:{{internal:{host: redpanda-0.redpanda.astradot.svc.cluster.local., port: 8082}}, {default:{host: redpanda-0, port: 30082}}}	- Rest API address and port to publish to client
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy.api_doc_dir:/usr/share/redpanda/proxy-api-doc	- API doc directory
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy.client_cache_max_size:10	- The maximum number of kafka clients in the LRU cache
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy.client_keep_alive:300000	- Time in milliseconds that an idle connection may remain open
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy.consumer_instance_timeout_ms:300000	- How long to wait for an idle consumer before removing it
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy.pandaproxy_api:{{internal:{host: 0.0.0.0, port: 8082}:<nullopt>}, {default:{host: 0.0.0.0, port: 8083}:<nullopt>}}	- Rest API listen address and port
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy.pandaproxy_api_tls:{}	- TLS configuration for Pandaproxy api
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy_client.broker_tls:{ enabled: 0 key/cert files: {nullopt} ca file: {nullopt} client_auth_required: 0 }	 TLS configuration for the brokers
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy_client.brokers:{{host: 0.0.0.0, port: 9093}}	- List of address and port of the brokers
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy_client.client_identifier:{pandaproxy_client}	- Identifier to use within the kafka request header
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy_client.consumer_heartbeat_interval_ms:500	- Interval (in milliseconds) for consumer heartbeats
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy_client.consumer_rebalance_timeout_ms:2000	- Timeout (in milliseconds) for consumer rebalance
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy_client.consumer_request_max_bytes:1048576	- Max bytes to fetch per request
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy_client.consumer_request_timeout_ms:100	- Interval (in milliseconds) for consumer request timeout
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy_client.consumer_session_timeout_ms:300000	- Timeout (in milliseconds) for consumer session
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy_client.produce_batch_delay_ms:100	- Delay (in milliseconds) to wait before sending batch
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy_client.produce_batch_record_count:1000	- Number of records to batch before sending to broker
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy_client.produce_batch_size_bytes:1048576	- Number of bytes to batch before sending to broker
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy_client.retries:5	- Number of times to retry a request to a broker
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy_client.retry_base_backoff_ms:100	- Delay (in milliseconds) for initial retry backoff
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy_client.sasl_mechanism:	- The SASL mechanism to use when connecting
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy_client.scram_password:	- Password to use for SCRAM authentication mechanisms
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - pandaproxy_client.scram_username:	- Username to use for SCRAM authentication mechanisms
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - schema_registry.api_doc_dir:/usr/share/redpanda/proxy-api-doc	- API doc directory
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - schema_registry.schema_registry_api:{{internal:{host: 0.0.0.0, port: 8081}:<nullopt>}, {default:{host: 0.0.0.0, port: 8084}:<nullopt>}}	- Schema Registry API listen address and port
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - schema_registry.schema_registry_api_tls:{}	- TLS configuration for Schema Registry API
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - schema_registry.schema_registry_replication_factor:{nullopt}	- Replication factor for internal _schemas topic.  If unset, defaults to `default_topic_replication`
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - schema_registry_client.broker_tls:{ enabled: 0 key/cert files: {nullopt} ca file: {nullopt} client_auth_required: 0 }	- TLS configuration for the brokers
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - schema_registry_client.brokers:{{host: 0.0.0.0, port: 9093}}	- List of address and port of the brokers
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - schema_registry_client.client_identifier:{schema_registry_client}	- Identifier to use within the kafka request header
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - schema_registry_client.consumer_heartbeat_interval_ms:500	- Interval (in milliseconds) for consumer heartbeats
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - schema_registry_client.consumer_rebalance_timeout_ms:2000	- Timeout (in milliseconds) for consumer rebalance
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - schema_registry_client.consumer_request_max_bytes:1048576	- Max bytes to fetch per request
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - schema_registry_client.consumer_request_timeout_ms:100	- Interval (in milliseconds) for consumer request timeout
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - schema_registry_client.consumer_session_timeout_ms:10000	- Timeout (in milliseconds) for consumer session
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - schema_registry_client.produce_batch_delay_ms:0	- Delay (in milliseconds) to wait before sending batch
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - schema_registry_client.produce_batch_record_count:0	- Number of records to batch before sending to broker
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - schema_registry_client.produce_batch_size_bytes:0	- Number of bytes to batch before sending to broker
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - schema_registry_client.retries:5	- Number of times to retry a request to a broker
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - schema_registry_client.retry_base_backoff_ms:100	- Delay (in milliseconds) for initial retry backoff
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - schema_registry_client.sasl_mechanism:	- The SASL mechanism to use when connecting
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - schema_registry_client.scram_password:	- Password to use for SCRAM authentication mechanisms
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:634 - schema_registry_client.scram_username:	- Username to use for SCRAM authentication mechanisms
INFO  2023-06-29 15:07:29,345 [shard 0] main - application.cc:414 - Setting abort_on_allocation_failure (abort on OOM): true
INFO  2023-06-29 15:07:29,350 [shard 0] syschecks - Writing pid file "/var/lib/redpanda/data/pid.lock"
INFO  2023-06-29 15:07:29,356 [shard 0] storage - directories.h:33 - Checking `/var/lib/redpanda/data` for supported filesystems
Reactor stalled for 42 ms on shard 0. Backtrace: 0x54ec8c3 0x54ec5fb 0x78b 0x5469ce3 0x5460517 0x1fbd18b 0x55061fb 0x5509063 0x5507147 0x543d997 0x543c193 0x1e1854b 0x57a543b 0x2b1c7 0x2b29f 0x1e12faf
INFO  2023-06-29 15:07:29,609 [shard 0] features - feature_table.cc:270 - Activating features from bootstrap version 7
INFO  2023-06-29 15:07:29,609 [shard 0] main - application.cc:1762 - Generated new UUID for node: {77fbc6e8-b01a-4425-a4a8-660af7ae5d2d}
INFO  2023-06-29 15:07:29,619 [shard 0] storage - segment.cc:673 - Creating new segment /var/lib/redpanda/data/redpanda/kvstore/0_0/0-0-v1.log
INFO  2023-06-29 15:07:29,620 [shard 0] main - application.cc:1792 - Started RPC server listening at {host: 0.0.0.0, port: 33145}
INFO  2023-06-29 15:07:29,621 [shard 0] cluster - cluster_discovery.cc:200 - Requesting cluster bootstrap info from {host: redpanda-2.redpanda.astradot.svc.cluster.local., port: 33145}
INFO  2023-06-29 15:07:29,621 [shard 0] cluster - cluster_discovery.cc:200 - Requesting cluster bootstrap info from {host: redpanda-1.redpanda.astradot.svc.cluster.local., port: 33145}
WARN  2023-06-29 15:07:29,649 [shard 0] cluster - cluster_discovery.cc:247 - Error requesting cluster bootstrap info from {host: redpanda-1.redpanda.astradot.svc.cluster.local., port: 33145}, retrying. std::__1::system_error (error system:111, Connection refused)
INFO  2023-06-29 15:07:29,650 [shard 0] cluster - cluster_discovery.cc:233 - Obtained cluster bootstrap info from {host: redpanda-2.redpanda.astradot.svc.cluster.local., port: 33145}
WARN  2023-06-29 15:07:30,713 [shard 0] cluster - cluster_discovery.cc:247 - Error requesting cluster bootstrap info from {host: redpanda-1.redpanda.astradot.svc.cluster.local., port: 33145}, retrying. std::__1::system_error (error system:111, Connection refused)
INFO  2023-06-29 15:07:32,164 [shard 0] cluster - cluster_discovery.cc:233 - Obtained cluster bootstrap info from {host: redpanda-1.redpanda.astradot.svc.cluster.local., port: 33145}
INFO  2023-06-29 15:07:32,164 [shard 0] cluster - Using index based node ID {0}
INFO  2023-06-29 15:07:32,164 [shard 0] main - application.cc:1819 - Starting Redpanda with node_id 0, cluster UUID {nullopt}
INFO  2023-06-29 15:07:32,164 [shard 0] raft - coordinated_recovery_throttle.cc:113 - Starting recovery throttle, rate: 104857600
INFO  2023-06-29 15:07:32,164 [shard 0] main - application.cc:1139 - Partition manager started
INFO  2023-06-29 15:07:32,176 [shard 0] cluster - raft0_utils.h:30 - Current node is a cluster founder
INFO  2023-06-29 15:07:32,200 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:1203 - Starting
INFO  2023-06-29 15:07:32,200 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:1237 - Starting with voted_for {id: {-2147483648}, revision: {-9223372036854775808}} term 0 initial_state true
INFO  2023-06-29 15:07:32,221 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:1283 - Read bootstrap state: data_seen 0 config_seen 0 eol false commit 0 term 0 prev_idx 0 prev_term 0 config_tracker -9223372036854775808 commit_base_tracker -9223372036854775808 configurations {}
INFO  2023-06-29 15:07:32,221 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:1284 - Current log offsets: {start_offset:-9223372036854775808, committed_offset:-9223372036854775808, committed_offset_term:-9223372036854775808, dirty_offset:-9223372036854775808, dirty_offset_term:-9223372036854775808, last_term_start_offset:-9223372036854775808}
INFO  2023-06-29 15:07:32,221 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:1308 - Truncating configurations at -9223372036854775808
INFO  2023-06-29 15:07:32,241 [shard 0] raft - [group_id:0, {redpanda/controller/0}] consensus.cc:1408 - started raft, log offsets: {start_offset:-9223372036854775808, committed_offset:-9223372036854775808, committed_offset_term:-9223372036854775808, dirty_offset:-9223372036854775808, dirty_offset_term:-9223372036854775808, last_term_start_offset:-9223372036854775808}, term: 0, configuration: {current: {voters: {{id: {0}, revision: {0}}, {id: {1}, revision: {0}}, {id: {2}, revision: {0}}}, learners: {}}, old:{nullopt}, revision: 0, update: {nullopt}, version: 4, brokers: {{id: 0, kafka_advertised_listeners: {{internal:{host: redpanda-0.redpanda.astradot.svc.cluster.local., port: 9093}}, {default:{host: redpanda-0, port: 31092}}}, rpc_address: {host: redpanda-0.redpanda.astradot.svc.cluster.local., port: 33145}, rack: {{us-east-1b}}, properties: {cores 1, mem_available 1, disk_available 54}}, {id: 1, kafka_advertised_listeners: {{internal:{host: redpanda-1.redpanda.astradot.svc.cluster.local., port: 9093}}, {default:{host: redpanda-1, port: 31092}}}, rpc_address: {host: redpanda-1.redpanda.astradot.svc.cluster.local., port: 33145}, rack: {{us-east-1a}}, properties: {cores 1, mem_available 1, disk_available 54}}, {id: 2, kafka_advertised_listeners: {{internal:{host: redpanda-2.redpanda.astradot.svc.cluster.local., port: 9093}}, {default:{host: redpanda-2, port: 31092}}}, rpc_address: {host: redpanda-2.redpanda.astradot.svc.cluster.local., port: 33145}, rack: {{us-east-1c}}, properties: {cores 1, mem_available 1, disk_available 54}}}}
INFO  2023-06-29 15:07:32,242 [shard 0] cluster - drain_manager.cc:21 - Drain manager starting
INFO  2023-06-29 15:07:32,242 [shard 0] cluster - members_manager.cc:88 - starting cluster::members_manager...
INFO  2023-06-29 15:07:32,242 [shard 0] cluster - controller.cc:354 - Controller log replay starting (to offset -9223372036854775808)
INFO  2023-06-29 15:07:32,242 [shard 0] cluster - controller.cc:366 - Controller log replay complete.
INFO  2023-06-29 15:07:33,793 [shard 0] raft - [group_id:0, {redpanda/controller/0}] vote_stm.cc:52 - Sending vote request to {id: {2}, revision: {0}} with timeout 1500
INFO  2023-06-29 15:07:33,793 [shard 0] raft - [group_id:0, {redpanda/controller/0}] vote_stm.cc:52 - Sending vote request to {id: {1}, revision: {0}} with timeout 1500
INFO  2023-06-29 15:07:33,803 [shard 0] raft - [group_id:0, {redpanda/controller/0}] vote_stm.cc:77 - vote reply from {id: {2}, revision: {0}} - {term:{1}, target_node_id{id: {0}, revision: {0}}, vote_granted: 1, log_ok:1}
INFO  2023-06-29 15:07:33,804 [shard 0] raft - [group_id:0, {redpanda/controller/0}] vote_stm.cc:265 - becoming the leader term:1
INFO  2023-06-29 15:07:33,804 [shard 0] storage - segment.cc:673 - Creating new segment /var/lib/redpanda/data/redpanda/controller/0_0/0-1-v1.log
INFO  2023-06-29 15:07:33,804 [shard 0] raft - [group_id:0, {redpanda/controller/0}] vote_stm.cc:77 - vote reply from {id: {1}, revision: {0}} - {term:{1}, target_node_id{id: {0}, revision: {0}}, vote_granted: 1, log_ok:1}
INFO  2023-06-29 15:07:33,847 [shard 0] cluster - node_status_backend.cc:61 - Node 1 has been discovered via members table
INFO  2023-06-29 15:07:33,847 [shard 0] cluster - node_status_backend.cc:61 - Node 2 has been discovered via members table
INFO  2023-06-29 15:07:33,847 [shard 0] raft - [group_id:0, {redpanda/controller/0}] vote_stm.cc:280 - became the leader term: 1
INFO  2023-06-29 15:07:33,945 [shard 0] cluster - controller.cc:591 - Creating cluster UUID {b9c1dc14-205a-458d-8571-30f9ec7fd4f6}
INFO  2023-06-29 15:07:33,946 [shard 0] cluster - bootstrap_backend.cc:88 - Applying update to bootstrap_manager
INFO  2023-06-29 15:07:33,946 [shard 0] features - feature_table.cc:270 - Activating features from bootstrap version 9
INFO  2023-06-29 15:07:33,946 [shard 0] cluster - feature_backend.cc:108 - Saving feature_table_snapshot at version 9...
INFO  2023-06-29 15:07:33,967 [shard 0] cluster - controller.cc:603 - Cluster UUID created {b9c1dc14-205a-458d-8571-30f9ec7fd4f6}
INFO  2023-06-29 15:07:33,968 [shard 0] cluster - feature_manager.cc:80 - Starting...
INFO  2023-06-29 15:07:33,968 [shard 0] cluster - leader_balancer.cc:98 - Leader balancer: controller leadership detected. Starting rebalancer in 30 seconds
INFO  2023-06-29 15:07:33,969 [shard 0] cluster - config_manager.cc:161 - Importing property compacted_log_segment_size:67108864
INFO  2023-06-29 15:07:33,969 [shard 0] cluster - config_manager.cc:161 - Importing property kafka_enable_authorization:{false}
INFO  2023-06-29 15:07:33,969 [shard 0] cluster - config_manager.cc:161 - Importing property log_segment_size_max:{268435456}
INFO  2023-06-29 15:07:33,969 [shard 0] cluster - config_manager.cc:161 - Importing property kafka_connection_rate_limit:{1000}
INFO  2023-06-29 15:07:33,969 [shard 0] cluster - config_manager.cc:161 - Importing property enable_rack_awareness:1
INFO  2023-06-29 15:07:33,969 [shard 0] cluster - config_manager.cc:161 - Importing property max_compacted_log_segment_size:536870912
INFO  2023-06-29 15:07:33,969 [shard 0] cluster - config_manager.cc:161 - Importing property log_segment_size_min:{16777216}
INFO  2023-06-29 15:07:33,969 [shard 0] cluster - config_manager.cc:161 - Importing property storage_min_free_bytes:2684354560
INFO  2023-06-29 15:07:33,969 [shard 0] cluster - config_manager.cc:161 - Importing property topic_partitions_per_shard:1000
WARN  2023-06-29 15:07:33,971 [shard 0] admin_api_server - admin_server.cc:389 - Insecure Admin API listener on 0.0.0.0:9645, consider enabling `admin_api_require_auth`
INFO  2023-06-29 15:07:33,971 [shard 0] admin_api_server - admin_server.cc:177 - Started HTTP admin service listening at {{internal:{host: 0.0.0.0, port: 9644}}, {default:{host: 0.0.0.0, port: 9645}}}
INFO  2023-06-29 15:07:33,972 [shard 0] main - application.cc:1855 - Started Pandaproxy listening at {{internal:{host: 0.0.0.0, port: 8082}:<nullopt>}, {default:{host: 0.0.0.0, port: 8083}:<nullopt>}}
INFO  2023-06-29 15:07:33,973 [shard 0] main - application.cc:1863 - Started Schema Registry listening at {{internal:{host: 0.0.0.0, port: 8081}:<nullopt>}, {default:{host: 0.0.0.0, port: 8084}:<nullopt>}}
INFO  2023-06-29 15:07:33,974 [shard 0] main - application.cc:2079 - Waiting for cluster membership
INFO  2023-06-29 15:07:33,974 [shard 0] main - application.cc:2088 - Started Kafka API server listening at {{internal:{host: 0.0.0.0, port: 9093}:{nullopt}}, {default:{host: 0.0.0.0, port: 9094}:{nullopt}}}
INFO  2023-06-29 15:07:33,974 [shard 0] main - application.cc:1870 - Successfully started Redpanda!
INFO  2023-06-29 15:07:33,977 [shard 0] cluster - config_manager.cc:133 - Completed bootstrap as leader
INFO  2023-06-29 15:07:33,977 [shard 0] cluster - config_manager.cc:113 - Bootstrap complete (version 1)
INFO  2023-06-29 15:07:34,241 [shard 0] cluster - health_monitor_backend.cc:561 - received node 1 health report, marking node as up
INFO  2023-06-29 15:07:34,242 [shard 0] cluster - health_monitor_backend.cc:561 - received node 2 health report, marking node as up
WARN  2023-06-29 15:07:34,447 [shard 0] pandaproxy - service.cc:225 - mitigate_error: kafka::client::broker_error ({ node: -1 }, { error_code: broker_not_available [8] })
WARN  2023-06-29 15:07:34,448 [shard 0] kafka/client - client.cc:159 - broker_error: kafka::client::broker_error ({ node: -1 }, { error_code: broker_not_available [8] })
INFO  2023-06-29 15:07:34,448 [shard 0] kafka/client - broker.cc:43 - connected to broker:-1 - 0.0.0.0:9093
INFO  2023-06-29 15:07:34,449 [shard 0] kafka/client - broker.cc:43 - connected to broker:1 - redpanda-1.redpanda.astradot.svc.cluster.local.:9093
INFO  2023-06-29 15:07:34,450 [shard 0] kafka/client - broker.cc:43 - connected to broker:0 - redpanda-0.redpanda.astradot.svc.cluster.local.:9093
INFO  2023-06-29 15:07:34,451 [shard 0] kafka/client - broker.cc:43 - connected to broker:2 - redpanda-2.redpanda.astradot.svc.cluster.local.:9093
INFO  2023-06-29 15:07:34,451 [shard 0] cluster - topics_frontend.cc:98 - Create topics {{configuration: { topic: {ns: {kafka}, topic: {_schemas}}, partition_count: 1, replication_factor: 3, properties: {compression: {none}, cleanup_policy_bitflags: {compact}, compaction_strategy: {nullopt}, retention_bytes: {disabled}, retention_duration_ms: {disabled}, segment_size: {nullopt}, timestamp_type: {nullopt}, recovery_enabled: {nullopt}, shadow_indexing: {disabled}, read_replica: {nullopt}, read_replica_bucket: {nullopt} remote_topic_properties: {nullopt}, batch_max_bytes: {nullopt}, retention_local_target_bytes: {disabled}, retention_local_target_ms: {disabled}, remote_delete: true, segment_ms: {}}}, custom_assignments: {}}}
INFO  2023-06-29 15:07:34,478 [shard 0] raft - [group_id:1, {kafka/_schemas/0}] consensus.cc:1203 - Starting
INFO  2023-06-29 15:07:34,478 [shard 0] raft - [group_id:1, {kafka/_schemas/0}] consensus.cc:1237 - Starting with voted_for {id: {-2147483648}, revision: {-9223372036854775808}} term 0 initial_state true
INFO  2023-06-29 15:07:34,498 [shard 0] offset_translator - ntp: {kafka/_schemas/0} - offset_translator.cc:105 - resetting offset translation state
INFO  2023-06-29 15:07:34,519 [shard 0] offset_translator - ntp: {kafka/_schemas/0} - offset_translator.cc:156 - started, state: {base offset/delta: {-9223372036854775808}/0, map size: 1, last delta: 0}, highest_known_offset: -9223372036854775808
INFO  2023-06-29 15:07:34,519 [shard 0] raft - [group_id:1, {kafka/_schemas/0}] consensus.cc:1283 - Read bootstrap state: data_seen 0 config_seen 0 eol false commit 0 term 0 prev_idx 0 prev_term 0 config_tracker -9223372036854775808 commit_base_tracker -9223372036854775808 configurations {}
INFO  2023-06-29 15:07:34,519 [shard 0] raft - [group_id:1, {kafka/_schemas/0}] consensus.cc:1284 - Current log offsets: {start_offset:-9223372036854775808, committed_offset:-9223372036854775808, committed_offset_term:-9223372036854775808, dirty_offset:-9223372036854775808, dirty_offset_term:-9223372036854775808, last_term_start_offset:-9223372036854775808}
INFO  2023-06-29 15:07:34,519 [shard 0] raft - [group_id:1, {kafka/_schemas/0}] consensus.cc:1308 - Truncating configurations at -9223372036854775808
INFO  2023-06-29 15:07:34,544 [shard 0] offset_translator - ntp: {kafka/_schemas/0} - offset_translator.cc:226 - synced with log, state: {base offset/delta: {-9223372036854775808}/0, map size: 1, last delta: 0}, highest_known_offset: -9223372036854775808
INFO  2023-06-29 15:07:34,544 [shard 0] raft - [group_id:1, {kafka/_schemas/0}] consensus.cc:1408 - started raft, log offsets: {start_offset:-9223372036854775808, committed_offset:-9223372036854775808, committed_offset_term:-9223372036854775808, dirty_offset:-9223372036854775808, dirty_offset_term:-9223372036854775808, last_term_start_offset:-9223372036854775808}, term: 0, configuration: {current: {voters: {{id: {1}, revision: {8}}, {id: {0}, revision: {8}}, {id: {2}, revision: {8}}}, learners: {}}, old:{nullopt}, revision: 8, update: {nullopt}, version: 4, brokers: {{id: 1, kafka_advertised_listeners: {{internal:{host: redpanda-1.redpanda.astradot.svc.cluster.local., port: 9093}}, {default:{host: redpanda-1, port: 31092}}}, rpc_address: {host: redpanda-1.redpanda.astradot.svc.cluster.local., port: 33145}, rack: {{us-east-1a}}, properties: {cores 1, mem_available 1, disk_available 54}}, {id: 0, kafka_advertised_listeners: {{internal:{host: redpanda-0.redpanda.astradot.svc.cluster.local., port: 9093}}, {default:{host: redpanda-0, port: 31092}}}, rpc_address: {host: redpanda-0.redpanda.astradot.svc.cluster.local., port: 33145}, rack: {{us-east-1b}}, properties: {cores 1, mem_available 1, disk_available 54}}, {id: 2, kafka_advertised_listeners: {{internal:{host: redpanda-2.redpanda.astradot.svc.cluster.local., port: 9093}}, {default:{host: redpanda-2, port: 31092}}}, rpc_address: {host: redpanda-2.redpanda.astradot.svc.cluster.local., port: 33145}, rack: {{us-east-1c}}, properties: {cores 1, mem_available 1, disk_available 54}}}}
INFO  2023-06-29 15:07:34,555 [shard 0] pandaproxy - service.cc:178 - Schema registry successfully initialized
INFO  2023-06-29 15:07:34,562 [shard 0] raft - [group_id:1, {kafka/_schemas/0}] consensus.cc:1647 - Received vote request with larger term from node {id: {1}, revision: {8}}, received 1, current 0
INFO  2023-06-29 15:07:34,598 [shard 0] storage - segment.cc:673 - Creating new segment /var/lib/redpanda/data/kafka/_schemas/0_8/0-1-v1.log
WARN  2023-06-29 15:07:34,755 [shard 0] pandaproxy - service.cc:225 - mitigate_error: kafka::client::partition_error ({topic_partition: {_schemas}:0}, { error_code: unknown_topic_or_partition [3] })
INFO  2023-06-29 15:07:38,971 [shard 0] cluster - members_backend.cc:893 - [update: {{node_id: 0, type: added, offset: 0}}] reconciliation loop - pending reallocation count: 0, finished: false
INFO  2023-06-29 15:07:38,971 [shard 0] cluster - members_backend.cc:586 - [update: {{node_id: 0, type: added, offset: 0}}] there are 0 replicas in -1 domain, requested to assign 0 replicas per node
INFO  2023-06-29 15:07:38,971 [shard 0] cluster - members_backend.cc:467 - [update: {{node_id: 0, type: added, offset: 0}}] unevenness error: 0, previous error: 1, improvement: 1
INFO  2023-06-29 15:07:38,971 [shard 0] cluster - members_backend.cc:586 - [update: {{node_id: 0, type: added, offset: 0}}] there are 3 replicas in 0 domain, requested to assign 1 replicas per node
INFO  2023-06-29 15:07:38,971 [shard 0] cluster - members_backend.cc:467 - [update: {{node_id: 0, type: added, offset: 0}}] unevenness error: 0, previous error: 1, improvement: 1
INFO  2023-06-29 15:07:38,971 [shard 0] cluster - members_backend.cc:908 - [update: {{node_id: 0, type: added, offset: 0}}] calculated reallocations: 0
INFO  2023-06-29 15:07:38,973 [shard 0] cluster - members_backend.cc:893 - [update: {{node_id: 1, type: added, offset: 0}}] reconciliation loop - pending reallocation count: 0, finished: false
INFO  2023-06-29 15:07:38,973 [shard 0] cluster - members_backend.cc:586 - [update: {{node_id: 1, type: added, offset: 0}}] there are 0 replicas in -1 domain, requested to assign 0 replicas per node
INFO  2023-06-29 15:07:38,973 [shard 0] cluster - members_backend.cc:467 - [update: {{node_id: 1, type: added, offset: 0}}] unevenness error: 0, previous error: 1, improvement: 1
INFO  2023-06-29 15:07:38,973 [shard 0] cluster - members_backend.cc:586 - [update: {{node_id: 1, type: added, offset: 0}}] there are 3 replicas in 0 domain, requested to assign 1 replicas per node
INFO  2023-06-29 15:07:38,973 [shard 0] cluster - members_backend.cc:467 - [update: {{node_id: 1, type: added, offset: 0}}] unevenness error: 0, previous error: 1, improvement: 1
INFO  2023-06-29 15:07:38,973 [shard 0] cluster - members_backend.cc:908 - [update: {{node_id: 1, type: added, offset: 0}}] calculated reallocations: 0
INFO  2023-06-29 15:07:38,974 [shard 0] cluster - members_backend.cc:893 - [update: {{node_id: 2, type: added, offset: 0}}] reconciliation loop - pending reallocation count: 0, finished: false
INFO  2023-06-29 15:07:38,974 [shard 0] cluster - members_backend.cc:586 - [update: {{node_id: 2, type: added, offset: 0}}] there are 0 replicas in -1 domain, requested to assign 0 replicas per node
INFO  2023-06-29 15:07:38,974 [shard 0] cluster - members_backend.cc:467 - [update: {{node_id: 2, type: added, offset: 0}}] unevenness error: 0, previous error: 1, improvement: 1
INFO  2023-06-29 15:07:38,974 [shard 0] cluster - members_backend.cc:586 - [update: {{node_id: 2, type: added, offset: 0}}] there are 3 replicas in 0 domain, requested to assign 1 replicas per node
INFO  2023-06-29 15:07:38,974 [shard 0] cluster - members_backend.cc:467 - [update: {{node_id: 2, type: added, offset: 0}}] unevenness error: 0, previous error: 1, improvement: 1
INFO  2023-06-29 15:07:38,974 [shard 0] cluster - members_backend.cc:908 - [update: {{node_id: 2, type: added, offset: 0}}] calculated reallocations: 0
INFO  2023-06-29 15:07:43,970 [shard 0] cluster - metrics_reporter.cc:320 - Initialized cluster_id to 0e6bcd9a-88c5-453c-a056-f1514c9885a6
INFO  2023-06-29 15:08:40,707 [shard 0] rpc - transport.cc:210 - RPC timeout (100 ms) to {host: redpanda-1.redpanda.astradot.svc.cluster.local., port: 33145}, method: node_status_rpc::node_status, correlation id: 662, 1 in flight, time since: {init: 100 ms, enqueue: 100 ms, dispatch: 100 ms, written: 100 ms}, flushed: true
INFO  2023-06-29 15:08:41,017 [shard 0] rpc - transport.cc:210 - RPC timeout (100 ms) to {host: redpanda-1.redpanda.astradot.svc.cluster.local., port: 33145}, method: node_status_rpc::node_status, correlation id: 664, 1 in flight, time since: {init: 100 ms, enqueue: 100 ms, dispatch: 100 ms, written: 100 ms}, flushed: true
INFO  2023-06-29 15:08:41,427 [shard 0] rpc - transport.cc:210 - RPC timeout (100 ms) to {host: redpanda-1.redpanda.astradot.svc.cluster.local., port: 33145}, method: node_status_rpc::node_status, correlation id: 667, 1 in flight, time since: {init: 100 ms, enqueue: 100 ms, dispatch: 100 ms, written: 100 ms}, flushed: true
INFO  2023-06-29 15:08:41,839 [shard 0] rpc - transport.cc:210 - RPC timeout (100 ms) to {host: redpanda-1.redpanda.astradot.svc.cluster.local., port: 33145}, method: node_status_rpc::node_status, correlation id: 670, 1 in flight, time since: {init: 100 ms, enqueue: 100 ms, dispatch: 100 ms, written: 100 ms}, flushed: true
INFO  2023-06-29 15:08:42,557 [shard 0] rpc - transport.cc:210 - RPC timeout (100 ms) to {host: redpanda-1.redpanda.astradot.svc.cluster.local., port: 33145}, method: node_status_rpc::node_status, correlation id: 676, 1 in flight, time since: {init: 100 ms, enqueue: 100 ms, dispatch: 100 ms, written: 100 ms}, flushed: true
INFO  2023-06-29 15:09:46,659 [shard 0] cluster - members_table.cc:121 - changing node 2 membership state to: draining
INFO  2023-06-29 15:09:46,660 [shard 0] cluster - members_backend.cc:893 - [update: {{node_id: 2, type: decommissioned, offset: 16}}] reconciliation loop - pending reallocation count: 0, finished: false
INFO  2023-06-29 15:09:46,660 [shard 0] cluster - members_backend.cc:908 - [update: {{node_id: 2, type: decommissioned, offset: 16}}] calculated reallocations: 1
INFO  2023-06-29 15:09:46,660 [shard 0] cluster - members_backend.cc:993 - [update: {{node_id: 2, type: decommissioned, offset: 16}}] decommissioning in progress. draining: true all_reallocations_finished: false, allocator_empty: false updates_in_progress:false
INFO  2023-06-29 15:09:51,661 [shard 0] cluster - members_backend.cc:893 - [update: {{node_id: 2, type: decommissioned, offset: 16}}] reconciliation loop - pending reallocation count: 1, finished: false
INFO  2023-06-29 15:09:51,661 [shard 0] cluster - members_backend.cc:908 - [update: {{node_id: 2, type: decommissioned, offset: 16}}] calculated reallocations: 1

[bharathv]

INFO 2023-06-29 15:09:51,661 [shard 0] cluster - members_backend.cc:893 - [update: {{node_id: 2, type: decommissioned, offset: 16}}] reconciliation loop - pending reallocation count: 1, finished: false

Its trying to reallocate one partition from the dead/decommissioned node, can you share the output of ..

curl http://<broker-0-host>:9644/v1/partitions/reconfigurations --> lists the reallocations in progress

and

rpk topic list -d --> lists topics and replica layout.

[mdfaizsiddiqui]

Output -

kubectl -n astradot exec -ti redpanda-0 -c redpanda -- \
  rpk topic list -d
_schemas, 1 partitions, 3 replicas
      PARTITION  LEADER  EPOCH  REPLICAS
      0          1       1      [0 1 2]

kubectl port-forward service/redpanda 9644:9644
curl http:/localhost:9644/v1/partitions/reconfigurations
[]%