v21.8.1

[dswang]

Features

• Automatic leadership rebalancing: Automatically rebalance Raft leadership of partitions across nodes in the cluster when a node becomes unavailable or available again.
• CreatePartitions API: Support the ability to add partitions to a topic
• Schema registry: Support a schema registry, with support for the Avro format.
• Transactions: Support the Kafka Transactions API

Improvements

• Operator: On k8s, by default we set transaction_coordinator_replication and id_allocator_replication to 3 when having at least 3 replicas.
• Operator: The Kubernetes operator now runs only one process in the main redpanda container which should prevent OOMs
• Raft: A new configuration option raft_heartbeat_disconnect_failures is added, defaulting to three missed heartbeats before connections are closed (i.e. reconnected). This should not need manual adjustment unless running on extremely overloaded systems which are emitting many Closed unresponsive connection log messages.

Bug fixes

• Automatic partition rebalancing: Bug fixes for corner cases
• Node decommissioning: Bug fixes for corner cases
• Seastar: Bug fix for hanging I/Os under heavy load
• Schema Registry:
  • Support enum defaults for avro compatibility checks
  • Allow deserializing 3rdparty record format
  • Introduce POST /subjects/{subject}
  • Introduce GET /schemas/ids/{id}/versions
  • Soft-deleting a subject no longer returns soft-deleted versions
  • Set the default compatibility to BACKWARDS
  • Allow set_compatibility on non-existent subject
  • Add tests for union
  • Weaken AVRO resolve check to support integer promotion on e.g., arrays
  • Fix error_code and message for subject_version_not_found
  • Fix error_code for schema_id_not_found
  • Remove delete_subject_value::version - this is a breaking change, but it hasn't been released yet
  • Permanently deleting a subject_version twice should return subject_version_not_found, instead of subject_not_found
  • Validate schema on insert to prevent invalid schemas reaching the topic

---

This discussion was created from the release v21.8.1.