Skip to content

Commit

Permalink
Merge pull request #356 from Altinity/add_binary_encoding_configuration
Browse files Browse the repository at this point in the history 
Updated Debezium to 2.5.0.Alpha1(Latest release), Added mariadb docke…
  • Loading branch information
@subkanthi
subkanthi authored Nov 14, 2023
2 parents f4bc992 + 29a1ade commit 1e4cdf1
Show file tree
Hide file tree
Showing 28 changed files with 310,397 additions and 30 deletions.
2 changes: 1 addition & 1 deletion .github/workflows/sink-connector-lightweight-tests.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ on:

jobs:
build:
runs-on: [self-hosted, linux, x64]
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
- name: Set up JDK 11
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/testflow-sink-connector-lightweight.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ on:
description: "sink connector version"
required: true
type: string
default: 2023-08-11
default: 2023-11-03

env:
# SINK_CONNECTOR_VERSION: "${{ inputs.sink_version }}"
Expand Down
12 changes: 8 additions & 4 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -128,14 +128,14 @@ GLOBAL OPTIONS:
|---------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| database.hostname | Source Database HostName |
| database.port | Source Database Port number |
| database.user | Source Database Username(user needs to have replication permission, Refer https://debezium.io/documentation/reference/stable/connectors/mysql.html) GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'user' IDENTIFIED BY 'password'; |
| database.user | Source Database Username(user needs to have replication permission, Refer https://debezium.io/documentation/reference/stable/connectors/mysql.html) GRANT SELECT, RELOAD, SHOW DATABASES, REPLICATION SLAVE, REPLICATION CLIENT ON *.* TO 'user' IDENTIFIED BY 'password'; |
| database.password | Source Database Password |
| database.include.list | List of databases to be included in replication. |
| table.include.list | List of tables to be included in replication. |
| clickhouse.server.url | ClickHouse URL, For TLS(use `https` and set port to `8443`) |
| clickhouse.server.url | ClickHouse URL, For TLS(use `https` and set port to `8443`) |
| clickhouse.server.user | ClickHouse username |
| clickhouse.server.password | ClickHouse password |
| clickhouse.server.port | ClickHouse port, For TLS(use the correct port `8443` or `443` |
| clickhouse.server.password | ClickHouse password |
| clickhouse.server.port | ClickHouse port, For TLS(use the correct port `8443` or `443` |
| clickhouse.server.database | ClickHouse destination database |
| snapshot.mode | "initial" -> Data that already exists in source database will be replicated. "schema_only" -> Replicate data that is added/modified after the connector is started.\<br/> MySQL: https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-property-snapshot-mode \ <br/>PostgreSQL: https://debezium.io/documentation/reference/stable/connectors/postgresql.html#postgresql-property-snapshot-mode <br/> MongoDB: initial, never. https://debezium.io/documentation/reference/stable/connectors/mongodb.html |
| connector.class | MySQL -> "io.debezium.connector.mysql.MySqlConnector" <br/> PostgreSQL -> <br/> Mongo -> <br/> |
Expand All @@ -145,6 +145,10 @@ GLOBAL OPTIONS:
| disable.ddl | **Optional**, Default: false, if DDL execution needs to be disabled |
| enable.ddl.snapshot | **Optional**, Default: false, If set to true, the DDL that is passed as part of snapshot process will be executed. Default behavior is DROP/TRUNCATE as part of snapshot is disabled. |
| database.allowPublicKeyRetrieval | **Optional**, MySQL specific: true/false |
| auto.create.tables | When True, connector will create tables(transformed DDL from source) |
| persist.raw.bytes | Debezium.BYTES data(usually UUID) is persisted as raw bytes(CH String) if set to true. |
| connectionTimeZone | Specify MySQL timezone for DATETIME conversions.https://debezium.io/documentation/reference/stable/connectors/mysql.html#mysql-temporal-types |
| enable.snapshot.ddl | When true, pre-existing DDL statements from source(MySQL) will be executed. Warning: This might run DROP TABLE commands. |



Expand Down
4 changes: 4 additions & 0 deletions build_docker.sh
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
#!/bin/bash

cd sink-connector
mvn clean install

cd ..
cd sink-connector-lightweight
mvn clean install -DskipTests=true
today_date=$(date +%F)
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@
<maven.compiler.target>11</maven.compiler.target>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<version.debezium>2.2.0.Alpha1</version.debezium>
<version.debezium>2.4.0</version.debezium>
<version.junit>5.9.1</version.junit>
<version.checkstyle.plugin>3.1.1</version.checkstyle.plugin>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
Expand Down
Binary file modified sink-connector-client/sink-connector-client
View file
Binary file not shown.
6 changes: 3 additions & 3 deletions sink-connector-lightweight/dependency-reduced-pom.xml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,7 @@
<plugin>
<groupId>org.antlr</groupId>
<artifactId>antlr4-maven-plugin</artifactId>
<version>4.9.2</version>
<version>4.10.1</version>
<executions>
<execution>
<goals>
Expand All @@ -115,7 +115,7 @@
<dependency>
<groupId>io.debezium</groupId>
<artifactId>debezium-connector-mongodb</artifactId>
<version>2.2.0.Alpha1</version>
<version>2.5.0.Alpha1</version>
<scope>test</scope>
<exclusions>
<exclusion>
Expand Down Expand Up @@ -316,7 +316,7 @@
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
<version.checkstyle.plugin>3.1.1</version.checkstyle.plugin>
<maven.compiler.target>11</maven.compiler.target>
<version.debezium>2.2.0.Alpha1</version.debezium>
<version.debezium>2.5.0.Alpha1</version.debezium>
<quarkus.platform.group-id>io.quarkus.platform</quarkus.platform.group-id>
</properties>
</project>
1 change: 1 addition & 0 deletions sink-connector-lightweight/docker/config.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,4 +42,5 @@ schema.history.internal.jdbc.schema.history.table.ddl: "CREATE TABLE if not exis

schema.history.internal.jdbc.schema.history.table.name: "altinity_sink_connector.replicate_schema_history"
enable.snapshot.ddl: "true"
persist.raw.bytes: "false"
auto.create.tables: "true"
Loading

0 comments on commit 1e4cdf1

Please sign in to comment.