-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Databricks on JDK 21 fails with "Object 'DATABASECHANGELOGLOCK' already exists. #5786
Comments
Hi @pradeep-general-motors-canada, thank you for reporting this issue. I have been able to replicate it using the latest liquibase and liquibase-databricks versions.
We will be looking further into this. Do let us know if you come across other problems or if you have any further information. Thank you, CC: @kevin-atx |
Hi @tati-qalified If my understanding is correct. There is no workaround to get the Liquibase working with databricks, unless this particular bug is addressed. Have I understood it right? |
@pradeep-general-motors-canada luckily that's not the case - being able to generate the SQL script correctly means that you can run that script directly onto your database, and for any future changes just run That would be the workaround until the bug is fixed. |
Thanks for the clarity. Would love to use the fixed version once its ready :) |
Search first
Description
Our Team is trying to use Liquibase for Databricks Schema migration .
Unfortunately, Every time I run the Liquibase
status
orupdate
command, It shows that the DATABASECHANGELOGLOCK tables already exists and keep failing the command. This happens even at the first run.I even tried to change the table name for the lock table, but no luck.
There is a similar question in the Liquibase forum but it has not been resolved. I did add a reply on that post. But I do not see, there are some activities on that ticket.
https://forum.liquibase.org/t/liquibase-creating-datachangeloglock-multiple-times-in-databricks-catalog-table/8967
System Setup:
OS : Windows 11
JAVA : 21
Liquibase : 4.27.0 ( even tried with 4.25 and 4.26 )
libraries : liquibase-databricks-1.1.3, DatabricksJDBC42(2.6.36)
Unit Catalog Enabled : Yes
Databricks : Azure Databricks
ERROR LOG:
FINE: CREATE TABLE uap_innovation.liquibase_example.databasechangelognew (ID VARCHAR(255) NOT NULL, AUTHOR VARCHAR(255) NOT NULL, FILENAME VARCHAR(255) NOT NULL, DATEEXECUTED TIMESTAMP NOT NULL, ORDEREXECUTED INT NOT NULL, EXECTYPE VARCHAR(10) NOT NULL, MD5SUM VARCHAR(35), DESCRIPTION VARCHAR(255), COMMENTS VARCHAR(255), TAG VARCHAR(255), LIQUIBASE VARCHAR(20), CONTEXTS VARCHAR(255), LABELS VARCHAR(255), DEPLOYMENT_ID VARCHAR(10))
Apr 10, 2024 10:37:21 AM liquibase.executor
FINE: -1 row(s) affected
Apr 10, 2024 10:37:21 AM liquibase.executor
FINE: Changelog query completed.
Apr 10, 2024 10:37:21 AM liquibase.servicelocator
FINE: Loaded liquibase.lockservice.LockService instance liquibase.lockservice.LockServiceImpl
Apr 10, 2024 10:37:21 AM liquibase.servicelocator
FINE: Loaded liquibase.lockservice.LockService instance liquibase.lockservice.MockLockService
Apr 10, 2024 10:37:21 AM liquibase.servicelocator
FINE: Loaded liquibase.lockservice.LockService instance liquibase.lockservice.OfflineLockService
Apr 10, 2024 10:37:21 AM liquibase.servicelocator
FINE: Loaded liquibase.lockservice.LockService instance liquibase.lockservice.StandardLockService
Apr 10, 2024 10:37:21 AM liquibase.servicelocator
FINE: Loaded liquibase.database.LiquibaseTableNames instance liquibase.database.StandardLiquibaseTableNames
Apr 10, 2024 10:37:21 AM liquibase.servicelocator
FINE: Loaded liquibase.database.LiquibaseTableNames instance com.datical.liquibase.ext.database.ProLiquibaseTableNames
Apr 10, 2024 10:37:21 AM liquibase.configuration
FINE: No configuration value for liquibase.dbclhistory.tableName aka databasechangeloghistory.tableName found
Apr 10, 2024 10:37:21 AM liquibase.configuration
FINE: Configuration liquibase.dbclhistory.tableName is using the default value of DATABASECHANGELOGHISTORY
Apr 10, 2024 10:37:21 AM liquibase.configuration
FINE: No configuration value for liquibase.sql.showSqlWarnings found
Apr 10, 2024 10:37:21 AM liquibase.configuration
FINE: Configuration liquibase.sql.showSqlWarnings is using the default value of true
Apr 10, 2024 10:37:21 AM liquibase.executor
FINE: Create Database Lock Table
Apr 10, 2024 10:37:21 AM liquibase.executor
FINE: CREATE TABLE uap_innovation.liquibase_example.databasechangelocknew (ID INT NOT NULL, LOCKED BOOLEAN NOT NULL, LOCKGRANTED TIMESTAMP, LOCKEDBY VARCHAR(255), CONSTRAINT PK_DATABASECHANGELOCKNEW PRIMARY KEY (ID))
Apr 10, 2024 10:37:30 AM liquibase.executor
FINE: -1 row(s) affected
Apr 10, 2024 10:37:30 AM liquibase.executor
FINE: Changelog query completed.
Apr 10, 2024 10:37:30 AM liquibase.lockservice
FINE: Created database lock table with name: uap_innovation.liquibase_example.databasechangelocknew
Apr 10, 2024 10:37:31 AM liquibase.lockservice
FINE: Failed to create or initialize the lock table, trying again, iteration 1 of 10
liquibase.exception.UnexpectedLiquibaseException: liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: Error executing SQL SELECT COUNT() FROM uap_innovation.liquibase_example.databasechangelocknew: [Databricks]JDBCDriver Error caught in BackgroundFetcher. Foreground thread ID: 1. Background thread ID: 35. Error caught: Could not initialize class com.databricks.client.jdbc42.internal.apache.arrow.memory.util.MemoryUtil.
at liquibase.lockservice.StandardLockService.isDatabaseChangeLogLockTableInitialized(StandardLockService.java:219)
at liquibase.lockservice.StandardLockService.init(StandardLockService.java:135)
at liquibase.command.core.helpers.DatabaseChangelogCommandStep.checkLiquibaseTables(DatabaseChangelogCommandStep.java:145)
at liquibase.command.core.helpers.DatabaseChangelogCommandStep.run(DatabaseChangelogCommandStep.java:91)
at liquibase.command.CommandScope.execute(CommandScope.java:219)
at liquibase.integration.commandline.CommandRunner.call(CommandRunner.java:55)
at liquibase.integration.commandline.CommandRunner.call(CommandRunner.java:24)
at picocli.CommandLine.executeUserObject(CommandLine.java:1953)
at picocli.CommandLine.access$1300(CommandLine.java:145)
at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2179)
at picocli.CommandLine.execute(CommandLine.java:2078)
at liquibase.integration.commandline.LiquibaseCommandLine.lambda$execute$2(LiquibaseCommandLine.java:395)
at liquibase.Scope.child(Scope.java:199)
at liquibase.Scope.child(Scope.java:175)
at liquibase.integration.commandline.LiquibaseCommandLine.lambda$execute$3(LiquibaseCommandLine.java:370)
at liquibase.Scope.child(Scope.java:199)
at liquibase.Scope.child(Scope.java:175)
at liquibase.integration.commandline.LiquibaseCommandLine.execute(LiquibaseCommandLine.java:367)
at liquibase.integration.commandline.LiquibaseCommandLine.main(LiquibaseCommandLine.java:104)
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
at java.base/java.lang.reflect.Method.invoke(Method.java:580)
at liquibase.integration.commandline.LiquibaseLauncher.main(LiquibaseLauncher.java:116)
Caused by: liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: Error executing SQL SELECT COUNT() FROM uap_innovation.liquibase_example.databasechangelocknew: [Databricks]JDBCDriver Error caught in BackgroundFetcher. Foreground thread ID: 1. Background thread ID: 35. Error caught: Could not initialize class com.databricks.client.jdbc42.internal.apache.arrow.memory.util.MemoryUtil.
at liquibase.executor.jvm.ChangelogJdbcMdcListener.query(ChangelogJdbcMdcListener.java:62)
at liquibase.lockservice.StandardLockService.isDatabaseChangeLogLockTableInitialized(StandardLockService.java:216)
... 24 more
Caused by: liquibase.exception.DatabaseException: Error executing SQL SELECT COUNT(*) FROM uap_innovation.liquibase_example.databasechangelocknew: [Databricks]JDBCDriver Error caught in BackgroundFetcher. Foreground thread ID: 1. Background thread ID: 35. Error caught: Could not initialize class com.databricks.client.jdbc42.internal.apache.arrow.memory.util.MemoryUtil.
at liquibase.executor.jvm.JdbcExecutor.execute(JdbcExecutor.java:101)
at liquibase.executor.jvm.JdbcExecutor.query(JdbcExecutor.java:227)
at liquibase.executor.jvm.JdbcExecutor.query(JdbcExecutor.java:235)
at liquibase.executor.jvm.JdbcExecutor.queryForObject(JdbcExecutor.java:243)
at liquibase.executor.jvm.JdbcExecutor.queryForObject(JdbcExecutor.java:258)
at liquibase.executor.jvm.JdbcExecutor.queryForInt(JdbcExecutor.java:279)
at liquibase.executor.jvm.JdbcExecutor.queryForInt(JdbcExecutor.java:274)
at liquibase.lockservice.StandardLockService.lambda$isDatabaseChangeLogLockTableInitialized$2(StandardLockService.java:216)
at liquibase.executor.jvm.ChangelogJdbcMdcListener.lambda$query$1(ChangelogJdbcMdcListener.java:55)
at liquibase.Scope.child(Scope.java:199)
at liquibase.Scope.child(Scope.java:175)
at liquibase.executor.jvm.ChangelogJdbcMdcListener.query(ChangelogJdbcMdcListener.java:55)
... 25 more
Caused by: java.sql.SQLException: [Databricks]JDBCDriver Error caught in BackgroundFetcher. Foreground thread ID: 1. Background thread ID: 35. Error caught: Could not initialize class com.databricks.client.jdbc42.internal.apache.arrow.memory.util.MemoryUtil.
at com.databricks.client.hivecommon.dataengine.BackgroundFetcher.run(Unknown Source)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
Caused by: com.databricks.client.support.exceptions.GeneralException: [Databricks]JDBCDriver Error caught in BackgroundFetcher. Foreground thread ID: 1. Background thread ID: 35. Error caught: Could not initialize class com.databricks.client.jdbc42.internal.apache.arrow.memory.util.MemoryUtil.
... 5 more
Caused by: java.lang.NoClassDefFoundError: Could not initialize class com.databricks.client.jdbc42.internal.apache.arrow.memory.util.MemoryUtil
at com.databricks.client.jdbc42.internal.apache.arrow.memory.ArrowBuf.getDirectBuffer(ArrowBuf.java:229)
at com.databricks.client.jdbc42.internal.apache.arrow.memory.ArrowBuf.nioBuffer(ArrowBuf.java:224)
at com.databricks.client.jdbc42.internal.apache.arrow.vector.ipc.ReadChannel.readFully(ReadChannel.java:87)
at com.databricks.client.jdbc42.internal.apache.arrow.vector.ipc.message.MessageSerializer.readMessageBody(MessageSerializer.java:728)
at com.databricks.client.jdbc42.internal.apache.arrow.vector.ipc.message.MessageSerializer.deserializeRecordBatch(MessageSerializer.java:363)
at com.databricks.client.spark.arrow.ArrowBuffer.deserializeBatch(Unknown Source)
at com.databricks.client.spark.arrow.ArrowBuffer.handleInitializeBuffer(Unknown Source)
at com.databricks.client.hivecommon.api.HiveServer2BaseBuffer.initializeBuffer(Unknown Source)
at com.databricks.client.hivecommon.api.RowsetBuffer.initializeBuffer(Unknown Source)
at com.databricks.client.hivecommon.api.HS2Client.getRowSetInformation(Unknown Source)
at com.databricks.client.hivecommon.api.HS2Client.fetchFromServer(Unknown Source)
at com.databricks.client.spark.jdbc.DownloadableFetchClient.fetchNRows(Unknown Source)
at com.databricks.client.hivecommon.api.HS2Client.fetchRows(Unknown Source)
at com.databricks.client.hivecommon.dataengine.BackgroundFetcher.run(Unknown Source)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.lang.ExceptionInInitializerError: Exception java.lang.RuntimeException: Failed to initialize MemoryUtil. [in thread "pool-1-thread-1"]
at com.databricks.client.jdbc42.internal.apache.arrow.memory.util.MemoryUtil.(MemoryUtil.java:136)
... 19 more
Apr 10, 2024 10:37:32 AM liquibase.executor
FINE: Create Database Lock Table
Apr 10, 2024 10:37:32 AM liquibase.executor
FINE: CREATE TABLE uap_innovation.liquibase_example.databasechangelocknew (ID INT NOT NULL, LOCKED BOOLEAN NOT NULL, LOCKGRANTED TIMESTAMP, LOCKEDBY VARCHAR(255), CONSTRAINT PK_DATABASECHANGELOCKNEW PRIMARY KEY (ID))
Apr 10, 2024 10:37:32 AM liquibase.lockservice
FINE: Failed to create or initialize the lock table, trying again, iteration 2 of 10
liquibase.exception.DatabaseException: liquibase.exception.DatabaseException: [Databricks]JDBCDriver ERROR processing query/statement. Error Code: 0, SQL state: 42P07, Query: CREATE TAB***, Error message from Server: org.apache.hive.service.cli.HiveSQLException: Error running query: [TABLE_OR_VIEW_ALREADY_EXISTS] org.apache.spark.sql.catalyst.analysis.TableAlreadyExistsException: [TABLE_OR_VIEW_ALREADY_EXISTS] Cannot create table or view
liquibase_example
.databasechangelocknew
because it already exists.Choose a different name, drop or replace the existing object, add the IF NOT EXISTS clause to tolerate pre-existing objects, or add the OR REFRESH clause to refresh the existing streaming table.
at org.apache.spark.sql.hive.thriftserver.HiveThriftServerErrors$.runningQueryError(HiveThriftServerErrors.scala:48)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.$anonfun$execute$1(SparkExecuteStatementOperation.scala:697)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.unity.UCSEphemeralState$Handle.runWith(UCSEphemeralState.scala:45)
at com.databricks.unity.HandleImpl.runWith(UCSHandle.scala:103)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.org$apache$spark$sql$hive$thriftserver$SparkExecuteStatementOperation$$execute(SparkExecuteStatementOperation.scala:574)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.$anonfun$run$2(SparkExecuteStatementOperation.scala:423)
at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
at com.databricks.logging.UsageLogging.withAttributionContext(UsageLogging.scala:420)
at com.databricks.logging.UsageLogging.withAttributionContext$(UsageLogging.scala:418)
at com.databricks.spark.util.PublicDBLogging.withAttributionContext(DatabricksSparkUsageLogger.scala:27)
at com.databricks.logging.UsageLogging.withAttributionTags(UsageLogging.scala:472)
at com.databricks.logging.UsageLogging.withAttributionTags$(UsageLogging.scala:455)
at com.databricks.spark.util.PublicDBLogging.withAttributionTags(DatabricksSparkUsageLogger.scala:27)
at com.databricks.spark.util.PublicDBLogging.withAttributionTags0(DatabricksSparkUsageLogger.scala:72)
at com.databricks.spark.util.DatabricksSparkUsageLogger.withAttributionTags(DatabricksSparkUsageLogger.scala:172)
at com.databricks.spark.util.UsageLogging.$anonfun$withAttributionTags$1(UsageLogger.scala:491)
at com.databricks.spark.util.UsageLogging$.withAttributionTags(UsageLogger.scala:603)
at com.databricks.spark.util.UsageLogging$.withAttributionTags(UsageLogger.scala:612)
at com.databricks.spark.util.UsageLogging.withAttributionTags(UsageLogger.scala:491)
at com.databricks.spark.util.UsageLogging.withAttributionTags$(UsageLogger.scala:489)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withAttributionTags(SparkExecuteStatementOperation.scala:65)
at org.apache.spark.sql.hive.thriftserver.ThriftLocalProperties.$anonfun$withLocalProperties$8(ThriftLocalProperties.scala:161)
at com.databricks.spark.util.IdentityClaim$.withClaim(IdentityClaim.scala:48)
at org.apache.spark.sql.hive.thriftserver.ThriftLocalProperties.withLocalProperties(ThriftLocalProperties.scala:160)
at org.apache.spark.sql.hive.thriftserver.ThriftLocalProperties.withLocalProperties$(ThriftLocalProperties.scala:51)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation.withLocalProperties(SparkExecuteStatementOperation.scala:65)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:401)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2$$anon$3.run(SparkExecuteStatementOperation.scala:386)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at org.apache.spark.sql.hive.thriftserver.SparkExecuteStatementOperation$$anon$2.run(SparkExecuteStatementOperation.scala:435)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:750)
Steps To Reproduce
Followed the below documentation to setup Liquibase on Windows and Run the status and update command.
liquibase-databricks
CHANGLOG FILE NAME :
example-changelog.sql
CHANGELOG FILE CONTENET:
LIQUIBASE PROPERTIES FILE NAME:
liquibase.properties
LIQUIBASE PROPERTIES FILE CONTENT:
COMMAD TO RUN:
Expected/Desired Behavior
Expecting tables mentioned in the changelog files gets deployed to databricks unity catalog
Liquibase Version
4.27
Database Vendor & Version
Databricks Unity Catalog
Liquibase Integration
CLI
Liquibase Extensions
liquibase-databricks-1.1.3, DatabricksJDBC42(2.6.36)
OS and/or Infrastructure Type/Provider
Windows 11
Additional Context
I am using Azure Databricks. Hence I created a Microsoft support Ticket and they connected me with Databricks Team who owns the databricks JDBC connector.
I did connect with the Databricks team to check, if the JDBC connector is having some issue. But Databricks Teams thinks its something to do with the Liquibase CLI and not the connector.
Databricks Team is ready to connect with anyone of the support engineer from Liquibase and work together .
NOTE:
If this issue required a collaboration between Liquibase and databricks. I would be really happy to setup a call with required engineers.
Are you willing to submit a PR?
The text was updated successfully, but these errors were encountered: