[Bug] Paimon catalog can't correctly capture schema changes #35024
Comments
|
we will fix it and update pr |
I have tried version-2.1.3, it not only occurs with the type 'decimal', adding other type's column also triggers the same error. |
|
The bug will be fixed by #35309 |
morningman
pushed a commit
that referenced
this issue
May 28, 2024
Issue Number: close #35024 This bug is because the fe incorrectly sets the update time of paimon catalog, causing the be to be unable to update paimon's schema in time. ```c++ private void initTable() { PaimonTableCacheKey key = new PaimonTableCacheKey(ctlId, dbId, tblId, paimonOptionParams, dbName, tblName); TableExt tableExt = PaimonTableCache.getTable(key); if (tableExt.getCreateTime() < lastUpdateTime) { LOG.warn("invalidate cache table:{}, localTime:{}, remoteTime:{}", key, tableExt.getCreateTime(), lastUpdateTime); PaimonTableCache.invalidateTableCache(key); tableExt = PaimonTableCache.getTable(key); } this.table = tableExt.getTable(); paimonAllFieldNames = PaimonScannerUtils.fieldNames(this.table.rowType()); if (LOG.isDebugEnabled()) { LOG.debug("paimonAllFieldNames:{}", paimonAllFieldNames); } } ```
yiguolei
pushed a commit
that referenced
this issue
May 28, 2024
Issue Number: close #35024 This bug is because the fe incorrectly sets the update time of paimon catalog, causing the be to be unable to update paimon's schema in time. ```c++ private void initTable() { PaimonTableCacheKey key = new PaimonTableCacheKey(ctlId, dbId, tblId, paimonOptionParams, dbName, tblName); TableExt tableExt = PaimonTableCache.getTable(key); if (tableExt.getCreateTime() < lastUpdateTime) { LOG.warn("invalidate cache table:{}, localTime:{}, remoteTime:{}", key, tableExt.getCreateTime(), lastUpdateTime); PaimonTableCache.invalidateTableCache(key); tableExt = PaimonTableCache.getTable(key); } this.table = tableExt.getTable(); paimonAllFieldNames = PaimonScannerUtils.fieldNames(this.table.rowType()); if (LOG.isDebugEnabled()) { LOG.debug("paimonAllFieldNames:{}", paimonAllFieldNames); } } ```
dataroaring
pushed a commit
that referenced
this issue
May 31, 2024
Issue Number: close #35024 This bug is because the fe incorrectly sets the update time of paimon catalog, causing the be to be unable to update paimon's schema in time. ```c++ private void initTable() { PaimonTableCacheKey key = new PaimonTableCacheKey(ctlId, dbId, tblId, paimonOptionParams, dbName, tblName); TableExt tableExt = PaimonTableCache.getTable(key); if (tableExt.getCreateTime() < lastUpdateTime) { LOG.warn("invalidate cache table:{}, localTime:{}, remoteTime:{}", key, tableExt.getCreateTime(), lastUpdateTime); PaimonTableCache.invalidateTableCache(key); tableExt = PaimonTableCache.getTable(key); } this.table = tableExt.getTable(); paimonAllFieldNames = PaimonScannerUtils.fieldNames(this.table.rowType()); if (LOG.isDebugEnabled()) { LOG.debug("paimonAllFieldNames:{}", paimonAllFieldNames); } } ```
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Search before asking
Version
2.1.2
What's Wrong?
When paimon table change schema (add a decimal column), after refresh paimon catalog's metadata, the table couldn't be queried and report a error in be.INFO.log:
What You Expected?
Paimon catalog can be queried normally after synchronizing schema changes.
How to Reproduce?
create paimon catalog like:
after a table
test_database.test1from mysql add a column(type decimal) and synchronize to paimon by cdc action, refresh paimon catalog's metadata.query the table again,
would report a error:
Anything Else?
No response
Are you willing to submit PR?
Code of Conduct
The text was updated successfully, but these errors were encountered: