Code

Deltalake#1256

Code

Merged

ghislainfourny merged 162 commits intomasterRumbleDB/rumble:masterfrom deltalakeRumbleDB/rumble:deltalakeCopy head branch name to clipboardSep 23, 2024

+7,977-2,204

Commits

Commits on May 28, 2023

• Create tests for delta-file function

  

  dloughlin

  committed
  179dfbc

• Create tests for delta-file function with transform

  

  dloughlin

  committed
  d049873

• Implement delta-file function

  

  dloughlin

  committed
  d5f8f00

• Add delta as output option

  

  dloughlin

  committed
  95d135f

• Add delta as output option

  

  dloughlin

  committed
  85c7342

• Add delta to configs

  

  dloughlin

  committed
  75d6c3f

• Create test data for delta

  

  dloughlin

  committed
  e907de3

• Create test to check non-delta-file values and non copied values are immutable

  

  dloughlin

  committed
  29177c5

• Create error for attempt to modify immutable values

  

  dloughlin

  committed
  a6e9503

• Make default mutability level -1

  

  dloughlin

  committed
  38c01f2

• Adjust error semantics to check for mutability level of -1

  

  dloughlin

  committed
  a17d345

• Adjust test to new immutable modify error semantics

  

  dloughlin

  committed
  c083e9a

• Add delta file to alter

  

  dloughlin

  committed
  24cdb58

• Create tests for semantics of updating delta files directly

  

  dloughlin

  committed
  32e5e01

• Add mutabilityLevel getters and setters to itemType hierarchy

  

  dloughlin

  committed
  27281a6

• Implement setting of mutabilityLevel for dataframes with itemtypes

  

  dloughlin

  committed
  f95d301

• Apple spotless

  

  dloughlin

  committed
  9f5b847

Commits on May 31, 2023

• Remove err debug

  

  dloughlin

  committed
  85fd7d3

Commits on Jun 1, 2023

• Remove json keyword from updating exprs

  

  dloughlin

  committed
  0434570

• Fix copy classnotfound exception when using repl

  

  dloughlin

  committed
  ca0bdd5

• Add this. to solve unqualified acceses

  

  dloughlin

  committed
  0fb6e57

• Add updating functions lexing

  

  dloughlin

  committed
  72ae3a2

• Update functiondecl visit to incorporate isUpdating and isExternal booleans

  

  dloughlin

  committed
  3eb57d7

• Update FunctionSignature to include isUpdating flag for StaticContextVisitor

  

  dloughlin

  committed
  cc5112c

• Implement ExpressionClassificationVisitor visits for InlineFunctions and FunctionCalls along with the errors

  

  dloughlin

  committed
  681d30c

Commits on Jun 2, 2023

• Create simple tests for updating functions

  

  dloughlin

  committed
  dcdfe63

• Extend function iterators to account for updating functions and implement getPUL methods

  

  dloughlin

  committed
  772727c

• Reorganise visitor list to ensure function calls are set to updating before type inference

  

  dloughlin

  committed
  7a0ebd9

• Make isUpdating non transient to ensure it retains after deppCopy

  

  dloughlin

  committed
  6d10219

• Include isUpdating to function signatures in InferTypeVisitor

  

  dloughlin

  committed
  9f80e1d

• Account for built-in functions when checking signatures

  

  dloughlin

  committed
  85e814b

• Add error tests for XUST0001, XUST0002, and XUST0028 for updating functions

  

  dloughlin

  committed
  d51d324

• Add error check for unknown functions in ExpressionClassificationVisitor

  

  dloughlin

  committed
  e42ed3a

Commits on Jun 3, 2023

• Refactor mutabilityLevel to be put inside dataframe and parsed during row to item parsing

  

  dloughlin

  committed
  6432b20

• Remove mutabilityLevel from itemType hierarchy

  

  dloughlin

  committed
  862ddb8

• Fix test to correct error code

  

  dloughlin

  committed
  77b0ee6

• Add mutabilityLevel to serialisation of ObjectItem and ArrayItem

  

  dloughlin

  committed
  78636a8

• Correct errorCode required

  

  dloughlin

  committed
  9f4b722

• Add row id and path in dataframe to item hierarchy

  

  dloughlin

  committed
  1fe3c25

• Change rowID to long

  

  dloughlin

  committed
  69bd06d

Commits on Jun 7, 2023

• Introduce RBDY0007 ErrorCode and refactor expected test outputs

  

  dloughlin

  committed
  17506e3

Commits on Jun 21, 2023

• Add getSparkSQLValue and getTableLocation functions to Item interface

  

  dloughlin

  committed
  9e01001

• Implement applyItem and applyDelta functions for UpdatePrimitive interface

  

  dloughlin

  committed
  ba0120f

• Make default table location "null" to allow kryo copy

  

  dloughlin

  committed
  b2fd52d

• Alter apply method to use applyItem when table location is "null" too

  

  dloughlin

  committed
  627c8dd

Commits on Jun 26, 2023

• Fix bug in query creation to allow for simple updates in delta

  

  dloughlin

  committed
  a6065b7

Commits on Jun 27, 2023

• Implement method for getting SQL type of an item in item interface

  

  dloughlin

  committed
  cafd00d

Commits on Jun 28, 2023

• Fix bugs in applyDelta methods: remove "." and add spaces

  

  dloughlin

  committed
  d976d99

• Implement extension to pathIn field for delta table dataframes

  

  dloughlin

  committed
  91cc89a

• Add temporary fix to stop copied dataframes being considered as delta

  

  dloughlin

  committed
  daba332

• Implement getSparkSQLValue func for AnnotatedItem

  

  dloughlin

  committed
  882af35

• Fix comma bug in getSparkSQLValue func for ArrayItem

  

  dloughlin

  committed
  74947f9

• Fix query string bugs in applyDelta funcs

  

  dloughlin

  committed
  37acbf8

• Implement dataframe delta table info for object lookups resulting in array items

  

  dloughlin

  committed
  9bf9619

Commits on Jul 1, 2023

• Add getSparkSQLValue function with ItemType param and implement the methods to account for nullable fields

  

  dloughlin

  committed
  e70b977

• Add getSparkSQLType func to ItemType interface to account for nullable fields

  

  dloughlin

  committed
  13fba96

• Remove originalArray field from dataframes and rework object key lookup positioning in pathIn field

  

  dloughlin

  committed
  ca69a07

• Create default arrayIndexingApplyDelta method for update primitives to account for common task of overwriting an array

  

  dloughlin

  committed
  b403a1b

• Implement getSparkSQLType for DerivedAtomicItemType

  

  dloughlin

  committed
  8d3e561

• Implement applyDelta branching for case when pathIn has an array lookup

  

  dloughlin

  committed
  583cf6a

• Fix bug duplicating DeleteFromObject primitives

  

  dloughlin

  committed
  1ebfe3e

• Apply spotless

  

  dloughlin

  committed
  7d52f94

Commits on Jul 7, 2023

• Implement altering schema in arrays for delta

  

  dloughlin

  committed
  4582e4e

• Fix object lookup bug with placement of "."

  

  dloughlin

  committed
  04e0520

Commits on Jul 19, 2023

• Create test suite for delta updates

  

  dloughlin

  committed
  ca5e53c

• Add tests for non nested delta updates

  

  dloughlin

  committed
  7ef44e6

Commits on Jul 20, 2023

• Fix bug causing tests to not run

  

  dloughlin

  committed
  03599de

• Add new tests to test nested delta updates

  

  dloughlin

  committed
  cc351ab

• Remove unnecessary test

  

  dloughlin

  committed
  ab07f5c

• Correct simple tests -- wrong naming

  

  dloughlin

  committed
  abc078d

• Remove comment

  

  dloughlin

  committed
  c4be336

• Add typing to the generation of arrays for SparkSQL from JSONiq to account for arrays of structs

  

  dloughlin

  committed
  1f6b419

• Refactor checkTableCreation

  

  dloughlin

  committed
  1adde98

• Add parquet file to test deeply nested updates in delta stores

  

  dloughlin

  committed
  74de11b

• Remove unnecessary delta files

  

  dloughlin

  committed
  b1a706e

• Apply spotless

  

  dloughlin

  committed
  de4d571

• Replace delta-file for runtime tests

  

  dloughlin

  committed
  91a74b6

• Add missing file

  

  dloughlin

  committed
  0077caa

Commits on Jul 24, 2023

• Create tests for R xQuery use case

  

  dloughlin

  committed
  37a5055

• Add CreateTable annotation

  

  dloughlin

  committed
  4aa46d8

• Refactor DeltaUpdateRuntimeTests for CreateTable annotation

  

  dloughlin

  committed
  e64f460

• Use CreateTable annotation

  

  dloughlin

  committed
  585aaf2

• Create tests for multirow delta updates

  

  dloughlin

  committed
  ba8b0fe

• Fix control flow for updating dataframes in return clause

  

  dloughlin

  committed
  5df8a5a

• Account for delta table identifications in dfs

  

  dloughlin

  committed
  c2d02b9

• Allow null ItemType in ItemParser

  

  dloughlin

  committed
  d1f4047

• Refactor Update Primitive Item ordering in PUL to include delta items

  

  dloughlin

  committed
  22e24f5

• Ensure Sequence from Updating iterator is empty

  

  dloughlin

  committed
  a6ceec7

• Create dataset for R use case

  

  dloughlin

  committed
  554dc8a

Commits on Jul 25, 2023

• Add tests for multirow delta tables and updating funcs

  

  dloughlin

  committed
  d9a2920

Commits on Aug 3, 2023

• pretty format

  

  dloughlin

  committed
  b52c529

• Add "this." to solve warning

  

  dloughlin

  committed
  f19193d

Commits on Aug 9, 2023

• Add test to check array insert is at valid index and add said semantics

  

  dloughlin

  committed
  05ace8e

• Make mergeUpdates a non-static method and fix bug regarding not removing renames nor replaces during merge

  

  dloughlin

  committed
  46211cf

• Fix error message

  

  dloughlin

  committed
  a17d934

Commits on Aug 10, 2023

• Create test for replaces before deletes in arrays and fix bug causing deletes to then be considered duplicate replaces

  

  dloughlin

  committed
  3646afa

Commits on Aug 17, 2023

• Create benchmark jsoniq queries

  

  dloughlin

  committed
  9c5c044

• Create benchmark class

  

  dloughlin

  committed
  7cbdbcc

• Stop Sequences with updating iterator from producing Sequence

  

  dloughlin

  committed
  a320298

• Add key exists check to InsertIntoObjectPrimitive

  

  dloughlin

  committed
  e535acd

• Ensure set delta id methods propagate to child items of structured items

  

  dloughlin

  committed
  9f9c414

• Remove redundant file

  

  dloughlin

  committed
  4fe6afc

• Ensure append

  

  dloughlin

  committed
  06adf01

• Add metadata

  

  dloughlin

  committed
  6550d5f

• Remove unused method

  

  dloughlin

  committed
  0baf27a

• Add tests to check delta ids propagate to child items of structured items

  

  dloughlin

  committed
  9cd90df

• Add tests for mutability scoping in return clause of transform

  

  dloughlin

  committed
  a941dbf

• Add tests to ensure snapshot isolation

  

  dloughlin

  committed
  7e82bc6

• Adapt applyUpdates to use TreeMap to ensure order of array selectors

  

  dloughlin

  committed
  3387e84

• apply spotless

  

  dloughlin

  committed
  94f0864

• fix bug with null pointers

  

  dloughlin

  committed
  ccd9533

Commits on Sep 3, 2023

• Set inferred static type of transform to return

  

  dloughlin

  committed
  26c30d6

• Add benchmark data

  

  dloughlin

  committed
  cc04085

• Refactor transform with delta tests into delta runtime

  

  dloughlin

  committed
  d096d71

• Remove comments

  

  dloughlin

  committed
  a308ea1

• Add test datasets

  

  dloughlin

  committed
  d4101c2

• Apply spotless

  

  dloughlin

  committed
  a6997c6

• Add delta tests to CI

  

  dloughlin

  committed
  51ed8d1

• Extend pom with delta dependencies

  

  dloughlin

  committed
  84629de

Commits on Jul 22, 2024

• Merge Delta into master.

  

  Ghislain Fourny

  committed
  1edd929

• Fix tests.

  

  Ghislain Fourny

  committed
  17b119e

• Fix grammar.

  

  Ghislain Fourny

  committed
  0094d4e

Commits on Jul 23, 2024

• Fix grammar.

  

  Ghislain Fourny

  committed
  0889cb1

• Solve dependency conflict on math3.

  

  Ghislain Fourny

  committed
  329b75e

Commits on Jul 24, 2024

• Upload test dataset.

  

  Ghislain Fourny

  committed
  866906d

• Fix syntax.

  

  Ghislain Fourny

  committed
  1ab2e36

• Reorder output in test.

  

  Ghislain Fourny

  committed
  7d6cc56

• Fix keys() function on DFs.

  

  Ghislain Fourny

  committed
  2cac215

• Spotless.

  

  Ghislain Fourny

  committed
  7dabc79

Commits on Jul 31, 2024

• Fix test.

  

  Ghislain Fourny

  committed
  dfa8c7b

• Fix mutability.

  

  Ghislain Fourny

  committed
  69d968e

• Remove debug output.

  

  Ghislain Fourny

  committed
  acd5235

• Fix test.

  

  Ghislain Fourny

  committed
  684d588

• Fix tests.

  

  Ghislain Fourny

  committed
  f225718

• Make new arrays mutable.

  

  Ghislain Fourny

  committed
  57c0e2f

• Fix test.

  

  Ghislain Fourny

  committed
  a87ed88

• Fix test.

  

  Ghislain Fourny

  committed
  ff247cf

• Fix syntax.

  

  Ghislain Fourny

  committed
  4b51c0d

Commits on Aug 2, 2024

• Fix test.

  

  Ghislain Fourny

  committed
  3eb58a9

• Allow updating without % and define default annotations namespace.

  

  Ghislain Fourny

  committed
  8d324c9

• Remove an prefix and make default.

  

  Ghislain Fourny

  committed
  58c6c0d

• Fix 3 tests.

  

  Ghislain Fourny

  committed
  afb8696

• Spotless.

  

  Ghislain Fourny

  committed
  9750f37

• Update metadata.

  

  Ghislain Fourny

  committed
  90e0702

• Fix test.

  

  Ghislain Fourny

  committed
  55c60f5

• Fix test.

  

  Ghislain Fourny

  committed
  b07be2d

• Fix test.

  

  Ghislain Fourny

  committed
  3ad52a2

• Add delta tests.

  

  Ghislain Fourny

  committed
  ca9bc04

• Fix delta tests.

  

  Ghislain Fourny

  committed
  c3cb597

• Fix tests.

  

  Ghislain Fourny

  committed
  22b98b3

• Fix column name to rowID.

  

  Ghislain Fourny

  committed
  684dd96

• Revert column names.

  

  Ghislain Fourny

  committed
  dcd15fe

Commits on Aug 23, 2024

• Run delta tests.

  

  Ghislain Fourny

  committed
  e70d2f9

• Fix test.

  

  Ghislain Fourny

  committed
  41053cc

Commits on Sep 18, 2024

• extend SequenceOfItems with functions to apply PUL and determine if materialisation is possible

  David-C-L

  committed
  24ffb46

• add command line arg for updates and adjust execution flow

  David-C-L

  committed
  0da26c1

• adjust tests containing update tests with cli parameter and SeqOfItems api

  David-C-L

  committed
  7912579

• add isSequential and isUpdating methods to program and StatementsWithOptionalExpr iterators

  David-C-L

  committed
  ea53993

• allow treat exprs to be updating

  David-C-L

  committed
  b02e732

Commits on Sep 20, 2024

• Merge pull request #1260 from David-C-L/deltalake-scripting-updates

  Show description for 7ee88c3
  ghislainfourny

  authored
  7ee88c3

• run mvn spotless

  David-C-L

  committed
  4fc8cc6

Commits on Sep 23, 2024

• Merge pull request #1261 from David-C-L/deltalake-scripting-updates

  Show description for 3520ad7
  ghislainfourny

  authored
  3520ad7