Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parquet failure on latest release #469

Open
GJMcClintock opened this issue Dec 19, 2024 · 2 comments
Open

Parquet failure on latest release #469

GJMcClintock opened this issue Dec 19, 2024 · 2 comments

Comments

@GJMcClintock
Copy link

Issue Description

Only env change was upgrading to 1.3.3 - I'm getting this error now any time I try to write to parquet.

  • Description of the issue:
~ failure running replication (see docs @ https://docs.slingdata.io/sling-cli)
--------------------------- schema.stream ---------------------------



~ error writing buffer
panic occurred! reflect: call of reflect.Value.Bool on string Value
goroutine 292 [running]:
runtime/debug.Stack()
        /Users/github/gh-runner/sling-cli/_work/_tool/go/1.22.8/arm64/src/runtime/debug/stack.go:24 +0x64
github.com/slingdata-io/sling-cli/core/dbio/iop.(*ParquetWriter).writeBuffer.func1()
        /Users/github/gh-runner/sling-cli/_work/sling-cli/sling-cli/core/dbio/iop/parquet.go:283 +0x3c
panic({0x106743600?, 0x14001618a08?})
        /Users/github/gh-runner/sling-cli/_work/_tool/go/1.22.8/arm64/src/runtime/panic.go:770 +0x124
reflect.flag.mustBe(...)
        /Users/github/gh-runner/sling-cli/_work/_tool/go/1.22.8/arm64/src/reflect/value.go:233
reflect.Value.panicNotBool({0x106631a80?, 0x14002e86120?, 0x1400153b558?})
        /Users/github/gh-runner/sling-cli/_work/_tool/go/1.22.8/arm64/src/reflect/value.go:302 +0x8c
reflect.Value.Bool(...)
        /Users/github/gh-runner/sling-cli/_work/_tool/go/1.22.8/arm64/src/reflect/value.go:296
github.com/parquet-go/parquet-go.makeValue(0x0, 0x0, {0x10670d1a0?, 0x14000df0b90?, 0x14000df0b90?})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/value.go:287 +0x1c0
github.com/parquet-go/parquet-go.deconstructFuncOfLeaf.func1({0x14000b10808, 0xf8, 0x1400153b6f8?}, {0x80?, 0x1a?, 0x63?}, {0x10670d1a0?, 0x14000df0b90?, 0x1400153b738?})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/row.go:542 +0x74
github.com/parquet-go/parquet-go.deconstructFuncOfOptional.func1({0x14000b10808?, 0xf8?, 0xf8?}, {0xe0?, 0x5?, 0x9b?}, {0x10670d1a0?, 0x14000df0b90?, 0x1099b05b8?})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/row.go:439 +0xdc
github.com/parquet-go/parquet-go.deconstructFuncOfGroup.func1({0x14000b10808, 0xf8, 0xf8}, {0xe0?, 0x8e?, 0xaf?}, {0x106783500?, 0x14001c4c388?, 0x1400153b701?})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/row.go:519 +0x134
github.com/parquet-go/parquet-go.(*Schema).deconstructValueToColumns(0x14000f11ec0, {0x14000b10808, 0xf8, 0xf8}, {0x1065d1d40?, 0x14001c4c388?, 0x700?})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/schema.go:240 +0x120
github.com/parquet-go/parquet-go.(*Schema).Deconstruct(0x14000f11ec0, {0x0, 0x0, 0x0}, {0x1065d1d40, 0x14001c4c388})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/schema.go:228 +0x15c
github.com/parquet-go/parquet-go.(*GenericWriter[...]).writeRows(0x106dc8360, {0x14001c4c388, 0x40, 0x1400153ba18})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/writer.go:234 +0x168
github.com/parquet-go/parquet-go.(*GenericWriter[...]).Write.func1(0x20?)
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/writer.go:178 +0x60
github.com/parquet-go/parquet-go.(*writer).writeRows(0x14001abd760, 0x64, 0x1400153ba78)
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/writer.go:1015 +0xb4
github.com/parquet-go/parquet-go.(*GenericWriter[...]).Write(0x10304f9f8?, {0x14001c4c388?, 0x106783500?, 0x1400343a3c8?})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/writer.go:177 +0x58
github.com/slingdata-io/sling-cli/core/dbio/iop.(*ParquetWriter).writeBuffer(0x14000c99bc0)
        /Users/github/gh-runner/sling-cli/_work/sling-cli/sling-cli/core/dbio/iop/parquet.go:288 +0x74
github.com/slingdata-io/sling-cli/core/dbio/iop.(*ParquetWriter).WriteRec(0x14000c99bc0, {0x140019fe008, 0xf8, 0x0?})
        /Users/github/gh-runner/sling-cli/_work/sling-cli/sling-cli/core/dbio/iop/parquet.go:270 +0x92c
github.com/slingdata-io/sling-cli/core/dbio/iop.(*Datastream).NewParquetReaderChnl.func1()
        /Users/github/gh-runner/sling-cli/_work/sling-cli/sling-cli/core/dbio/iop/datastream.go:2467 +0x218
created by github.com/slingdata-io/sling-cli/core/dbio/iop.(*Datastream).NewParquetReaderChnl in goroutine 291
        /Users/github/gh-runner/sling-cli/_work/sling-cli/sling-cli/core/dbio/iop/datastream.go:2412 +0x234

~ Could not write
~ error writing buffer
panic occurred! reflect: call of reflect.Value.Bool on string Value
goroutine 292 [running]:
runtime/debug.Stack()
        /Users/github/gh-runner/sling-cli/_work/_tool/go/1.22.8/arm64/src/runtime/debug/stack.go:24 +0x64
github.com/slingdata-io/sling-cli/core/dbio/iop.(*ParquetWriter).writeBuffer.func1()
        /Users/github/gh-runner/sling-cli/_work/sling-cli/sling-cli/core/dbio/iop/parquet.go:283 +0x3c
panic({0x106743600?, 0x14001618a08?})
        /Users/github/gh-runner/sling-cli/_work/_tool/go/1.22.8/arm64/src/runtime/panic.go:770 +0x124
reflect.flag.mustBe(...)
        /Users/github/gh-runner/sling-cli/_work/_tool/go/1.22.8/arm64/src/reflect/value.go:233
reflect.Value.panicNotBool({0x106631a80?, 0x14002e86120?, 0x1400153b558?})
        /Users/github/gh-runner/sling-cli/_work/_tool/go/1.22.8/arm64/src/reflect/value.go:302 +0x8c
reflect.Value.Bool(...)
        /Users/github/gh-runner/sling-cli/_work/_tool/go/1.22.8/arm64/src/reflect/value.go:296
github.com/parquet-go/parquet-go.makeValue(0x0, 0x0, {0x10670d1a0?, 0x14000df0b90?, 0x14000df0b90?})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/value.go:287 +0x1c0
github.com/parquet-go/parquet-go.deconstructFuncOfLeaf.func1({0x14000b10808, 0xf8, 0x1400153b6f8?}, {0x80?, 0x1a?, 0x63?}, {0x10670d1a0?, 0x14000df0b90?, 0x1400153b738?})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/row.go:542 +0x74
github.com/parquet-go/parquet-go.deconstructFuncOfOptional.func1({0x14000b10808?, 0xf8?, 0xf8?}, {0xe0?, 0x5?, 0x9b?}, {0x10670d1a0?, 0x14000df0b90?, 0x1099b05b8?})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/row.go:439 +0xdc
github.com/parquet-go/parquet-go.deconstructFuncOfGroup.func1({0x14000b10808, 0xf8, 0xf8}, {0xe0?, 0x8e?, 0xaf?}, {0x106783500?, 0x14001c4c388?, 0x1400153b701?})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/row.go:519 +0x134
github.com/parquet-go/parquet-go.(*Schema).deconstructValueToColumns(0x14000f11ec0, {0x14000b10808, 0xf8, 0xf8}, {0x1065d1d40?, 0x14001c4c388?, 0x700?})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/schema.go:240 +0x120
github.com/parquet-go/parquet-go.(*Schema).Deconstruct(0x14000f11ec0, {0x0, 0x0, 0x0}, {0x1065d1d40, 0x14001c4c388})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/schema.go:228 +0x15c
github.com/parquet-go/parquet-go.(*GenericWriter[...]).writeRows(0x106dc8360, {0x14001c4c388, 0x40, 0x1400153ba18})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/writer.go:234 +0x168
github.com/parquet-go/parquet-go.(*GenericWriter[...]).Write.func1(0x20?)
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/writer.go:178 +0x60
github.com/parquet-go/parquet-go.(*writer).writeRows(0x14001abd760, 0x64, 0x1400153ba78)
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/writer.go:1015 +0xb4
github.com/parquet-go/parquet-go.(*GenericWriter[...]).Write(0x10304f9f8?, {0x14001c4c388?, 0x106783500?, 0x1400343a3c8?})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/writer.go:177 +0x58
github.com/slingdata-io/sling-cli/core/dbio/iop.(*ParquetWriter).writeBuffer(0x14000c99bc0)
        /Users/github/gh-runner/sling-cli/_work/sling-cli/sling-cli/core/dbio/iop/parquet.go:288 +0x74
github.com/slingdata-io/sling-cli/core/dbio/iop.(*ParquetWriter).WriteRec(0x14000c99bc0, {0x140019fe008, 0xf8, 0x0?})
        /Users/github/gh-runner/sling-cli/_work/sling-cli/sling-cli/core/dbio/iop/parquet.go:270 +0x92c
github.com/slingdata-io/sling-cli/core/dbio/iop.(*Datastream).NewParquetReaderChnl.func1()
        /Users/github/gh-runner/sling-cli/_work/sling-cli/sling-cli/core/dbio/iop/datastream.go:2467 +0x218
created by github.com/slingdata-io/sling-cli/core/dbio/iop.(*Datastream).NewParquetReaderChnl in goroutine 291
        /Users/github/gh-runner/sling-cli/_work/sling-cli/sling-cli/core/dbio/iop/datastream.go:2412 +0x234


~ error writing buffer
panic occurred! reflect: call of reflect.Value.Bool on string Value
goroutine 292 [running]:
runtime/debug.Stack()
        /Users/github/gh-runner/sling-cli/_work/_tool/go/1.22.8/arm64/src/runtime/debug/stack.go:24 +0x64
github.com/slingdata-io/sling-cli/core/dbio/iop.(*ParquetWriter).writeBuffer.func1()
        /Users/github/gh-runner/sling-cli/_work/sling-cli/sling-cli/core/dbio/iop/parquet.go:283 +0x3c
panic({0x106743600?, 0x14001618a08?})
        /Users/github/gh-runner/sling-cli/_work/_tool/go/1.22.8/arm64/src/runtime/panic.go:770 +0x124
reflect.flag.mustBe(...)
        /Users/github/gh-runner/sling-cli/_work/_tool/go/1.22.8/arm64/src/reflect/value.go:233
reflect.Value.panicNotBool({0x106631a80?, 0x14002e86120?, 0x1400153b558?})
        /Users/github/gh-runner/sling-cli/_work/_tool/go/1.22.8/arm64/src/reflect/value.go:302 +0x8c
reflect.Value.Bool(...)
        /Users/github/gh-runner/sling-cli/_work/_tool/go/1.22.8/arm64/src/reflect/value.go:296
github.com/parquet-go/parquet-go.makeValue(0x0, 0x0, {0x10670d1a0?, 0x14000df0b90?, 0x14000df0b90?})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/value.go:287 +0x1c0
github.com/parquet-go/parquet-go.deconstructFuncOfLeaf.func1({0x14000b10808, 0xf8, 0x1400153b6f8?}, {0x80?, 0x1a?, 0x63?}, {0x10670d1a0?, 0x14000df0b90?, 0x1400153b738?})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/row.go:542 +0x74
github.com/parquet-go/parquet-go.deconstructFuncOfOptional.func1({0x14000b10808?, 0xf8?, 0xf8?}, {0xe0?, 0x5?, 0x9b?}, {0x10670d1a0?, 0x14000df0b90?, 0x1099b05b8?})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/row.go:439 +0xdc
github.com/parquet-go/parquet-go.deconstructFuncOfGroup.func1({0x14000b10808, 0xf8, 0xf8}, {0xe0?, 0x8e?, 0xaf?}, {0x106783500?, 0x14001c4c388?, 0x1400153b701?})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/row.go:519 +0x134
github.com/parquet-go/parquet-go.(*Schema).deconstructValueToColumns(0x14000f11ec0, {0x14000b10808, 0xf8, 0xf8}, {0x1065d1d40?, 0x14001c4c388?, 0x700?})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/schema.go:240 +0x120
github.com/parquet-go/parquet-go.(*Schema).Deconstruct(0x14000f11ec0, {0x0, 0x0, 0x0}, {0x1065d1d40, 0x14001c4c388})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/schema.go:228 +0x15c
github.com/parquet-go/parquet-go.(*GenericWriter[...]).writeRows(0x106dc8360, {0x14001c4c388, 0x40, 0x1400153ba18})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/writer.go:234 +0x168
github.com/parquet-go/parquet-go.(*GenericWriter[...]).Write.func1(0x20?)
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/writer.go:178 +0x60
github.com/parquet-go/parquet-go.(*writer).writeRows(0x14001abd760, 0x64, 0x1400153ba78)
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/writer.go:1015 +0xb4
github.com/parquet-go/parquet-go.(*GenericWriter[...]).Write(0x10304f9f8?, {0x14001c4c388?, 0x106783500?, 0x1400343a3c8?})
        /Users/fritz/go/pkg/mod/github.com/parquet-go/[email protected]/writer.go:177 +0x58
github.com/slingdata-io/sling-cli/core/dbio/iop.(*ParquetWriter).writeBuffer(0x14000c99bc0)
        /Users/github/gh-runner/sling-cli/_work/sling-cli/sling-cli/core/dbio/iop/parquet.go:288 +0x74
github.com/slingdata-io/sling-cli/core/dbio/iop.(*ParquetWriter).WriteRec(0x14000c99bc0, {0x140019fe008, 0xf8, 0x0?})
        /Users/github/gh-runner/sling-cli/_work/sling-cli/sling-cli/core/dbio/iop/parquet.go:270 +0x92c
github.com/slingdata-io/sling-cli/core/dbio/iop.(*Datastream).NewParquetReaderChnl.func1()
        /Users/github/gh-runner/sling-cli/_work/sling-cli/sling-cli/core/dbio/iop/datastream.go:2467 +0x218
created by github.com/slingdata-io/sling-cli/core/dbio/iop.(*Datastream).NewParquetReaderChnl in goroutine 291
        /Users/github/gh-runner/sling-cli/_work/sling-cli/sling-cli/core/dbio/iop/datastream.go:2412 +0x234```
- Sling version (`sling --version`): 

- Operating System (`linux`, `mac`, `windows`): 
mac
- Replication Configuration: 

```yaml
source: <source>
target: <target>
streams:

defaults:
object: {stream_schema}/{stream_table}/*.parquet
file_format: parquet
  ...
  • Log Output (please run command with -d):
Paste log here.
@flarco
Copy link
Collaborator

flarco commented Dec 19, 2024

That's strange, parquet logic wasn't changed.

  1. What version were you using prior?
  2. What is the source connection type?

@GJMcClintock
Copy link
Author

GJMcClintock commented Dec 19, 2024 via email

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@flarco @GJMcClintock