[Go][Parquet] slice bounds out of range - panic in writeDenseArrow

[thoro]

Describe the bug, including details regarding any error messages, version, and platform.

I'm getting this error when writing large amounts of strings:

> panic: runtime error: slice bounds out of range [:-2147360192]

  goroutine 186374 [running]:
  github.com/apache/arrow/go/v17/parquet/pqarrow.writeDenseArrow(0xddae04a9b0, {0x20e96a8, 0xc4ccf9ef00}, {0x20e5320, 0xc4d1a6e5f0}, {0x0, 0x0, 0x0},
  {0x0, 0x0, ...}, ...)
      /go/pkg/mod/github.com/apache/arrow/go/v17@v17.0.0/parquet/pqarrow/encode_arrow.go:491 +0x2d2d
  github.com/apache/arrow/go/v17/parquet/pqarrow.WriteArrowToColumn({0x20c7298, 0xdeb4627020}, {0x20e96a8, 0xc4ccf9ef00}, {0x20e5320, 0xc4d1a6e5f0},
  {0x0, 0x0, 0x0}, {0x0, ...}, ...)
      /go/pkg/mod/github.com/apache/arrow/go/v17@v17.0.0/parquet/pqarrow/encode_arrow.go:232 +0x325
  github.com/apache/arrow/go/v17/parquet/pqarrow.(*arrowColumnWriter).Write(0xc4fb951710, {0x20c7298, 0xdeb4627020})
      /go/pkg/mod/github.com/apache/arrow/go/v17@v17.0.0/parquet/pqarrow/encode_arrow.go:193 +0x533
  github.com/apache/arrow/go/v17/parquet/pqarrow.(*FileWriter).WriteColumnChunked(0xdeb4634a10, 0x31b3220?, 0xc4fb9517d8?, 0x1?)
      /go/pkg/mod/github.com/apache/arrow/go/v17@v17.0.0/parquet/pqarrow/file_writer.go:330 +0xe5
  github.com/apache/arrow/go/v17/parquet/pqarrow.(*FileWriter).WriteColumnData(0xdeb4634a10, {0x20e5320, 0xc1c0963360})
      /go/pkg/mod/github.com/apache/arrow/go/v17@v17.0.0/parquet/pqarrow/file_writer.go:339 +0xc5
  github.com/apache/arrow/go/v17/parquet/pqarrow.(*FileWriter).Write(0xdeb4634a10, {0x20e1c20, 0xc4d1a16060})
      /go/pkg/mod/github.com/apache/arrow/go/v17@v17.0.0/parquet/pqarrow/file_writer.go:242 +0x272

haven't exactly pinpointed what my data looks like at that moment, but this shouldn't happen regardless.

Component(s)

Go

[qiushido]

我也遇到了这个问题，构建arrow record时使用多个large string 和 binary 类型字段，我分析调试我的程序试了很久，依旧panic，直到我把写入parquet的行数从10000改成了5000，这个panic就消失了

[zeroshade]

This was fixed in v18.4.0 or around there, the issue filed is from v17. Gonna close this for now. Feel free to reopen if you can reproduce on the latest version

[qiushido]

This was fixed in v18.4.0 or around there, the issue filed is from v17. Gonna close this for now. Feel free to reopen if you can reproduce on the latest version

当修改为v18.4.0以后，我以为这个问题解决了，但是我又遇到了类似的问题

[zeroshade]

@qiushido do you have a reproducer?

[qiushido]

@qiushido do you have a reproducer?

I am not entirely sure I understand the specific point you are raising. My knowledge of Apache Arrow isn't deeply specialized, but I can describe my use case for more context.
Since Parquet requires a predefined schema for writing, I used a workaround for my data export tasks to avoid manually defining a schema for every single job. I consume messages from Kafka, parse them into map[string]any, and use reflection to map values to a few fixed Arrow-supported types:

• int64
• float64
• string -> largeString
• []byte -> largeBinary
• Other types outside this range -> json.Marshal -> largeString

then use Arrow to write this data to Parquet. During this process, I’ve noticed that if a message is very large (tens or hundreds of MBs)—which I assume means a specific field value is extremely long—writing multiple rows into a single Parquet file triggers this panic.
To circumvent this, I currently have to reduce the number of rows per Parquet file (e.g., from 1,000 rows down to 100 rows per file).

I consulted Gemini, and it suggested adding the configuration parquet.WithDictionaryDefault(false) to disable dictionary encoding and prevent index overflow with large datasets. However, this doesn't seem to have resolved the issue.

Additionally, I encountered the error panic: bytes.Buffer.Grow: negative count. This happened in a similar data export task, and I also resolved it by reducing the number of rows per Parquet file. Currently, with the row count limited to 100 per file and ZSTD (level 1) compression enabled, the resulting Parquet files are generally under 500MB.

I am not sure what the best practices are for Parquet write configurations; if there are recommended settings or parameters to handle this more gracefully, please let me know. Thank you very much!

[zeroshade]

I am not entirely sure I understand the specific point you are raising. My knowledge of Apache Arrow isn't deeply specialized, but I can describe my use case for more context.

Could you share any minimum amount of code and synthetic data so that I could attempt to trigger the error myself locally?

During this process, I’ve noticed that if a message is very large (tens or hundreds of MBs)—which I assume means a specific field value is extremely long—writing multiple rows into a single Parquet file triggers this panic.

Are you able to share one of these messages?

Other types outside this range -> json.Marshal -> largeString

Perhaps use variant?

Your use case seems normal and nothing out of the ordinary, so I'm surprised you're hitting this issue. I'd like to try to recreate the error myself so I can try to debug it. If you could share sample code and data that would be very helpful.

[qiushido]

Could you share any minimum amount of code and synthetic data so that I could attempt to trigger the error myself locally?

I can share my code as it does not contain any private user data, but please understand that I am unable to share the actual dataset. In my observation, the issue isn't necessarily triggered by a specific record, but rather occurs when a certain volume of data accumulates. Therefore, I'm not entirely sure which subset of data would be most helpful to export for reproduction, but I hope this code provides enough context to be useful.
arrow2parquet.go

[zeroshade]

I dug into this a bit more @qiushido, can you confirm for me that when this issue occurs, it corresponds with the uncompressed data for a single string column exceeding 2GB in a batch that you write? If so, then it would correspond with what I've found and I might have a way forward for you.

It looks like the primary issue is that the Parquet Spec doesn't allow for a DataPage's uncompressed size to be larger than what can fit in an int32 (i.e. ~2GB). That said, you wouldn't want a single page to be that large anyways..... but if you're writing, say 50 rows that each have a 50MB string, you'd easily exceed that. In the current implementation of arrow-go's Parquet writer, we only check the size of the page after a batch is written. Which means if you write multiple rows at once as a batch, and the total uncompressed size of that single batch is larger than 2GB we'll just fail rather than properly split the values into separate pages, which is the bug.

While this is something I'll need to fix regardless, I want to confirm that this is the source of the issue you're running into by confirming the size of the data you're writing. Can you confirm this for me?

[qiushido]

@zeroshade Thank you for finding the problem and deciding to fix it. I think your analysis is the root cause of the problem, because my panic usually occurs when a certain field is relatively large. I will try to reproduce it, which may take some time.