fix: chunk large DataFrames in register to avoid duckdb-rs panic

[cpsievert]

Summary

DuckDBReader::register panics on any DataFrame with more than 2048 rows. This affects all callers — including the Python bindings (PyDuckDBReader.register) — making it impossible to register moderately-sized datasets.

Root cause

dataframe_to_arrow_params concatenates all Arrow IPC batches into a single RecordBatch, then passes it to duckdb-rs's arrow() table function. That function's ArrowVTab::func writes the entire RecordBatch into a single DuckDB DataChunk in one call — but DataChunk vectors have a fixed capacity of STANDARD_VECTOR_SIZE (2048). When the RecordBatch has more rows than that, FlatVector::copy hits assert!(data.len() <= self.capacity()) and the process aborts.

Fix

Chunk DataFrames larger than 2048 rows before passing them to dataframe_to_arrow_params. The first chunk creates the table (CREATE TEMP TABLE ... AS SELECT * FROM arrow(?, ?)), and subsequent chunks insert into it (INSERT INTO ... SELECT * FROM arrow(?, ?)). Small DataFrames (≤2048 rows) take the existing single-batch path with no overhead.

This is a workaround for a bug in duckdb-rs's ArrowVTab implementation rather than a fundamental limitation of DuckDB itself.

Test plan

• Added test_register_large_dataframe (3000 rows with int, float, and string columns — verifies row count and data integrity at chunk boundaries)
• All existing register tests continue to pass

🤖 Generated with Claude Code

[georgestagg]

Great spot, thanks!

This looks like it would work, but before we start managing the chunking ourselves can we look into the appender API for an alternative to loading the arrow datasets into duckdb. I think it's gated behind the appender-arrow feature, rather than vtab-arrow.

It looks like there was a fix in the appender API to support chunking for large arrow datasets, so if we can use that we'd get the chunking without having to manage it ourselves.

If it is no good, or it ends up being to complex a change to switch, we'll stick with this method.

[georgestagg]

Oh it's a shame we have to do this mapping ourselves...

Bah, both solutions have drawbacks.

Which did you prefer between chunking the data ourselves or this appender API, if you have a preference?

[cpsievert]

Yeah, I think I prefer the original approach in hindsight. 4cb2c4c reverts to that approach and
4f63f69 explains in more detail where the constant is coming from.