Support non-UTF-8 encoded CSV files by Rafferty97 · Pull Request #20626 · apache/datafusion

Rafferty97 · 2026-03-01T05:30:09Z

Which issue does this PR close?

Closes #20473

Rationale for this change

CSV is a ubiquitous file format, and many are encoded in Windows-1252 and other encodings. It would be useful to have the option to read them in datafusion.

What changes are included in this PR?

Adds a configuration option to the CSV reader to specify an encoding
Adds an optional dependency on encoding_rs to do the actual decoding
Refactored CsvSource somewhat to aid the implementation
Removed the return value from DecoderDeserializer::digest as it was misleading (call sites were ignoring it)

Are these changes tested?

I have added one unit test that attempts to read a SHIFT-JIS encoded CSV file. More tests are probably needed, but I may need some guidance on this. I'm also running into issues getting the test suite to run locally on my Windows machine.

Are there any user-facing changes?

Adds a new field to CsvOptions.

Rafferty97 · 2026-03-18T02:36:59Z

@alamb Hey, I think this would be a useful addition to Datafusion, as CSV files in non-UTF-8 encodings are quite common. I'm wondering if there's anything I can do to make review easier?

alamb · 2026-03-18T19:25:16Z

@alamb Hey, I think this would be a useful addition to Datafusion, as CSV files in non-UTF-8 encodings are quite common. I'm wondering if there's anything I can do to make review easier?

I just need to find time (or we need to find some other reviewers)

See

[DISCUSSION] PR Review Process / States / "How to find the PRs in the list that need review" #21038

github-actions · 2026-05-18T02:18:20Z

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.

code slightly

make schema inference charset-aware

Rafferty97 · 2026-05-18T11:02:41Z

bump

github-actions · 2026-05-18T12:45:24Z

Thank you for opening this pull request!

Reviewer note: cargo-semver-checks reported the current version number is not SemVer-compatible with the changes in this pull request (compared against the base branch).

Details

     Cloning apache/main
    Building datafusion v53.1.0 (current)
       Built [ 101.454s] (current)
     Parsing datafusion v53.1.0 (current)
      Parsed [   0.038s] (current)
    Building datafusion v53.1.0 (baseline)
       Built [  96.137s] (baseline)
     Parsing datafusion v53.1.0 (baseline)
      Parsed [   0.037s] (baseline)
    Checking datafusion v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.985s] 222 checks: 221 pass, 1 fail, 0 warn, 30 skip

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field CsvReadOptions.charset in /home/runner/work/datafusion/datafusion/datafusion/core/src/datasource/file_format/options.rs:89
  field CsvReadOptions.charset in /home/runner/work/datafusion/datafusion/datafusion/core/src/datasource/file_format/options.rs:89
  field CsvReadOptions.charset in /home/runner/work/datafusion/datafusion/datafusion/core/src/datasource/file_format/options.rs:89

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [ 201.677s] datafusion
    Building datafusion-common v53.1.0 (current)
       Built [  31.585s] (current)
     Parsing datafusion-common v53.1.0 (current)
      Parsed [   0.059s] (current)
    Building datafusion-common v53.1.0 (baseline)
       Built [  31.638s] (baseline)
     Parsing datafusion-common v53.1.0 (baseline)
      Parsed [   0.063s] (baseline)
    Checking datafusion-common v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.948s] 222 checks: 221 pass, 1 fail, 0 warn, 30 skip

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field CsvOptions.charset in /home/runner/work/datafusion/datafusion/datafusion/common/src/config.rs:3163

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  65.672s] datafusion-common
    Building datafusion-datasource v53.1.0 (current)
       Built [  34.621s] (current)
     Parsing datafusion-datasource v53.1.0 (current)
      Parsed [   0.032s] (current)
    Building datafusion-datasource v53.1.0 (baseline)
       Built [  34.662s] (baseline)
     Parsing datafusion-datasource v53.1.0 (baseline)
      Parsed [   0.034s] (baseline)
    Checking datafusion-datasource v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.360s] 222 checks: 221 pass, 1 fail, 0 warn, 30 skip

--- failure trait_method_now_returns_unit: pub trait method now returns unit ---

Description:
A trait method that used to return a value now returns `()`.
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/trait_method_now_returns_unit.ron
Failed in:
  datafusion_datasource::decoder::BatchDeserializer::digest in /home/runner/work/datafusion/datafusion/datafusion/datasource/src/decoder.rs:42

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  71.038s] datafusion-datasource
    Building datafusion-datasource-csv v53.1.0 (current)
       Built [  34.942s] (current)
     Parsing datafusion-datasource-csv v53.1.0 (current)
      Parsed [   0.013s] (current)
    Building datafusion-datasource-csv v53.1.0 (baseline)
       Built [  34.928s] (baseline)
     Parsing datafusion-datasource-csv v53.1.0 (baseline)
      Parsed [   0.012s] (baseline)
    Checking datafusion-datasource-csv v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   0.146s] 222 checks: 222 pass, 30 skip
     Summary no semver update required
    Finished [  71.735s] datafusion-datasource-csv
    Building datafusion-proto v53.1.0 (current)
       Built [  51.931s] (current)
     Parsing datafusion-proto v53.1.0 (current)
      Parsed [   0.141s] (current)
    Building datafusion-proto v53.1.0 (baseline)
       Built [  51.892s] (baseline)
     Parsing datafusion-proto v53.1.0 (baseline)
      Parsed [   0.144s] (baseline)
    Checking datafusion-proto v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   2.337s] 222 checks: 221 pass, 1 fail, 0 warn, 30 skip

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field CsvOptions.charset in /home/runner/work/datafusion/datafusion/datafusion/proto/src/generated/datafusion_proto_common.rs:703
  field CsvOptions.charset in /home/runner/work/datafusion/datafusion/datafusion/proto/src/generated/datafusion_proto_common.rs:703

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [ 109.012s] datafusion-proto
    Building datafusion-proto-common v53.1.0 (current)
       Built [  20.047s] (current)
     Parsing datafusion-proto-common v53.1.0 (current)
      Parsed [   0.045s] (current)
    Building datafusion-proto-common v53.1.0 (baseline)
       Built [  20.161s] (baseline)
     Parsing datafusion-proto-common v53.1.0 (baseline)
      Parsed [   0.049s] (baseline)
    Checking datafusion-proto-common v53.1.0 -> v53.1.0 (no change; assume patch)
     Checked [   1.516s] 222 checks: 221 pass, 1 fail, 0 warn, 30 skip

--- failure constructible_struct_adds_field: externally-constructible struct adds field ---

Description:
A pub struct constructible with a struct literal has a new pub field. Existing struct literals must be updated to include the new field.
        ref: https://doc.rust-lang.org/reference/expressions/struct-expr.html
       impl: https://github.com/obi1kenobi/cargo-semver-checks/tree/v0.47.0/src/lints/constructible_struct_adds_field.ron

Failed in:
  field CsvOptions.charset in /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/generated/prost.rs:703
  field CsvOptions.charset in /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/generated/prost.rs:703
  field CsvOptions.charset in /home/runner/work/datafusion/datafusion/datafusion/proto-common/src/generated/prost.rs:703

     Summary semver requires new major version: 1 major and 0 minor checks failed
    Finished [  43.012s] datafusion-proto-common

github-actions Bot added core Core DataFusion crate common Related to common crate proto Related to proto crate datasource Changes to the datasource crate documentation Improvements or additions to documentation labels Mar 1, 2026

Rafferty97 force-pushed the non-utf8-csv2 branch 4 times, most recently from 5bab23f to 09a2357 Compare March 5, 2026 00:24

github-actions Bot added the Stale PR has not had any activity for some time label May 18, 2026

Rafferty97 added 10 commits May 18, 2026 20:50

Add "encoding" option to CsvOptions and protobuf schemas

944bcd9

Implement charset decoding in the CSV reader, and refactor surrounding

c8b35bb

code slightly

Fix clippy issues

124f0fc

Add SHIFT-JIS test

02b2c6e

Add license header

b0926ac

Rename "encoding" to "charset" for clarity, add in missing functions,

442ac2b

make schema inference charset-aware

Factor out CharsetDecoder, fix bugs, add feature flag to README

e3d6c8c

Fix module name

34a0417

Fix more naming inconsistencies

b9f3202

Fix clippy issue

902d7e0

Rafferty97 force-pushed the non-utf8-csv2 branch from 09a2357 to 902d7e0 Compare May 18, 2026 11:00

github-actions Bot added the auto detected api change Auto detected API change label May 18, 2026

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support non-UTF-8 encoded CSV files#20626

Support non-UTF-8 encoded CSV files#20626
Rafferty97 wants to merge 10 commits into
apache:mainfrom
Rafferty97:non-utf8-csv2

Rafferty97 commented Mar 1, 2026

Uh oh!

Rafferty97 commented Mar 18, 2026

Uh oh!

alamb commented Mar 18, 2026

Uh oh!

github-actions Bot commented May 18, 2026

Uh oh!

Rafferty97 commented May 18, 2026

Uh oh!

github-actions Bot commented May 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

Rafferty97 commented Mar 1, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

Rafferty97 commented Mar 18, 2026

Uh oh!

alamb commented Mar 18, 2026

Uh oh!

github-actions Bot commented May 18, 2026

Uh oh!

Rafferty97 commented May 18, 2026

Uh oh!

github-actions Bot commented May 18, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants