Skip to content

GH-3482: parquet-hadoop tests to work behind a web proxy#3483

Open
alexeyroytman wants to merge 6 commits intoapache:masterfrom
alexeyroytman:master
Open

GH-3482: parquet-hadoop tests to work behind a web proxy#3483
alexeyroytman wants to merge 6 commits intoapache:masterfrom
alexeyroytman:master

Conversation

@alexeyroytman
Copy link
Copy Markdown

@alexeyroytman alexeyroytman commented Apr 16, 2026

Rationale for this change

Allow parquet-hadoop tests to work behind a web proxy. Otherwise their failures stop some other jar files from being build.

What changes are included in this PR?

Analysis of system properties parquet.https.proxyHost and parquet.https.proxyPort, and if set, use a Proxy for OkHttpClient creation.
We use a new different JVM property set and not a widely used one, because CI may define JVM properties https.proxyHost and https.proxyPort and that proxy won't support some compressions (e.g. gzip/snappy on github.com CI).

Are these changes tested?

The ./mvn clean build now passes when I'm behind a web proxy.

Are there any user-facing changes?

No.

Copy link
Copy Markdown
Contributor

@Fokko Fokko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @alexeyroytman This seems reasonable to add

@alexeyroytman
Copy link
Copy Markdown
Author

alexeyroytman commented Apr 16, 2026

It looks like that on CI Hadoop the JVM properties https.proxyHost and https.proxyPort are set (I don't see their values, but I suppose they are set), and this specific proxy does not allow good gzip/snappy work.
Strangely, for JDK 11 tests passed, but for JDK 17 they failed...

I'll try to work on my environment with OpenJDK 17, and if it passes, I'm planning to change the code to use different JVM properties, e.g. parquet.https.proxyHost and parquet.https.proxyPort. Thus they won't be occasionally defined for CI. And occasionally not working for certain compression method.

@alexeyroytman
Copy link
Copy Markdown
Author

BTW, how can I find out which CI job has JVM properties https.proxyHost and https.proxyPort set ?

@alexeyroytman
Copy link
Copy Markdown
Author

Or shall we rename the properties to something referring to github.com ?.. I mean, it's parquet, hadoop, tests/junit, and for https://github.com ...

@alexeyroytman
Copy link
Copy Markdown
Author

I believe it shall pass CI now.

@alexeyroytman
Copy link
Copy Markdown
Author

@Fokko, may I ask you to try to run the CI again ?

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR aims to make parquet-hadoop test downloads from github.com/apache/parquet-testing work in environments that require an HTTPS proxy by creating an OkHttpClient that can be configured with a proxy via JVM properties.

Changes:

  • Added a helper in InterOpTester to build an OkHttpClient optionally configured with a proxy.
  • Updated InterOpTester to use the proxy-aware client for parquet-testing downloads.
  • Updated TestInteropBloomFilter to reuse the same proxy-aware client creation.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File Description
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestInteropBloomFilter.java Uses the new shared proxy-aware OkHttpClient factory instead of creating a direct client.
parquet-hadoop/src/test/java/org/apache/parquet/hadoop/InterOpTester.java Introduces proxy-aware OkHttpClient creation and switches interop download logic to use it.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

alexeyroytman and others added 2 commits May 6, 2026 15:19
offered by Copilot

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants