GH-3482: parquet-hadoop tests to work behind a web proxy#3483
GH-3482: parquet-hadoop tests to work behind a web proxy#3483alexeyroytman wants to merge 6 commits intoapache:masterfrom
Conversation
Fokko
left a comment
There was a problem hiding this comment.
Thanks @alexeyroytman This seems reasonable to add
|
It looks like that on CI Hadoop the JVM properties I'll try to work on my environment with OpenJDK 17, and if it passes, I'm planning to change the code to use different JVM properties, e.g. |
|
BTW, how can I find out which CI job has JVM properties |
|
Or shall we rename the properties to something referring to github.com ?.. I mean, it's |
|
I believe it shall pass CI now. |
|
@Fokko, may I ask you to try to run the CI again ? |
There was a problem hiding this comment.
Pull request overview
This PR aims to make parquet-hadoop test downloads from github.com/apache/parquet-testing work in environments that require an HTTPS proxy by creating an OkHttpClient that can be configured with a proxy via JVM properties.
Changes:
- Added a helper in
InterOpTesterto build anOkHttpClientoptionally configured with a proxy. - Updated
InterOpTesterto use the proxy-aware client for parquet-testing downloads. - Updated
TestInteropBloomFilterto reuse the same proxy-aware client creation.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| parquet-hadoop/src/test/java/org/apache/parquet/hadoop/TestInteropBloomFilter.java | Uses the new shared proxy-aware OkHttpClient factory instead of creating a direct client. |
| parquet-hadoop/src/test/java/org/apache/parquet/hadoop/InterOpTester.java | Introduces proxy-aware OkHttpClient creation and switches interop download logic to use it. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
offered by Copilot Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Rationale for this change
Allow
parquet-hadooptests to work behind a web proxy. Otherwise their failures stop some other jar files from being build.What changes are included in this PR?
Analysis of system properties
parquet.https.proxyHostandparquet.https.proxyPort, and if set, use aProxyforOkHttpClientcreation.We use a new different JVM property set and not a widely used one, because CI may define JVM properties
https.proxyHostandhttps.proxyPortand that proxy won't support some compressions (e.g. gzip/snappy on github.com CI).Are these changes tested?
The
./mvn clean buildnow passes when I'm behind a web proxy.Are there any user-facing changes?
No.