GH-36411: [C++][Python] Use meson-python for PyArrow build system #45854
Conversation
|
|
python/meson.build
Outdated
There was a problem hiding this comment.
Before doing this, include the following code:
# no-op placeholder
arrow_dep = dependency('', required: false)
if get_option('wrap_mode') != 'forcefallback'
arrow_dep = dependency('arrow', 'Arrow', modules: ['Arrow::arrow_shared'], required: false)
endifAnd then shift the rest to look like this:
if not arrow_dep.found()
cmake = import('cmake')
# further lookups
# ...
arrow_dep = arrow_proj.dependency('arrow_shared')
endifThere was a problem hiding this comment.
What this does:
- check build options for wrap_mode, which is a builtin meson option allowing you to choose whether you wish to resolve bundled dependencies or look for system dependencies. It defaults to finding system dependencies, but when users run meson with
--wrap-mode=forcefallbackthey are asking to explicitly avoid system deps - first try to find an arrow dependency, using both names it might be available as:
- "arrow" (pkgconfig)
- "Arrow" (cmake, yes capitalization does matter), with
modules:ensuring we pick up the correct cmake find_package() variable
- if it is not available,
required: falsemeans we continue to import the cmake subproject as a fallback
There was a problem hiding this comment.
It is possible to avoid doing if/else checks:
$ cat subprojects/arrow.wrap
[wrap-file]
directory = arrow
method = cmake
[provide]
arrow = arrow_static_dep
However, using wrap files with method=cmake doesn't (currently) allow you to pass your add_cmake_defines. If you didn't need any defines, then you could simply do this:
arrow_dep = dependency('arrow', 'Arrow', modules: ['Arrow::arrow_shared'])and you would not need any if/else, it would automatically build the cmake subproject if either:
- wrap-mode=forcefallback
- no system arrow was found
There was a problem hiding this comment.
How does that wrap file work? The directory to the cpp source is in arrow/cpp whereas the wrap file itself will be located in arrow/python/subprojects - how would that resolve to the right directory?
There was a problem hiding this comment.
Since you included a symlink anyway, I chose not to bother including any mechanism for downloading the wrap contents. Meson skips over that because the directory already exists with the correct content.
The key benefit of the wrap file is that it allows specifying in ini syntax:
- the subproject should use the
method=cmakeautomatically, when used viadependency() - the autogenerated
arrow_static_dep(maybe this should bearrow_shared_depinstead?) will fulfilldependency('arrow')
Again, it's missing the necessary cmake defines so it may not be worth pursuing further.
WillAyd
force-pushed
the
use-meson-python
branch
2 times, most recently
from
cf5b610 to
b902e1d
Compare
github-actions
bot
added
awaiting committer review
WillAyd
force-pushed
the
use-meson-python
branch
6 times, most recently
from
eabf11f to
7be3f7b
Compare
python/pyproject.toml
Outdated
There was a problem hiding this comment.
You could still use setuptools-scm with meson if you want.
project(
'pyarrow',
# ....,
version: run_command('python3', '-m', 'setuptools_scm', '--force-write-version-files', check: true).stdout().strip(),
)
WillAyd
force-pushed
the
use-meson-python
branch
2 times, most recently
from
05ff60d to
6e0a5fe
Compare
|
@kou I have made some offline progress on this, but one of the things I am getting stuck on is how the pyarrow C++ modules are being compiled. From what I understand, the current build process will compile Cython modules first (at least lib.pyx) and from that auto-generate Assuming that understanding is correct, where in the process are lib.h and lib_api.h being generated? I found the CMake command that copies them from the source to the build folder, but I can't figure out where they come from in the first place. Any guidance would be appreciated. |
|
The following codes may be related: arrow/cpp/cmake_modules/UseCython.cmake Lines 120 to 126 in 5e9fce4 Line 674 in 5e9fce4 |
|
Ah nevermind I think I have figured it out. So it looks like Cython generates the header files in the build directory when compiling lib.pyx, so the idea is to copy those header files to a directory structure in the build directory that the sources can resolve to. I'll have to think about the best way to accomplish that via Meson. |
WillAyd
force-pushed
the
use-meson-python
branch
2 times, most recently
from
d67b903 to
ba8b276
Compare
WillAyd
force-pushed
the
use-meson-python
branch
6 times, most recently
from
a2d07ad to
4ff818e
Compare
|
Wow thanks for finding that! Feel free to push directly - thanks @raulcd ! |
github-actions
bot
added
awaiting change review
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
This comment was marked as outdated.
github-actions
bot
added
awaiting changes
raulcd
force-pushed
the
use-meson-python
branch
from
ce85d06 to
ddca98c
Compare
github-actions
bot
added
awaiting change review
raulcd
force-pushed
the
use-meson-python
branch
from
ddca98c to
da9453d
Compare
|
I have been able to fix the sdist failures and all the wheels applying the changes on this commit: I think we can apply the commit to this branch, validate CI and continue with review discussions. Thanks @WillAyd for your help during the process of fixing all the issues that we've found during the last couple of weeks! |
raulcd
force-pushed
the
use-meson-python
branch
from
d853855 to
b5ee816
Compare
|
@github-actions crossbow submit -g python -g wheel |
|
Revision: b5ee816 Submitted crossbow builds: ursacomputing/crossbow @ actions-616f6b7be9
|
|
From a quick glance at some of the crossbow failures, looks like we are missing meson-python in |
|
That's curious, I wonder why it worked on the PR here (#48882 (comment)): Edit: Sorry! I've just realized. I've moved to |
|
In the log you linked it looks like there is no In the crossbow failures, |
|
@github-actions crossbow submit wheel-*cp313-cp313-amd64 |
|
Revision: d8844b4 Submitted crossbow builds: ursacomputing/crossbow @ actions-6e13bde35b
|
7 participants
Rationale for this change
This helps simplify the steps to build pyarrow by leveraging Meson, a build system strongly inspired by Python's syntax. In it's current form, it requires Arrow to be installed on the host system, but in the future we may even be able to have PyArrow build Arrow as a subproject, as needed
What changes are included in this PR?
This PR adds Meson configuration files to the Python code base within Arrow.
Are these changes tested?
Yes
Are there any user-facing changes?
We may want to deprecate the traditional setup.py way of building PyArrow alongside this.