[Question] - How do you acess the "@rules_python//python/cc:current_py_cc_headers" when working with multiple versions?

[tenspd137]

I have C++ module that I use pybind11_bazel to compile and then pass on to python rules and eventually generate a wheel. For one version - works great. The docs and examples show that you can use multiple versions of python using the extension with multiple calls to python.toolchain:

python = use_extension("@rules_python//python/extensions:python.bzl", "python")
python.toolchain(
    configure_coverage_tool = True,
    python_version = "3.8",
)
python.toolchain(
    configure_coverage_tool = True,
    # Only set when you have mulitple toolchain versions.
    is_default = True,
    python_version = "3.9",
)
python.toolchain(
    configure_coverage_tool = True,
    python_version = "3.10",
)
python.toolchain(
    configure_coverage_tool = True,
    python_version = "3.11",
)
use_repo(
    python,
    python = "python_versions",
)

and then use like this:

load("@python//3.8:defs.bzl", py_binary_3_8 = "py_binary", py_test_3_8 = "py_test")
....
py_binary_3_8(
    name = "version_3_8",
    srcs = ["version.py"],
    main = "version.py",
)

This makes sense - but I can't seem to figure out how to access the equivalent of "@rules_python//python/cc:current_py_cc_headers" for different versions, which you need to pass on to pybind11 in order to have the correct headers for multi-version module builds. Any ideas? I can't seem to find any examples, and I am learning Bazel, so the source code still doesn't make sense to me yet beyond how to use it (limited).

Thanks!

[rickeylev]

Thanks for giving current_py_cc_headers a try!

It's the same target regardless of version.

Using py_binary_3_8 basically says, "Only use toolchains that support python 3.8". Later, when current_py_cc_headers is evaluated, it inherits that same requirement because it's a dependency of the py_binary. It then asks, "Find me the Python C++ toolchain that supports 3.8", and gets the headers for 3.8

If you run bazel cquery deps(//:my_python_3_8_target), then you should see it list out something like @python_3_8_linux_x86_64//:headers (I forget the exact name).

[tenspd137]

Thanks for the explanation! I just figured out the right thing to do though after reading the docs a bit closer - this seems to get me there or at least - I don't get errors:

python.toolchain(
    python_version = "3.11",
)
python.toolchain(
    python_version = "3.8",
)
use_repo(python, "python_3_11") <-----  These are the trick
use_repo(python, "python_3_8") <------
use_repo(python, "python_versions")

Then later:

pybind_extension(
    name = "_axraw",
    srcs = glob(["pybind11/*.*pp"], allow_empty = False,),
    deps = ["//:axraw","@python_3_11//:python_headers"], <----- tp get the right headers
    linkstatic = True,
)

I have some more experimenting to do, and I'd like to figure out how to wrap this up into a module extension so I could just do something like "pybind11.version = 3.8" and "pybind11.version=3.11" - but obviously - pybind11_bazel is a different project - I didn't want to ask to many questions about it here, but this, I think, is key to the interop. Thanks again - this is great work - BTW! Hopefully this little nugget helps someone.

First steps would be duplication - similar to the multi-version example here. Later - figure out how to abstract it so I can just write "version=x.y"

[rickeylev]

Using @python_X_Y//:python_headers works, but is exactly what current_py_cc_headers was created to replace :). The issue is that using @python_X_Y hard-codes the Python version of the headers you're using.

If you're OK with having all consuming py_binary's closely coupled to the python version your pybind library, then it'll work ok. If you have potentially different (or unknowable in advance) python versions consuming your pybind library, then your best option is to use current_py_cc_headers to have the headers selected automatically based on the python version being built for.

do something like "pybind11.version = 3.8" and "pybind11.version=3.11"

The thing you want to do is use a transition to set the //python/config_setting:python_version flag. The pybind rules themselves shouldn't have such a setting because they're libraries -- they don't know what python version is going to consume them. It's the responsibility of a higher level (i.e. a target depending on the library setting the flag, or the user setting the command line flag) to set the python version.

HTH

[tenspd137]

best option is to use current_py_cc_headers to have the headers selected automatically based on the python version being built for.

I totally agree with this. What I can't seem to figure out how to do is "switch" between python versions when building. For example - I'd like to create a wheel for python3.8 and a wheel for python3.11 - but I don't know how to get it to build for anything other than one of the python toolchains I declare. I can't seem to make the connection that allows me to switch toolchains. In the module, I can declare which versions I want to use, I see how to use it to switch between py_binary_X_Y, but I don't see how to pass that to pybind11.

PYBIND_DEPS = [
    Label("@pybind11//:pybind11"),
    "@rules_python//python/cc:current_py_cc_headers",
]

def pybind_extension(
        name,
        copts = [],
        features = [],
        linkopts = [],
        
        .......
        

BTW - thank you again for entertaining my questions - I realize I am basically asking how to use 2 rule sets that are not related maintainer-wise.

[rickeylev]

To switch, set the command line or use a transition.

Command line: Set bazel build --@rules_python//python/config_settings:python_version=3.11 //my:pybind

Transition: Right now, the only easy way to use the transitions is through the e.g. 3.8.defs%py_binary wrappers. Ideally, py_package/py_wheel would let you set the version directly via an attribute (PRs welcome to add such functionality). Otherwise, you have to write your own wrapper rule to do the transition; you can look at python/config_settings/transition.bzl as an example (its what the py_binary wrappers use).

Note that you can only switch to a version that a toolchain has been declared for.

[tenspd137]

The command line worked perfectly. A few config settings that set the python wheel tags:

config_setting(
    name = "py38",
    flag_values = {"@rules_python//python/config_settings:python_version":"3.8"}
)

config_setting(
    name = "py311",
    flag_values = {"@rules_python//python/config_settings:python_version":"3.11"}
)
....
    python_tag      = select({
        ":py38":"cp38",
        ":py311":"cp311",
        "//conditions:default":"cp311"
    }),
    abi             = select({
        ":py38":"cp38",
        ":py311":"cp311",
        "//conditions:default":"cp311"

With a default value of 3.11, and I can build multiple versions! Thanks for your help getting me started. I can refine as I learn more - but this is really good - at least in my mind..... Thanks!!!