multiprocessing.conenction.Connection.fileno does not return a valid fd on windows

[ergoithz]

Bug report

Bug description:

multiprocessing.conenction.Connection.fileno is returning a raw Windows file handle (unusable for os operations) instead of the corresponding python file descriptor from msvcrt.open_osfhandle.

This is somewhat documented as fileno() Return the file descriptor or handle used by the connection. (emphasis on handle), but this quirk is probably unjustified as msvcrt supports going back an forth between a proper fd for named pipes (actual file objects on windows, unlike sockets) so this is just a quirk maybe meant to support socket handles.

I think the right behavior for a file object on windows would be returning a python file descriptor instead of the raw handle (which isn't supported by most of stdlib), and so a fundamentally different case to winsock2 socket handles (which is directly supported by socket, asyncio, etc).

Simple test to reproduce.

import errno
import msvcrt  # windows-only
import multiprocessing
import os
import stat

conn_a, conn_b = multiprocessing.Pipe()

handle = conn_a.fileno()
try:
    os.fstat(handle)

except OSError as e:
    print(f'{errno.errorcode[e.errno]}: handle {handle} is not a valid fd!')

fd = msvcrt.open_osfhandle(handle, os.O_BINARY)
if stat.S_ISFIFO(os.fstat(fd).st_mode):
    print(f'fd {fd} from handle {handle} points to a pipe')

Test output.

EBADF: handle 132 is not a valid fd!
fd 3 from handle 132 points to a pipe

Since this is Windows-only, I'm using docker and wine to reproduce this myself in a proper OS.

docker run --volume .:/src --name pywine --rm -it --entrypoint wine tobix/pywine:3.14 python /src/connection_fileno_bug.py

CPython versions tested on:

3.14

Operating systems tested on:

Windows

[picnixz]

As you observed, the docs say that fileno() returns "the file descriptor or handle used by the connection" so it's fine. Changing this would break all existing code and I don't think it's worth it.

cc @zooba @gpshead

[gpshead]

This is a Windows question - the API's behavior has been what it is forever since multiprocessing was introduced. If this returns something different than io module .fileno() methods, we should start by just documenting that.

If we need a new method to match io behavior, one could be added.

Internally in Lib/multiprocessing/connection.py you'll see the .fileno() result being used via the _winapi.DuplicateHandle and _winapi.DuplicateHandle APIs as well as socket.fromfd().

[ergoithz]

@picnixz

Changing this would break all existing code and I don't think it's worth it.

I know there must be code relying on this platform-specific quirk, but there is also code out there that remains posix-only because of quirks like this (based on personal experience).

Also, while this would be a breaking change, since I didn't find previous reports about this I wouldn't consider it as high-impact, nor a widely used method at all, especially considering a handle cannot be used anywhere but msvcrt and ctypes.

@picnixz @gpshead
I would argue fileno is a protocol, with the well-known caveat of winsocks2-backed sockets, and breaking it here doesn't seem justified given it's backed by a completely valid file object.

[zooba]

Unfortunately, the caveats are much worse and way more subtle than "sockets", and there's an unfortunately high chance that using open_osfhandle (whether we do it or the user does it) is going to mess things up. The sample code given above is probably fine, since nobody else is using the pipe but I expect there are more complex scenarios in multiprocessing as well.

The order of making a change here is document, deprecate, warn, wait, change. In that time, it's probably pretty easy to add an optional method that returns whether fileno will return from the C runtime or OS,¹ which at least means a caller can reliably choose how to wrap it (since file numbers and handles can collide).

I'd still love to get file descriptors out of the Windows version entirely and use handles everywhere. The emulation provided by the CRT is more limited than what we offer through Python, which makes it hard to hide the rough edges from people developing on actual POSIX systems. But that's a much bigger change (though certainly long-term preferable to dumbing down the whole runtime to match the CRT).

Footnotes

1. Presumably this only varies by type and not by instance, but we shouldn't make people look up the type or have a list of "known weird" types, so give them a method where "absent" indicates FD. ↩

[picnixz]

Considering my fellow core devs' comment, I will close this as not planned.