Linux: Nuke Stat bits in favour of Statx #25639

The-King-of-Toasters · 2025-10-19T22:43:14Z

Maintaining the POSIX stat bits for Zig is a pain. Not only is struct stat incompatable between architectures, but maddingly annoying so; timestamps are specified as machine longs or fixed-width ints, members can be signed or unsigned. The libcs deal with this by introducing the own version of struct stat and copying the kernel structure members to
it. In the case of glibc, they did it twice thanks to the largefile transition!

In practice, the project needs to maintain three versions of struct stat:

What the kernel defines.
What musl wants for struct stat.
What glibc wants for struct stat64. Make sure to use fstatat64!

And it's not as simple as running zig translate-c. In #21440 I had to create test programs in C and use pahole to dump the structure of stat for each arch, and I was constantly running into issue regarding padding and signed/unsigned ints. The fact that so many target checks in the linux and posix tests exist is most likely due to writing to padding bits and failing later.

The solution to this madness is statx(2):

It takes a single structure that is the same for all arches AND libcs.
It uses a custom timestamp format, but it is 64-bit ready.
It gives the same info as fstatat(2) and more!
Unlike fstatat(2), you can request a subset of the info required based on passing a mask.

It's so good that modern Linux arches (e.g. riscv) don't even implement stat, with the libcs using a generic struct stat and copying from struct statx.

Therefore, this PR rips out all the stat bits from std.os.linux and std.c. std.posix.Stat is now void, and calling std.posix.*stat is an compile-time error. A wrapper around statx has been added to std.os.linux, and callers have been upgraded to use it. Tests have also been updated to use statx where possible.

While I was here, I converted the mask and file attributes to be packed struct bitfields. A nice side effect is checking that you actually received the members you asked for via Statx.mask, which I have used by adding asserts at specific callsites.

Progress towards #21738

The-King-of-Toasters · 2025-10-19T23:58:19Z

Forgot to compile the compiler and added a call to MappedFile. Perhaps fs.File.Stat should add these members? I imagine it would be useful to know.

rootbeer

Looks reasonable to me. I've got some questions, and ideas, but feel free to ignore them.

lib/std/os/linux/test.zig

lib/std/os/linux.zig

rootbeer · 2025-10-19T23:51:19Z

lib/std/os/linux.zig

+            .{ .major = 2, .minor = 28, .patch = 0 });
+        const sys = if (use_c) std.c else std.os.linux;
+
+        var stx = std.mem.zeroes(Statx);


I'm a little surprised to see this wrapper is stack allocating and zero'ing a statx buffer, instead of taking a pointer to a statx buffer and letting the caller decide if/how/when to allocate and zero it. AFAICT, none of the other syscall wrappers in here do anything quite like that.

I did this because

Most call sites used posix.*stat.

posix.*stat did the same thing.

My understanding is that the wrappers act like posix in that they translate errors and allocate structures for you. Happy to change if not.

rootbeer · 2025-10-19T23:52:06Z

lib/std/posix/test.zig

-test "linkat with different directories" {
-    if ((builtin.cpu.arch == .riscv32 or builtin.cpu.arch.isLoongArch()) and builtin.os.tag == .linux and !builtin.link_libc) return error.SkipZigTest; // No `fstatat()`.
-    if (builtin.cpu.arch.isMIPS64()) return error.SkipZigTest; // `nstat.nlink` assertion is failing with LLVM 20+ for unclear reasons.
+fn getLinkInfo(fd: posix.fd_t) !struct { posix.ino_t, posix.nlink_t } {


lib/std/posix.zig

The-King-of-Toasters · 2025-10-20T00:34:00Z

@rootbeer I'll take your comments and update the PR later today (Sydney time).
@alexrp How would you handle the failure in glibc_runtime_check.zig? Perhaps adding the stat function definitions but making the struct stat part a byte buffer instead.

alexrp · 2025-10-20T02:01:53Z

@alexrp How would you handle the failure in glibc_runtime_check.zig? Perhaps adding the stat function definitions but making the struct stat part a byte buffer instead.

That seems reasonable for this test.

alexrp · 2025-10-20T10:50:09Z

Would you mind opening a mirror PR on Codeberg for LoongArch and RISC-V CI?

alexrp · 2025-10-20T11:46:04Z

Also, it doesn't have to be done in this PR, but it would be nice to decouple std.c.dev_t (& friends) completely from the corresponding types in std.os.linux since, for some of them, they're not the same size or even representation.

The-King-of-Toasters · 2025-10-21T09:44:21Z

Would you mind opening a mirror PR on Codeberg for LoongArch and RISC-V CI?

Done.

alexrp · 2025-10-22T21:16:55Z

The std.fs changes here will cause some merge conflicts with #25592. They look relatively minor to me, but probably best for @andrewrk to confirm whether we should hold off on merging this until #25592 lands.

bernardassan · 2025-10-23T10:37:09Z

@The-King-of-Toasters could you adapt the structure I used in #25394 for the typed flags changes in IoUring.zig for Statx.Attr and Statx.Mask

zig/lib/std/os/linux.zig

Line 7064 in 321a739

pub const Mask = packed struct(u32) {

and

zig/lib/std/os/linux.zig

Line 7122 in 321a739

pub const Attr = packed struct(u64) {

? In my opinion, that has better namespacing, and the doc comment allows users familiar with STATX_ATTR_* and STATX_MASK_* constants to easily find it when searching for them on https://ziglang.org/documentation/master/std/ which I think seems to be the main motivation for most of the identifiers been UPPER_CASE in linux.zig. We can do better in Zig 😄 than C, and this would also help to reduce conflicts when #25394 is been merged.

alexrp · 2025-10-28T09:18:06Z

The std.fs changes here will cause some merge conflicts with #25592. They look relatively minor to me, but probably best for @andrewrk to confirm whether we should hold off on merging this until #25592 lands.

Andrew confirmed on Zulip that he'd prefer for this to wait until #25592 lands.

The-King-of-Toasters · 2025-10-30T01:55:29Z

Now that std.Io has landed, let's get some questions answered before I rebase:

Should the statx wrapper exist?
- If so, should the structure be allocated & zeroed for you?
Should @bernardassan's naming style be preferred over the UPPERCASE way I have?

alexrp · 2025-10-30T07:23:07Z

Should the statx wrapper exist?

If so, should the structure be allocated & zeroed for you?

I think yes. We can always revisit that later, but let's stick with existing practice for now.

Should @bernardassan's naming style be preferred over the UPPERCASE way I have?

Seems fine I guess? But I don't have a strong opinion one way or the other.

The-King-of-Toasters · 2025-11-02T06:00:47Z

Don't understand why the wasi errors are tripping now 🤷

alexrp · 2025-11-02T12:11:21Z

https://github.com/The-King-of-Toasters/zig/blob/a0f479ad2b321bbcfc85206a8d5aff01d2aa706c/lib/std/posix.zig#L3950-L3959

Which is being used here: https://github.com/ziglang/zig/pull/25639/files#diff-8472743cd38adbeb3a546025c048641d73c51e8373defeb3ea16f9e44a91bef6R199

The-King-of-Toasters · 2025-11-03T05:28:23Z

Ahh, I see it got removed in 3bf0ce6. Guess I'll add a check against wasi nolibc.

Closes ziglang#26035

HEAD/TRACE are bodyless, but PUT responses are body-capable per RFC 7231.

Hybrid KEMs combine a post-quantum secure KEM with a traditional elliptic curve Diffie-Hellman key exchange. The hybrid construction provides security against both classical and quantum adversaries: even if one component is broken, the combined scheme remains secure as long as the other component holds. The implementation follows the IETF CFRG draft specification for concrete hybrid KEMs: https://datatracker.ietf.org/doc/draft-irtf-cfrg-concrete-hybrid-kems/

Migrated from ziglang#21419 Co-authored-by: Jonathan Hallstrom <lmj.hallstrom@gmail.com> Reviewed-on: https://codeberg.org/ziglang/zig/pulls/30005

…n NixOS like environment. Basically detect `-idirafter` flag in `NIX_CFLAGS_COMPILE` and treat it like `-isystem`, also detect `NIX_CFLAGS_LINK` environment variable and treat it like the `NIX_LDFLAGS` . Reference: https://github.com/NixOS/nixpkgs/blob/74eefb4210ef545cebd6a7a25356734b0b49854f/pkgs/build-support/build-fhsenv-chroot/env.nix#L83

…ey encapsulation' (#30010) from jedisct1/zig:hybridkem into master Reviewed-on: https://codeberg.org/ziglang/zig/pulls/30010

Otherwise the pthread_cancel can affect unrelated tasks.

It doesn't support setting the "canceled" status to false, so once a thread has been canceled, all operations on the thread start permanently failing.

If ECANCELED occurs, it's from pthread_cancel which will *permanently* set that thread to be in a "canceling" state, which means the cancel cannot be ignored. That means it cannot be retried, like EINTR. It must be acknowledged.

…om cancellation into master Reviewed-on: https://codeberg.org/ziglang/zig/pulls/30033

Also reduce the memory required by tests. 4GB for every test is way too much and doesn't provide much benefits in testing the algorithms.

…edisct1/zig:argon2 into master Reviewed-on: https://codeberg.org/ziglang/zig/pulls/30084

These are already built and run on x86_64-linux.

pulsar is much faster than george so we don't need to do this anymore.

…enity

…enbsd

This avoids pessimizing concurrency on all machines due to e.g. the macOS machine having high memory usage across the board due to 16K page size. This also adds max_rss to test-unit and test-c-abi since those tend to eat a decent chunk of memory too.

All of our runners now define this. When running a CI script locally, this will not be set, so we default to 0, aka "all available system memory".

These excessive timeouts should no longer be necessary with the recent tuning of job capacity and maxrss on these machines.

…nstead of the non-existant .nanoseconds

The main goal here was to avoid allocating padding and header space if `malloc` already guarantees the alignment we need via `max_align_t`. Previously, the compiler was using `std.heap.raw_c_allocator` as its GPA in some cases depending on `std.c.max_align_t`, but that's pretty fragile (it meant we had to encode our alignment requirements into `src/main.zig`!). Perhaps more importantly, that solution is unnecessarily restrictive: since Zig's `Allocator` API passes the `Alignment` not only to `alloc`, but also to `free` etc, we are able to use a different strategy depending on its value. So `c_allocator` can simply compare the requested align to `Alignment.of(std.c.max_align_t)`, and use a raw `malloc` call (no header needed!) if it will guarantee a suitable alignment (which, in practice, will be true the vast majority of the time). So in short, this makes `std.heap.c_allocator` more memory efficient, and probably removes any incentive to use `std.heap.raw_c_allocator`. I also refactored the `c_allocator` implementation while doing this, just to neaten things up a little.

Also removes the blank lines between members, and a comptime sizeOf check.

Maintaining the POSIX `stat` bits for Zig is a pain. The order and bit-length of members differ between all architectures, and int types can be signed or unsigned. The libcs deal with this by introducing the own version of `struct stat` and copying the kernel structure members to it. In the case of glibc, they did it twice thanks to the largefile transition! In practice, the project needs to maintain three versions of `struct stat`: - What the kernel defines. - What musl wants for `struct stat`. - What glibc wants for `struct stat64`. Make sure to use `fstatat64`! This isn't as simple as running `zig translate-c`. In ziglang#21440 I had to: - Compile toolchains for each arch+glibc/musl combo. - Create a test `fstat` program with/without `FILE_OFFSET_BITS=64`. - Dump the value for `struct stat`. - Stare at `std.os.linux`/`std.c` and cry. - Add some missing padding. The fact that so many target checks in the `linux` and `posix` tests exist is most likely due to writing to padding bits and failing later. The solution to this madness is `statx(2)`: - It takes a single structure that is the same for all arches AND libcs. - It uses a custom timestamp format, but it is 64-bit ready. - It gives the same info as `fstatat(2)` and more! - Unlike `fstatat(2)`, you can request a subset of the info required based on passing a mask. It's so good that modern Linux arches (e.g. riscv) don't even implement `stat`, with the libcs using a generic `struct stat` and copying from `struct statx`. Therefore, this commit rips out all the `stat` bits from `std.os.linux` and `std.c`. `std.posix.Stat` is now `void`, and calling `std.posix.*stat` is an compile-time error. A wrapper around `statx` has been added to `std.os.linux`, and callers have been upgraded to use it. Tests have also been updated to use `statx` where possible. While I was here, I converted the mask and file attributes to be packed struct bitfields. A nice side effect is checking that you actually recieved the members you asked for via `Statx.mask`, which I have used by adding `assert`s at specific callsites.

Commit #fc7a5f2 moved many of the `_t` types up a level, but didn't remove them from arch_bits. Since `Stat` is gone, all but `time_t` can be removed.

alexrp · 2025-12-07T10:47:49Z

https://codeberg.org/ziglang/zig/pulls/30122

alexrp self-assigned this Oct 19, 2025

The-King-of-Toasters force-pushed the statx branch from 07f169b to 2550570 Compare October 19, 2025 23:53

rootbeer reviewed Oct 20, 2025

View reviewed changes

The-King-of-Toasters force-pushed the statx branch 3 times, most recently from 09ec5b7 to f7a6449 Compare October 20, 2025 07:56

The-King-of-Toasters force-pushed the statx branch 2 times, most recently from ce81ddb to a0f479a Compare November 1, 2025 22:54

jacobly0 and others added 8 commits November 27, 2025 05:05

x86_64: fix abi of a struct that partially fits in registers

a1827d5

Closes ziglang#26035

http.requestHasBody: a PUT response can have a body

854774d

HEAD/TRACE are bodyless, but PUT responses are body-capable per RFC 7231.

std.sort.partitionPoint: faster implementation (#30005)

e427ba9

Migrated from ziglang#21419 Co-authored-by: Jonathan Hallstrom <lmj.hallstrom@gmail.com> Reviewed-on: https://codeberg.org/ziglang/zig/pulls/30005

ci: pin actions/checkout to current ziglang/checkout@HEAD

fbafbd4

Merge pull request 'std.crypto: add hybrid post-quantum/traditional k…

e6d19a0

…ey encapsulation' (#30010) from jedisct1/zig:hybridkem into master Reviewed-on: https://codeberg.org/ziglang/zig/pulls/30010

issue templates: use anchors in URLs

e19c61a

andrewrk and others added 27 commits December 1, 2025 19:17

std.Io.Threaded: musl: handle ECANCELED same as EINTR

bf0ffc4

Otherwise the pthread_cancel can affect unrelated tasks.

std.Io.Threaded: don't use pthread_cancel with musl

cf82064

It doesn't support setting the "canceled" status to false, so once a thread has been canceled, all operations on the thread start permanently failing.

std.Io.Threaded: separate out ECANCELED handling again

bb3f56d

If ECANCELED occurs, it's from pthread_cancel which will *permanently* set that thread to be in a "canceling" state, which means the cancel cannot be ignored. That means it cannot be retried, like EINTR. It must be acknowledged.

Merge pull request 'std.Io.Threaded: rework cancellation' (#30033) fr…

52ad126

…om cancellation into master Reviewed-on: https://codeberg.org/ziglang/zig/pulls/30033

Argon2: use the std.Io interface

6fe95c2

Also reduce the memory required by tests. 4GB for every test is way too much and doesn't provide much benefits in testing the algorithms.

ci: set a sensible maxrss in x86_64-windows scripts

be9649f

std.process.ArgIteratorWasi: fix no-args deinit

cb115cf

Merge pull request 'Argon2: use the std.Io interface' (#30084) from j…

d73fbcc

…edisct1/zig:argon2 into master Reviewed-on: https://codeberg.org/ziglang/zig/pulls/30084

build: add some missing darwin tags

ad9a518

build: add -Dskip-spirv and -Dskip-wasm options

e2a9e56

ci: skip spirv and wasm tests on x86_64-freebsd

c21ce53

These are already built and run on x86_64-linux.

ci: don't skip release mode tests on x86_64-windows-debug

e270c97

pulsar is much faster than george so we don't need to do this anymore.

std.process.Child: enable rusage collection for freebsd, illumos, ser…

9eed87f

…enity

std.c: add rusage for dragonfly, netbsd, openbsd

2659fad

std.process.Child: enable rusage collection for dragonfly, netbsd, op…

4454380

…enbsd

ci: set maxrss from $ZSF_MAX_RSS if provided by the runner

78cba86

All of our runners now define this. When running a CI script locally, this will not be set, so we default to 0, aka "all available system memory".

ci: reduce x86_64-linux timeouts

c166bb3

These excessive timeouts should no longer be necessary with the recent tuning of job capacity and maxrss on these machines.

make Io.net.sendMany compile

58e3c2c

Change github links to codeberg

adc5a39

std.Io.Timestamp: when creating a Clock.Timestamp actually set .raw i…

032e3c9

…nstead of the non-existant .nanoseconds

std.debug: skip manage resources correctly with cbe

ea94ac5

build: bump test-zigc max_rss to 767_483_904 on x86_64-linux

1e61609

Linux: Update Statx structure

9abf055

Also removes the blank lines between members, and a comptime sizeOf check.

Linux: Dedupe generic decls

f4f268b

Commit #fc7a5f2 moved many of the `_t` types up a level, but didn't remove them from arch_bits. Since `Stat` is gone, all but `time_t` can be removed.

The-King-of-Toasters force-pushed the statx branch from a0f479a to f4f268b Compare December 6, 2025 09:28

alexrp closed this Dec 7, 2025

Uh oh!

Linux: Nuke Stat bits in favour of Statx #25639

Linux: Nuke Stat bits in favour of Statx #25639

Uh oh!

Conversation

The-King-of-Toasters commented Oct 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

The-King-of-Toasters commented Oct 19, 2025

Uh oh!

rootbeer left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rootbeer Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

The-King-of-Toasters Oct 20, 2025

Choose a reason for hiding this comment

Uh oh!

rootbeer Oct 19, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

The-King-of-Toasters commented Oct 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

alexrp commented Oct 20, 2025

Uh oh!

alexrp commented Oct 20, 2025

Uh oh!

alexrp commented Oct 20, 2025

Uh oh!

The-King-of-Toasters commented Oct 21, 2025

Uh oh!

alexrp commented Oct 22, 2025

Uh oh!

bernardassan commented Oct 23, 2025

Uh oh!

alexrp commented Oct 28, 2025

Uh oh!

The-King-of-Toasters commented Oct 30, 2025

Uh oh!

alexrp commented Oct 30, 2025

Uh oh!

The-King-of-Toasters commented Nov 2, 2025

Uh oh!

alexrp commented Nov 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

The-King-of-Toasters commented Nov 3, 2025

Uh oh!

alexrp commented Dec 7, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

16 participants

The-King-of-Toasters commented Oct 19, 2025 •

edited

Loading

The-King-of-Toasters commented Oct 20, 2025 •

edited

Loading

alexrp commented Nov 2, 2025 •

edited

Loading