virtio/net/unixgram: Retry on ENOBUFS

[nirs]

When running iperf3 with gvproxy or vmnet-helper, krunkit breaks randomly with:

[2026-02-19T02:53:41Z ERROR devices::virtio::net::worker] Failed to process rx:
Backend(Internal(ENOBUFS)) (triggered by backend socket readable)

macOS returns ENOBUFS when the kernel socket buffer is full, rather than blocking or returning EAGAIN on non-blocking sockets. This is handled by gvproxy and vmnet-helper by retrying the write.

Changes

1. Retry on ENOBUFS — Map ENOBUFS to WriteError::NothingWritten so the write is retried instead of treated as a fatal error.

2. Stop tx loop on WrittenNothing — When a backend cannot write, stop the tx loop and turn to the event loop. The event loop will wakes when the socket is writable.

3. Microsecond timestamps — Use format_timestamp_micros() in both logger init paths so timing of events can be understood from the logs.

Test results

Tested 600 seconds bidirectional iperf3 runs with 8 streams:

iperf3 -c addr --bidir -P 8 --json -t 600

Configuration:

• gvproxy:
  • podman 5.8.0 rootful, 4 cpus, 2048m memory
  • gvproxy: tap: reduce lock contention in switch packet forwarding gvisor-tap-vsock#613
  • iperf3: https://hub.docker.com/r/networkstatic/iperf3
• vmnet-helper:
  • Fedora 43, 4 cpus, 2048m memory
  • vmnet-helper: Add minikube performance tests nirs/vmnet-helper#179
  • iperf3: iperf3-3.19.1-1.fc43.src.rpm
• host:
  macOS 26.3
  iperf3: 3.20

Fixes #555

[gemini-code-assist]

Summary of Changes

Hello @nirs, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request addresses a critical network stability and performance issue on macOS when using gvproxy with iperf3. It introduces specific error handling for ENOBUFS (no buffer space available) errors, which macOS returns when the kernel socket buffer is full. By treating this condition as a signal to retry the write operation, the PR prevents crashes and enables successful, stable network stress testing, significantly improving the robustness of the virtio network backend on macOS.

Highlights

• macOS ENOBUFS Handling: Modified the write_frame function in virtio/net/unixgram to explicitly handle nix::Error::ENOBUFS on macOS. This error, which occurs when the kernel socket buffer is full, is now mapped to WriteError::NothingWritten, allowing the caller to retry the send operation instead of treating it as a fatal error.
• Improved Network Stability: This change prevents gvproxy from crashing during iperf3 stress tests on macOS, which previously occurred due to unhandled ENOBUFS errors. The system can now gracefully handle transient buffer full conditions.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• src/devices/src/virtio/net/unixgram.rs
  • Modified the write_frame function to use a match statement for the send system call result.
  • Added a specific case to catch nix::Error::ENOBUFS and return WriteError::NothingWritten.
  • Included comments explaining the macOS-specific behavior of ENOBUFS and referencing a similar handling in vmnet-helper.

Activity

• The author identified a critical issue where iperf3 stress tests would fail within seconds on macOS due to Backend(Internal(ENOBUFS)) errors when using gvproxy.
• Detailed test results were provided, demonstrating that the change enables successful iperf3 bidirectional stress testing for 600 seconds, with consistent performance using vmnet-helper and improved stability for gvproxy.
• The author noted that without this change, performance testing was impossible as the stress test would immediately break.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request correctly handles the ENOBUFS error on macOS for unixgram sockets by mapping it to WriteError::NothingWritten, which allows the caller to retry sending the frame. This prevents a fatal error when the kernel socket buffer is full. I've added one suggestion to also handle EAGAIN and EWOULDBLOCK in the same way to improve portability and make the error handling more robust across different Unix-like systems.

[slp]

@mtjhrc PTAL

[mtjhrc]

The change LGTM, but what did you mean by this in the commit message:

macOS returns ENOBUFS when the kernel socket buffer is full, rather
than blocking or returning EAGAIN. The kernel does not provide any
readiness notification for this condition, so the only recovery is
to retry the send.

Wouldn't we get notified via kqueue the socket is writable again? (same as EAGAIN on Linux...)

[nirs]

The change LGTM, but what did you mean by this in the commit message:

macOS returns ENOBUFS when the kernel socket buffer is full, rather
than blocking or returning EAGAIN. The kernel does not provide any
readiness notification for this condition, so the only recovery is
to retry the send.

Wouldn't we get notified via kqueue the socket is writable again? (same as EAGAIN on Linux...)

I could not find any evidence that we can use non-blocking io to detect if the socket is writable or not. The only info I could find was in FreeBSD mailing list thread explaining that the only way to recover is retrying.

Looking at early experiments with vmnet-helper I did not try non-blocking io, or maybe I did not commit this since it did not work.

[mtjhrc]

Hmm, but we are already using a non-blocking socket here. Notably we also use EVFILT_WRITE | EV_CLEAR, which should be similar to the edge triggered epoll behavior on Linux.

What I am wondering is that, If receiving an ENOBUFS meant we won't get woken up by EVFILT_WRITE, this could still stall temporarily - suppose there is a burst of network packets, causing ENOBUFS, followed by period of no packets from the guest, that could mean we wouldn't get kicked from guest or from the socket until guest tries to send more packets. (on Linux we would get notified the socket became writable again and we would continue processing the queue).

[nirs]

What I am wondering is that, If receiving an ENOBUFS meant we won't get woken up by EVFILT_WRITE, this could still stall temporarily - suppose there is a burst of network packets, causing ENOBUFS, followed by period of no packets from the guest, that could mean we wouldn't get kicked from guest or from the socket until guest tries to send more packets. (on Linux we would get notified the socket became writable again and we would continue processing the queue).

I'm not sure that edge triggered behavior works for datagram socket. The socket is probably always writable since we never get EAGAIN. But this is a separate issue to investigate, retrying ENOBUFS works.

It can be interesting to count retries and be able to get stats so we have some visibility on this issue. Maybe add debug log in this case?

[mtjhrc]

I mean this is definitely good, even if it likely isn't a full fix (like discussed above) - the code change LGTM.

It's just the commit message seems misleading: "Map ENOBUFS to WriteError::NothingWritten so the caller retries the frame instead of treating it as a fatal error." - the caller doesn't retry on WriteError::NothingWritten, it relies on a writable kqueue notification (which the commit message implies doesn't arrive) or a kick from the guest.

[nirs]

I mean this is definitely good, even if it likely isn't a full fix (like discussed above) - the code change LGTM.

It's just the commit message seems misleading: "Map ENOBUFS to WriteError::NothingWritten so the caller retries the frame instead of treating it as a fatal error." - the caller doesn't retry on WriteError::NothingWritten, it relies on a writable kqueue notification (which the commit message implies doesn't arrive) or a kick from the guest.

Right, "caller retries" is not a good description. The write is retried when the socket is considered writable (need to test if this happen) or the guest kicks us.

I'll try to experiment more and have a more correct description how this is handled.

[mtjhrc]

I feel like we should be getting the EV_WRITE events. I made a simple test and it seemed to work:
https://gist.github.com/mtjhrc/2a8583cfc82cf913a8402114dda1bb9a

[nirs]

I feel like we should be getting the EV_WRITE events. I made a simple test and it seemed to work: https://gist.github.com/mtjhrc/2a8583cfc82cf913a8402114dda1bb9a

Interesting, but there are no timestamps so it is not clear if we got the event immediately after the read.

[nirs]

@mtjhrc I'm trying the EV_WRITE way, it seems like a simple change.

[nirs]

@mtjhrc Current version should be a complete fix. Tested with vment-helper and podman/gvproxy. I will post updated results later.