bug: GPU gateway fails on DGX Spark - missing cgroup controller

[lupinrider]

Agent Diagnostic

GPU gateway fails on DGX Spark — missing cgroup controllers (kubepods)

Summary

The GPU-enabled gateway fails to start on a DGX Spark (Founders Edition) when NemoClaw runs openshell gateway start --name nemoclaw --gpu. The non-GPU gateway works perfectly — openshell sandbox create succeeds and I can connect to the sandbox without issues.

Environment

• Hardware: NVIDIA DGX Spark (Founders Edition), GB10 Grace Blackwell Superchip, 128GB unified memory
• Architecture: aarch64
• OS: DGX OS (Ubuntu-based)
• Kernel: (run uname -r and paste here)
• OpenShell version: 0.0.6
• Docker:
  • Cgroup Driver: systemd
  • Cgroup Version: 2
• NVIDIA Container Toolkit: 1.19.0
• NVIDIA GPU: 1 GPU detected, 124610 MB VRAM

Steps to Reproduce

1. Install OpenShell v0.0.6 via curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh
2. Confirm non-GPU gateway works: openshell sandbox create → succeeds, sandbox created and connectable
3. Stop the existing gateway: openshell gateway stop
4. Clone NemoClaw: git clone https://github.com/NVIDIA/NemoClaw.git
5. Run cd NemoClaw && ./install.sh
6. Installer reaches step [2/7] and runs openshell gateway start --name nemoclaw --gpu

Expected Behaviour

GPU-enabled gateway starts successfully.

Actual Behaviour

Gateway container starts but the K3s cluster inside it fails at kubelet startup with missing cgroup controllers:

Error:   × K8s namespace not ready
  ╰─▶ gateway container is not running while waiting for namespace 'openshell': container exited
      (status=EXITED, exit_code=1)

Key error from container logs:

E0316 21:10:05.545118     118 cgroup_manager_linux.go:406] "Unhandled Error" err="cgroup manager.Set failed: openat2 /sys/fs/cgroup/kubepods/pids.max: no such file or directory"
E0316 21:10:05.545156     118 kubelet.go:1745] "Failed to start ContainerManager" err="failed to initialize top level QOS containers: error validating root container [kubepods] : cgroup [\"kubepods\"] has some missing controllers: cpu, cpuset, hugetlb, memory, pids"

Additional Context

• The DGX Spark uses cgroup v2 with the systemd driver (stat -fc %T /sys/fs/cgroup/ returns cgroup2fs).
• The non-GPU gateway works fine on this system, so the issue appears specific to the --gpu gateway variant.
• The cgroup error mentions /sys/fs/cgroup/kubepods/pids.max which looks like a cgroup v1 path — the kubelet inside the GPU gateway container may not be correctly configured for cgroup v2.
• NVIDIA Container Toolkit 1.19.0 is installed and functional.
• This was tested on the day of the NemoClaw GTC 2026 announcement (16 March 2026).

Description

Expected Behaviour
GPU-enabled gateway starts successfully.

Actual Behaviour
Gateway container starts but the K3s cluster inside it fails at kubelet startup with missing cgroup controllers:
Error: × K8s namespace not ready
╰─▶ gateway container is not running while waiting for namespace 'openshell': container exited
(status=EXITED, exit_code=1)
Key error from container logs:
E0316 21:10:05.545118 118 cgroup_manager_linux.go:406] "Unhandled Error" err="cgroup manager.Set failed: openat2 /sys/fs/cgroup/kubepods/pids.max: no such file or directory"
E0316 21:10:05.545156 118 kubelet.go:1745] "Failed to start ContainerManager" err="failed to initialize top level QOS containers: error validating root container [kubepods] : cgroup ["kubepods"] has some missing controllers: cpu, cpuset, hugetlb, memory, pids"

Reproduction Steps

Steps to Reproduce

Install OpenShell v0.0.6 via curl -LsSf https://raw.githubusercontent.com/NVIDIA/OpenShell/main/install.sh | sh
Confirm non-GPU gateway works: openshell sandbox create → succeeds, sandbox created and connectable
Stop the existing gateway: openshell gateway stop
Clone NemoClaw: git clone https://github.com/NVIDIA/NemoClaw.git
Run cd NemoClaw && ./install.sh
Installer reaches step [2/7] and runs openshell gateway start --name nemoclaw --gpu

Environment

Environment

Hardware: NVIDIA DGX Spark (Founders Edition), GB10 Grace Blackwell Superchip, 128GB unified memory
Architecture: aarch64
OS: DGX OS (Ubuntu-based)
Kernel: (run uname -r and paste here)
OpenShell version: 0.0.6
Docker:

Cgroup Driver: systemd
Cgroup Version: 2

NVIDIA Container Toolkit: 1.19.0
NVIDIA GPU: 1 GPU detected, 124610 MB VRAM

Logs

Agent-First Checklist

• I pointed my agent at the repo and had it investigate this issue
• I loaded relevant skills (e.g., debug-openshell-cluster, debug-inference, openshell-cli)
• My agent could not resolve this — the diagnostic above explains why

[brianwtaylor]

This is a timing issue — the fix landed on main shortly after v0.0.6 was released.

Timeline (UTC, March 16):

• 14:57 — v0.0.6 published
• 15:20 — PR fix(bootstrap): use host cgroup namespace for gateway container #329 merged (cgroupns_mode: host in HostConfig)
• 17:36 — PR fix(bootstrap): support cgroup v1 hosts by disabling kubelet failCgroupV1 check #360 merged (cgroup v1 kubelet compat in entrypoint)
• 21:10 — This issue filed

PR #329 is the exact fix for this error. On Docker Engine 28.x+ with cgroup v2, the default cgroupns mode is private. In a private cgroup namespace, cgroup.subtree_control at the container's cgroup root is empty — kubelet can't delegate cpu, cpuset, hugetlb, memory, pids into kubepods, so it exits with the error you're seeing. Setting cgroupns_mode: host exposes the host's cgroup hierarchy, which is what k3s needs.

Workaround for v0.0.6 (no rebuild required):

Set the daemon-level default so all containers use the host cgroup namespace:

sudo mkdir -p /etc/docker
# If you have an existing daemon.json, add "default-cgroupns-mode": "host" to it.
# If not:
echo '{ "default-cgroupns-mode": "host" }' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker

Then retry openshell gateway start --gpu. This is safe — host cgroupns was the default before Docker 20.10, and the gateway container already runs with privileged: true.

Verified on hardware: Two DGX Spark units (aarch64, Ubuntu 24.04.4, Docker 28.x, cgroup v2) both exhibit this exact failure without the workaround and succeed with it.

Re: non-GPU gateway working: The code path for cgroup namespace is identical between GPU and non-GPU — ensure_container() uses the same HostConfig. It's possible the non-GPU test was run with a prior Docker daemon state or an earlier build. The underlying cause affects both paths equally on cgroup v2.

A v0.0.7 release would resolve this for everyone on v0.0.6.

[johntmyers]

I am closing this as we are at v7+ now so hopefully this is working for you.