Implement Runtime NVMe Instance Storage Discovery Using AWS EBS Symlinks

[neddp]

Problem

On AWS Nitro-based instances with NVMe devices, the kernel's PCIe enumeration order is non-deterministic. This means:

• /dev/nvme0n1 could be the root EBS volume OR instance storage
• /dev/nvme1n1 could be instance storage OR the root EBS volume
• The order varies between boots and instance types
• There is no guaranteed ordering

Solution

Implemented runtime discovery to reliably identify instance storage by excluding EBS volumes.

Discovery Algorithm

1. Glob all NVMe devices: /dev/nvme*n1
2. Glob EBS symlinks: /dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_*
3. Resolve each symlink to its target device
4. Subtract EBS devices from all NVMe devices = instance storage
5. Validate count matches CPI expectations
6. Partition only the discovered instance storage devices

Why EBS Symlinks Are Reliable

AWS automatically creates persistent symlinks for all EBS volumes via udev rules:

/dev/disk/by-id/nvme-Amazon_Elastic_Block_Store_vol{volume_id}

Backwards Compatibility

Non-NVMe instances: No changes to behavior

• Traditional Xen instances (/dev/xvdb, /dev/sdb) use CPI paths directly
• Paravirtual instances work as before

This must be merged together with the CPI changes - cloudfoundry/bosh-aws-cpi-release#196

---

Pair @Ivaylogi98

[rkoster]

In general I would have expected this logic to go into the https://github.com/cloudfoundry/bosh-agent/tree/main/infrastructure/devicepathresolver package.

[rkoster]

This should somehow be passed in via the agent config in the stemcell builder, because it is IaaS specific.

[Ivaylogi98]

Created a new component instanceStorageResolver for the linux platform. It has 3 variants:

• autoDetectingInstanceStorageResolver - the linux platform uses this one, it detects which storage resolver to use
• awsNVMeInstanceStorageResolver - detects what are the instance storage devices; contains the new logic
• identityInstanceStorageResolver - simple resolver for the traditional naming used before this change

[neddp]

In general I would have expected this logic to go into the https://github.com/cloudfoundry/bosh-agent/tree/main/infrastructure/devicepathresolver package.

Thank you for the review! That's was a big oversight on my end, I'll look into it.

[rkoster]

No worries 🙂

[beyhan]

We discussed this during the FI WG meeting and this have to relay on the stemcell agent settings and agent strategy for disc handling.

[rkoster]

As discussed during the working group meeting, focus is now on validating: cloudfoundry/bosh-aws-cpi-release#196 (comment)