Unable to deleted support bundles in "Failed" state

[wfchandler]

On dogfood we have a large number of support bundles in Failed state:

$ oxide --profile=dogfood bundle list
[
  {
    "id": "62180262-5748-43fe-932f-cd90f3d2fa1f",
    "reason_for_creation": "Created by external API",
    "reason_for_failure": "Nexus managing this bundle no longer exists",
    "state": "failed",
    "time_created": "2025-02-28T18:45:18.306219Z"
  }, {
    "id": "acfdd1ac-9ccb-42a7-a4a8-e8c4dbea6ce2",
    "reason_for_creation": "Created by internal API",
    "reason_for_failure": "Nexus managing this bundle no longer exists",
    "state": "failed",
    "time_created": "2025-05-01T20:07:18.363585Z"
  }
...

Attempting to delete these bundles fails with the following:

$ oxide --debug--profile=dogfood bundle delete --bundle-id 62180262-5748-43fe-932f-cd90f3d2fa1f
{"timestamp":"2025-12-22T15:15:30.203547Z","level":"DEBUG","message":"request succeeded","url":"https://oxide.sys.rack2.eng.oxide.computer/experimental/v1/system/support-bundles/62180262-5748-43fe-932f-cd90f3d2fa1f","path":"/experimental/v1/system/support-bundles/62180262-5748-43fe-932f-cd90f3d2fa1f","operation_id":"support_bundle_delete","remote_addr":"172.20.26.9:443","http.request.method":"DELETE","http.response.content_length":169,"http.response.status_code":400,"start_time":"2025-12-22T15:15:29.355531Z","duration_ms":847,"oxide.request_id":"2fa9cdc2-f26b-40c0-baa6-5767180f784e","target":"oxide"}
error
Error Response: status: 400 Bad Request; headers: {"content-type": "application/json", "x-request-id": "2fa9cdc2-f26b-40c0-baa6-5767180f784e", "content-length": "169", "date": "Mon, 22 Dec 2025 15:15:30 GMT"}; value: Error { error_code: Some("InvalidRequest"), message: "Cannot update support bundle state from Failed to Destroying", request_id: "2fa9cdc2-f26b-40c0-baa6-5767180f784e" }

The chain of operations is:

• NexusExternalApi::support_bundle_delete
• Nexus::support_bundle_delete
• Datastore::support_bundle_update

This final function fails because it is attempting to update the bundle state to Destroying, but Failed is a terminal state. Datastore::support_bundle_delete would succeed for these bundles, but we never reach that point.

Based on the "Nexus managing this bundle no longer exists" message, these bundles should have transitioned to Failing state, so it's not immediately obvious why the SupportBundleCollector background task didn't remove them, as it checks for items in Destroying and Failing state.

[smklein]

I checked on dogfood - the most recent recent "failed support bundle" is from the end of october, 2025, which I believe pre-dates #9286 landing. Therefore: I think all the existing failed bundles are expected.

Now, onward to this issue: I think this is a real issue in Nexus::support_bundle_delete, as you mentioned, because the Failed -> Destroying pathway should not happen. This is a case we should handle explicitly, by removing a bundle that has already been marked Failed - after all, Failed exists so that users can see the impact of failures (rather than dealing with magically disappearing bundles).

I'll get a PR (and regression test!) up soon