Introduce mock-based unit tests for cloud provider integrations

[dangrondahl]

Problem

Our AWS, Azure, and Docker integration tests all call real external services. This causes:

• Flaky CI — transient rate limits (e.g. AWS Lambda 429s), network issues, and external outages cause test failures unrelated to code changes
• Slow feedback — waiting on real API calls adds significant time to test runs
• Credential dependency — tests can't run without real credentials and infrastructure (Lambda functions, ECS clusters, S3 buckets, Azure apps)
• Poor edge case coverage — hard to test error paths like "what happens when Lambda returns a 500 on the 3rd page of results?"

Proposed approach

Introduce a two-layer testing strategy:

Layer 1: Mock/unit tests (fast, run on every PR)

• Introduce interfaces for cloud service clients (e.g. a LambdaAPI interface with ListFunctions and GetFunctionConfiguration)
• Write mock implementations for use in tests
• Cover logic, edge cases, and error paths: pagination, concurrent goroutine behavior, filtering, partial failures

Layer 2: Integration tests (slow, run on main/daily)

• Keep a small set of smoke tests per cloud provider that prove real wiring works (credentials, API compatibility, serialization)
• These can run less frequently — on pushes to main or via daily-cli-tests.yml

Packages to target (in priority order)

1. internal/aws — Lambda, ECS, S3 (highest flake rate due to rate limiting)
2. internal/azure — Azure Apps
3. internal/docker — Docker/OCI
4. internal/kube — Kubernetes

Context

We recently switched AWS to adaptive retry mode (PR #757) which mitigates rate limiting, but the underlying problem remains: we have no fast, isolated test layer for cloud integrations.

[tooky]

Great issue Dan — the problem statement and the two-layer approach are spot on. A couple of thoughts to sharpen the approach:

Terminology: Fakes rather than Mocks

Small but worth being precise about: what's described here as "mock implementations" are really Fakes — lightweight, in-memory implementations of the interfaces (e.g. a FakeLambdaAPI that returns canned data from a slice). A Mock in the test-double sense is something that verifies interactions (e.g. "was ListFunctions called exactly twice?"). Fakes are what we want here — they simulate real behaviour without the coupling to call counts and ordering that mocks bring. This distinction matters because Fakes tend to be more resilient to refactoring, which is exactly what we need for this kind of test.

Contract tests to keep the Fakes honest

The risk with Fakes is drift — the Fake behaves one way, the real service behaves another, and your tests pass while production breaks. Contract tests solve this.

The key insight is that contract tests don't need to test the entire external API — they only need to test the behaviour we actually depend on, which is scoped by the interface we define. If our LambdaAPI interface only exposes ListFunctions and GetFunctionConfiguration, then the contract tests only verify how those two operations behave — pagination, error codes, response shapes, etc. We're testing our expectations of the dependency, not the dependency itself.

In practice:

1. Define a shared test suite written against the interface (e.g. LambdaAPI)
2. Run that suite against both the Fake and the real implementation
3. If both pass the same contract, you have confidence the Fake is a valid stand-in

This keeps the contract test surface small and focused. If a contract test fails against the real implementation but passes against the Fake, we know our assumptions have drifted and the Fake needs updating.

Proposed layers (refined)

• Layer 1: Fake-backed unit tests — fast, every PR. Cover logic, edge cases, error paths. This is where the bulk of our tests live.
• Layer 2: Contract tests — run against both the Fake and the real implementation. On main/daily for the real side. These establish that the Fake is a trustworthy substitute, and they only cover the slice of the dependency's behaviour that we actually use.
• Layer 3 (optional): A handful of smoke tests — fully integrated, end-to-end. These can give us confidence the full wiring works, but we don't need many — the contract tests already cover API compatibility.

This way we get fast feedback on PRs, and the contract tests give us a structured way to know when our Fakes have drifted from reality — rather than finding out the hard way.