Hedged Requests + Alt-Endpoint Failover + Priority Hints

[hoangsonww]

Feature: Hedged Requests + Alt-Endpoint Failover + Priority Hints

Summary
Reduce tail latency and improve reliability by (1) issuing hedged requests (a second request after a small delay if the first stalls), (2) failing over to configured alternate endpoints/regions when primary hosts are degraded, and (3) supporting priority hints to inform the scheduler and hedging policy.

Motivation

• Long-tail outliers (cold starts, noisy neighbors) hurt UX even when averages look fine.
• Many APIs expose multiple regions/hosts; automatic failover avoids manual retries.
• Not all requests are equal—priority should shape timeouts, hedging delays, and retry budgets.

Goals

1. Hedging: launch a backup request after hedgeDelayMs if the first hasn’t received headers; cancel the loser on winner resolve.
2. Alt-Endpoint Failover: per-host list of alternates (e.g., api-us, api-eu, api-ap); choose next on network error, 5xx, or breaker-open; preserve method/body/headers.
3. Priority Hints: priority: "high" | "normal" | "low" adjusts default timeoutMs, hedgeDelayMs, and retry/backoff caps.

API (additive)

import { fastFetch, configureFastFetch } from "@hoangsonw/fast-fetch";

configureFastFetch({
  hedging: {
    enabled: true,
    hedgeDelayMs: 200,           // when to launch hedge if no headers yet
    maxHedges: 1,                // number of backup attempts per call
    cancelLoser: true            // abort the slower in-flight
  },
  failover: {
    enabled: true,
    policy: "sequential",        // "sequential" | "round-robin" | "random"
    map: {
      "https://api.example.com": [
        "https://api-eu.example.com",
        "https://api-ap.example.com"
      ]
    },
    failOnStatuses: [500, 502, 503, 504, 522, 524], // overridable
    respectCircuitBreaker: true
  },
  priorities: {
    high:   { timeoutMs: 8000,  hedgeDelayMs: 120, retryBudget: { maxAttempts: 5 } },
    normal: { timeoutMs: 15000, hedgeDelayMs: 200, retryBudget: { maxAttempts: 4 } },
    low:    { timeoutMs: 25000, hedgeDelayMs: 350, retryBudget: { maxAttempts: 2 } }
  }
});

Per-request overrides:

await fastFetch("/orders?id=123", {
  priority: "high",
  hedging: { hedgeDelayMs: 100 },    // override
  failover: { policy: "round-robin" } // override
});

Behavior Details

• Hedging lifecycle:

  • Start request A → if no headers by hedgeDelayMs, start request B (same endpoint).
  • First to receive headers “wins”; losing request is aborted (if cancelLoser).
  • Hedging counts toward retry budget; integrates with existing backoff/timeout logic.

• Failover:

  • On qualifying failure (network, breaker-open, or failOnStatuses), swap base URL with next alternate and replay request.
  • Preserves method/body; re-signing hooks can be supported via user-provided beforeSend callback (future).

• Priority:

  • Sets sensible defaults for timeout, hedging delay, and retry budget.
  • Users can still override per-request.

Examples

Kill tail latency on product page

const res = await fastFetch("/api/product/42", {
  priority: "high",
  hedging: { hedgeDelayMs: 120 }
});

Global failover map

configureFastFetch({
  failover: {
    enabled: true,
    map: {
      "https://api.mycorp.com": [
        "https://api-eu.mycorp.com",
        "https://api-ap.mycorp.com"
      ]
    }
  }
});

Acceptance Criteria

• Hedged requests launch after delay and cancel the loser; only one Response is resolved to the caller.
• Failover switches to alternates on qualifying errors and respects circuit-breaker state.
• Priorities adjust defaults as configured; per-request overrides take precedence.
• Works with existing dedup, timeouts, backoff, and (optional) circuit breaker features.
• Unit tests: hedging win/lose paths, failover sequencing, priority defaulting, and interaction with AbortSignals.
• README updated with guidance and caveats (idempotency, signing, metrics).

Notes & Caveats

• Idempotency: Hedging duplicates requests—safe for GET/HEAD; for mutating verbs require idempotency keys or disable hedging.
• Auth/Signing: If headers or bodies are time-bound (HMAC, nonce), expose a beforeSend(req) hook (follow-up) to re-sign on failover/hedge.
• Metrics: Consider optional event hooks (onHedge, onFailover, onWinner) for observability (separate issue).