Concurrent Lambda invocations generate byte-identical JWTs causing transient 404 on installation token endpoint

[vegardx]

Description

When multiple scale-up Lambda invocations run concurrently (common during burst workloads with 100+ workflow_job events), the GitHub App JWT generation produces byte-identical tokens. GitHub rejects these duplicates (likely replay protection), causing POST /app/installations/{id}/access_tokens to return HTTP 404. This triggers silent batch dropping (see related issue: silent batch drop) and permanently loses the affected jobs.

Root Cause

The @octokit/auth-app library (via universal-github-app-jwt) generates JWTs with only { iat, exp, iss } claims — no jti (JWT ID). The iat uses seconds precision (Math.floor(Date.now() / 1000)). When multiple Lambda invocations generate JWTs within the same second using the same App ID and private key, they produce byte-identical tokens.

GitHub rejects duplicate JWTs, causing the POST /app/installations/{id}/access_tokens request to be treated as unauthenticated. An unauthenticated request to this endpoint returns HTTP 404 (GitHub won't confirm resource existence to unauthorized callers).

The 404 is transient — the same installation ID succeeds on subsequent requests seconds later.

Observed Error

{
  "level": "ERROR",
  "message": "Error processing batch (size: 4): Not Found, ignoring batch",
  "error": {
    "name": "HttpError",
    "status": 404,
    "request": {
      "method": "POST",
      "url": "https://api.<redacted>.ghe.com/app/installations/<redacted>/access_tokens"
    },
    "response": { "status": 404, "data": { "message": "Not Found" } }
  }
}

This repeats for multiple concurrent invocations in rapid succession while other invocations at the same time succeed with the same installation ID.

Impact

• Combined with the silent batch drop issue, this permanently loses SQS messages and their corresponding jobs.
• The failure rate correlates with burst size and number of runner type configurations. More configurations = more concurrent Lambdas = higher probability of same-second JWT generation.
• Small workloads work fine; large matrix workflows trigger this consistently.

Environment

• Module version: ~> 7.3
• GitHub: Enterprise Cloud with Data Residency (ghe.com)
• Multi-runner module with 12+ runner type configurations

Suggested Fixes

Fix 1: Add jti claim to JWT generation

Add a unique jti (JWT ID) claim to the JWT payload to prevent byte-identical tokens. This can be done via @octokit/auth-app's createJwt callback:

import { randomUUID } from 'node:crypto';

const auth = createAppAuth({
  appId,
  createJwt: async ({ appId, privateKey }) => {
    const now = Math.floor(Date.now() / 1000);
    const payload = { iat: now - 60, exp: now + 600, iss: appId, jti: randomUUID() };
    // ... sign with privateKey
  },
});

This eliminates the root cause of the transient 404s.

Alternatively, this could be fixed upstream in universal-github-app-jwt by always including a jti claim.

Fix 2: Retry installation token API call with backoff

As a defense-in-depth measure, treat HTTP 404 on POST /app/installations/{id}/access_tokens as a transient error and retry within createGithubInstallationAuth():

const installationId = await getInstallationId(githubAppClient, enableOrgLevel, payload);
let ghAuth, retries = 0;
while (retries < 3) {
  try {
    ghAuth = await createGithubInstallationAuth(installationId, ghesApiUrl);
    break;
  } catch (e) {
    if (e.status === 404 && retries < 2) {
      retries++;
      await new Promise(r => setTimeout(r, 1000 * retries));
      continue;
    }
    throw e;
  }
}

Our Workaround

We have applied both fixes locally:

1. Custom createJwt callback with jti claim via crypto.randomUUID()
2. Retry with backoff on 404 in createGithubInstallationAuth()

[vegardx]

After digging into this, I found that @octokit/auth-app v8.1.2 (already used by this project) has a first-class createJwt callback option that completely bypasses universal-github-app-jwt. This is cleaner than either of the fixes I originally proposed.

How it works

createAppAuth accepts either privateKey or createJwt (mutually exclusive):

type CreateJwt = (
  appId: string | number,
  timeDifference?: number
) => Promise<{ jwt: string; expiresAt: string }>;

When createJwt is provided, @octokit/auth-app calls it instead of universal-github-app-jwt for all JWT generation — both for { type: 'app' } and internally for { type: 'installation' } auth.

Proposed change

In auth.ts — createAuth(), replace passing privateKey to createAppAuth with a createJwt callback that signs JWTs using node:crypto and includes a jti (JWT ID) claim:

import { createSign, randomUUID } from 'node:crypto';

function signJwt(payload: Record<string, unknown>, privateKey: string): string {
  const header = { alg: 'RS256', typ: 'JWT' };
  const encode = (obj: unknown) =>
    Buffer.from(JSON.stringify(obj)).toString('base64url');
  const message = `${encode(header)}.${encode(payload)}`;
  const signature = createSign('RSA-SHA256')
    .update(message)
    .sign(privateKey, 'base64url');
  return `${message}.${signature}`;
}

async function createAuth(installationId, ghesApiUrl): Promise<AuthInterface> {
  const appId = parseInt(await getParameter(process.env.PARAMETER_GITHUB_APP_ID_NAME));
  const privateKey = Buffer.from(
    await getParameter(process.env.PARAMETER_GITHUB_APP_KEY_BASE64_NAME),
    'base64',
  )
    .toString()
    .replace('/[\\n]/g', String.fromCharCode(10));

  const createJwt = async (appId: string | number, timeDifference?: number) => {
    const now = Math.floor(Date.now() / 1000) + (timeDifference ?? 0);
    const iat = now - 30;
    const exp = iat + 600;
    const jwt = signJwt({ iat, exp, iss: appId, jti: randomUUID() }, privateKey);
    return { jwt, expiresAt: new Date(exp * 1000).toISOString() };
  };

  let authOptions = { appId, createJwt };
  // ... rest unchanged (installationId, ghesApiUrl handling)

  return createAppAuth(authOptions);
}

Why this is better than the originally proposed fixes

	Fix 1 (original)	Fix 2 (original)	This approach
Addresses root cause	✅ Conceptually, but no supported injection point was identified	❌ Symptom-level retry	✅ Uses official createJwt API
New dependencies	Would need jsonwebtoken or manual JWT signing	None	None — node:crypto is built-in
API stability	Custom patching of internals	Stable	createJwt is a typed, documented option in @octokit/auth-app
Scope of change	Unknown	createGithubInstallationAuth only	createAuth only — all callers benefit automatically
PKCS#1 support	❌ universal-github-app-jwt only supports PKCS#8	N/A	✅ node:crypto.createSign supports both PKCS#1 and PKCS#8

Details

• Zero new dependencies — node:crypto.createSign and randomUUID are built-in on Node.js 24.x (the Lambda runtime)
• universal-github-app-jwt becomes unused — it remains as a transitive dependency of @octokit/auth-app but is never called
• timeDifference (clock drift correction used by @octokit/auth-app) is properly forwarded
• The iat/exp logic mirrors universal-github-app-jwt exactly: 30s safety margin, 10-minute expiry
• The \n replacement logic for private keys stored as single-line env vars is preserved

Retry as defense-in-depth

The retry-on-404 fix (Fix 2) is still worth considering as a separate, additive defense-in-depth measure. It would catch any other transient 404 scenarios unrelated to JWT collisions. But with jti in place, the root cause is eliminated and retries become a safety net rather than the primary fix.