feat(webapp,redis): handle UNBLOCKED during ElastiCache role change (#3549)

## Summary

When ElastiCache demotes a primary to replica — during a Multi-AZ
failover or a vertical node-type change — the demoting primary issues an
`UNBLOCKED` reply to any in-flight blocking commands (`BLPOP`, `BRPOP`,
`BLMOVE`, `XREADGROUP ... BLOCK`, etc.) to clear them before the role
flips. ioredis surfaces these as `ReplyError` to caller code.

The shared `defaultReconnectOnError` added in #3548 only matches
`READONLY` and `LOADING`. This extends it to `UNBLOCKED` so the
disconnect-reconnect-retry cycle handles BLPOP-shaped errors the same
way the existing two cases handle non-blocking-command errors.

## Fix

```ts
export function defaultReconnectOnError(err: Error): boolean | 1 | 2 {
  const msg = err.message ?? "";
  if (
    msg.startsWith("READONLY") ||
    msg.startsWith("LOADING") ||
    msg.startsWith("UNBLOCKED")
  ) {
    return 2;
  }
  return false;
}
```

Returning `2` tells ioredis to disconnect, reconnect, and re-issue the
command. For a BLPOP that means a fresh BLPOP against the new primary
instead of the `UNBLOCKED` error escaping to the caller.

## Test plan

- [ ] CI green
- [ ] Trigger a Multi-AZ failover or a vertical scale event on an
ElastiCache replication group whose clients are running blocking
commands and confirm no `UNBLOCKED` errors surface to caller code during
the cutover.

Author: ericallam

.server-changes/redis-reconnect-on-unblocked.md

@@ -0,0 +1,6 @@
+---
+area: webapp
+type: improvement
+---
+
+Extend the shared ioredis `reconnectOnError` hook (PR #3548) to also match `UNBLOCKED` reply errors so blocking commands like BLPOP transparently reconnect-and-retry when the ElastiCache primary forces them to unblock during a node role change.

internal-packages/redis/src/index.ts

@@ -9,6 +9,12 @@ export { Redis, type Callback, type RedisOptions, type Result, type RedisCommand
  * reply errors to caller code over a healthy TCP/TLS connection (the
  * client keeps talking to a node whose role swapped underneath it).
  *
+ * UNBLOCKED is the BLPOP-shaped case: the Redis primary forcibly
+ * unblocks any blocking command on a connection whose node is about
+ * to be demoted, returning an UNBLOCKED reply. Surfaced 65 times on
+ * engine/v1/worker-actions/dequeue at the cutover instant during the
+ * TRI-8873 test-cloud scale-up dry-run.
+ *
  * Returning 2 tells ioredis to disconnect, reconnect, and retry the
  * command that triggered the error. After reconnect, DNS / SG routing
  * should land on a writable primary.
@@ -18,7 +24,13 @@ export { Redis, type Callback, type RedisOptions, type Result, type RedisCommand
  */
 export function defaultReconnectOnError(err: Error): boolean | 1 | 2 {
   const msg = err.message ?? "";
-  if (msg.startsWith("READONLY") || msg.startsWith("LOADING")) return 2;
+  if (
+    msg.startsWith("READONLY") ||
+    msg.startsWith("LOADING") ||
+    msg.startsWith("UNBLOCKED")
+  ) {
+    return 2;
+  }
   return false;
 }