Skip to content

Commit cba00a1

Browse files
vdusekclaude
andcommitted
fix: Add exponential backoff for shared RQ propagation delay in integration tests
In shared request queue mode, the Apify API has an eventual consistency propagation delay before newly added or reclaimed requests become visible in fetch_next_request(). Many integration tests were missing the call_with_exp_backoff() retry pattern, causing sporadic failures. Also added rq_access_mode parameter to call_with_exp_backoff() so callers can pass it directly instead of wrapping every call in an if/else block. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
1 parent d808305 commit cba00a1

File tree

2 files changed

+170
-120
lines changed

2 files changed

+170
-120
lines changed

tests/integration/_utils.py

Lines changed: 11 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -13,13 +13,23 @@
1313
T = TypeVar('T')
1414

1515

16-
async def call_with_exp_backoff(fn: Callable[[], Awaitable[T]], *, max_retries: int = 3) -> T | None:
16+
async def call_with_exp_backoff(
17+
fn: Callable[[], Awaitable[T]],
18+
*,
19+
rq_access_mode: str | None = None,
20+
max_retries: int = 3,
21+
) -> T | None:
1722
"""Call an async callable with exponential backoff retries until it returns a truthy value.
1823
1924
In shared request queue mode, there is a propagation delay before newly added, reclaimed, or handled requests
2025
become visible in the API (see https://github.com/apify/apify-sdk-python/issues/808). This helper retries with
2126
exponential backoff to handle that delay in integration tests.
27+
28+
When `rq_access_mode` is provided and is not `'shared'`, the function is called once without retries.
2229
"""
30+
if rq_access_mode is not None and rq_access_mode != 'shared':
31+
return await fn()
32+
2333
result = None
2434

2535
for attempt in range(max_retries):

0 commit comments

Comments
 (0)