Skip to content

Commit 6b986ae

Browse files
committed
Simplify job reservation: no locking, rely on make() transaction
- Remove SELECT FOR UPDATE locking from job reservation - Conflicts (rare) resolved by make() transaction's duplicate key error - Second worker catches error and moves to next job - Simpler code, better performance on high-traffic jobs table
1 parent f0b7cd8 commit 6b986ae

File tree

1 file changed

+25
-19
lines changed

1 file changed

+25
-19
lines changed

docs/src/design/autopopulate-2.0-spec.md

Lines changed: 25 additions & 19 deletions
Original file line numberDiff line numberDiff line change
@@ -204,11 +204,11 @@ class JobsTable(Table):
204204
"""
205205
Attempt to reserve a job for processing.
206206
207-
Uses SELECT FOR UPDATE to prevent race conditions.
208-
Only reserves jobs with status='pending' and scheduled_time <= now.
207+
Updates status to 'reserved' if currently 'pending' and scheduled_time <= now.
208+
No locking is used; rare conflicts are resolved by the make() transaction.
209209
210210
Returns:
211-
True if reservation successful, False if already taken.
211+
True if reservation successful, False if job not found or not pending.
212212
"""
213213
...
214214

@@ -435,32 +435,38 @@ FilteredImage.jobs.refresh()
435435

436436
The jobs table is created with the appropriate primary key structure matching the target table's foreign-key-derived attributes.
437437

438-
### Race Condition Handling
438+
### Conflict Resolution
439439

440-
Job reservation uses database-level locking to prevent race conditions:
440+
Job reservation does **not** use transaction-level locking for simplicity and performance. Instead, conflicts are resolved at the `make()` transaction level:
441441

442-
```sql
443-
-- Reserve a job atomically
444-
START TRANSACTION;
445-
SELECT * FROM `_my_table__jobs`
446-
WHERE status = 'pending'
447-
AND scheduled_time <= NOW()
448-
ORDER BY priority DESC, scheduled_time ASC
449-
LIMIT 1
450-
FOR UPDATE SKIP LOCKED;
451-
452-
-- If row found, update it
442+
```python
443+
# Simple reservation (no locking)
453444
UPDATE `_my_table__jobs`
454445
SET status = 'reserved',
455446
reserved_time = NOW(),
456447
user = CURRENT_USER(),
457448
host = @@hostname,
458449
pid = CONNECTION_ID()
459-
WHERE <primary_key_match>;
460-
461-
COMMIT;
450+
WHERE status = 'pending'
451+
AND scheduled_time <= NOW()
452+
ORDER BY priority DESC, scheduled_time ASC
453+
LIMIT 1;
462454
```
463455

456+
**Conflict scenario** (rare):
457+
1. Two workers reserve the same job nearly simultaneously
458+
2. Both run `make()` for the same key
459+
3. First worker's `make()` transaction commits, inserting the result
460+
4. Second worker's `make()` transaction fails with duplicate key error
461+
5. Second worker catches the error and moves to the next job
462+
463+
**Why this is acceptable**:
464+
- Conflicts are rare in practice (requires near-simultaneous reservation)
465+
- The `make()` transaction already guarantees data integrity
466+
- Duplicate key error is a clean, expected signal
467+
- Avoids locking overhead on the high-traffic jobs table
468+
- Wasted computation is minimal compared to locking complexity
469+
464470
### Stale Reserved Job Detection
465471

466472
Reserved jobs that have been running too long may indicate crashed workers:

0 commit comments

Comments
 (0)