Issue 51 - Event Sweeper & Consistency Reconciliation Job (backend)

[23abdul23]

Event Sweeper & Consistency Reconciliation Job

Issue Type: First-come
Points: 30

Description

Implement a scheduled background job that ensures eventual consistency for SnapMap’s event system. This job acts as a safety net to handle edge cases such as missed Kafka events, exhausted retries, or photos that remain unassigned to any event.

The sweeper does not perform clustering itself. Instead, it detects inconsistencies and re-triggers the existing Kafka-based clustering flow.

---

Scope

• Periodically detect photos without an assigned eventId
• Requeue eligible photos for clustering via Kafka
• Finalize inactive events based on time thresholds
• Ensure the system converges to a consistent state over time

---

Folder & Files

jobs/
  eventSweeper.js

services/
  eventSweeperService.js

---

File Responsibilities

jobs/eventSweeper.js

Responsible for scheduling and orchestration only.

• Runs on a fixed interval (or cron)
• Calls service-layer functions
• Ensures the job can safely re-run multiple times

Main function:

• runEventSweeper()

---

services/eventSweeperService.js

Contains database queries and reconciliation logic.

Key functions to implement:

• findUnassignedPhotos()
• requeuePhotosForClustering(photos)
• finalizeInactiveEvents()

Uses:

• models/Photo.js
• models/Event.js
• Kafka producer (to requeue photos)

---

Function Responsibilities & Call Flow

runEventSweeper()

Entry point for the sweeper job.

Flow:

1. Calls findUnassignedPhotos()
2. If unassigned photos are found, calls requeuePhotosForClustering(photos)
3. Calls finalizeInactiveEvents()
4. Exits safely until the next scheduled run

---

findUnassignedPhotos()

Finds photos that do not have an eventId and are still eligible for clustering.

What this function does:

• Queries the Photo collection with bounded, time-based filters
• Avoids full or unbounded database scans
• Returns photos that should be reconsidered

Called by:

• runEventSweeper()

---

requeuePhotosForClustering(photos)

Re-emits Kafka messages for photos that were missed or not clustered correctly.

What this function does:

• Iterates over unassigned photos
• Publishes metadata-only Kafka messages to the photo-uploads topic
• Allows the Event Clustering Worker (Issue 3) to reprocess them

Called by:

• runEventSweeper()

---

finalizeInactiveEvents()

Finalizes events that have been inactive beyond a configured time threshold.

What this function does:

• Finds ACTIVE events with no recent photo activity
• Marks them as FINALIZED
• Prevents further clustering into finalized events

Called by:

• runEventSweeper()

---

Acceptance Criteria

• Job runs without blocking application startup
• No full or unbounded database scans are performed
• Unassigned photos are eventually reprocessed
• Inactive events are finalized correctly
• Job is safe to run multiple times

---

Notes

• This job is not the primary clustering mechanism
• It complements Kafka-based processing
• Must be configurable via environment variables

Ask me things if Confused on Discord, and refer to ARCHITECTURE.md in root folder.