Fix failed rollouts counted as completed in AgentModeDaemon

[Mr-Neutr0n]

Summary

Fixes #355

• When vLLM returns HTTP 400 errors, rollouts end up with None reward and empty triplets. These were previously counted as "completed" in _async_run_until_finished, causing the completed count to exceed _total_tasks_queued (e.g. "Completed 33/32 tasks") and triggering assertion failures in get_train_data_batch.
• Added _is_failed_rollout() helper that detects rollouts with no reward and no triplets, identifying them as backend failures rather than real results.
• Failed rollouts are now skipped instead of stored, allowing tasks to be re-claimed and retried by the store's timeout mechanism.
• For v1 mode, wait_for_rollouts now only polls for rollout IDs that have not yet successfully completed, so successful retries are properly picked up.
• Removed the early "already processed" skip that could prevent a successful retry from overwriting a stale failed entry.

Test plan

• Deploy with a vLLM backend that intermittently returns HTTP 400 errors and verify that failed rollouts are retried rather than counted as completed
• Verify get_train_data_batch no longer hits the assert len(self._completed_rollouts_v0) == self._total_tasks_queued assertion
• Confirm that "Completed N/M tasks" log messages never show N > M
• Run existing training loop end-to-end to verify no regression in normal (no-error) flow

[Mr-Neutr0n]

Friendly bump! Let me know if there's anything I should update or improve to help move this forward.

[ultmaster]

I think we already have retried the failed rollouts. Why this change?

[Mr-Neutr0n]

@ultmaster — good question. The bug is more subtle than just "should we count failed rollouts": without this change, the first failed rollout is added to _completed_rollouts_v0 and then the wait_for_rollouts call passes list(self._task_id_to_original_sample.keys()) (all IDs, including ones already in the completed set) on the next loop. The retry infrastructure in the store is fine — it re-enqueues the failed task — but the loop is racing with it: the next wait_for_rollouts returns the same failed rollout, the loop sees it as already-completed, and the "Completed X/Y" counter increments past Y.

Concretely: the symptom in the PR description is "Completed 33/32 tasks", which only happens when failed rollouts have been added to the completed set. So the change is doing two things at once:

1. pending_ids filters out already-completed IDs, so the loop doesn't re-consider them.
2. _is_failed_rollout skips counting a failed result, so the retry actually has room to deliver a successful result.

Happy to split into two PRs if that's clearer; the two changes do interact (without #1, #2 alone doesn't help; without #2, #1 just shifts the off-by-one to the next failure).