feat(inference): WIP inference lifecycle prototype — discover, validate, register, serve

[bussyjd]

Draft PR for issue #300 — Hermes Agent automated inference lifecycle.

What's here

1,942 lines of Python prototype across 8 files in internal/inference/lifecycle/:

File	Lines	Purpose
registry.py	253	SQLite model registry with state machine
hardware.py	188	GPU/system auto-detection, optimal ngl
discover.py	218	x/LocalLLaMA signal sourcing via x-cli
validate.py	347	llama-bench + ToolCall-15 eval pipeline
serve.py	316	systemd service mgmt, hot-swap + rollback
api.py	272	FastAPI routes for all lifecycle endpoints
test_inference_lifecycle.py	348	Unit tests for all modules

Context

Built during prototyping session where we:

• Set up llama.cpp + TurboQuant on SilverMesh (AMD RX 6800 XT)
• Benchmarked Qwen3.5-27B Q6_K → 15/15 on ToolCall-15
• Validated x-cli for community signal sourcing
• Tested llama-server as systemd service with auto-restart

Status

WIP — needs review, integration with obol-stack ServiceOffer CRD, and x402 payment wiring from PR #288.

Refs: #300

In general, to fix log injection you should sanitize any user‑controlled strings before logging: strip or replace newline and carriage‑return characters (and optionally other control characters), and avoid logging raw request bodies or other large, untrusted text without normalization. For structured logs that end up as plain text, removing \r and \n is typically sufficient to prevent forging extra log lines.

For this specific case, the best minimal fix is to sanitize model_id before logging it in Registry.update_status. The model_id value can be arbitrarily controlled via the promote_model and retire_model endpoints, and is logged at line 163. We can defensively normalize it right before logging by replacing any \r and \n characters with safe placeholders (e.g., a space or empty string) in a local variable used only for logging. This keeps the database operations and functional behavior unchanged, because all real uses of model_id (SQL queries, state transitions) continue to use the original string; only the log message sees the sanitized version.

Concretely in internal/inference/lifecycle/registry.py, inside update_status, we will:

• Introduce a local variable safe_model_id right before the logger.info call.
• Set safe_model_id to a sanitized version of model_id, e.g. model_id.replace("\r", "").replace("\n", "").
• Change the logger.info call to log safe_model_id instead of model_id.

No additional imports are needed; we rely only on standard string methods. This single change addresses all three CodeQL alert variants because they all trace to the same logging sink.

---

In general, to fix log injection issues, sanitize any user-controlled strings before logging them, especially removing or neutralizing newline and carriage-return characters (and optionally other control characters). When using structured/log-framework formatting (like %s in logger.info), keep that pattern but pass in a sanitized version of the untrusted value.

For this specific case, the best targeted fix is to sanitize model_id before it is logged in start_serving. Since we cannot assume how Registry constrains IDs and we do not want to change functional behavior, we will introduce a small helper function in serve.py that strips \r and \n (and, for completeness, any ASCII control characters) from strings used in logs. We then apply this helper when logging model IDs in start_serving and hot_swap. This preserves all existing functionality (the registry operations use the original model_id), while ensuring that the string sent to the logger cannot inject extra log lines.

Concretely:

• In internal/inference/lifecycle/serve.py, define a helper _sanitize_for_log(value: str) -> str near the top of the file.
• Replace logger.info("Model %s (%s) now serving on port %d", model_id, model.name, port) with the same log message but passing sanitized versions of model_id and model.name to be conservative.
• Replace logger.error("Hot swap to %s failed: %s", new_model_id, e) and logger.info("Rolling back to %s", old_model_id) similarly, so that any logged model IDs are sanitized.
• No changes are required in api.py since it does not log the tainted data directly.

No new imports are necessary; the helper can be implemented with basic string methods and a simple comprehension.

In general, to fix log injection when logging user-controlled values, sanitize or normalize those values before logging. For plain-text logs, the most important step is to remove or neutralize newline and carriage-return characters (and optionally other non-printable control characters) so that a single log call cannot create multiple apparent log entries or otherwise corrupt log structure.

For this specific case, the best minimal fix without changing behavior is to ensure that any model_id value logged by hot_swap is sanitized first. We can achieve this by introducing a small helper function within internal/inference/lifecycle/serve.py that strips \r and \n from a string (and safely handles None and non-strings), and then using that helper when passing new_model_id and old_model_id into log calls. This keeps the functional behavior of the system intact while ensuring that any untrusted input appearing in the logs cannot inject extra log lines.

Concretely:

• Add a helper such as _sanitize_for_log(value: Optional[str]) -> str near the top of serve.py (after logger definition, for example). It should:
  • Return "" for None or non-string values.
  • Replace \r and \n with empty strings (or space).
• Update logger.error("Hot swap to %s failed: %s", new_model_id, e) to use the sanitized value: logger.error("Hot swap to %s failed: %s", _sanitize_for_log(new_model_id), e).
• Optionally, for completeness and consistency, also sanitize old_model_id when logging “Rolling back to %s”.

No new external dependencies are required; we only use standard Python string methods.

---