generation_config.json path override per LLM node

[korund]

Component: LLM continuous batching, LLMCalculatorOptions / mediapipe registration
OVMS version: 2026.1.0.72cc0624 (OpenVINO backend 2026.1.0, OpenVINO GenAI backend 2026.1.0.0)

Context

When several deployments share the same on-disk model directory but need different generation defaults (e.g. different num_assistant_tokens, temperature, or sampling settings per served endpoint), the only current option is to duplicate the model directory — including the weights — because OVMS reads generation_config.json from a fixed name inside models_path. For multi-gigabyte LLMs this is impractical.
The same problem exists for graph.pbtxt, but is already solved there: graph_path in the mediapipe config entry lets one model directory back several deployments with different graphs. There is no equivalent for generation_config.json.

Related to #4221

Question

Would it be feasible to add a per-LLM-node override for the generation-config file path — analogous to graph_path? A natural shape would be either:

• a generation_config_path field in LLMCalculatorOptions (next to models_path), absolute or relative to models_path; or
• a sibling field at the mediapipe config-entry level (next to graph_path).
  From a quick read of openvino.genai, ContinuousBatchingPipeline accepts an optional GenerationConfig at construction and exposes set_config() post-construction, so the underlying mechanism appears to be already in place. The work seems contained within src/llm/language_model/continuous_batching/servable_initializer.cpp on the OVMS side.

Use case

Multiple served names backed by the same model weights, each with its own generation defaults. Without per-entry generation-config selection, each variant requires a full copy of the model directory on disk.

Open questions

• Is there a reason this hasn't been exposed yet — for example, a planned different mechanism (per-deployment overrides through some other channel), or an interaction with model auto-detection/conversion that I'm missing?
• Is one of the placement options preferred from the architecture side?

[mzegla]

Well, I don't think you missed anything. Current logic assumes model catalog holds configuration and defaults and loads pipeline with them. As you correctly identified, you can specify path to graph.pbtxt allowing for deployment with different LLM node options.
It works that way due to the fact that this file has been used by OVMS before LLMs support has been added - it's handled by MediaPipe framework that was used even before v3 OpenAI-compatible endpoints. So the capability of spawning different deployments from different graph.pbtxt that lead to the same model was there from the beginning.

Now generation_config.json is different as it is sort of treated as integral part of the model. Same as chat template and other configuration files from model catalog. Graph configuration only read model_path and that's the lowest granularity we have for LLMs. We didn't not make it configurable as it was never raised as an issue or request and we did not want to expand configuration with new options without strong justification.

With that said I can agree that it might be useful to completely decouple model weights (that are heavy and we don't want to have multiple copies of the same model on the filesystem) from configuration in some cases. How about having symbolic links? Like create catalog with generation_config.json and link everything else from original model catalog, so no copies are required.

[korund]

Thanks - symlinks (hardlinks on Windows, in my case) will work as a workaround, and I will go with that for now. Appreciated.

That said, I think there is a real gap worth surfacing here, and it becomes visible when this issue is read together with #4221.

In #4221 you drew a clear line between the two config surfaces:

calculator options in graph.pbtxt should have pipeline configuration that is shared between all requests across entire pipeline lifecycle [...]

and

generation parameters default values in generation_config.json should hold recommended configuration for per request parameters, so parameters that are independently owned by the request [...]

That taxonomy makes sense to me and I am not arguing with it. But if I apply it to my current situation, I end up with no place to put the value:

• graph.pbtxt is per-deployment, but by your own rule it should not carry per-request parameters - so num_assistant_tokens does not belong there.
• generation_config.json is exactly the right bucket for the parameter, but it lives inside the model catalog and there is only one per model - so it cannot express "two deployments of the same model with different defaults".

So for a parameter that your own taxonomy classifies as "per-request with a per-deployment default", the config has no per-deployment slot to actually store that default. The symlink trick works around this by fabricating a second "model" to get a second "deployment", which is why it feels off: it solves the problem in the wrong layer.

It also has a practical cost for the operator. Once a deployment is backed by a directory that looks like a real checkpoint but is actually a thin overlay of links plus one overridden file, routine cleanup becomes risky - it is no longer obvious which directory holds the actual weights and which is a derivative, and deleting what looks like a redundant copy can silently break a deployment.

To be clear, I am not asking for a generic override mechanism for everything in the model catalog. Chat template, tokenizer and so on really are integral parts of the model, and I agree the config surface should not grow without strong justification. generation_config.json is the one file that, by the definition you gave in #4221, holds per-request defaults rather than model-intrinsic configuration - so it is the one case where a per-deployment override in the server config would fill an existing gap rather than expand the taxonomy.

[mzegla]

Yes, generation_config.json is the right place for your parameter and I'm not denying it could be useful to parametrize generation config path somewhere in a deployment configuration. With that said, we would need to consider where to place it and whether that would be all we should expose - for example I can see how someone would like to run the same model with different chat templates for testing or other purposes.

I think the idea is valid overall.