[BUG] Inline loading of models at Symlinks causes models to unload and reload

[jrruethe]

OS

Linux

GPU Library

CUDA 12.x

Python version

3.12

Describe the bug

I set inline_model_loading: true so I could load models dynamically from the models directory by name.

I saved the model from huggingface.co, such as ArtusDev_Qwen_Qwen3-Coder-30B-A3B-Instruct-EXL3, then I created a symlink named coder to that directory.

The goal was to be able to use "coder" as my model name for the API call while keeping a descriptive directory name on disk. This also lets me re-symlink to a different model without changing the clients making the calls.

However, I think it resolves the symlink, properly loads the model but thinks the model is called ArtusDev_Qwen_Qwen3-Coder-30B-A3B-Instruct-EXL3 instead of coder, then when the next API call comes in for model coder it unloads the model, then immediately reloads it (its the same model).

Reproduction steps

1. Set inline_model_loading: true
2. Save a model to the models directory. I used ArtusDev_Qwen_Qwen3-Coder-30B-A3B-Instruct-EXL3
3. ln -s ./ArtusDev_Qwen_Qwen3-Coder-30B-A3B-Instruct-EXL3 coder
4. Make an API call with the model name "coder"
5. The model will unload, then reload

Expected behavior

The model would stay loaded and be used.

Logs

No response

Additional context

This is super low priority

Acknowledgements

• I have looked for similar issues before submitting this one.
• I have read the disclaimer, and this issue is related to a code bug. If I have a question, I will use the Discord server.
• I understand that the developers have lives and my issue will be answered when possible.
• I understand the developers of this program are human, and I will ask my questions politely.

[jrruethe]

I'll add another thing I noticed related to this.

So lets say that I have the symlink coder pointing to a real model like ArtusDev_Qwen_Qwen3-Coder-30B-A3B-Instruct-EXL3.

In the config.yml file, I set the model_name: coder and max_seq_len: 32768.

Then, in ArtusDev_Qwen_Qwen3-Coder-30B-A3B-Instruct-EXL3/tabby_config.yml I set max_seq_len: 65535

Then I start up Tabby API. It will see "coder" in the config, and load "ArtusDev_Qwen_Qwen3-Coder-30B-A3B-Instruct-EXL3" (due to the symlink). It will see the currently loaded model as "ArtusDev_Qwen_Qwen3-Coder-30B-A3B-Instruct-EXL3".

At this point, there are two different paths I can take, and both show a bug:

Path A) I can make an API request with model name coder. This will not match the currently loaded name of "ArtusDev_Qwen_Qwen3-Coder-30B-A3B-Instruct-EXL3", so the model is unloaded and reloaded.

Path B) I can make an API request with the model name ArtusDev_Qwen_Qwen3-Coder-30B-A3B-Instruct-EXL3. The model is NOT unloaded, however it was loaded with a max_seq_len of 32768, and is not honoring the value from ArtusDev_Qwen_Qwen3-Coder-30B-A3B-Instruct-EXL3/tabby_config.yml.

I guess what I am saying is that when doing the initial load, if model/tabby_config.yml is present, those values should override any in the main config.yml (especially when inline loading is enabled). This might be a different bug not related to the symlink, if so I can open a different issue.

[kingbri1]

Regarding your second comment, the initial load from config.yml is equivalent to that of providing CLI args to the program launch command. The flow is as follows:

Highest priority
CLI args
config.yml
inline_config
hardcoded defaults
Lowest priority

If you want the inline settings to be used, please comment out the config.yml values. There could be a case made for subsequent API loads, but I'd need to look into it more.