sd: sync to master-646-0baf721

[wbruna]

On top of #2213 . LTX 2.3 support, some sampler changes, and a vocab file refactor.

Several added files:

otherarch/sdcpp/guidance.cpp
otherarch/sdcpp/guidance.h
otherarch/sdcpp/ltx_audio_vae.h
otherarch/sdcpp/ltx_vae.hpp
otherarch/sdcpp/tokenizers/gemma_tokenizer.cpp
otherarch/sdcpp/tokenizers/gemma_tokenizer.h
otherarch/sdcpp/tokenizers/vocab/clip_merges.hpp
otherarch/sdcpp/tokenizers/vocab/gemma_merges.hpp
otherarch/sdcpp/tokenizers/vocab/gemma_vocab.hpp
otherarch/sdcpp/tokenizers/vocab/mistral_merges.hpp
otherarch/sdcpp/tokenizers/vocab/mistral_vocab.hpp
otherarch/sdcpp/tokenizers/vocab/qwen_merges.hpp
otherarch/sdcpp/tokenizers/vocab/t5.hpp

and a few removed:

otherarch/sdcpp/tokenizers/vocab/clip_t5.hpp
otherarch/sdcpp/tokenizers/vocab/mistral.hpp
otherarch/sdcpp/tokenizers/vocab/qwen.hpp

so I ended up sorting and rebuilding the source file list.

To fix the linking errors, I've added load_gemma_merges and load_gemma_vocab_json functions with search+replace from the mistral ones.

Edit: updated to master-633-5b0267e to get a few new fixes.

Edit 2: updated to master-637-ef92a00 for more LTX stuff, and a LoRA offloading performance fix.

Edit 3: updated to master-642-3a8788c, with more LTX stuff, and a small refactor.

Edit 4: updated to master-646-0baf721 for more LTX changes.

[LostRuins]

Alright now that the other PR is merged, this is next on the list after I test out the gemma vocab extraction and minifications

[henk717]

Does this already enable LTX or will that require additionals?

[wbruna]

It's just the sync for now. It'll still need at least the gemma external merges and embed files, support for two additional models (embeddings connectors and audio VAE), and likely a file remapping for the llm.

[wbruna]

@LostRuins , I'm hitting a crash bug on the current concedo_experimental version related to offloading with LoRAs; I believe the LoRA caching code is interacting badly with the recent backend changes. But apparently the "LoRA offloading performance fix" from this PR fixes it, too.

Do you intend to merge this soon? I can make a separate PR for the fix if you don't.

[LostRuins]

I was planning to defer this PR until I could look at LTX 2.3, but I can merge it early if you prefer. Would that be easier?

[wbruna]

I was planning to defer this PR until I could look at LTX 2.3, but I can merge it early if you prefer. Would that be easier?

It would be a bit easier to test, since we'd be in sync with upstream. There is a risk of regressions, but I believe it would be better for us to find them sooner anyway.