feat: Support for DRY and XTC high level sampler API (already existing in llama.cpp)...

[onzag]

Feature Description

Currently the samplers for top_p temperature, etc... are included by default, but I don't see anywhere, in the docs or in the type definitions ot have DRY and XTC data, which appear clearly in the llama.cpp and doesn't seem like they'd be too hard to implement as they are natively supported.

abetlen/llama-cpp-python#1813

Similar comments about this in llama-cpp-python.

The Solution

Adding the options to dry_sampler and xtc_sampler with the given options for generateCompletition and other functions.

dry_sampler

• enabled boolean
• multiplier number,
• base number,
• allowed_length number
• penalty_last_n number
• seq_breakers: Array

xtc_sampler

• enabled boolean
• probability number
• threshold: number
• min_keep: number
• seed?: number

Considered Alternatives

I couldn't find any workaround, checked grammar defintions, checked for any mention of dry or xtc and couldn't find it.

Additional Context

No response

Related Features to This Feature Request

• Metal support
• CUDA support
• Vulkan support
• Grammar
• Function calling

Are you willing to resolve this issue by submitting a Pull Request?

Yes, I have the time, but I don't know how to start. I would need guidance.

[onzag]

https://github.com/withcatai/node-llama-cpp/blob/master/llama/addon/AddonSampler.cpp

I assume it takes on from here.

[onzag]

I am trying but not a C coder :( nor a NodeJS API native extensions developer.

I think I could figure it out eventually, but it would take me a while to grasp the native side of NApi.

Got trying to get the seq_breakers and num_breakers from what is supposed to be an array list, also the function is defined as

llama_sampler_init_dry(const struct llama_vocab * vocab, int32_t n_ctx_train, float dry_multiplier, float dry_base, int32_t dry_allowed_length, int32_t dry_penalty_last_n, const char** seq_breakers, size_t num_breakers) {

no idea where I'd get the vocab or what that ctx_train is meant to be.

Tho I found this reference llama_sampler_init_dry(vocab, llama_model_n_ctx_train(model), params.dry_multiplier, params.dry_base, params.dry_allowed_length, params.dry_penalty_last_n, c_breakers.data(), c_breakers.size())

if (config.Has("drySampling")) {
        Napi::Value drySamplingValue = config.Get("drySampling");
        if (drySamplingValue.IsObject()) {
            Napi::Object drySampling = drySamplingValue.As<Napi::Object>();
            bool enabled = drySampling.Has("enabled") ? drySampling.Get("enabled").As<Napi::Boolean>().Value() : false;

            if (enabled) {
                auto multiplierDrySampling = drySampling.Has("multiplier")
                    ? drySampling.Get("multiplier").As<Napi::Number>().FloatValue()
                    : 1.0f;
                auto baseDrySampling = drySampling.Has("base")
                    ? drySampling.Get("base").As<Napi::Number>().FloatValue()
                    : 0.0f;
                auto allowedLengthDrySampling = drySampling.Has("allowedLength")
                    ? drySampling.Get("allowedLength").As<Napi::Number>().Int32Value()
                    : 0;
                auto penaltyLastNDrySampling = drySampling.Has("penaltyLastN")
                    ? drySampling.Get("penaltyLastN").As<Napi::Number>().Int32Value()
                    : 0;

                 // TODO
                
            } else if (drySampler != nullptr) {
                freeChain();
                llama_sampler_free(drySampler);
                drySampler = nullptr;
            }
        } else if (drySampler != nullptr) {
            freeChain();
            llama_sampler_free(drySampler);
            drySampler = nullptr;
        }
    } else if (drySampler != nullptr) {
        freeChain();
        llama_sampler_free(drySampler);
        drySampler = nullptr;
    }