[Feature Request] Add the ability to manually control caching rather than relying upon the smart cache during generation

[matthock]

Describe the Issue
I would like the ability to more directly control KV cache slots during a koboldcpp API Generate call. Specifically, I would like to be able to include a field as part of the generate API call giving one or more cache slot IDs and have the back end use either that slot, or smart cache among the listed slots, rather than using full smart cache over the entire set of cache slots.

Additional Information:
My immediate desire for this relates to using SillyTavern with the MessageSummarize extension. This produces a message by message summary entry via a call after each response to the back end, rather than the standard Summarize extension that will process at long periods. I am finding with this setup, the smart cache almost never finds valid cache when it switches back to the regular chat. This change would need to be paired with one made otherwise to the SillyTavern connection for KoboldCPP to allow optionally specifying the slots as part of the connection profile. Then, two different connection profiles with a different subset of slots could be used to allow MessageSummarize to operate either in one fixed slot, or depending on feature implementation, tell it to avoid caching the response at all, as each of these will necessarily be a completely bespoke and short call without any real overlap.

A potentially easier first implementation step would be to add a no-cache flag to the generate call that prompts KoboldCPP to store the current cache, perform the generate, then immediately reload what was swapped out. This wouldn't fit some other ideas I had for how to use it (which involved using the slots to represent individual characters within a multi-character chat, allowing each character's stream of thought and information to be independent of others), but it would solve this particular issue.

[LostRuins]

Are you able to use your extension to call koboldcpp's API directly? KoboldCpp actually has a semi-hidden API for managing cache slots, check out the 3 endpoints at https://lite.koboldai.net/koboldcpp_api#/api%2Fextra%2Fdata/post_api_extra_data_save , that allows you to manipulate individual slots via the API.

[henk717]

I take this as a request for us not to remove that concedo, since I know you were originally considering it obsolete. Still, it may make more sense in a different menu rather than the admin menu.

[matthock]

@LostRuins I did see those commands when I was looking through the API. It's not clear how those overlap with SmartCache detection, though. If I were to call one of those, would it pin the cache slot and override SmartCache detection? That was part of why I envisioned it as part of the Generate call, it would be unambiguous that it's THIS Generate that belongs in the cache. This also potentially has interactions with Multi User mode, where if you're doing admin cache swaps via the API, it's not clear how it would handle timing relative to generate calls and queued user generate calls. Multiuser in particular is relevant because that's how I was planning on configuring SillyTavern to use it - the extension in question allows you to specify an alternative ST connection endpoint for just that feature, so I was planning on adding the KCPP instance to ST multiple times as different users, with each user given a subset of KV slots.

[LostRuins]

If you do use this caching feature you should disable smartcache. They're not meant to be used together. You'd manually trigger the cache save/load from your extension to whatever slot you want, every time you want it. It's full, flexible manual cache save/load.