VLM chat aplication via python jinja#4235
Open
dkalinowski wants to merge 5 commits into
Open
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the Visual Language Model (VLM) servables to support applying chat templates via a Python/Jinja processor (when Python support is enabled), including injecting <ov_genai_image_*> tags into the request JSON before template rendering, and adds exception handling around the C++ tokenizer chat-template path.
Changes:
- Added RapidJSON-based rewriting of the request JSON to prepend image tags into
messages[*].contentbefore callingPyJinjaTemplateProcessor::applyChatTemplate(Python-enabled builds). - Wrapped
tokenizer.apply_chat_template(...)intry/catchand improved error handling for invalid/missing chat templates (Python-disabled builds). - Added validation that the final prompt after template application is non-empty.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| src/llm/visual_language_model/legacy/servable.cpp | Adds Python/Jinja chat-template path with image-tag injection and improves error handling for chat-template application. |
| src/llm/visual_language_model/continuous_batching/servable.cpp | Mirrors the Python/Jinja chat-template path and image-tag injection for continuous batching, plus exception handling around tokenizer template application. |
Comments suppressed due to low confidence (2)
src/llm/visual_language_model/legacy/servable.cpp:322
msg.HasMember("content")also asserts if themessages[chatTurnIndex]element is not an object. Guard withmsg.IsObject()(or usemsg.GetObject().FindMember) beforeHasMemberto avoid crashes on unexpected request shapes.
if (chatTurnIndex < messages.Size()) {
auto& msg = messages[chatTurnIndex];
if (msg.HasMember("content") && msg["content"].IsString()) {
std::string newContent = imageTagString + msg["content"].GetString();
msg["content"].SetString(newContent.c_str(), newContent.length(), jsonDoc.GetAllocator());
}
src/llm/visual_language_model/continuous_batching/servable.cpp:126
msg.HasMember("content")asserts ifmessages[chatTurnIndex]is not an object. Guard withmsg.IsObject()(or usemsg.GetObject().FindMember) beforeHasMemberto avoid crashes on unexpected request shapes.
if (chatTurnIndex < messages.Size()) {
auto& msg = messages[chatTurnIndex];
if (msg.HasMember("content") && msg["content"].IsString()) {
std::string newContent = imageTagString + msg["content"].GetString();
msg["content"].SetString(newContent.c_str(), newContent.length(), jsonDoc.GetAllocator());
}
Comment on lines
+314
to
+319
| if (!jsonDoc.HasParseError() && jsonDoc.HasMember("messages") && jsonDoc["messages"].IsArray()) { | ||
| auto& messages = jsonDoc["messages"]; | ||
| for (const auto& [chatTurnIndex, imageTagString] : imageTags) { | ||
| if (chatTurnIndex < messages.Size()) { | ||
| auto& msg = messages[chatTurnIndex]; | ||
| if (msg.HasMember("content") && msg["content"].IsString()) { |
Comment on lines
+118
to
+123
| if (!jsonDoc.HasParseError() && jsonDoc.HasMember("messages") && jsonDoc["messages"].IsArray()) { | ||
| auto& messages = jsonDoc["messages"]; | ||
| for (const auto& [chatTurnIndex, imageTagString] : imageTags) { | ||
| if (chatTurnIndex < messages.Size()) { | ||
| auto& msg = messages[chatTurnIndex]; | ||
| if (msg.HasMember("content") && msg["content"].IsString()) { |
Comment on lines
+25
to
+34
| #pragma warning(push) | ||
| #pragma warning(disable : 4005 4309 6001 6385 6386 6326 6011 4005 4456 6246) | ||
| #pragma GCC diagnostic push | ||
| #pragma GCC diagnostic ignored "-Wdeprecated-declarations" | ||
| #include <rapidjson/document.h> | ||
| #include <rapidjson/stringbuffer.h> | ||
| #include <rapidjson/writer.h> | ||
| #pragma GCC diagnostic pop | ||
| #pragma warning(pop) | ||
|
|
99c9346 to
40dcbf7
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.