Move getAdjustedTokenCountFromModel from CLI to core

[RomneyDa]

Summary

This PR moves the getAdjustedTokenCountFromModel function from the CLI tokenizer to core/llm/countTokens.ts and integrates it with core's token counting logic.

Changes

• Moved function: Extracted getAdjustedTokenCountFromModel from CLI to core
• Integration: Updated core's countTokens() to apply model-specific multipliers
• Tests: Added comprehensive test suite for the function in core
• CLI Update: Updated CLI tokenizer to import from core instead of duplicating code
• Cleanup: Removed duplicate constants and implementation from CLI

Why?

• DRY: Eliminates code duplication between CLI and core
• Consistency: Ensures token counting uses the same multipliers everywhere
• Maintainability: Single source of truth for token adjustment logic
• Better coverage: The llama tokenizer (~= gpt tokenizer) now applies safety multipliers in core as well

Model Multipliers

The function applies these multipliers based on empirical data:

• Claude: 1.23x (Anthropic tokenizer produces ~23% more tokens)
• Gemini: 1.18x (Gemini tokenizer produces ~18% more tokens)
• Mistral family: 1.26x (includes mistral, mixtral, devstral, codestral)

Testing

Added unit tests covering:

• Base cases (no multiplier)
• All model-specific multipliers
• Edge cases (zero tokens, undefined model names)
• Case-insensitivity

Breaking Changes

None - this is backward compatible. The function signature changed slightly (takes model name string instead of ModelConfig) but all call sites have been updated.

---

Continue Tasks: ▶️ 1 queued · ✅ 2 no changes — View all

---

Summary by cubic

Moves getAdjustedTokenCountFromModel to core and applies model-specific multipliers inside core token counting. This unifies token estimates across core and CLI.

• Refactors
  • Centralized getAdjustedTokenCountFromModel in core/llm/getAdjustedTokenCount.ts and integrated it into countTokens().
  • Applied model multipliers: Claude 1.23x, Gemini 1.18x, Mistral family 1.26x.
  • CLI now imports from core; removed duplicate logic and constants.
  • Added unit tests for multipliers and edge cases. No breaking changes.

^{Written for commit 7c32e1e. Summary will update on new commits.}

[github-actions]

⚠️ PR Title Format

Your PR title doesn't follow the conventional commit format, but this won't block your PR from being merged. We recommend using this format for better project organization.

Expected Format:

<type>[optional scope]: <description>

Examples:

• feat: add changelog generation support
• fix: resolve login redirect issue
• docs: update README with new instructions
• chore: update dependencies

Valid Types:

feat, fix, docs, style, refactor, perf, test, build, ci, chore, revert

This helps with:

• 📝 Automatic changelog generation
• 🚀 Automated semantic versioning
• 📊 Better project history tracking

This is a non-blocking warning - your PR can still be merged without fixing this.

[github-actions]

✅ Review Complete

Code Review Summary

⚠️ Continue API authentication failed. Please check your CONTINUE_API_KEY.

---

[cubic-dev-ai]

2 issues found across 3 files

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="extensions/cli/src/util/tokenizer.ts">

<violation number="1" location="extensions/cli/src/util/tokenizer.ts:3">
P2: CLI now imports getAdjustedTokenCountFromModel from core/llm/countTokens.js, which pulls in heavy tokenizer dependencies at module load time, likely regressing CLI startup/memory for a simple multiplier helper.</violation>
</file>

<file name="core/llm/countTokens.ts">

<violation number="1" location="core/llm/countTokens.ts:166">
P2: countTokensAsync does not apply model-specific token multipliers, so async token counts can be lower than countTokens for Claude/Gemini/Mistral models, affecting limit checks like readFileLimit.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cubic-dev-ai]

1 issue found across 1 file (changes from recent commits).

Prompt for AI agents (all issues)

Check if these issues are valid — if so, understand the root cause of each and fix them.


<file name="core/llm/getAdjustedTokenCount.ts">

<violation number="1" location="core/llm/getAdjustedTokenCount.ts:31">
P2: Regression: Mistral-family models like “Ministral”/“Mathstral” no longer match the multiplier because the broad `includes("stral")` check was replaced with a strict allowlist.</violation>
</file>

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cubic-dev-ai]

P2: Regression: Mistral-family models like “Ministral”/“Mathstral” no longer match the multiplier because the broad includes("stral") check was replaced with a strict allowlist.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At core/llm/getAdjustedTokenCount.ts, line 31:

<comment>Regression: Mistral-family models like “Ministral”/“Mathstral” no longer match the multiplier because the broad `includes("stral")` check was replaced with a strict allowlist.</comment>

<file context>
@@ -27,8 +27,13 @@ export function getAdjustedTokenCountFromModel(
-  } else if (lowerModelName.includes("stral")) {
-    // devstral, mixtral, mistral, etc
+  } else if (
+    lowerModelName.includes("mistral") ||
+    lowerModelName.includes("mixtral") ||
+    lowerModelName.includes("codestral") ||
</file context>