Skip to content

Commit 38ee9e8

Browse files
authored
CI - Fix hint for fixing llm benchmarks (#4040)
# Description of Changes I believe that local users do not have API tokens for OpenAI, so the existing hint was not helpful. Apparently the correct path is to post `/update-llm-benchmark` on the PR and let the CI take care of it. # API and ABI breaking changes None # Expected complexity level and risk 1 # Testing None --------- Co-authored-by: Zeke Foppa <bfops@users.noreply.github.com>
1 parent f011750 commit 38ee9e8

2 files changed

Lines changed: 5 additions & 3 deletions

File tree

docs/DEVELOP.md

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,8 @@ This document explains how to configure the environment, run the LLM benchmark t
1616

1717
Use this single command to quickly unblock CI by regenerating hashes and running only GPT-5 for the minimal Rust + C# passes. This is not the full benchmark suite.
1818

19+
**Note: You will need OpenAI API keys to run this locally**. Alternatively, any SpacetimeDB member can comment `/update-llm-benchmark` on a PR to start a CI job to do this.
20+
1921
`cargo llm ci-quickfix`
2022
What this does:
2123
1. Runs Rust rustdoc_json pass for GPT-5 only.

tools/xtask-llm-benchmark/src/bin/llm_benchmark.rs

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -283,7 +283,7 @@ fn cmd_ci_check(args: CiCheckArgs) -> Result<()> {
283283
};
284284

285285
// Debug hint for how to (re)generate entries
286-
let hint_for = |_lang: Lang| -> &'static str { "cargo llm ci-quickfix" };
286+
let hint_for = |_lang: Lang| -> &'static str { "Check DEVELOP.md for instructions on how to proceed." };
287287

288288
// Load docs-benchmark summary to compare hashes against
289289
let summary_path = docs_benchmark_summary();
@@ -298,13 +298,13 @@ fn cmd_ci_check(args: CiCheckArgs) -> Result<()> {
298298
match xtask_llm_benchmark::context::resolve_mode_paths(mode) {
299299
Ok(paths) if !paths.is_empty() => {}
300300
Ok(_) => bail!(
301-
"CI check FAILED: {}/{} resolved to 0 paths.\nTry: {}",
301+
"CI check FAILED: {}/{} resolved to 0 paths.\n→ {}",
302302
mode,
303303
lang_str,
304304
hint_for(lang)
305305
),
306306
Err(e) => bail!(
307-
"CI check FAILED: {}/{} not available: {}.\nTry: {}",
307+
"CI check FAILED: {}/{} not available: {}.\n→ {}",
308308
mode,
309309
lang_str,
310310
e,

0 commit comments

Comments
 (0)