Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
a0a1809
cookies not proxy
bobular Oct 18, 2025
d4b51be
getting started
bobular Oct 18, 2025
95b694a
added some genes
bobular Oct 18, 2025
0119dba
fetching working nicely, thanks Claude
bobular Oct 18, 2025
db28d0f
add skipFetch option to sites.json
bobular Oct 18, 2025
abcaa7e
remove useProxy unused prop
bobular Oct 18, 2025
caba1a2
compare and condense steps done
bobular Oct 19, 2025
8db4dff
simplify condensation merging by swapping B-A to A-B in second input
bobular Oct 19, 2025
aca3cab
types consolidation
bobular Oct 20, 2025
12e9376
aggregate phase 3
bobular Oct 20, 2025
3fed74a
standard loadSitesConfig and skip=true behaviour
bobular Oct 20, 2025
b96ed3c
full gene list and gene name feature
bobular Oct 20, 2025
f57700c
phase 1 configurable analysis_model
bobular Oct 21, 2025
55f1c63
update Claude notes
bobular Oct 21, 2025
d6907f5
first pass openai support
bobular Oct 21, 2025
fdf36e7
aiClient OO refactor
bobular Oct 21, 2025
d6b5182
docs update
bobular Oct 21, 2025
c1b8b77
remove unused polling code and documentation
bobular Oct 21, 2025
3f2ff51
max_completion_tokens for openai
bobular Oct 21, 2025
d949b6d
handle GPT-5 reasoning issue
bobular Oct 21, 2025
caded98
skip already performed comparisons
bobular Oct 21, 2025
f33b214
add deterministic topic size stats
bobular Oct 21, 2025
47307d6
html reports
bobular Oct 22, 2025
b2d3a92
modal rep
bobular Oct 22, 2025
d7b63b4
add simple gene summary pages
bobular Oct 22, 2025
ec48721
nicer popup link-outs
bobular Oct 22, 2025
700256e
better, conditional gene page links
bobular Oct 22, 2025
39f2843
verbiage tweak on links
bobular Oct 22, 2025
af8d4dc
AI notes and confidence column
bobular Oct 22, 2025
9449e26
add dissenting genes lists
bobular Oct 22, 2025
cda2c11
add headline style analysis
bobular Oct 23, 2025
e898a7e
add index.html step
bobular Oct 23, 2025
e99704b
prompt comparison aggregations to continue to refer to Summary A and …
bobular Nov 11, 2025
9401988
presentation by Claude Code and pptx skillz
bobular Nov 19, 2025
d26f132
add more detail for comparison phase
bobular Nov 19, 2025
24c3113
add promptfoo slide
bobular Nov 19, 2025
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,8 @@ tmp/
node_modules/
dist/
data/
!comparison/data/

*~

.yarn
Expand Down
329 changes: 278 additions & 51 deletions CLAUDE-ai-provider-comparison.md

Large diffs are not rendered by default.

65 changes: 65 additions & 0 deletions comparison/config/sites.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
{
"public_site": {
"name": "VectorBase",
"base_url": "https://vectorbase.org/vectorbase"
},
"_analysis_model": {
"model_string": "claude-sonnet-4-5-20250929",
"name": "claude4_5",
"model": "Claude Sonnet 4.5",
"platform": "anthropic"
},
"__analysis_model": {
"model_string": "claude-sonnet-4-20250514",
"name": "claude4",
"model": "Claude Sonnet 4",
"platform": "anthropic"
},
"analysis_model": {
"model_string": "gpt-5-2025-08-07",
"name": "gpt5",
"model": "GPT-5 (minimal reasoning)",
"platform": "openai"
},
"sites": [
{
"name": "claude4_5",
"hostname": "bmaccallum.vectorbase.org",
"appPath": "vectorbase.bmaccallum",
"model": "Claude Sonnet 4.5",
"skip": true,
"disableGenePageLinks": true
},
{
"name": "claude4_5thinking",
"hostname": "bmaccallum.vectorbase.org",
"appPath": "vectorbase.bmaccallum",
"model": "Claude Sonnet 4.5 Thinking",
"skip": false,
"disableGenePageLinks": false
},
{
"name": "claude4",
"hostname": "bmaccallum.vectorbase.org",
"appPath": "vectorbase.bmaccallum",
"model": "Claude Sonnet 4",
"skip": true
},
{
"name": "gpt5",
"hostname": "bmaccallum-b.vectorbase.org",
"appPath": "vectorbase.bmaccallum-b",
"model": "GPT-5",
"skip": false
},
{
"name": "gpt4o",
"hostname": "qa.vectorbase.org",
"appPath": "vectorbase.b69",
"model": "GPT-4o",
"skip": true
}
],
"endpoint": "/service/record-types/gene/searches/single_record_question_GeneRecordClasses_GeneRecordClass/reports/aiExpression",
"projectId": "VectorBase"
}
10 changes: 10 additions & 0 deletions comparison/data/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
# Ignore most intermediate data files
summaries/
comparisons/
condensed/
*.json
*.zip

# But DO check in the final aggregate reports (~20MB)
# These are the final research outputs worth preserving
!aggregate-reports/

Large diffs are not rendered by default.

367 changes: 367 additions & 0 deletions comparison/data/aggregate-reports/claude4/claude4-gpt4o-report.html

Large diffs are not rendered by default.

387 changes: 387 additions & 0 deletions comparison/data/aggregate-reports/claude4/claude4-gpt5-report.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

357 changes: 357 additions & 0 deletions comparison/data/aggregate-reports/claude4/claude4_5-gpt4o-report.html

Large diffs are not rendered by default.

387 changes: 387 additions & 0 deletions comparison/data/aggregate-reports/claude4/claude4_5-gpt5-report.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

377 changes: 377 additions & 0 deletions comparison/data/aggregate-reports/claude4/gpt4o-gpt5-report.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1,390 changes: 1,390 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP000610-gpt4o.html

Large diffs are not rendered by default.

1,446 changes: 1,446 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP000610-gpt5.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1,390 changes: 1,390 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP000693-gpt4o.html

Large diffs are not rendered by default.

1,558 changes: 1,558 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP000693-gpt5.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1,362 changes: 1,362 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP000999-gpt4o.html

Large diffs are not rendered by default.

1,418 changes: 1,418 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP000999-gpt5.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1,474 changes: 1,474 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP001212-gpt4o.html

Large diffs are not rendered by default.

1,614 changes: 1,614 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP001212-gpt5.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1,474 changes: 1,474 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP001713-gpt4o.html

Large diffs are not rendered by default.

1,586 changes: 1,586 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP001713-gpt5.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1,418 changes: 1,418 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP001826-gpt4o.html

Large diffs are not rendered by default.

1,530 changes: 1,530 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP001826-gpt5.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1,334 changes: 1,334 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP002851-gpt4o.html

Large diffs are not rendered by default.

1,446 changes: 1,446 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP002851-gpt5.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1,334 changes: 1,334 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP003751-gpt4o.html

Large diffs are not rendered by default.

1,418 changes: 1,418 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP003751-gpt5.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1,390 changes: 1,390 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP004036-gpt4o.html

Large diffs are not rendered by default.

1,418 changes: 1,418 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP004036-gpt5.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1,418 changes: 1,418 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP004203-gpt4o.html

Large diffs are not rendered by default.

1,474 changes: 1,474 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP004203-gpt5.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1,446 changes: 1,446 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP006414-gpt4o.html

Large diffs are not rendered by default.

1,614 changes: 1,614 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP006414-gpt5.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1,446 changes: 1,446 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP006747-gpt4o.html

Large diffs are not rendered by default.

1,446 changes: 1,446 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP006747-gpt5.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1,390 changes: 1,390 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP006796-gpt4o.html

Large diffs are not rendered by default.

1,474 changes: 1,474 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP006796-gpt5.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1,390 changes: 1,390 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP009551-gpt4o.html

Large diffs are not rendered by default.

1,474 changes: 1,474 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP009551-gpt5.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1,362 changes: 1,362 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP009646-gpt4o.html

Large diffs are not rendered by default.

1,390 changes: 1,390 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP009646-gpt5.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1,390 changes: 1,390 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP011294-gpt4o.html

Large diffs are not rendered by default.

1,474 changes: 1,474 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP011294-gpt5.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1,362 changes: 1,362 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP011434-gpt4o.html

Large diffs are not rendered by default.

1,362 changes: 1,362 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP011434-gpt5.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1,446 changes: 1,446 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP011630-gpt4o.html

Large diffs are not rendered by default.

1,558 changes: 1,558 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP011630-gpt5.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1,334 changes: 1,334 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP012703-gpt4o.html

Large diffs are not rendered by default.

1,390 changes: 1,390 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP012703-gpt5.html

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

Large diffs are not rendered by default.

1,362 changes: 1,362 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP013543-gpt4o.html

Large diffs are not rendered by default.

1,474 changes: 1,474 additions & 0 deletions comparison/data/aggregate-reports/claude4/html-summaries/AGAP013543-gpt5.html

Large diffs are not rendered by default.

76 changes: 76 additions & 0 deletions comparison/data/aggregate-reports/claude4/index.html
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Model Comparison Reports - Claude Sonnet 4</title>
<script src="https://cdn.tailwindcss.com"></script>
</head>
<body class="bg-gray-100 min-h-screen py-8">
<div class="container mx-auto px-4 max-w-6xl">
<div class="bg-white shadow-md rounded-lg p-6 mb-6">
<h1 class="text-3xl font-bold text-gray-800 mb-2">
VEuPathDB AI Gene Expression Summaries
</h1>
<p class="text-gray-600">
<span class="font-semibold">Analysis by:</span> Claude Sonnet 4
</p>
</div>

<div class="bg-white shadow-md rounded-lg p-6 mb-6">
<h2 class="text-2xl font-bold text-gray-800 mb-4">Model Comparison Matrix</h2>
<div class="bg-blue-50 border-l-4 border-blue-400 p-3 mb-4">
<p class="text-sm text-gray-700">
<span class="font-semibold">How to read this matrix:</span> Each cell shows links to comparison reports
for the corresponding model pair. The matrix shows only the upper triangle since comparisons are symmetric.
Click <strong>HTML</strong> for the formatted report or <strong>JSON</strong> for the raw data.
</p>
</div>

<div class="overflow-x-auto">
<table class="min-w-full bg-white shadow-md rounded-lg">
<thead>
<tr class="bg-gray-100 border-b border-gray-300">
<th class="px-4 py-3 text-left text-sm font-semibold text-gray-700">Model</th>
<th class="px-4 py-3 text-center text-sm font-semibold text-gray-700">claude4</th><th class="px-4 py-3 text-center text-sm font-semibold text-gray-700">claude4_5</th><th class="px-4 py-3 text-center text-sm font-semibold text-gray-700">claude4_5thinking</th><th class="px-4 py-3 text-center text-sm font-semibold text-gray-700">gpt4o</th><th class="px-4 py-3 text-center text-sm font-semibold text-gray-700">gpt5</th>
</tr>
</thead>
<tbody>

<tr class="border-t border-gray-200">
<th class="px-4 py-3 text-left text-sm font-semibold text-gray-700 bg-gray-50">claude4</th>
<td class="px-4 py-3 bg-gray-200 text-center text-gray-400">—</td><td class="px-4 py-3 text-center text-sm"><a href="claude4-claude4_5-report.html" class="text-blue-600 hover:text-blue-800 font-medium">HTML</a> / <a href="claude4-claude4_5-report.json" class="text-gray-600 hover:text-gray-800 font-medium">JSON</a></td><td class="px-4 py-3 text-center text-gray-400 text-sm">—</td><td class="px-4 py-3 text-center text-sm"><a href="claude4-gpt4o-report.html" class="text-blue-600 hover:text-blue-800 font-medium">HTML</a> / <a href="claude4-gpt4o-report.json" class="text-gray-600 hover:text-gray-800 font-medium">JSON</a></td><td class="px-4 py-3 text-center text-sm"><a href="claude4-gpt5-report.html" class="text-blue-600 hover:text-blue-800 font-medium">HTML</a> / <a href="claude4-gpt5-report.json" class="text-gray-600 hover:text-gray-800 font-medium">JSON</a></td>
</tr>

<tr class="border-t border-gray-200">
<th class="px-4 py-3 text-left text-sm font-semibold text-gray-700 bg-gray-50">claude4_5</th>
<td class="px-4 py-3 bg-gray-50"></td><td class="px-4 py-3 bg-gray-200 text-center text-gray-400">—</td><td class="px-4 py-3 text-center text-sm"><a href="claude4_5-claude4_5thinking-report.html" class="text-blue-600 hover:text-blue-800 font-medium">HTML</a> / <a href="claude4_5-claude4_5thinking-report.json" class="text-gray-600 hover:text-gray-800 font-medium">JSON</a></td><td class="px-4 py-3 text-center text-sm"><a href="claude4_5-gpt4o-report.html" class="text-blue-600 hover:text-blue-800 font-medium">HTML</a> / <a href="claude4_5-gpt4o-report.json" class="text-gray-600 hover:text-gray-800 font-medium">JSON</a></td><td class="px-4 py-3 text-center text-sm"><a href="claude4_5-gpt5-report.html" class="text-blue-600 hover:text-blue-800 font-medium">HTML</a> / <a href="claude4_5-gpt5-report.json" class="text-gray-600 hover:text-gray-800 font-medium">JSON</a></td>
</tr>

<tr class="border-t border-gray-200">
<th class="px-4 py-3 text-left text-sm font-semibold text-gray-700 bg-gray-50">claude4_5thinking</th>
<td class="px-4 py-3 bg-gray-50"></td><td class="px-4 py-3 bg-gray-50"></td><td class="px-4 py-3 bg-gray-200 text-center text-gray-400">—</td><td class="px-4 py-3 text-center text-gray-400 text-sm">—</td><td class="px-4 py-3 text-center text-sm"><a href="claude4_5thinking-gpt5-report.html" class="text-blue-600 hover:text-blue-800 font-medium">HTML</a> / <a href="claude4_5thinking-gpt5-report.json" class="text-gray-600 hover:text-gray-800 font-medium">JSON</a></td>
</tr>

<tr class="border-t border-gray-200">
<th class="px-4 py-3 text-left text-sm font-semibold text-gray-700 bg-gray-50">gpt4o</th>
<td class="px-4 py-3 bg-gray-50"></td><td class="px-4 py-3 bg-gray-50"></td><td class="px-4 py-3 bg-gray-50"></td><td class="px-4 py-3 bg-gray-200 text-center text-gray-400">—</td><td class="px-4 py-3 text-center text-sm"><a href="gpt4o-gpt5-report.html" class="text-blue-600 hover:text-blue-800 font-medium">HTML</a> / <a href="gpt4o-gpt5-report.json" class="text-gray-600 hover:text-gray-800 font-medium">JSON</a></td>
</tr>

<tr class="border-t border-gray-200">
<th class="px-4 py-3 text-left text-sm font-semibold text-gray-700 bg-gray-50">gpt5</th>
<td class="px-4 py-3 bg-gray-50"></td><td class="px-4 py-3 bg-gray-50"></td><td class="px-4 py-3 bg-gray-50"></td><td class="px-4 py-3 bg-gray-50"></td><td class="px-4 py-3 bg-gray-200 text-center text-gray-400">—</td>
</tr>

</tbody>
</table>
</div>

</div>

<footer class="text-center text-gray-500 text-sm mt-8">
Generated on 11/11/2025, 4:28:23 PM
</footer>
</div>
</body>
</html>
Loading