Architecture Benchmarks – Review & Extension

[jlarson4]

Re-ran all benchmarks against a collection of models (gpt2, neo, pythia, OPT, Qwen2, Bloom, OpenELM), and resolved any new discrepancies caused by updated to transformers v5, and all of our other latest changes.

• Added new Text Quality benchmark, which runs a generation of text and scores it with GPT2 to ensure that we are generating valid human-readable text

• Stabilizing float types – some benchmark comparisons were converting back and forth between float16 and float32 due to the source type of the model. These updates stabilize those types to allow for more accurate testing of the model's accuracy when loaded via TransformerBridge

• Resolved bugs discovered in bloom_attention

• Resolved deprecation issues caused by transformers v5 in T5

• the following models were failing to properly generate or pass the other benchmarks, updated Architecture adapter to pass base benchmarks & new generate benchmark

  • BERT
  • Bloom
  • Neo
  • Pythia

• Cleaned up any duplicate code or unused benchmark functions that have been superseded by newer, better testing

• Bug fix (non-breaking change which fixes an issue)

• New feature (non-breaking change which adds functionality)

Checklist:

• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes
• I have not rewritten tests relating to key interfaces which would affect backward compatibility