diff --git a/_layouts/page.html b/_layouts/page.html index 4e0d4eb..72ad5d7 100644 --- a/_layouts/page.html +++ b/_layouts/page.html @@ -3,6 +3,8 @@ ---
-

{{ page.title }}

+ {% if page.notitle != true %} +

{{ page.title }}

+ {% endif %} {{ content }}
diff --git a/index.md b/index.md index e6007fb..6a2fdc1 100644 --- a/index.md +++ b/index.md @@ -5,9 +5,10 @@ title: Home

MLBench: Distributed Machine Learning Benchmark

- - - + + + + Documentation Status diff --git a/public/css/hyde.css b/public/css/hyde.css index 213e232..0ea5ec7 100644 --- a/public/css/hyde.css +++ b/public/css/hyde.css @@ -248,3 +248,10 @@ a.sidebar-nav-item:focus { .theme-base-0f .related-posts li a:hover { color: #8f5536; } + + +table.wide { + max-width: 200%; + width: 200%; + font-size: 15px; +} \ No newline at end of file diff --git a/results/index.md b/results/index.md new file mode 100644 index 0000000..22f3a4b --- /dev/null +++ b/results/index.md @@ -0,0 +1,39 @@ +--- +layout: page +notitle: true +title: Results +datatable: true +--- + +# MLBench Official Training Results +--- + +

+Here we present the official training results obtained by running the benchmark tasks. +It provides a reference and comparison point for different implementations. +

+ + +

+The table below provides a comparison table for results. Times are in seconds. The baseline represents the 1 worker case, and we +only present the results for the best obtained speedups. + +For more detailed results for each run, please check the documentation. + +

+ +--- + +|Task | Dataset | Model | Aggregation scheme | Metric Name | Metric Goal | Framework | Baseline | Best Speedup | Best Speedup Workers | Description | Implementation | Results | +|:-------------------:|:--------:|:---------------------:|:------------------:|:----------------------:|:-----------:|:------------:|:--------------:|:--------------:|:--------------------:|:-----------:|:--------------:|:-------:| +| Image Recognition | CIFAR10 | ResNet20 | All-Reduce | Validation Accuracy | 80% | Torch 1.7.0 | 171.15s | 6.6 (total), 11.9 (compute) | 16 | [details](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#task-1-image-classification)| [code](https://github.com/mlbench/mlbench-benchmarks/tree/develop/pytorch/imagerecognition/cifar10-resnet20-all-reduce) | [results](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#id17) | +| | | | PyTorch DDP | | | Torch 1.7.0 | 182.81s | 4.4 | 8 | [details](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#task-1-image-classification)| [code](https://github.com/mlbench/mlbench-benchmarks/tree/develop/pytorch/imagerecognition/cifar10-resnet20-distributed-data-parallel) | | +|---------------------+----------+-----------------------+--------------------+------------------------+-------------+--------------+----------------+----------------+----------------------+-------------+---------| +| Language Modelling | Wikitext2| AWD-LSTM | All-Reduce | Validation Perplexity | 70 | Torch 1.7.0 | 87,401.33s | | | [details](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#task-3-language-modelling) | [code]() | | +| Language Modelling | | BERT +|---------------------+----------+-----------------------+--------------------+------------------------+-------------+--------------+----------------+----------------+----------------------+----------------+---------| +| Machine Translation | WMT16 | LSTM (GNMT) | All-Reduce | Validation BLEU-score | 24 | Torch 1.7.0 | 65,206.62s | 2.8 (total), 15.8 (compute) | 16 | [details](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#a-lstm-wmt16-en-de) | [code](https://github.com/mlbench/mlbench-benchmarks/tree/develop/pytorch/nlp/translation/wmt16-gnmt-all-reduce) | [results](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#id30) | +|---------------------+----------+-----------------------+--------------------+------------------------+-------------+--------------+----------------+----------------+----------------------+----------------+---------| +| Machine Translation | WMT17 | Transformer | All-Reduce | Validation BLEU-score | 25 | Torch 1.7.0 | 37,594.21s | 3.0 (total), 18.3 (compute) | 16 | [details](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#b-transformer-wmt17-en-de) | [code](https://github.com/mlbench/mlbench-benchmarks/tree/develop/pytorch/nlp/translation/wmt17-transformer-all-reduce) | [results](https://mlbench.readthedocs.io/en/latest/benchmark-tasks.html#id32) | +{:.wide} +