bake: distributed builds #22
Conversation
crazy-max
force-pushed
the
bake-distrib
branch
2 times, most recently
from
5171c81 to
a98d2d3
Compare
| pattern: ${{ needs.bake-local.outputs.artifact-name }}* | ||
| merge-multiple: true |
There was a problem hiding this comment.
Now artifacts are pushed per platform so we need to merge them:
| break; | ||
| case 'local': | ||
| outputOverride = `*.output=type=local,dest=${inpLocalExportDir}`; | ||
| outputOverride = `*.output=type=local,platform-split=true,dest=${inpLocalExportDir}`; |
There was a problem hiding this comment.
Need to opt-in platform-split=true for local artifacts to make output consistent and avoid merge conflicts with @actions/download-artifact.
| # needs predefined outputs as we can't use dynamic ones atm: https://github.com/actions/runner/pull/2477 | ||
| # 20 is the maximum number of platforms supported by our matrix strategy | ||
| result_0: ${{ steps.result.outputs.result_0 }} | ||
| result_1: ${{ steps.result.outputs.result_1 }} | ||
| result_2: ${{ steps.result.outputs.result_2 }} | ||
| result_3: ${{ steps.result.outputs.result_3 }} | ||
| result_4: ${{ steps.result.outputs.result_4 }} | ||
| result_5: ${{ steps.result.outputs.result_5 }} | ||
| result_6: ${{ steps.result.outputs.result_6 }} | ||
| result_7: ${{ steps.result.outputs.result_7 }} | ||
| result_8: ${{ steps.result.outputs.result_8 }} | ||
| result_9: ${{ steps.result.outputs.result_9 }} | ||
| result_10: ${{ steps.result.outputs.result_10 }} | ||
| result_11: ${{ steps.result.outputs.result_11 }} | ||
| result_12: ${{ steps.result.outputs.result_12 }} | ||
| result_13: ${{ steps.result.outputs.result_13 }} | ||
| result_14: ${{ steps.result.outputs.result_14 }} | ||
| result_15: ${{ steps.result.outputs.result_15 }} | ||
| result_16: ${{ steps.result.outputs.result_16 }} | ||
| result_17: ${{ steps.result.outputs.result_17 }} | ||
| result_18: ${{ steps.result.outputs.result_18 }} | ||
| result_19: ${{ steps.result.outputs.result_19 }} |
There was a problem hiding this comment.
Maximum of 20 jobs to align with max jobs concurrency for GitHub hosted runners https://docs.github.com/en/actions/reference/limits#job-concurrency-limits-for-github-hosted-runners
I don't think people will build more than 20 platforms anyway 😅
| platforms.forEach((platform, index) => { | ||
| let runner = inpRunner; | ||
| if (runner === 'auto') { | ||
| runner = platform.startsWith('linux/arm') ? 'ubuntu-24.04-arm' : 'ubuntu-latest'; | ||
| } | ||
| includes.push({ | ||
| index: index, | ||
| platform: platform, | ||
| runner: runner | ||
| }); | ||
| }); |
There was a problem hiding this comment.
Will choose best-matching runner to use based on target platform to build. cc @dvdksn
.github/workflows/bake.yml
Outdated
| on: | ||
| workflow_call: | ||
| inputs: | ||
| runner: |
There was a problem hiding this comment.
wdyt about reusing the same input name?
| runner: | |
| runs-on: |
There was a problem hiding this comment.
Was thinking of reusing the same but here we have some logic if set to auto to choose best matching one. Maybe we could default to empty instead of auto, I'm taking a look.
There was a problem hiding this comment.
Changes to runs-on and now defaults to empty instead of auto.
crazy-max
force-pushed
the
bake-distrib
branch
2 times, most recently
from
6d79c77 to
0f73820
Compare
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
crazy-max
force-pushed
the
bake-distrib
branch
from
0f73820 to
ebba68f
Compare
crazy-max
force-pushed
the
bake-distrib
branch
from
ebba68f to
d1ea56b
Compare
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
Signed-off-by: CrazyMax <1951866+crazy-max@users.noreply.github.com>
crazy-max
force-pushed
the
bake-distrib
branch
from
d1ea56b to
f74a6b7
Compare
needs #20relates to #2
Allows to distribute builds across multiple runners per platform to speed up builds and avoid storage space issues like docker/buildx#3520 (comment)