feat: generate kinds in parallel across multiple processes

[bhearsum]

This is a cleaned up and slightly improved version of @ahal's original patch in #717. Most notably, it uses wait to resubmit new kinds as soon as they become available (instead of waiting for all kinds in each round to be completed). This means that if a slow kind gets submitted before all other (non-downstream) kinds have been submitted, that it won't block them. In the case of Gecko, the effect of this is that the test kind begins to process very quickly, and all other kinds are finished processing before that has completed.

Locally, this took ./mach taskgraph tasks from 1m26s to 1m9s (measured from command start to the final "Generated xxx tasks" message.

On try the results were a bit more mixed. The minimum time I observed without this patch was 140s, while the maximum was 291s (which seems to have been caused by bugbug slowness...which I'm willing to throw out). Outside of that outlier, the maximum was 146s and the mean was 143s. The minimum time I observed with this patch was 130s, while the maximum was 144s and the mean was 138s.

I presume the difference in results locally vs. Try is that locally I'm on a 64-core SSD machine, and the decision tasks run on lowered powered machines on Try, so there ends up being some resource contention (I/O, I suspect, because the ProcessPoolExecutor will only run one process per CPU core) when we process kinds in parallel there.

Despite this disappointing result on Try, this may still be worth taking, as ./mach taskgraph runs twice in the critical path of many try pushes (once on a developer's machine, and again in the decision task).

raw data:
Over 5 runs on try I got, without this patch: 291s, 146s, 146s, 140s, 140s

In each of those, there were 241s, 92s, 94s, 90s, 90s between "Loading tasks for kind test" and "Generated xxxxxx tasks for kind test"

Which means we spent the following amount of time doing non-test kind things in the critical path: 50s, 54s, 52s, 50s, 50s

With this patch: 130s, 141s, and 144s, 140s, 135s

In each of those, there were 105s, 114s, 115s, 114s, 109s between "Loading tasks for kind test" and "Generated xxxxxx tasks for kind test"

Which means we spent the following amount of time doing non-test kind things, but it was almost entirely out of the critical path: 25s, 27s, 29s, 26s, 26s

[Eijebong]

Having a graph fail (for example because some dependency is missing in a transform), exhibits 3 issues.

1. The error is printed when it happens, which might be in the middle of the graph generation which makes it hard to see because in big graphs it'll be hidden in the middle of the rest
2. When a kind errors out it doesn't stop graph generation. So if you have a kind that fails in 0.5, but a second kind that takes 30s to complete, the command will take 30s. This is easily reproducible by adding a time.sleep(30) in a random transform and having another one in another kind fail
3. taskgraph full still exits with 0 despite the graph generation having failed. (on main it properly stops immediately and exits with 1).

[bhearsum]

Having a graph fail (for example because some dependency is missing in a transform), exhibits 3 issues.

1. The error is printed when it happens, which might be in the middle of the graph generation which makes it hard to see because in big graphs it'll be hidden in the middle of the rest

2. When a kind errors out it doesn't stop graph generation. So if you have a kind that fails in 0.5, but a second kind that takes 30s to complete, the command will take 30s. This is easily reproducible by adding a `time.sleep(30)` in a random transform and having another one in another kind fail

3. `taskgraph full` still exits with `0` despite the graph generation having failed. (on main it properly stops immediately and exits with 1).

Thanks for catching these. As it turns out, all three are addressed by actually checking for exceptions from the futures and shutting down the executor and re-raising the exception.

[Eijebong]

Code looks fine. Tested on a bunch of mozilla and non mozilla project and didn't see major differences in the output. The only thing that differed was the order of some dependencies in the dependency list but if that has an impact on anything downstream I'd argue that it's not taskgraph's problem...

Perf wise it's slower on my machine for tiny graphs (30% slower for a graph generating 26 tasks from 5 kinds, with one kind generating 21 of those, 0.172s to 0.232s). I had some better results on a graph generating 2000 tasks, 500 per kind, where it went from 0.8s to 0.5s. This makes sense to me and I think is an acceptable drawback because for those tasks, taskgraph generation was never going to be the main bottleneck anyway.