Rebalance shards when ingester status changes

[ncoiffier-celonis]

Description

Attempt to fix #6158

Following @guilload's suggestion here, this PR:

• gossip the ingester status over chit chat
• update the ingester pool when ingester status changes
• update the indexer pool too when ingester status changes (to fix no open shard found on ingester error)
• have the control plane rebalance the shards when the ingester status changes

With this approach, even if we have some 10s propagation delay before decomissioning, it is still possible to fail to ingest some documents if the chitchat takes longer than expected to gossip the ingester status to the control-plane.

Any feedback is welcome!!

How was this PR tested?

In addition of the unit and integration tests, I've run it against a local cluster with 2 indexer and observed that the number of errors reported in #6158 decreases from a few 100 to no errors.

Other approches

• Exclude decomissioning nodes when opening new shards, using chitchat #6165
• Exclude decomissioning nodes when opening new shards, using gRPC stream #6166

This PR is fairly identical to the branch guilload/ingester-status, rebased on main and with some additional bugfixes:

• fix bug in timeout_after being always 0, causing to not wait
• update ingester pool when IngesterStatus change (not only indexer pool)
• more unit and integration tests