Merge planner part 1: Metastore

[nadav-govari]

Description

This is part 1 of the new planner for split compaction.

The design principles are as follows:

• the Merge planner scans the metastore for all immature splits from the last 24 hours.
  • After this first pass, it keeps a cursor for the most recent timestamp before which all splits were already fetched, so that each following fetch is a delta.
• It has a struct that tracks index configs- doc mappers, merge policies, etc.
  • Get or fetch from the metastore.
• The splits it got from the metastore it inserts these into its split tracker state.
• It groups all splits by index/source/partition/doc mapper uid.
• Splits that are mature are tossed.
• Splits that are already tracked are tossed.
• for each split partition, the planner uses the index metadata to get the merge policy.
• it feeds these into the merge policy, which creates merge operations.
• The splits in the merge operations get added into an inflight splits set in the state tracker.
• The splits in the merge operations get removed from the need merge set in the state tracker.

In the next PR, the splits that are candidates for merge will be sent to compactors for merging.

How was this PR tested?

Unit tests.

Once everything is wired up, manual testing and functional testing.