s3 killer: paginate to find complete list of files to delete

[abannon]

Description

For segments stored in s3, the segment killer looks for files to remove within the segment prefix by calling listObjectsV2, which returns at most 1000 items. In the case where there are more than 1000 files per segment prefix, the segment killer did not find and delete all of the files. The fix is to add pagination.

Release note

Fixed: Kill task for s3 will remove all segment files, instead of up to 1000 files.

---

Key changed/added classes in this PR

• Changed S3DataSegmentKiller

---

This PR has:

• been self-reviewed.
• a release note entry in the PR description.
• added Javadocs for most classes and all non-trivial methods. Linked related entities via Javadoc links.
• added comments explaining the "why" and the intent of the code wherever would not be obvious for an unfamiliar reader.
• added unit tests or modified existing tests to cover new code paths, ensuring the threshold for code coverage is met.
• added integration tests.
• been tested in a test Druid cluster.

[kfaraz]

Thanks a lot for the changes, @abannon !
I have left a small suggestion.