[cloudflare_logpush] Add support for the data deduplication toggle for AWS S3 input

[brijesh-elastic]

Proposed commit message

AWS S3 input does "_id-Based Deduplication"[1] and generates "_id" by default.
This enhancement introduces "Data Deduplication" integration setting to allow users
to deduplicate data at the cost of indexing.
When users disable this setting, the "_id" generated by the input should
also be deleted. This PR does that.

[1]: https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-aws-s3#_document_id_generation

Checklist

• I have reviewed tips for building integrations and this pull request is aligned with them.
• I have verified that all data streams collect metrics or logs.
• I have added an entry to my package's changelog.yml file.
• I have verified that Kibana version constraints are current according to guidelines.
• I have verified that any added dashboard complies with Kibana's Dashboard good practices

How to test this PR locally

• Clone integrations repo.
• Install elastic package locally.
• Start elastic stack using elastic-package.
• Move to integrations/packages/cloudflare_logpush directory.
• Run the following command to run tests.

elastic-package test

Related issues

• Relates [Cloudflare Logpush] Allow the user to disable the custom _id when the input is AWS S3 #16520

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[elastic-vault-github-plugin-prod]

🚀 Benchmarks report

To see the full report comment with /test benchmark fullreport

[ShourieG]

LGTM from my end but can you get this reviewed by @kcreddy also before merging

[kcreddy]

Minor suggestions. LGTM overall. Thanks!

[kcreddy]

Suggested change

	Add support for the data deduplication toggle for AWS S3 input. By default, it is enabled.
	This setting prevents duplicates but may result in a lower indexing rate
	because Elasticsearch must check for existing documents before indexing.
	Add `Enable Data Deduplication` toggle for AWS S3 input. By default, it is enabled.
	When enabled, this setting prevents duplicates but may result in a lower indexing rate
	because Elasticsearch must check for existing documents before indexing.

[elasticmachine]

💚 Build Succeeded

• Buildkite Build
• Commit: 931c8c4

History

• 💚 Build #35938 succeeded 4a10acf
• 💚 Build #35859 succeeded df34c3e

cc @brijesh-elastic

[elastic-vault-github-plugin-prod]

Package cloudflare_logpush - 1.42.0 containing this change is available at https://epr.elastic.co/package/cloudflare_logpush/1.42.0/

[elastic-vault-github-plugin-prod]

Package cloudflare_logpush - 1.43.0 containing this change is available at https://epr.elastic.co/package/cloudflare_logpush/1.43.0/