ti_threatconnect: update pipeline to lowercase all hashes#17455
ti_threatconnect: update pipeline to lowercase all hashes#17455GShepherdTC wants to merge 3 commits intoelastic:mainfrom
Conversation
ReviewersBuildkite won't run for external contributors automatically; you need to add a comment:
NOTE: https://github.com/elastic/integrations/blob/main/.buildkite/pull-requests.json contains all those details. |
efd6
changed the title
efd6
left a comment
efd6
left a comment
There was a problem hiding this comment.
Suggested commit message:
ti_threatconnect: lowercase all hash values in indicator processing
Hash values should be normalized to lowercase for consistent matching
and deduplication across threat intelligence sources.
though this probably needs expansion. I think we want to know what visualisation and saved search impacts this will have.
andrewkroh
left a comment
andrewkroh
left a comment
There was a problem hiding this comment.
I opened a ECS issue to advise using lowercase for the hashes.
While reviewing I noticed that ECS threat.indicator.file.size and threat.indicator.geo.location were present in the TC data but completely missing from the ECS mapping. Can you please map those as well to improve ECS coverage?
| copy_from: threat_connect.indicator.md5 | ||
| ignore_empty_value: true | ||
| if: ctx.threat_connect?.indicator?.type instanceof String && ctx.threat_connect.indicator.type.contains('File') | ||
| - lowercase: |
There was a problem hiding this comment.
This needs to occur before the value is copied into the ECS threat.indicator.file.hash.md5 field.
Please check the sha1 and sha256 get the same treatment if they need it.
There was a problem hiding this comment.
Thanks. Will look at incorporating file size as that makes sense and its not specific to hash type so I believe I just have to add the mapping once. Seems they didn't map it originally.
But geo location is another story. Thats not a first party attribute in our system but keyed to attributes and I see they didn't map any attributes over from what I can tell. So I'm not going to touch that one.
andrewkroh
added
Integration:ti_threatconnect
|
Pinging @elastic/security-service-integrations (Team:Security-Service Integrations) |
ti_threatconnect: lowercase all hash values in indicator processing Hash values should be normalized to lowercase for consistent matching and deduplication across threat intelligence sources. Co-authored-by: Dan Kortschak <dan.kortschak@elastic.co>
4 participants
Proposed commit message
Updated pipeline to lowercase all hashes
Checklist
changelog.ymlfile.Author's Checklist
How to test this PR locally
Run the pipeline test. I updated the response data to match.
Related issues
Screenshots
No changes to UI.