JSON Parsing - ProofPoint

L2 Linker

JSON Parsing - ProofPoint

Has anyone been able to get ProofPoint TAP logs into MineMeld?  I think the issue I'm having is with my JSON configuration.  Here's what I have so far but it's not pulling any indicators.  I've tested my query on http://jmespath.org/ with sucessful results.  The field I'm trying to extract is the URL in the threat field - badsite.zz in the example below.

 

json.JPG 

 

 

Sample Log:

{

"messagesDelivered":[
{
"GUID":"c26dbea0-80d5-463b-b93c-4e8b708219ce",
"QID":"r2FNwRHF004109",
"ccAddresses":[
"bruce.wayne@university-of-education.zz"
],
"clusterId":"pharmtech_hosted",
"completelyRewritten":"true",
"fromAddress":"badguy@evil.zz",
"headerCC":"\"Bruce Wayne\" <bruce.wayne@university-of-education.zz>",
"headerFrom":"\"A. Badguy\" <badguy@evil.zz>",
"headerReplyTo":null,
"headerTo":"\"Clark Kent\" <clark.kent@pharmtech.zz>; \"Diana Prince\" <diana.prince@pharmtech.zz>",
"impostorScore":0,
"malwareScore":100,
"messageID":"20160624211145.62086.mail@evil.zz",
"messageParts":[
{
"contentType":"text/plain",
"disposition":"inline",
"filename":"text.txt",
"md5":"008c5926ca861023c1d2a36653fd88e2",
"oContentType":"text/plain",
"sandboxStatus":"unsupported",
"sha256":"85738f8f9a7f1b04b5329c590ebcb9e425925c6d0984089c43a022de4f19c281"
},
{
"contentType":"application/pdf",
"disposition":"attached",
"filename":"Invoice for Pharmtech.pdf",
"md5":"5873c7d37608e0d49bcaa6f32b6c731f",
"oContentType":"application/pdf",
"sandboxStatus":"threat",
"sha256":"2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca"
}
],
"messageTime":"2016-06-24T21:18:38.000Z",
"modulesRun":[
"pdr",
"sandbox",
"spam",
"urldefense"
],
"phishScore":46,
"policyRoutes":[
"default_inbound",
"executives"
],
"quarantineFolder":"Attachment Defense",
"quarantineRule":"module.sandbox.threat",
"recipient":[
"clark.kent@pharmtech.zz",
"diana.prince@pharmtech.zz"
],
"replyToAddress":null,
"sender":"e99d7ed5580193f36a51f597bc2c0210@evil.zz",
"senderIP":"192.0.2.255",
"spamScore":4,
"subject":"Please find a totally safe invoice attached.",
"threatsInfoMap":[
{
"campaignId":"46e01b8a-c899-404d-bcd9-189bb393d1a7",
"classification":"MALWARE",
"threat":"badsite.zz",
"threatId":"3ba97fc852c66a7ba761450edfdfb9f4ffab74715b591294f78b5e37a76481aa",
"threatTime":"2016-06-24T21:18:07.000Z",
"threatType":"URL",
"threatUrl":"https://threatinsight.proofpoint.com/#/73aa0499-dfc8-75eb-1de8-a471b24a2e75/threat/u/3ba97fc852c66a7..."
}
],
"toAddresses":[
"clark.kent@pharmtech.zz",
"diana.prince@pharmtech.zz"
]
},
"xmailer":"Spambot v2.5"
],
"queryEndTime":"2016-06-24T21:36:00Z"
}

]

}

L5 Sessionator

Re: JSON Parsing - ProofPoint

Hi @jt1025,

 

could you, please, check example log? The one that you have copied is an invalid JSON document.

 

As "messagesDelivered" proterty is a list of objects, its second item should be '{"xmailer":"Spambot v2.5"}' instead of '"xmailer":"Spambot v2.5"'

 

JSON miner will produce unpredictible results for non-valid JSON documents

 

Xavi

L2 Linker

Re: JSON Parsing - ProofPoint

Thanks xhoms.  You are correct.  The sample log I provided was incorrect.  Here is a sanitized log I pulled directly from the API which I believe is correctly formatted.  I've also tried messagesDelivered[*].threatsInfoMap[*] for the extractor.

 

{
"queryEndTime": "2018-01-29T18:57:00Z",
"messagesDelivered": [
{
"spamScore": 4,
"phishScore": 46,
"threatsInfoMap": [
{
"threatID": "2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca",
"threatStatus": "active",
"classification": "MALWARE",
"threatUrl": "https://threatinsight.proofpoint.com/#/73aa0499-dfc8-75eb-1de8-a471b24a2e75/threat/u/2fab740f143fc1a...",
"threatTime": "2018-01-29T18:41:20.000Z",
"threat": "badsite.zz",
"campaignID": "46e01b8a-c899-404d-bcd9-189bb393d1a7",
"threatType": "URL"
}
],
"messageTime": "2018-01-23T15:44:07.000Z",
"impostorScore": 0,
"malwareScore": 0,
"cluster": "pharmtech_hosted",
"subject": "Please find a totally safe invoice attached.",
"quarantineFolder": "Attachment Defense",
"quarantineRule": "module.sandbox.threat",
"policyRoutes": [
"default_inbound"
],
"modulesRun": [
"sandbox",
"spam",
"pdr"
],
"messageSize": 6191,
"headerFrom": "A. Badguy <badguy@evil.zz>",
"headerReplyTo": null,
"fromAddress": [
"badguy@evil.zz"
],
"ccAddresses": [
"bruce.wayne@university-of-education.zz"
],
"replyToAddress": null,
"toAddresses": [
"clark.kent@pharmtech.zz",
"diana.prince@pharmtech.zz"
],
"xmailer": "Spambot v2.5",
"messageParts": [
{
"disposition": "inline",
"sha256": "85738f8f9a7f1b04b5329c590ebcb9e425925c6d0984089c43a022de4f19c281",
"md5": "008c5926ca861023c1d2a36653fd88e2",
"filename": "text.txt",
"sandboxStatus": "UNSUPPORTED_TYPE",
"oContentType": "text/plain",
"contentType": "text/plain"
},
{
"disposition": "inline",
"sha256": "2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca",
"md5": "5873c7d37608e0d49bcaa6f32b6c731f",
"filename": "text.html",
"sandboxStatus": "UNSUPPORTED_TYPE",
"oContentType": "text/html",
"contentType": "text/html"
}
],
"completelyRewritten": "true",
"QID": "r2FNwRHF004109",
"GUID": "c26dbea0-80d5-463b-b93c-4e8b708219ce",
"sender": "e99d7ed5580193f36a51f597bc2c0210@evil.zz",
"recipient": [
"clark.kent@pharmtech.zz"
],
"senderIP": "192.0.2.255",
"messageID": "20160624211145.62086.mail@evil.zz"
}
]
}

L5 Sessionator

Re: JSON Parsing - ProofPoint

@jt1025,

 

your JMESPath expression is projecting a List of Lists. And the expected result must be a list.

 

One option is to use the following flatten projection:

extractor = 'messagesDelivered[].threatsInfoMap[].{"indicator":threat}'

Another option is to flatten the threatsInfoMap object and then enrich the indicator with the the additional attributes you want to collect. For instance:

extractor = 'messagesDelivered[].threatsInfoMap[]'
indicator = 'threat'
fields = [ 'threatID', 'threatStatus', 'classification', 'campaignID', 'threatType'] 
L2 Linker

Re: JSON Parsing - ProofPoint

Thanks again xhoms.  I tried both options as seen below but I'm still not pulling any indictors. 

flatten.JPGfields.JPG

L5 Sessionator

Re: JSON Parsing - ProofPoint

@jt1025,

 

could you check URL, username and password with curl command?

curl -u <username>:<password> -o output.json "<URL>"

if it works then I'd love to get access to your output.json to reproduce your issue in my lab

L2 Linker

Re: JSON Parsing - ProofPoint

The curl was sucessful. I was able to create a miner using regex to pull the indicators as a workaround.  Is there a way to share files directly through the community?

L5 Sessionator

Re: JSON Parsing - ProofPoint

@jt1025,

 

yes. You can. Use the attachements section bellow the text area.

 

If you do not want to share it publicly in the community then just drom me an email message (xhoms@paloaltonetworks.com)

L2 Linker

Re: JSON Parsing - ProofPoint

For any one interested here is a working config provided by xhoms

 

works.JPG

L2 Linker

Re: JSON Parsing - ProofPoint

That's a very interesting one. Is there any way we can have this in the predefined set of prototypes so we don't have to manually create it?

 

 

Anyway, I've deployed this one and seems is not working for me. Connection seems successful, but it just doesn't get any indicator.

 

This is the config:

 

2TAP.png

 

This is the status:

 

1TAP.png

 

@jt1025 @xhoms, any idea on how to troubleshoot? Does minemeld host logs give more information?

 

I've tried manually running the CURL and I get results. The only thing is I use the "/siem/all" instead of the "/siem/messages/delivered", but I've tried both options.

 

curl -u (myuser):(mypass) -o output.json "https://tap-api-v2.proofpoint.com/v2/siem/all?format=json&sinceSeconds=3600"

 

 

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!