JSON Parsing - ProofPoint

[jt1025]

Has anyone been able to get ProofPoint TAP logs into MineMeld?  I think the issue I'm having is with my JSON configuration.  Here's what I have so far but it's not pulling any indicators.  I've tested my query on http://jmespath.org/ with sucessful results.  The field I'm trying to extract is the URL in the threat field - badsite.zz in the example below.

 

Sample Log:

{

"messagesDelivered":[
{
"GUID":"c26dbea0-80d5-463b-b93c-4e8b708219ce",
"QID":"r2FNwRHF004109",
"ccAddresses":[
"bruce.wayne@university-of-education.zz"
],
"clusterId":"pharmtech_hosted",
"completelyRewritten":"true",
"fromAddress":"badguy@evil.zz",
"headerCC":"\"Bruce Wayne\" <bruce.wayne@university-of-education.zz>",
"headerFrom":"\"A. Badguy\" <badguy@evil.zz>",
"headerReplyTo":null,
"headerTo":"\"Clark Kent\" <clark.kent@pharmtech.zz>; \"Diana Prince\" <diana.prince@pharmtech.zz>",
"impostorScore":0,
"malwareScore":100,
"messageID":"20160624211145.62086.mail@evil.zz",
"messageParts":[
{
"contentType":"text/plain",
"disposition":"inline",
"filename":"text.txt",
"md5":"008c5926ca861023c1d2a36653fd88e2",
"oContentType":"text/plain",
"sandboxStatus":"unsupported",
"sha256":"85738f8f9a7f1b04b5329c590ebcb9e425925c6d0984089c43a022de4f19c281"
},
{
"contentType":"application/pdf",
"disposition":"attached",
"filename":"Invoice for Pharmtech.pdf",
"md5":"5873c7d37608e0d49bcaa6f32b6c731f",
"oContentType":"application/pdf",
"sandboxStatus":"threat",
"sha256":"2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca"
}
],
"messageTime":"2016-06-24T21:18:38.000Z",
"modulesRun":[
"pdr",
"sandbox",
"spam",
"urldefense"
],
"phishScore":46,
"policyRoutes":[
"default_inbound",
"executives"
],
"quarantineFolder":"Attachment Defense",
"quarantineRule":"module.sandbox.threat",
"recipient":[
"clark.kent@pharmtech.zz",
"diana.prince@pharmtech.zz"
],
"replyToAddress":null,
"sender":"e99d7ed5580193f36a51f597bc2c0210@evil.zz",
"senderIP":"192.0.2.255",
"spamScore":4,
"subject":"Please find a totally safe invoice attached.",
"threatsInfoMap":[
{
"campaignId":"46e01b8a-c899-404d-bcd9-189bb393d1a7",
"classification":"MALWARE",
"threat":"badsite.zz",
"threatId":"3ba97fc852c66a7ba761450edfdfb9f4ffab74715b591294f78b5e37a76481aa",
"threatTime":"2016-06-24T21:18:07.000Z",
"threatType":"URL",
"threatUrl":"https://threatinsight.proofpoint.com/#/73aa0499-dfc8-75eb-1de8-a471b24a2e75/threat/u/3ba97fc852c66a7..."
}
],
"toAddresses":[
"clark.kent@pharmtech.zz",
"diana.prince@pharmtech.zz"
]
},
"xmailer":"Spambot v2.5"
],
"queryEndTime":"2016-06-24T21:36:00Z"
}

]

}

[xhoms]

Hi @jt1025,

could you, please, check example log? The one that you have copied is an invalid JSON document.

As "messagesDelivered" proterty is a list of objects, its second item should be '{"xmailer":"Spambot v2.5"}' instead of '"xmailer":"Spambot v2.5"'

JSON miner will produce unpredictible results for non-valid JSON documents

Xavi

[jt1025]

Thanks xhoms.  You are correct.  The sample log I provided was incorrect.  Here is a sanitized log I pulled directly from the API which I believe is correctly formatted.  I've also tried messagesDelivered[*].threatsInfoMap[*] for the extractor.

{
"queryEndTime": "2018-01-29T18:57:00Z",
"messagesDelivered": [
{
"spamScore": 4,
"phishScore": 46,
"threatsInfoMap": [
{
"threatID": "2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca",
"threatStatus": "active",
"classification": "MALWARE",
"threatUrl": "https://threatinsight.proofpoint.com/#/73aa0499-dfc8-75eb-1de8-a471b24a2e75/threat/u/2fab740f143fc1a...",
"threatTime": "2018-01-29T18:41:20.000Z",
"threat": "badsite.zz",
"campaignID": "46e01b8a-c899-404d-bcd9-189bb393d1a7",
"threatType": "URL"
}
],
"messageTime": "2018-01-23T15:44:07.000Z",
"impostorScore": 0,
"malwareScore": 0,
"cluster": "pharmtech_hosted",
"subject": "Please find a totally safe invoice attached.",
"quarantineFolder": "Attachment Defense",
"quarantineRule": "module.sandbox.threat",
"policyRoutes": [
"default_inbound"
],
"modulesRun": [
"sandbox",
"spam",
"pdr"
],
"messageSize": 6191,
"headerFrom": "A. Badguy <badguy@evil.zz>",
"headerReplyTo": null,
"fromAddress": [
"badguy@evil.zz"
],
"ccAddresses": [
"bruce.wayne@university-of-education.zz"
],
"replyToAddress": null,
"toAddresses": [
"clark.kent@pharmtech.zz",
"diana.prince@pharmtech.zz"
],
"xmailer": "Spambot v2.5",
"messageParts": [
{
"disposition": "inline",
"sha256": "85738f8f9a7f1b04b5329c590ebcb9e425925c6d0984089c43a022de4f19c281",
"md5": "008c5926ca861023c1d2a36653fd88e2",
"filename": "text.txt",
"sandboxStatus": "UNSUPPORTED_TYPE",
"oContentType": "text/plain",
"contentType": "text/plain"
},
{
"disposition": "inline",
"sha256": "2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca",
"md5": "5873c7d37608e0d49bcaa6f32b6c731f",
"filename": "text.html",
"sandboxStatus": "UNSUPPORTED_TYPE",
"oContentType": "text/html",
"contentType": "text/html"
}
],
"completelyRewritten": "true",
"QID": "r2FNwRHF004109",
"GUID": "c26dbea0-80d5-463b-b93c-4e8b708219ce",
"sender": "e99d7ed5580193f36a51f597bc2c0210@evil.zz",
"recipient": [
"clark.kent@pharmtech.zz"
],
"senderIP": "192.0.2.255",
"messageID": "20160624211145.62086.mail@evil.zz"
}
]
}

[xhoms · Accepted solution]

@jt1025,

your JMESPath expression is projecting a List of Lists. And the expected result must be a list.

One option is to use the following flatten projection:

extractor = 'messagesDelivered[].threatsInfoMap[].{"indicator":threat}'

Another option is to flatten the threatsInfoMap object and then enrich the indicator with the the additional attributes you want to collect. For instance:

extractor = 'messagesDelivered[].threatsInfoMap[]'
indicator = 'threat'
fields = [ 'threatID', 'threatStatus', 'classification', 'campaignID', 'threatType'] 

[jt1025]

Thanks again xhoms.  I tried both options as seen below but I'm still not pulling any indictors. 

[xhoms]

@jt1025,

could you check URL, username and password with curl command?

curl -u <username>:<password> -o output.json "<URL>"

if it works then I'd love to get access to your output.json to reproduce your issue in my lab

[jt1025]

The curl was sucessful. I was able to create a miner using regex to pull the indicators as a workaround.  Is there a way to share files directly through the community?

[xhoms]

@jt1025,

yes. You can. Use the attachements section bellow the text area.

If you do not want to share it publicly in the community then just drom me an email message (xhoms@paloaltonetworks.com)

[jt1025]

For any one interested here is a working config provided by xhoms

[MarcelST]

That's a very interesting one. Is there any way we can have this in the predefined set of prototypes so we don't have to manually create it?

Anyway, I've deployed this one and seems is not working for me. Connection seems successful, but it just doesn't get any indicator.

This is the config:

This is the status:

@jt1025 @xhoms, any idea on how to troubleshoot? Does minemeld host logs give more information?

I've tried manually running the CURL and I get results. The only thing is I use the "/siem/all" instead of the "/siem/messages/delivered", but I've tried both options.

curl -u (myuser):(mypass) -o output.json "https://tap-api-v2.proofpoint.com/v2/siem/all?format=json&sinceSeconds=3600"

[xhoms]

Hi @MarcelST,

SimpleJSON class is basically a JMESPath engine. What I use to do to troubleshot it is to push the JSON document to http://jmespath.org/tutorial.html and to test the extractor expression there (is must provide a simple list of objects)

[jt1025]

The service principal is the username and secret is the password.  In your screenshot you have it reversed but you should have recieved an error in the last run field if that was the case.  

[mikealanni]

@xhoms

Hello,

Trying to get tap info but I cannot see the class minemeld.ft.json.SimpleJSON. please can you let me know who to add this class?

Mike

[xhoms]

Hi @mikealanni,

just create a new prototype out of any prototype that already uses the SimpleJSON class (i.e. the aws.AMAZON one)

[mikealanni]

@xhoms

Thanks, I did it and I can get indectors, but I don't know which process I need to use? I used aggrigator URL but it show zero indicators 

EDIT:

Nevermind, got to set it as a domin not URL