topic Re: JSON Parsing - ProofPoint in General Topics

JSON Parsing - ProofPoint

jt1025 — Wed, 31 Jan 2018 13:39:51 GMT

Has anyone been able to get ProofPoint TAP logs into MineMeld? I think the issue I'm having is with my JSON configuration. Here's what I have so far but it's not pulling any indicators. I've tested my query on http://jmespath.org/ with sucessful results. The field I'm trying to extract is the URL in the threat field - badsite.zz in the example below.

Sample Log:

{

"messagesDelivered":[
{
"GUID":"c26dbea0-80d5-463b-b93c-4e8b708219ce",
"QID":"r2FNwRHF004109",
"ccAddresses":[
"bruce.wayne@university-of-education.zz"
],
"clusterId":"pharmtech_hosted",
"completelyRewritten":"true",
"fromAddress":"badguy@evil.zz",
"headerCC":"\"Bruce Wayne\" <bruce.wayne@university-of-education.zz>",
"headerFrom":"\"A. Badguy\" <badguy@evil.zz>",
"headerReplyTo":null,
"headerTo":"\"Clark Kent\" <clark.kent@pharmtech.zz>; \"Diana Prince\" <diana.prince@pharmtech.zz>",
"impostorScore":0,
"malwareScore":100,
"messageID":"20160624211145.62086.mail@evil.zz",
"messageParts":[
{
"contentType":"text/plain",
"disposition":"inline",
"filename":"text.txt",
"md5":"008c5926ca861023c1d2a36653fd88e2",
"oContentType":"text/plain",
"sandboxStatus":"unsupported",
"sha256":"85738f8f9a7f1b04b5329c590ebcb9e425925c6d0984089c43a022de4f19c281"
},
{
"contentType":"application/pdf",
"disposition":"attached",
"filename":"Invoice for Pharmtech.pdf",
"md5":"5873c7d37608e0d49bcaa6f32b6c731f",
"oContentType":"application/pdf",
"sandboxStatus":"threat",
"sha256":"2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca"
}
],
"messageTime":"2016-06-24T21:18:38.000Z",
"modulesRun":[
"pdr",
"sandbox",
"spam",
"urldefense"
],
"phishScore":46,
"policyRoutes":[
"default_inbound",
"executives"
],
"quarantineFolder":"Attachment Defense",
"quarantineRule":"module.sandbox.threat",
"recipient":[
"clark.kent@pharmtech.zz",
"diana.prince@pharmtech.zz"
],
"replyToAddress":null,
"sender":"e99d7ed5580193f36a51f597bc2c0210@evil.zz",
"senderIP":"192.0.2.255",
"spamScore":4,
"subject":"Please find a totally safe invoice attached.",
"threatsInfoMap":[
{
"campaignId":"46e01b8a-c899-404d-bcd9-189bb393d1a7",
"classification":"MALWARE",
"threat":"badsite.zz",
"threatId":"3ba97fc852c66a7ba761450edfdfb9f4ffab74715b591294f78b5e37a76481aa",
"threatTime":"2016-06-24T21:18:07.000Z",
"threatType":"URL",
"threatUrl":"https://threatinsight.proofpoint.com/#/73aa0499-dfc8-75eb-1de8-a471b24a2e75/threat/u/3ba97fc852c66a7ba761450edfdfb9f4ffab74715b591294f78b5e37a76481aa"
}
],
"toAddresses":[
"clark.kent@pharmtech.zz",
"diana.prince@pharmtech.zz"
]
},
"xmailer":"Spambot v2.5"
],
"queryEndTime":"2016-06-24T21:36:00Z"
}

]

}

Re: JSON Parsing - ProofPoint

xhoms — Wed, 31 Jan 2018 06:14:21 GMT

Hi @jt1025,

could you, please, check example log? The one that you have copied is an invalid JSON document.

As "messagesDelivered" proterty is a list of objects, its second item should be '{"xmailer":"Spambot v2.5"}' instead of '"xmailer":"Spambot v2.5"'

JSON miner will produce unpredictible results for non-valid JSON documents

Xavi

Re: JSON Parsing - ProofPoint

jt1025 — Wed, 31 Jan 2018 13:47:22 GMT

Thanks xhoms. You are correct. The sample log I provided was incorrect. Here is a sanitized log I pulled directly from the API which I believe is correctly formatted. I've also tried messagesDelivered[*].threatsInfoMap[*] for the extractor.

{
"queryEndTime": "2018-01-29T18:57:00Z",
"messagesDelivered": [
{
"spamScore": 4,
"phishScore": 46,
"threatsInfoMap": [
{
"threatID": "2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca",
"threatStatus": "active",
"classification": "MALWARE",
"threatUrl": "https://threatinsight.proofpoint.com/#/73aa0499-dfc8-75eb-1de8-a471b24a2e75/threat/u/2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca",
"threatTime": "2018-01-29T18:41:20.000Z",
"threat": "badsite.zz",
"campaignID": "46e01b8a-c899-404d-bcd9-189bb393d1a7",
"threatType": "URL"
}
],
"messageTime": "2018-01-23T15:44:07.000Z",
"impostorScore": 0,
"malwareScore": 0,
"cluster": "pharmtech_hosted",
"subject": "Please find a totally safe invoice attached.",
"quarantineFolder": "Attachment Defense",
"quarantineRule": "module.sandbox.threat",
"policyRoutes": [
"default_inbound"
],
"modulesRun": [
"sandbox",
"spam",
"pdr"
],
"messageSize": 6191,
"headerFrom": "A. Badguy <badguy@evil.zz>",
"headerReplyTo": null,
"fromAddress": [
"badguy@evil.zz"
],
"ccAddresses": [
"bruce.wayne@university-of-education.zz"
],
"replyToAddress": null,
"toAddresses": [
"clark.kent@pharmtech.zz",
"diana.prince@pharmtech.zz"
],
"xmailer": "Spambot v2.5",
"messageParts": [
{
"disposition": "inline",
"sha256": "85738f8f9a7f1b04b5329c590ebcb9e425925c6d0984089c43a022de4f19c281",
"md5": "008c5926ca861023c1d2a36653fd88e2",
"filename": "text.txt",
"sandboxStatus": "UNSUPPORTED_TYPE",
"oContentType": "text/plain",
"contentType": "text/plain"
},
{
"disposition": "inline",
"sha256": "2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca",
"md5": "5873c7d37608e0d49bcaa6f32b6c731f",
"filename": "text.html",
"sandboxStatus": "UNSUPPORTED_TYPE",
"oContentType": "text/html",
"contentType": "text/html"
}
],
"completelyRewritten": "true",
"QID": "r2FNwRHF004109",
"GUID": "c26dbea0-80d5-463b-b93c-4e8b708219ce",
"sender": "e99d7ed5580193f36a51f597bc2c0210@evil.zz",
"recipient": [
"clark.kent@pharmtech.zz"
],
"senderIP": "192.0.2.255",
"messageID": "20160624211145.62086.mail@evil.zz"
}
]
}

Re: JSON Parsing - ProofPoint

xhoms — Wed, 31 Jan 2018 14:39:17 GMT

@jt1025,

your JMESPath expression is projecting a List of Lists. And the expected result must be a list.

One option is to use the following flatten projection:

extractor = 'messagesDelivered[].threatsInfoMap[].{"indicator":threat}'

Another option is to flatten the threatsInfoMap object and then enrich the indicator with the the additional attributes you want to collect. For instance:

extractor = 'messagesDelivered[].threatsInfoMap[]'
indicator = 'threat'
fields = [ 'threatID', 'threatStatus', 'classification', 'campaignID', 'threatType']

Re: JSON Parsing - ProofPoint

jt1025 — Wed, 31 Jan 2018 16:06:14 GMT

Thanks again xhoms. I tried both options as seen below but I'm still not pulling any indictors.

Re: JSON Parsing - ProofPoint

xhoms — Wed, 31 Jan 2018 16:13:58 GMT

@jt1025,

could you check URL, username and password with curl command?

curl -u <username>:<password> -o output.json "<URL>"

if it works then I'd love to get access to your output.json to reproduce your issue in my lab

Re: JSON Parsing - ProofPoint

jt1025 — Wed, 31 Jan 2018 20:16:12 GMT

The curl was sucessful. I was able to create a miner using regex to pull the indicators as a workaround. Is there a way to share files directly through the community?

Re: JSON Parsing - ProofPoint

xhoms — Wed, 31 Jan 2018 20:46:11 GMT

@jt1025,

yes. You can. Use the attachements section bellow the text area.

If you do not want to share it publicly in the community then just drom me an email message (xhoms@paloaltonetworks.com)

Re: JSON Parsing - ProofPoint

jt1025 — Thu, 01 Feb 2018 17:17:18 GMT

For any one interested here is a working config provided by xhoms

Re: JSON Parsing - ProofPoint

MarcelST — Wed, 29 Aug 2018 09:03:51 GMT

That's a very interesting one. Is there any way we can have this in the predefined set of prototypes so we don't have to manually create it?

Anyway, I've deployed this one and seems is not working for me. Connection seems successful, but it just doesn't get any indicator.

This is the config:

This is the status:

@jt1025 @xhoms, any idea on how to troubleshoot? Does minemeld host logs give more information?

I've tried manually running the CURL and I get results. The only thing is I use the "/siem/all" instead of the "/siem/messages/delivered", but I've tried both options.

curl -u (myuser):(mypass) -o output.json "https://tap-api-v2.proofpoint.com/v2/siem/all?format=json&sinceSeconds=3600"

Re: JSON Parsing - ProofPoint

xhoms — Wed, 29 Aug 2018 10:17:45 GMT

Hi @MarcelST,

SimpleJSON class is basically a JMESPath engine. What I use to do to troubleshot it is to push the JSON document to http://jmespath.org/tutorial.html and to test the extractor expression there (is must provide a simple list of objects)

Re: JSON Parsing - ProofPoint

jt1025 — Wed, 29 Aug 2018 14:16:47 GMT

The service principal is the username and secret is the password. In your screenshot you have it reversed but you should have recieved an error in the last run field if that was the case.

Re: JSON Parsing - ProofPoint

mikealanni — Fri, 19 Oct 2018 17:23:14 GMT

@xhoms

Hello,

Trying to get tap info but I cannot see the class minemeld.ft.json.SimpleJSON. please can you let me know who to add this class?

Mike

Re: JSON Parsing - ProofPoint

xhoms — Thu, 25 Oct 2018 18:17:39 GMT

Hi @mikealanni,

just create a new prototype out of any prototype that already uses the SimpleJSON class (i.e. the aws.AMAZON one)

Re: JSON Parsing - ProofPoint

mikealanni — Mon, 29 Oct 2018 18:21:27 GMT

@xhoms

Thanks, I did it and I can get indectors, but I don't know which process I need to use? I used aggrigator URL but it show zero indicators

EDIT:

Nevermind, got to set it as a domin not URL