JSON Parsing - ProofPoint

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

JSON Parsing - ProofPoint

L2 Linker

Has anyone been able to get ProofPoint TAP logs into MineMeld?  I think the issue I'm having is with my JSON configuration.  Here's what I have so far but it's not pulling any indicators.  I've tested my query on http://jmespath.org/ with sucessful results.  The field I'm trying to extract is the URL in the threat field - badsite.zz in the example below.

 

json.JPG 

 

 

Sample Log:

{

"messagesDelivered":[
{
"GUID":"c26dbea0-80d5-463b-b93c-4e8b708219ce",
"QID":"r2FNwRHF004109",
"ccAddresses":[
"bruce.wayne@university-of-education.zz"
],
"clusterId":"pharmtech_hosted",
"completelyRewritten":"true",
"fromAddress":"badguy@evil.zz",
"headerCC":"\"Bruce Wayne\" <bruce.wayne@university-of-education.zz>",
"headerFrom":"\"A. Badguy\" <badguy@evil.zz>",
"headerReplyTo":null,
"headerTo":"\"Clark Kent\" <clark.kent@pharmtech.zz>; \"Diana Prince\" <diana.prince@pharmtech.zz>",
"impostorScore":0,
"malwareScore":100,
"messageID":"20160624211145.62086.mail@evil.zz",
"messageParts":[
{
"contentType":"text/plain",
"disposition":"inline",
"filename":"text.txt",
"md5":"008c5926ca861023c1d2a36653fd88e2",
"oContentType":"text/plain",
"sandboxStatus":"unsupported",
"sha256":"85738f8f9a7f1b04b5329c590ebcb9e425925c6d0984089c43a022de4f19c281"
},
{
"contentType":"application/pdf",
"disposition":"attached",
"filename":"Invoice for Pharmtech.pdf",
"md5":"5873c7d37608e0d49bcaa6f32b6c731f",
"oContentType":"application/pdf",
"sandboxStatus":"threat",
"sha256":"2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca"
}
],
"messageTime":"2016-06-24T21:18:38.000Z",
"modulesRun":[
"pdr",
"sandbox",
"spam",
"urldefense"
],
"phishScore":46,
"policyRoutes":[
"default_inbound",
"executives"
],
"quarantineFolder":"Attachment Defense",
"quarantineRule":"module.sandbox.threat",
"recipient":[
"clark.kent@pharmtech.zz",
"diana.prince@pharmtech.zz"
],
"replyToAddress":null,
"sender":"e99d7ed5580193f36a51f597bc2c0210@evil.zz",
"senderIP":"192.0.2.255",
"spamScore":4,
"subject":"Please find a totally safe invoice attached.",
"threatsInfoMap":[
{
"campaignId":"46e01b8a-c899-404d-bcd9-189bb393d1a7",
"classification":"MALWARE",
"threat":"badsite.zz",
"threatId":"3ba97fc852c66a7ba761450edfdfb9f4ffab74715b591294f78b5e37a76481aa",
"threatTime":"2016-06-24T21:18:07.000Z",
"threatType":"URL",
"threatUrl":"https://threatinsight.proofpoint.com/#/73aa0499-dfc8-75eb-1de8-a471b24a2e75/threat/u/3ba97fc852c66a7..."
}
],
"toAddresses":[
"clark.kent@pharmtech.zz",
"diana.prince@pharmtech.zz"
]
},
"xmailer":"Spambot v2.5"
],
"queryEndTime":"2016-06-24T21:36:00Z"
}

]

}

1 accepted solution

Accepted Solutions

@jt1025,

 

your JMESPath expression is projecting a List of Lists. And the expected result must be a list.

 

One option is to use the following flatten projection:

extractor = 'messagesDelivered[].threatsInfoMap[].{"indicator":threat}'

Another option is to flatten the threatsInfoMap object and then enrich the indicator with the the additional attributes you want to collect. For instance:

extractor = 'messagesDelivered[].threatsInfoMap[]'
indicator = 'threat'
fields = [ 'threatID', 'threatStatus', 'classification', 'campaignID', 'threatType'] 

View solution in original post

14 REPLIES 14

L5 Sessionator

Hi @jt1025,

 

could you, please, check example log? The one that you have copied is an invalid JSON document.

 

As "messagesDelivered" proterty is a list of objects, its second item should be '{"xmailer":"Spambot v2.5"}' instead of '"xmailer":"Spambot v2.5"'

 

JSON miner will produce unpredictible results for non-valid JSON documents

 

Xavi

Thanks xhoms.  You are correct.  The sample log I provided was incorrect.  Here is a sanitized log I pulled directly from the API which I believe is correctly formatted.  I've also tried messagesDelivered[*].threatsInfoMap[*] for the extractor.

 

{
"queryEndTime": "2018-01-29T18:57:00Z",
"messagesDelivered": [
{
"spamScore": 4,
"phishScore": 46,
"threatsInfoMap": [
{
"threatID": "2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca",
"threatStatus": "active",
"classification": "MALWARE",
"threatUrl": "https://threatinsight.proofpoint.com/#/73aa0499-dfc8-75eb-1de8-a471b24a2e75/threat/u/2fab740f143fc1a...",
"threatTime": "2018-01-29T18:41:20.000Z",
"threat": "badsite.zz",
"campaignID": "46e01b8a-c899-404d-bcd9-189bb393d1a7",
"threatType": "URL"
}
],
"messageTime": "2018-01-23T15:44:07.000Z",
"impostorScore": 0,
"malwareScore": 0,
"cluster": "pharmtech_hosted",
"subject": "Please find a totally safe invoice attached.",
"quarantineFolder": "Attachment Defense",
"quarantineRule": "module.sandbox.threat",
"policyRoutes": [
"default_inbound"
],
"modulesRun": [
"sandbox",
"spam",
"pdr"
],
"messageSize": 6191,
"headerFrom": "A. Badguy <badguy@evil.zz>",
"headerReplyTo": null,
"fromAddress": [
"badguy@evil.zz"
],
"ccAddresses": [
"bruce.wayne@university-of-education.zz"
],
"replyToAddress": null,
"toAddresses": [
"clark.kent@pharmtech.zz",
"diana.prince@pharmtech.zz"
],
"xmailer": "Spambot v2.5",
"messageParts": [
{
"disposition": "inline",
"sha256": "85738f8f9a7f1b04b5329c590ebcb9e425925c6d0984089c43a022de4f19c281",
"md5": "008c5926ca861023c1d2a36653fd88e2",
"filename": "text.txt",
"sandboxStatus": "UNSUPPORTED_TYPE",
"oContentType": "text/plain",
"contentType": "text/plain"
},
{
"disposition": "inline",
"sha256": "2fab740f143fc1aa4c1cd0146d334c5593b1428f6d062b2c406e5efe8abe95ca",
"md5": "5873c7d37608e0d49bcaa6f32b6c731f",
"filename": "text.html",
"sandboxStatus": "UNSUPPORTED_TYPE",
"oContentType": "text/html",
"contentType": "text/html"
}
],
"completelyRewritten": "true",
"QID": "r2FNwRHF004109",
"GUID": "c26dbea0-80d5-463b-b93c-4e8b708219ce",
"sender": "e99d7ed5580193f36a51f597bc2c0210@evil.zz",
"recipient": [
"clark.kent@pharmtech.zz"
],
"senderIP": "192.0.2.255",
"messageID": "20160624211145.62086.mail@evil.zz"
}
]
}

@jt1025,

 

your JMESPath expression is projecting a List of Lists. And the expected result must be a list.

 

One option is to use the following flatten projection:

extractor = 'messagesDelivered[].threatsInfoMap[].{"indicator":threat}'

Another option is to flatten the threatsInfoMap object and then enrich the indicator with the the additional attributes you want to collect. For instance:

extractor = 'messagesDelivered[].threatsInfoMap[]'
indicator = 'threat'
fields = [ 'threatID', 'threatStatus', 'classification', 'campaignID', 'threatType'] 

Thanks again xhoms.  I tried both options as seen below but I'm still not pulling any indictors. 

flatten.JPGfields.JPG

@jt1025,

 

could you check URL, username and password with curl command?

curl -u <username>:<password> -o output.json "<URL>"

if it works then I'd love to get access to your output.json to reproduce your issue in my lab

The curl was sucessful. I was able to create a miner using regex to pull the indicators as a workaround.  Is there a way to share files directly through the community?

@jt1025,

 

yes. You can. Use the attachements section bellow the text area.

 

If you do not want to share it publicly in the community then just drom me an email message (xhoms@paloaltonetworks.com)

For any one interested here is a working config provided by xhoms

 

works.JPG

That's a very interesting one. Is there any way we can have this in the predefined set of prototypes so we don't have to manually create it?

 

 

Anyway, I've deployed this one and seems is not working for me. Connection seems successful, but it just doesn't get any indicator.

 

This is the config:

 

2TAP.png

 

This is the status:

 

1TAP.png

 

@jt1025 @xhoms, any idea on how to troubleshoot? Does minemeld host logs give more information?

 

I've tried manually running the CURL and I get results. The only thing is I use the "/siem/all" instead of the "/siem/messages/delivered", but I've tried both options.

 

curl -u (myuser):(mypass) -o output.json "https://tap-api-v2.proofpoint.com/v2/siem/all?format=json&sinceSeconds=3600"

 

 

Hi @MarcelST,

 

SimpleJSON class is basically a JMESPath engine. What I use to do to troubleshot it is to push the JSON document to http://jmespath.org/tutorial.html and to test the extractor expression there (is must provide a simple list of objects)

L2 Linker

The service principal is the username and secret is the password.  In your screenshot you have it reversed but you should have recieved an error in the last run field if that was the case.  

@xhoms

 

Hello,

 

Trying to get tap info but I cannot see the class minemeld.ft.json.SimpleJSON. please can you let me know who to add this class?

 

Mike

Hi @mikealanni,

 

just create a new prototype out of any prototype that already uses the SimpleJSON class (i.e. the aws.AMAZON one)

 

2018-10-25_20-15-46.png

 

 

 

@xhoms

Thanks, I did it and I can get indectors, but I don't know which process I need to use? I used aggrigator URL but it show zero indicators 

 

EDIT:

Nevermind, got to set it as a domin not URL

 

 

  • 1 accepted solution
  • 10346 Views
  • 14 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!