PAN EDLs and Bitbucket

L1 Bithead

PAN EDLs and Bitbucket

I am currently working with EDLs that use Bitbucket repos as the external sources. I have 2 EDL types: IP and URL. Both are supposed to allow/whitelist.

 

The ask was for engineers to be able to pull/commit/push, etc to those repos when they want to allow an IP or url. This way they aren't dependant on a person to do the work. We'll have approvals in place of course. There are no specifics on what they use to do this. I am using terminal and/or accessing the repos themselves.

 

I currently have 2 repos with files containing the appropriatly formated IPs and urls.
https://bitbucket.com/projects/DEPT/repos/pan-edl-domain/urls.txt
https://bitbucket.com/projects/DEPT/repos/pan-edl-ip/ips.txt

 

I keep getting errors on the firewalls about not being able to fetch the lists:
Warnings:
Details:EDL(vsys1/TEST-EDL-IP ip) Unable to fetch external dynamic list. No error. Using old copy for refresh.
EDL(vsys1/TEST-EDL-IP ip) Refresh job success

Warnings:
Details:EDL(vsys1/TEST-EDL-URL url) Unable to fetch external dynamic list. No error. Using old copy for refresh.
EDL(vsys1/TEST-EDL-URL url) Refresh job success


If I use another source such as a docker linux server, there are no issues.
http://nginx-server.com/urls.txt
http://nginx-server.com/ips.txt

 

Details:EDL(vsys1/TEST-EDL-IP ip) Refresh job success
Details:EDL(vsys1/TEST-EDL-URL url) Refresh job success

 

Looking at the fiormat of the Bitbucket files I see:
$ file ips.txt
ips.txt: ASCII text

$ file urls.txt
urls.txt: ASCII text

 

Unfortunately our Birbucket admin left the company and no one is as knowledgeable. TAC cannot figure it out and keeps pointing at Bitbucket. I am inclined to agree, but I hope someone here has used Bitbucket with EDLs and maybe see what I could be missing. Thank you.

L4 Transporter

Re: PAN EDLs and Bitbucket

I know little about bitbucket, but your question is interesting enough to take a shot...

 

Is the repository publicly accessible?

https://confluence.atlassian.com/bitbucketserver/allowing-public-access-to-code-776639799.html

 

Is there anything in the audit logs?

https://confluence.atlassian.com/bitbucketserver/audit-logging-in-bitbucket-server-776640417.html

https://confluence.atlassian.com/bitbucketserverkb/how-to-read-the-bitbucket-server-log-formats-7791...

 

My shot-in-the-dark guess is that the Palo is getting a 401 unauthorized and does not know how to continue.

L7 Applicator

Re: PAN EDLs and Bitbucket

@rperez-mz,

What does your security settings look like on the Bitbucket repository? If I would have to guess the reason that this is getting denied is due to the repository not being public so the firewall can't actually view the text document in question. 

 

FYI,

highly recommend feeding all EDLs into a MineMeld instance and having the firewall pull from MineMeld. This still allows you to have the engineers work in the Bitbucket repo like they want, but it also allows you to ensure that there is never conflicting indicators included. If you pull in a blacklist at all and the EDL has the IPs whitelisted they simply don't get included in the list that the firewall pulls down; this prevents issues where it's simply whatever security policy you have first. 

L7 Applicator

Re: PAN EDLs and Bitbucket

@JoeAndreini,

The repository audit logs really aren't going to include access logs; they simply log what you've done with the repository. You'd have to look at the aduit logs on the server to get the access logs. 

L1 Bithead

Re: PAN EDLs and Bitbucket

@JoeAndreini
I mdae the project which contains both repos public and this time the error is something different:

 

Warnings:
Details:EDL(vsys1/TEST-EDL-URL url) Downloaded file is not a text file. Using old copy for refresh.
EDL(vsys1/TEST-EDL-URL url) Refresh job success

 

Warnings:
Details:EDL(vsys1/TEST-EDL-IP ip) Downloaded file is not a text file. Using old copy for refresh.
EDL(vsys1/TEST-EDL-IP ip) Refresh job success

 

I changed it back to private and it went back to:
Details:EDL(vsys1/TEST-EDL-IP ip) Unable to fetch external dynamic list. No error. Using old copy for refresh.
EDL(vsys1/TEST-EDL-IP ip) Refresh job success

 

Details:EDL(vsys1/TEST-EDL-URL url) Unable to fetch external dynamic list. No error. Using old copy for refresh.
EDL(vsys1/TEST-EDL-URL url) Refresh job success

 

So it has to be something on the BB side. <fist shake>

L1 Bithead

Re: PAN EDLs and Bitbucket

@BPry
By default, the project is not set to public. Access is to a few users including myself. All as admin. Both repos are the same. Not public and access to a few. There are no restrictions on brances.

 

I think the next steps are to post all this on a Bitbucket support site! It's not PAN as far as I can tell. I've read a bunch of article on MindMeld, but thought it was something cmpletely unrelated. I am waiting on Bitbucket support maybe they'll have something.

 

Just thinking out loud. So far it might be public access (as that gave a different error - see my reply to JoeAndreini) and/or BB file format. I thought a text file was a teaxt file. Apperently, BB doesn't think so. I'll keep this alive when I find the solution.

L7 Applicator

Re: PAN EDLs and Bitbucket

@rperez-mz

Making them "public" seems to be the solution. But what do you see in your webbrowser when you access the file? You probably see a bitbucket website with the txt file somewhere embedded in that website, right? If yes, is there somewhere the possibility that shows you ONLY this txt file without anything html related around that txt file?

L7 Applicator

Re: PAN EDLs and Bitbucket

@rperez-mz,

Correct which is why you are getting an error when the project isn't set to public, unless you are feeding it your username and password in the EDL URL the firewall wouldn't be authorized to view the document. 

As for the error that you are recieving when you set this to public; that isn't necissarly a bitbucket thing. That is a generic error essentially stating that the firewall doesn't know how to read the information you are feeding it. 

L7 Applicator

Re: PAN EDLs and Bitbucket

@rperez-mz

Going off of what @vsys_remo mentioned you are attempting to view the raw text file right? So it would be something like https://bitbucket/projects/project-name/repos/repo-name/raw/myedl.txt

 

L1 Bithead

Re: PAN EDLs and Bitbucket

@vsys_remo

Setting the repos to public do give more info thats for sure.

 

And yes, the URL I currently have in the EDL is https://bitbucket.com/projects/DEPT/repos/pan-edl-domain/browse/urls.txt, which does open up the BB site.

 

The other (non BB) source URL that I used is ONLY the txt file. I have thought that if I could get a BB url that does this, that it might work.

 

There is an option to use the raw file version which is
https://bitbucket.com/projects/DEPT/repos/pan-edl-domain/browse/urls.txt?at=refs%2Fheads%2Fmaster&ra..., but that hasn't worked either. Besides it seems ugly and not sure PAN likes that URL as a source.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!