PAN EDLs and Bitbucket

[rperez-mz]

I am currently working with EDLs that use Bitbucket repos as the external sources. I have 2 EDL types: IP and URL. Both are supposed to allow/whitelist.

The ask was for engineers to be able to pull/commit/push, etc to those repos when they want to allow an IP or url. This way they aren't dependant on a person to do the work. We'll have approvals in place of course. There are no specifics on what they use to do this. I am using terminal and/or accessing the repos themselves.

I currently have 2 repos with files containing the appropriatly formated IPs and urls.
https://bitbucket.com/projects/DEPT/repos/pan-edl-domain/urls.txt
https://bitbucket.com/projects/DEPT/repos/pan-edl-ip/ips.txt

I keep getting errors on the firewalls about not being able to fetch the lists:
Warnings:
Details:EDL(vsys1/TEST-EDL-IP ip) Unable to fetch external dynamic list. No error. Using old copy for refresh.
EDL(vsys1/TEST-EDL-IP ip) Refresh job success

Warnings:
Details:EDL(vsys1/TEST-EDL-URL url) Unable to fetch external dynamic list. No error. Using old copy for refresh.
EDL(vsys1/TEST-EDL-URL url) Refresh job success

If I use another source such as a docker linux server, there are no issues.
http://nginx-server.com/urls.txt
http://nginx-server.com/ips.txt

Details:EDL(vsys1/TEST-EDL-IP ip) Refresh job success
Details:EDL(vsys1/TEST-EDL-URL url) Refresh job success

Looking at the fiormat of the Bitbucket files I see:
$ file ips.txt
ips.txt: ASCII text

$ file urls.txt
urls.txt: ASCII text

Unfortunately our Birbucket admin left the company and no one is as knowledgeable. TAC cannot figure it out and keeps pointing at Bitbucket. I am inclined to agree, but I hope someone here has used Bitbucket with EDLs and maybe see what I could be missing. Thank you.

[JoeAndreini]

I know little about bitbucket, but your question is interesting enough to take a shot...

Is the repository publicly accessible?

https://confluence.atlassian.com/bitbucketserver/allowing-public-access-to-code-776639799.html

Is there anything in the audit logs?

https://confluence.atlassian.com/bitbucketserver/audit-logging-in-bitbucket-server-776640417.html

https://confluence.atlassian.com/bitbucketserverkb/how-to-read-the-bitbucket-server-log-formats-7791...

My shot-in-the-dark guess is that the Palo is getting a 401 unauthorized and does not know how to continue.

[BPry]

@rperez-mz,

What does your security settings look like on the Bitbucket repository? If I would have to guess the reason that this is getting denied is due to the repository not being public so the firewall can't actually view the text document in question. 

FYI,

I highly recommend feeding all EDLs into a MineMeld instance and having the firewall pull from MineMeld. This still allows you to have the engineers work in the Bitbucket repo like they want, but it also allows you to ensure that there is never conflicting indicators included. If you pull in a blacklist at all and the EDL has the IPs whitelisted they simply don't get included in the list that the firewall pulls down; this prevents issues where it's simply whatever security policy you have first. 

[BPry]

@JoeAndreini,

The repository audit logs really aren't going to include access logs; they simply log what you've done with the repository. You'd have to look at the aduit logs on the server to get the access logs. 

[rperez-mz]

@JoeAndreini
I mdae the project which contains both repos public and this time the error is something different:

Warnings:
Details:EDL(vsys1/TEST-EDL-URL url) Downloaded file is not a text file. Using old copy for refresh.
EDL(vsys1/TEST-EDL-URL url) Refresh job success

Warnings:
Details:EDL(vsys1/TEST-EDL-IP ip) Downloaded file is not a text file. Using old copy for refresh.
EDL(vsys1/TEST-EDL-IP ip) Refresh job success

I changed it back to private and it went back to:
Details:EDL(vsys1/TEST-EDL-IP ip) Unable to fetch external dynamic list. No error. Using old copy for refresh.
EDL(vsys1/TEST-EDL-IP ip) Refresh job success

Details:EDL(vsys1/TEST-EDL-URL url) Unable to fetch external dynamic list. No error. Using old copy for refresh.
EDL(vsys1/TEST-EDL-URL url) Refresh job success

So it has to be something on the BB side. <fist shake>

[rperez-mz]

@BPry
By default, the project is not set to public. Access is to a few users including myself. All as admin. Both repos are the same. Not public and access to a few. There are no restrictions on brances.

I think the next steps are to post all this on a Bitbucket support site! It's not PAN as far as I can tell. I've read a bunch of article on MindMeld, but thought it was something cmpletely unrelated. I am waiting on Bitbucket support maybe they'll have something.

Just thinking out loud. So far it might be public access (as that gave a different error - see my reply to JoeAndreini) and/or BB file format. I thought a text file was a teaxt file. Apperently, BB doesn't think so. I'll keep this alive when I find the solution.

[Remo]

@rperez-mz

Making them "public" seems to be the solution. But what do you see in your webbrowser when you access the file? You probably see a bitbucket website with the txt file somewhere embedded in that website, right? If yes, is there somewhere the possibility that shows you ONLY this txt file without anything html related around that txt file?

[BPry]

@rperez-mz,

Correct which is why you are getting an error when the project isn't set to public, unless you are feeding it your username and password in the EDL URL the firewall wouldn't be authorized to view the document. 

As for the error that you are recieving when you set this to public; that isn't necissarly a bitbucket thing. That is a generic error essentially stating that the firewall doesn't know how to read the information you are feeding it. 

[BPry]

@rperez-mz

Going off of what @Remo mentioned you are attempting to view the raw text file right? So it would be something like https://bitbucket/projects/project-name/repos/repo-name/raw/myedl.txt

[rperez-mz]

@Remo

Setting the repos to public do give more info thats for sure.

And yes, the URL I currently have in the EDL is https://bitbucket.com/projects/DEPT/repos/pan-edl-domain/browse/urls.txt, which does open up the BB site.

The other (non BB) source URL that I used is ONLY the txt file. I have thought that if I could get a BB url that does this, that it might work.

There is an option to use the raw file version which is
https://bitbucket.com/projects/DEPT/repos/pan-edl-domain/browse/urls.txt?at=refs%2Fheads%2Fmaster&ra..., but that hasn't worked either. Besides it seems ugly and not sure PAN likes that URL as a source.

[rperez-mz]

@BPry
WHile the repos were private, I added my username and pw in the EDL object without success. I haven't found a great article to what I actually need to add. Is it "rperez"? Or "domain\rperez"? I've tried both. BB is tied to our AD and I simply use "rperez". But no go on either.

What's interesting is this is the only option for the project or repo permissions:
"Public Access - Allow users without a Bitbucket account to clone and browse repositories within this project.
[] Enable"

I'm just spitballing here, but would PAN need an account? And if I did have the repos set to public, the line "Allow users without a BB account..." should mean no creds are needed in the object... right?

I just set the repos back to public and added credentials. I used rperez and domain/rperez. Results are the same:
Details:EDL(vsys1/TEST-EDL-URL url) Downloaded file is not a text file. Using old copy for refresh.
EDL(vsys1/TEST-EDL-URL url) Refresh job success

I now know that domain\rperez fails. What firewall log could I view to se more details than just the 'generic' response?

[rperez-mz · Accepted solution]

@BPry @Remo @JoeAndreini

Its working! So here is the working combo. This works as expected... unless someone says the repos can't be public.

1. Upgrading to PANOS 8.0.9+ (8.0.7 had known EDL bugs)
2. Setting the project permissions (under Default Permissions) to "Write" or "Read". Setting as public also works. Since this was a new project, the default perms were set to 'No Access" which the FW didnt like.
3. Adding Bitbucket creds in the EDL object (unless the repo is set to public, then no creds are needed)
4. Using the BB raw URL in the EDL source

Thank you to all for contributing and helping me figure this out. LIVE Community is awesome!!