topic Re: LetsEncrypt integration in Automation/API Discussions

LetsEncrypt integration

Brett_Hobbs — Wed, 01 Aug 2018 06:43:03 GMT

Hi,

While I know most would use an issued SSL certificate it would be great if PANOS supported LetsEncrypt for requesting SSL certificates for things like the management interface and GlobalProtect.

Re: LetsEncrypt integration

gfreeman — Fri, 05 Oct 2018 20:57:47 GMT

Hi Brett_Hobbs,

What would you be looking for in a Let's Encrypt integration from the workflow perspective?

If you still had to do the certbot renew from some linux box you controlled, then updating the certs on PAN-OS was provided as an Ansible or Terraform module, would that be helpful?

Re: LetsEncrypt integration

Brett_Hobbs — Sun, 07 Oct 2018 20:42:13 GMT

Hi,

That particular process would not work for us today (possibly in the future).

I was thinking that because GlobalProtect would have a DNS A record that having the certbot agent installed on the firewall we could support automatic verification and renewals.

MGMT interface woud take some aditional thought to solve either via your below method or some external DNS requirements.

Re: LetsEncrypt integration

gfreeman — Thu, 18 Oct 2018 20:32:11 GMT

For everyone that's interested in Let's Encrypt integration with PAN-OS:

Hi, my name is Garfield and I work here at Palo Alto Networks in the developer relations team. I'm wanting to get a feel for the interest and expectations of a Let's Encrypt integration. I'd very much appreciate anyone who's interested in a Let's Encrypt integration to respond to this thread with some information about their setup and expectations.

I'd like to separate this discussion into a few parts: what integrations today are doing, what can be done to help that in the short term, and what the expectation for the end result could look like.

Today: given that there is currently no native Let's Encrypt client on PAN-OS, people that are using Let's Encrypt certs on PAN-OS today are, to my knowledge, running a client on some (linux) host to renew the certs, then uploading the certs to their PAN-OS.

End-goal: I assume that the desired end-result is that PAN-OS runs Let's Encrypt natively, doing cert renewal automatically behind the scenes.

So here's the questions I have:

1) If you're currently using Let's Encrypt certs with PAN-OS and your workflow does not look like the above, can you briefly describe it?

2) Is your desired end goal that PAN-OS runs Let's Encrypt natively? If not, what is your desired end goal?

3) In between the end goal and now, would you want a stop-gap solution?

4) If you want a stop-gap solution, what form should it take? A standalone executable / script? Ansible module? Terraform resource? Tie-in to an existing Let's Encrypt client, such as certbot or acme.sh?

Thanks in advance for the feedback!

Re: LetsEncrypt integration

AK-X — Tue, 20 Nov 2018 15:05:04 GMT

1) If you're currently using Let's Encrypt certs with PAN-OS and your workflow does not look like the above, can you briefly describe it?

2) Is your desired end goal that PAN-OS runs Let's Encrypt natively? If not, what is your desired end goal?

3) In between the end goal and now, would you want a stop-gap solution?

1. We don't use Lets Encrypt certs with PAN-OS currently because it's a pita to manage cert renewal manually as you have to do it every 90 days. We do run certbot on our other web servers, it runs everyday and renew only when cert is near expiring, it also swap out certs and flush apache cache automatically. If there is any error, an email is sent to me.

2. Natively or not, I think making the process automatic and simple is what I would expect.

3. and 4. Yes. it doesn't really matter as long as it can automate the process, or at least automate as much as possible, so that functions in PAN-OS don't fail just because admin forgot to renew the certs.

Other comment:

Please also make domain ownership validation options flexible as everyone's setup is different.

In our case, xyz.com as well as DNS is controlled by headquarter, branchvpn.abc.com and branchvpn2.abc.com are issued to us. We won't be able to prove ownership of xyz.com but branchvpn.abc.com or branchvpn2.abc.com. And we can only use .well-known files method, and not DNS TXT method as we do not control DNS server.

Re: LetsEncrypt integration

WTSU — Mon, 25 Feb 2019 16:10:25 GMT

Hi,

I am just setting up LetsEncrypt certificates for a small Global Protect deployment and use pretty much the method that you suggest. I use a separate linux box to handle the certificate creation and renewal and have an upload script to upload the certificate via the api with a simple curl command.

This however does not currently work as the certificate gets imported via the API without the private key. If I use the web GUI, the certificate works fine, complete with the private key - is this a bug?

Native LE support would be great, however at least being able to upload the cert via the API would make life a lot easier (assuming that I am not just doing something wrong!).

Re: LetsEncrypt integration

WTSU — Mon, 25 Feb 2019 17:15:36 GMT

Doh! Just found the private-key API import command and realised that you have to import the cert first and then the private key afterwards! I assumed it was a single step process...

Re: LetsEncrypt integration

chris-oakops — Thu, 21 Mar 2019 18:01:23 GMT

1) The above is accurate for us.

2) No, having Terraform and Ansible support to manage certificates would be a better option in my opinion. If you integrate Lets Encrypt directly on the OS then that fixes cert management for LE users but not users of other CAs. If you had modules for Terraform and Ansible, that would cover all users and not just LE users. Or support LE natively but also have cert management modules.

3/4) No, we have a working solution.

EDIT: If you do integrate LE directly, please support all validation methods and don't limit it to just one.

Re: LetsEncrypt integration

fjwcash — Fri, 10 May 2019 21:57:06 GMT

@gfreeman wrote:
So here's the questions I have:

1) If you're currently using Let's Encrypt certs with PAN-OS and your workflow does not look like the above, can you briefly describe it?
2) Is your desired end goal that PAN-OS runs Let's Encrypt natively? If not, what is your desired end goal?
3) In between the end goal and now, would you want a stop-gap solution?
4) If you want a stop-gap solution, what form should it take? A standalone executable / script? Ansible module? Terraform resource? Tie-in to an existing Let's Encrypt client, such as certbot or acme.sh?

Thanks in advance for the feedback!

1. We run dehydrated on a Linux station that runs once a week and updates certs for our firewalls, panorama, and GlobalProtect portal domains. We use a self-signed CA root cert for GlobalProtect clients. (We run dehydrated on another Linux system that updates the cert on 50-odd Linux servers for use with Webmin, Apache, Lighttpd, CUPS, 3Ware GUI etc, automatically.)

2. Having a way to script the uploads of the certs into Panorama for pushing out to the firewalls, and into the GP Portal would be handy, and save the 10-15 minutes I spend every 60-odd days doing it manually. 🙂 (No, I haven't looked into the XML API as yet, it's on the Todo list, though.)

3. and 4. See 2. above.

Re: LetsEncrypt integration

fjwcash — Fri, 10 May 2019 21:56:24 GMT

@WTSU wrote:
Doh! Just found the private-key API import command and realised that you have to import the cert first and then the private key afterwards! I assumed it was a single step process...

Oooh, that's helpful. Now I have some reading to do to get our LE setup fully-automated. 😄

Re: LetsEncrypt integration

Alex_Samad — Fri, 26 Jul 2019 07:07:39 GMT

Just to add to the thread.

Yes I would like to use letsencrypt with PA.

No I don't want to manage the certs in PA. why - current management sucks - renew a cert with SAN attributes and they get lost - support tell me thats just how it is and I shouldn't be using the PA for cert management so (double checked with SE ..)

I do like current have a script for auth and distributing certs.

I would mind if somebody here could port the scripts to insert into PA.

By PA I mean Panorama which would then distribute it to the other PA's

so I wouild have a place holder name of say LE1 which could then assign to a PA management interface.

My script would renew the LE1 cert and then insert into PA (via api ?) which would overwrite the current LE1 and then somehow push from panorama to the PA's

Re: LetsEncrypt integration

TyronF — Sun, 28 Jul 2019 05:27:46 GMT

Having this integration would be amazing.

We manage around 100-odd PA-220's for small clients all with GP.

To answer you questions:

1) If you're currently using Let's Encrypt certs with PAN-OS and your workflow does not look like the above, can you briefly describe it?

We aren't using it because of the high maintenance.

2) Is your desired end goal that PAN-OS runs Let's Encrypt natively? If not, what is your desired end goal?

100% Natively would be the goal.

3) In between the end goal and now, would you want a stop-gap solution?

Depends on how complex.

Anything - but depends on how complex.

Re: LetsEncrypt integration

CLIq — Wed, 18 Dec 2019 17:11:55 GMT

@gfreeman

any update here?

1) I have a webserver behind the Palo for which I want to enable inbound ssl decryption, I use letsencrypt certs for this.

2.) endgoal is only to not have to reimport the cert into palo every x weeks, an integration into the autocertbot would be good

3) yes

4) standalone script or better tied into certbot

Re: LetsEncrypt integration

btorresgil — Wed, 18 Dec 2019 17:46:53 GMT

@panguyen wrote a LetsEncrypt integration for PAN-OS into the acme.sh client. The Pull Request is up for review here:

https://github.com/Neilpang/acme.sh/pull/2614

While the PR is getting reviewed and merged, you can use the integration by simply downloading the deployment file (deploy/panos.sh) into your own acme.sh installation. Here's a link to the file: https://github.com/Neilpang/acme.sh/pull/2614/files#diff-6ca80cd0349982033417d0bcd9b6952e

I know many people use Certbot, but we wanted a solution for internal and external firewalls that could be 100% automated, and we couldn't find a way to do that with Certbot. Acme.sh has many API-based domain verification capabilities that match well with the use case for internal firewall certs and automatic deployment. If anyone knows a viable way to integrate with Certbot, let us know.

Happy to answer any questions, and enjoy your free, auto-deployed firewall certificates!

-Brian

Re: LetsEncrypt integration

CLIq — Fri, 20 Dec 2019 10:08:31 GMT

Hi @btorresgil @panguyen @gfreeman

ok, so the deploy of this is done automatically and only when a new cert has been issued...

but this stores a superuser account name and password in cleartext... should be mentioned or better yet store only the api key accessible only by the user executing the command or something like that..

final solution should be somewhat secure, an exposed DMZ-host which might have a vulnerability giving superuser access to the firewall which can then open up access to the rest of the network? 😞

must be a better way

EDIT: giving API access for import and commit seems to be enough (still allows to import new config and gain full access this way but...)... now if it were possible to only store the api-key and store this in a safe way it would at least be an interim solution until something can be created that allows nothing but import certificates 😛

Re: LetsEncrypt integration

btorresgil — Fri, 20 Dec 2019 18:07:38 GMT

Hello,

Your concerns about security are important to us. Let me try to explain our logic around how this works and we can discuss if this meets your needs.

On first issue of the cert, acme.sh accepts the FW credentials via a temporary environment variable. However, to renew the cert and deploy it automatically, acme.sh needs to have the FW creds to deploy the cert, so we leverage the acme.sh credential storing mechanism to store the FW user/pass. acme.sh does store all the creds in clear text (PAN-OS and DNS host passwords alike) in a file called `account.conf`. The location and permissions of this file can be controlled through acme.sh settings, so it can be stored in a way that makes it inaccessible to other users. In fact, this is the default.

acme.sh (and other LetsEncrypt clients like certbot) also store the private keys for the certificates after they are issued, and these private keys are not password protected or encrypted. They are stored in the same secure location as the `account.conf` file, inaccessible to other users.

We recommend that a dedicated letsencrypt username be created on the firewall with only import and commit permissions. The deploy script is designed to commit only the changes made by this letsencrypt user, so it won't interfere with any uncommited changes that exist during the certificate renewal.

We decided that with the minimal permissions available to the firewall user, the secure nature of the acme.sh account credential file, and the fact that there are other valuable secrets like DNS host credentials and private keys stored in the same secure way, it would be acceptable to store the FW credentials according to the acme.sh standard.

If you disagree or this doesn't meet your needs, we're open to other methods of handling credentials during a certificate renewal. Please let us know what would work better for you.

Thanks!

-Brian

Re: LetsEncrypt integration

btorresgil — Fri, 20 Dec 2019 18:12:04 GMT

One other note since you mentioned a DMZ host. The nature of acme.sh is to verify domains without the need for incoming connections. This is a huge advantage, and a necessary feature for automated certificates on a firewall. This feature allows acme.sh to live on hosts anywhere in your network that has access to the internet and the firewall, which means it doesn't need to live in a DMZ or accept incoming connections of any kind. You can therefore place the acme.sh host on a secure vlan that does not allow anything inbound.

Re: LetsEncrypt integration

fjwcash — Fri, 20 Dec 2019 18:16:32 GMT

This is very handy, thanks for the link. Now I just need to figure out how to make it work with our dehydrated setup. 🙂 Although, since we do everything via Panorama, this might not be as useful; I'm not sure I'd want an automated push to 52 firewalls ...

Re: LetsEncrypt integration

panguyen — Mon, 30 Dec 2019 20:43:30 GMT

I want to make a correction in how acme.sh stores the credentials. acme.sh will base64encode the credentials and then save them.

Re: LetsEncrypt integration

hshawn — Thu, 02 Jan 2020 23:56:45 GMT

This is interesting, I am glad I found this thread because it appears this acme.sh script may be what works for my lab setup however reading through the docs I do not see an option that would work for a PAN firewall that is hosting a GP VPN (ports 80 and 443 are in use). There is mention in this thread about this working without needing inbound connections but I am not seeing that outlined here: https://github.com/Neilpang/acme.sh/wiki/How-to-issue-a-cert

I will comb through the docs more when I get off home tonight but am I missing something? Is this still in the works or should this give me a Let's Encrypt cert with auto renewal for the firewall (without needing a server behind the firewall exposed to the internet) as it is today?

Use case: Generate a valid cert for the GP VPN portal domain.