topic Terraform provider inconsistencies and issues with IAM role tags in Cloud NGFW for AWS Discussions

Terraform provider inconsistencies and issues with IAM role tags

SZanpure — Mon, 27 Jun 2022 18:40:57 GMT

Hi,

I am currently on provider version 1.0.4 (prior to this, I was trialling out CloudNGFW a month ago, on locally built version 1.0.0 of the provider). I do not think it's a provider bug as such, but the issues I am facing pertain to provider block configuration.

Before I list the issues, I would like to provide some info on the pre-requisite steps I have completed prior to deploying NGFW via terraform.

1. I have subscribed to CloudNGFW via AWS Marketplace.

2. I have added my sandbox account to CloudNGFW tenant via Web UI and I have run the Cloudformation template. I am using Customer managed endpoints.

3. I have created the Cloudwatch Log groups and log streams matching the names in the CT above.

4. I have switched on programmatic access and followed the guide to create IAM role and policy. I have added all 3 tags on my IAM role as shown below

With this in place, here are my findings:

Scenario 1: Running locally built provider version 1.0.0, I could bootstrap the cloudngfwaws provider as such

provider "cloudngfwaws" { json_config_file = "./cloudngfwaws_config.json" } cloudngfwaws_config.json below: { "host": "api.us-east-1.aws.cloudngfw.paloaltonetworks.com", "region": "us-east-1", "arn": "arn:aws:iam::account-id:role/palo-ngfw-admin-role", "logging": ["login", "get", "post", "put", "delete", "path", "send", "receive"] }

This worked flawlessly, and I did not have to specify any access_key or secret_key. The provider would lookup the AWS credentials automatically (not sure how. Possibly by looking at AWS_PROFILE environment variable or the likes).

Scenario 2: Fast forward a month, and today I started working on this project again, and upgraded to provider version 1.0.4 from Terraform registry, instead of building it locally. Below are the issues I noticed:

Issue 1: If access_key and secret_key are not supplied in provider_config (json file), I get the following error upon running terraform plan:

Error: error configuring Terraform AWS Provider: no valid credential sources for Terraform AWS Provider found. Deprecated

This issue was NOT for AWS provider, but for the cloudngfwaws provider. So I decided to add access_key and secret_key into the provider config block, which led me to issue 2

Issue 2: After supplying the info above, I ran terraform plan, and got the following error:

Account is not successfully onboarded by FMS. Programmatic Access for CloudNGFWGlobalRulestackAdmin role is not supported

The clue here is "CloudNGFWGlobalRulestackAdmin role is not supported". My terraform code is rather simple. It creates a standard NGFW, with a local rulestack, that has a local security rule defined underneath. I do not want or have a global rulestack. However, the IAM role created in the pre-requisite steps has the "CloudNGFWGlobalRulestackAdmin" tag associated with it. If that tag is defined, it seems to force a dependency on the NGFW being configured in AWS FMS with a matching policy. This is not mentioned anywhere in the docs as far as I can see, though I might have missed it.

Another thing to note here, is that I attempted to delete the tag from IAM policy in terraform and tried to run terraform plan, but the NGFW provider was not letting me apply that change (terraform plan failed preventing me from proceeding). So I had to manually login to AWS console and delete that tag from IAM policy. This leads me to the next issue.

Issue 3: After deleting the tag via AWS console, and deleting it in terraform code, I ran terraform plan again. This time I got a new error

Error: Error(-1): Unknown path: /v1/mgmt/tokens/cloudglobalrulestackadmin

This threw me off because I am not creating global rules anywhere and I had already fixed the IAM role above. So I read through the provider docs, and it turns out that if I supply the provider settings "arn" in config block, it tries to generate/refresh all tokens (localfirewalladmin, localrulestackadmin and globalrulestackadmin). So I removed "arn" and instead supplied "lfa_arn" and "lra_arn" in my config block. This led me to my next issue.

Issue 4: Now, after running terraform plan, I got the following error. Please note that my config was supplied in json file.

Error: No ARNs were specified

This is definitely incorrect, since the ARN's are specified.

{ "host": "api.us-east-1.aws.cloudngfw.paloaltonetworks.com", "region": "us-east-1", "lfa_arn": "arn:aws:iam::account-id:role/palo-ngfw-admin-role", "lra_arn": "arn:aws:iam::account-id:role/palo-ngfw-admin-role", "logging": ["login", "get", "post", "put", "delete", "path", "send", "receive"] }

So I read the provider docs again, and the config block resolution seems to look for properties in the provider block first, followed by ENV variables, and then in the json file. So I moved all the above config from json file to provider block directly, and at this point, terraform plan finally gave me an output. I have yet to run terraform apply.

The point of all this is to note that changes in backend functionality are breaking the provider, and the documentation does not match the backend. These nuances of access_key and secret_key being mandatory, when to use arn vs lfa/lra arn and IAM role tags effectively being used to suss out centralised vs distributed deployment (and AWS FMS therefore being needed for distributed deployment) need to be documented better.

Please let me know if I have missed some relevant documentation, and I will be happy to read it 🙂

Thanks,

Shreyas

Re: Terraform provider inconsistencies and issues with IAM role tags

SZanpure — Mon, 27 Jun 2022 19:33:20 GMT

Ok another update after running terraform apply. The customer managed VPC endpoints are not accepted immediately by the NGFW/Palo trusted account. In my case, after running terraform apply, it took around 5-6 minutes, for the status to change from PENDING to ACCEPTED in NGFW Web UI. This meant that my terraform apply failed when creating routes for the VPC endpoints

│ Error: error updating Route in Route Table (<route table name>) with destination (<cidr>): RouteNotSupported: Route table contains unsupported route target: <vpc endpoint name>. VPC Endpoints of this type cannot be used as route targets. │ status code: 400, request id: 6e873cbd-291f-4131-8107-ebf1cb3e3997 │ │ with module.firewall-vpce.aws_route.inspection-rtb-tgw-subnet-1, │ on modules/firewall-vpce/palo_ngfw_routes.tf line 27, in resource "aws_route" "inspection-rtb-tgw-subnet-1": │ 27: resource "aws_route" "inspection-rtb-tgw-subnet-1" {

Re: Terraform provider inconsistencies and issues with IAM role tags

SZanpure — Fri, 01 Jul 2022 11:06:05 GMT

Hi,

@gfreeman Tagging you. Hope that's ok.

I completely un-subscribed and re-subscribed via AWS Marketplace and started with a clean slate. On the very first run of my provider, I always get this error below

I believe I am getting this error because the creation of cloudngfwaws resource via terraform finishes way before the actual NGFW has finished creation in Palo Web UI. This is causing the VPC Endpoint Service to be in a "non-ready" state when terraform queries it. Can this please be looked at?

FYI, my terraform code for creating Palo NGFW is below

resource "cloudngfwaws_ngfw" "test-ngfw-terraform" { name = "test-ngfw-terraform" vpc_id = var.inspection-vpc-id account_id = var.account-id description = "Test NGFW created via terraform provider" endpoint_mode = "CustomerManaged" subnet_mapping { availability_zone = var.inspection-vpc-private-subnet-1-az-name } subnet_mapping { availability_zone = var.inspection-vpc-private-subnet-2-az-name } rulestack = cloudngfwaws_commit_rulestack.commit-test-rulestack-terraform.rulestack } resource "cloudngfwaws_ngfw_log_profile" "test-ngfw-terraform-log-profile" { ngfw = cloudngfwaws_ngfw.test-ngfw-terraform.name account_id = cloudngfwaws_ngfw.test-ngfw-terraform.account_id log_destination { destination_type = "CloudWatchLogs" destination = aws_cloudwatch_log_group.test-ngfw-terraform-loggroup.name log_type = "THREAT" } log_destination { destination_type = "CloudWatchLogs" destination = aws_cloudwatch_log_group.test-ngfw-terraform-loggroup.name log_type = "TRAFFIC" } log_destination { destination_type = "CloudWatchLogs" destination = aws_cloudwatch_log_group.test-ngfw-terraform-loggroup.name log_type = "DECRYPTION" } } resource "cloudngfwaws_commit_rulestack" "commit-test-rulestack-terraform" { rulestack = cloudngfwaws_rulestack.test-rulestack-terraform-module.name } resource "cloudngfwaws_rulestack" "test-rulestack-terraform-module" { name = "test-rulestack-terraform-module" scope = "Local" account_id = var.account-id description = "Test rulestack created via terraform provider" profile_config { anti_spyware = "BestPractice" anti_virus = "BestPractice" file_blocking = "BestPractice" vulnerability = "BestPractice" } } resource "cloudngfwaws_security_rule" "test-security-rule-terraform" { rulestack = cloudngfwaws_rulestack.test-rulestack-terraform-module.name rule_list = "LocalRule" priority = 1 name = "test-security-rule-terraform" description = "Test security rule created via terraform provider" source { cidrs = ["any"] } destination { cidrs = ["any"] } applications = ["any"] category {} action = "Allow" logging = true audit_comment = "initial config" } resource "aws_cloudwatch_log_group" "test-ngfw-terraform-loggroup" { name = "PaloAltoCloudNGFW" retention_in_days = 90 tags = { Name = "PaloAltoCloudNGFW" } } resource "aws_cloudwatch_log_stream" "test-ngfw-terraform-logstream" { name = "PaloAltoCloudNGFW" log_group_name = aws_cloudwatch_log_group.test-ngfw-terraform-loggroup.name }

The code which errors is below

data "cloudngfwaws_ngfw" "palo-ngfw" { name = var.ngfw-name } # Errors for this data resource data "aws_vpc_endpoint_service" "palo-ngfw-vpce-service" { service_name = data.cloudngfwaws_ngfw.palo-ngfw.endpoint_service_name }

One last thing to note, is that if I wait for the NGFW to finish creation, and re-run my terraform code, I get the error below. After this, no matter how many times I re-run the build, I keep getting the same result

Re: Terraform provider inconsistencies and issues with IAM role tags

priyer — Wed, 06 Jul 2022 23:50:33 GMT

Hello @SZanpure

Greetings from Palo Alto Networks!

Issue 1 We are working on removing the necessity to specify the access key / secret key in the provider block. Not sure yet when it will come out, but it's on our list.
Issue 2 - The API itself has changed how it behaves. We are working on an update for the provider.

Issue 3 - This happens because you removed the global rulestack tag, so it's going to error.
Issue 4 - Looks like you using underscores in the JSON config, if you want to do JSON config, you need to use hyphens.

Regards,
Prasanna Iyer
Product Specialist
Palo Alto Networks
live.paloaltonetworks.com/t5/cloud-ngfw-help-center/ct-p/Cloud_NGFW

Re: Terraform provider inconsistencies and issues with IAM role tags

SZanpure — Thu, 07 Jul 2022 12:01:22 GMT

Thanks Prasanna. Regarding Issue 4, can you please update the docs with this info? Currently, neither Github nor Terraform provider docs page state this.

https://github.com/PaloAltoNetworks/terraform-provider-cloudngfwaws/blob/main/docs/index.md

https://registry.terraform.io/providers/PaloAltoNetworks/cloudngfwaws/latest/docs

However, your suggestion is valid, because the code seems to be looking for hyphens

https://github.com/PaloAltoNetworks/terraform-provider-cloudngfwaws/blob/main/internal/provider/provider.go

Also, did you get a chance to get feedback from the backend team about the VPC Endpoint error I am facing above?

Thanks,

Shreyas

Re: Terraform provider inconsistencies and issues with IAM role tags

priyer — Fri, 08 Jul 2022 00:28:22 GMT

Hello @SZanpure ,

Greetings!

The docmentation has been updated with the information.

Kindly check the links.

https://github.com/PaloAltoNetworks/terraform-provider-cloudngfwaws/blob/main/docs/index.md

https://registry.terraform.io/providers/PaloAltoNetworks/cloudngfwaws/latest/docs

Regards,
Prasanna Iyer
Product Specialist
Palo Alto Networks
https://live.paloaltonetworks.com/t5/cloud-ngfw-help-center/ct-p/Cloud_NGFW

Re: Terraform provider inconsistencies and issues with IAM role tags

priyer — Wed, 13 Jul 2022 23:33:36 GMT

Hello @SZanpure ,

Greetings!

Regarding your VPC error:
The team has noticed in the documentation that a note mentions that cloudngfwaws_commit_rulestack resource should be placed in a separate plan as the plan that configures the rulestack and its contents. We suggest you put it in a separate plan.

Kindly refer to this link
https://registry.terraform.io/providers/PaloAltoNetworks/cloudngfwaws/latest/docs/resources/commit_rulestack

Regards,
Prasanna Iyer
Product Specialist
Palo Alto Networks
https://live.paloaltonetworks.com/t5/cloud-ngfw-help-center/ct-p/Cloud_NGFW

Re: Terraform provider inconsistencies and issues with IAM role tags

priyer — Tue, 26 Jul 2022 23:13:34 GMT

Hi @SZanpure,

I wanted to follow up with you to know if you still need any assistance on the issue.

Regards
Prasanna Iyer
Product Specialist
Palo Alto Networks
https://live.paloaltonetworks.com/t5/cloud-ngfw-help-center/ct-p/Cloud_NGFW
*Don’t forget to accept the solution provided!*

Re: Terraform provider inconsistencies and issues with IAM role tags

SZanpure — Wed, 27 Jul 2022 07:54:07 GMT

Hi Prasanna,

Sorry for the late response, but we have decided not to pursue with NGFW for now, since it still feels like a product in the works. I am happy for you to close this issue.

Thanks
Shreyas