Migrating Checkpoint R80

[alestevez]

With the new version of Checkpoint Smartcenter R80, the way to obtain the rules has changed. 

Exporting Security Rules

After you login to your SmartCenter, you have to run the following command to know what policies are avaialble for exportation:

mgmt_cli show access-layers

- uid: "xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxx"
 name: "Internet Security"
 type: "access-layer"
 domain:
 uid: "xxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx"
 name: "SMC User"
 domain-type: "domain"

In this case, "Internet Security" is my Rulebase to export for migration. Based on what we have seen from the Checkpoint API, in case we have more than 400 rules, we have to use an offload to generate exportations in chunks of 400 rules, which means if we have 650 security rules, we will have to do this:

mgmt_cli show access-rulebase offset 0 limit 400 name "Internet Security" details-level "full" use-object-dictionary true --format json > RuleSet_0_400.json
mgmt_cli show access-rulebase offset 401 limit 650 name "Internet Security" details-level "full" use-object-dictionary true --format json > RuleSet_401_650.json

After we created all the json files, we will need to create a new file, and we will name it "order" to put inside the filenames in the order they need to be read, like in this example:

Edit "order" file

RuleSet_0_400.json
RuleSet_401_650.json

After that, we have to ZIP it. To do it right, all these three files need to be under the same folder, and from within the folder, we can run the command from the CLI:

zip Rules.zip *

This will create a new Rules.zip only contaning the three files without any other folder inside. It's important to remember when you create the ZIP file to avoid having any folder inside the ZIP file—just the json and "order" files.

• If the Security rules are less than 400, we have to just export to a single json file. There is no need to ZIP it then.

 Exporting Nat Rules

For Nat rules, we have found the limitation is for 500 Nat Rules per export, so we can apply the samething we did with the Security Rules. For example, we are going to sat we have 600 Nat Rules.

Notice the name of the Ruleset, in this case, is Internet without Security at the end. You have to check from the SmartCenter to see the Nat ruleset Name.

mgmt_cli show nat-rulebase offset 0 limit 500 package "Internet" details-level "full" use-object-dictionary true --format json > NATRuleSet_0_500.json
mgmt_cli show nat-rulebase offset 501 limit 600 package "Internet" details-level "full" use-object-dictionary true --format json > NATRuleSet_501_600.json

After we created all the json files, we will need to create a new file, and we will name it "order" to put inside the filenames in the order they need to be read, like in this example:

Edit "order" file

NATRuleSet_0_500.json
NATRuleSet_501_600.json

After that, we have to ZIP it. To do it right, all these three files need to be under the same folder, and from within the folder, we can run the command from the CLI:

zip NatRules.zip *

• If the Nat rules are less than 500, we have to just export to a single json file. There is no need to ZIP it then.

Exporting Routing and interfaces

From the Firewall CLI, you can run the following:

netstat -nr > routes.txt

Export the routes.txt to where you store the NatRules.zip and Rules.zip

With all this information, we can go to Expedition, Create a new Project, enter the Project, and go to IMPORT > CHECKPOINT > VERSION R80.

1. Assign a name to your configuration such as "MyInternetGW"
2. Select the Rules.zip for Security Rules
3. Select the NatRules.zip for Nat rules
4. Select the routes.txt for the routes
5. Click UPLOAD

[SezgiDogan]

Hello,

I can not migrate sub policies in the security policies. Is there a fix for this?

Regards,

[Gun-Slinger]

This isn't working, keep getting the following error:

Running:

SMS01-CKPT> mgmt_cli show access-rulebase name "Test-Policy Security" details-level "full" use-object-dictionary true --format json > Test-Policy.json

Keep getting:

MGMT9000 {
"code" : "generic_err_invalid_parameter_name",
"message" : "Unrecognized parameter [>]"

[alestevez]

You have to be "expert" to run the command and redirect the output to a file

[Gun-Slinger]

I ran the command in expert mode and the file generated correctly, however I have tried to import into the Expedition tool and each time it states There are not Objects defined. Import Failed!

THis is being completed on a SMS R80.10 VM

[alestevez]

https://live.paloaltonetworks.com/t5/Expedition-Discussions/How-to-Upload-configuration-files-bigger...

[Gun-Slinger]

Completed all the config associated with that link How to Upload conf files with no luck. I have attempted to complete this instructions in this thread EXACTLY as perscribed with the most progress being the Import progress status flapping between Initializing and Reading Config.... of which it continues this back and forth for quite sometime.

Next I attempted this option outlined in this thread

Issue Getting Checkpoint R80 config to load

No luck there.

I only have 162 rules, no NAT rules, but a bunch of objects since this is an SMS with multiple policies.

Any ideas?

[Gun-Slinger]

RESOLVED: The issue was with the .json files and piping the output to a seperate file. The commands above are correct, however the output files were adding the following "Username" at the TOP of the file:

Username:
{
"uid" : "1cxxxx6-3bff-4xxxxxx8-7xxxxxxxxx",
"name" : "blah-blah",
"rulebase" : [ ],
"total" : 111
}

This appears to be an issue/bug within the R80.10 code when doing outputs to a file as this "Username" is coming from the prompt before launching the api call.

Once this "Username:" line was removed from the json file, the import worked as expected.

[Gun-Slinger]

I have confirmed there is a limitation on the Checkpoints of 500 rules when exporting through the API. With our environment, I had to break it up into groups of 200 due to time out and RAM issues.

I have yet to get the zip file with the orders.txt file to work as perscribed above. I am on the latest version of the expedition tool and when I upload it, it will transistion from initilizing to reading config files, but stalls from there.

Anyone else ran into issues like this or did something change with the new version?

[AR00473455]

Just want to ask you one question that suppose security and NAT policies count around 1500 rules then what is the best way to export policy in one file. Otherwise. I have to export the security policy in 400 rules count like below and zip it one folder.

mgmt_cli show access-rulebase offset 0 limit 400 name "Internet Security" details-level "full" use-object-dictionary true --format json > RuleSet_0_400.json

mgmt_cli show access-rulebase offset 401 limit 800 name "Internet Security" details-level "full" use-object-dictionary true --format json > RuleSet_401_800.json

mgmt_cli show access-rulebase offset 801 limit 1200 name "Internet Security" details-level "full" use-object-dictionary true --format json > RuleSet_801_1200.json

mgmt_cli show access-rulebase offset 1201 limit 1500 name "Internet Security" details-level "full" use-object-dictionary true --format json > RuleSet_1201_1500.json

[Gun-Slinger]

@AR00473455

It is my understanding that the limitations are 500 for any of the rule sets according to Checkpoint. NAT and Security are treated as 2 SEPERATE groups. For example if you have 1000 security rules and 200 NATs; you will need 2 seperate security rules files of 500 each and 1 NAT rule file.

This being said I have yet to get the zip file proceedure to work and actually had to export out my security policies in seperate files as perscribed, but then manually merge the files into 1 .json file using notepad++.

Won't lie, it would be great if the zip proceedure worked

[alestevez]

To let the ZIP procedure work you have to be sure at the time to create the zip there are no Folders inside it, so create the zip like

cd wherealljsonfiles_plus_order_are
zip checkpoint.zip *

[cpsantos]

Hi all,

There are a few misconceptions of what each command does in your procedure, first of all "limit" is the number of results after the specified offset, for instance you set offset to 0 and limit to 400 you get rules 1 to 400.

Now the next set of rules you should consider offset 400 instead of 401, if 0=1 then 400=401 in the offset counting.

Also it's a pain to always have to request the customer for username and password and the command you wrote "hides" username request, my suggestion to update and works like a charm, do has you like:

Step1

#mgmt_cli login (when asked the customer should type his username and password here)

Step2

Copy output of the result login to a file, for the sake of my example I used id.txt

This id lets you reuse the logged in session id, with an idle timeout of 10 minutes, more then enough to run all queries, even for quite big rulebases like in my test scenario 2569 rules (don't ask...)

Step 3 run your instructions with slit differences...limit as I explained should be set to 400, you are right 400 is good number, although it allows up to 500, in rulebase it normally crashes the api or it times out, so:

Also it's good idea to set limit and the default value is 50 so you don't want to think you have it all, but the end file will tell you a tip.

Myscript:

mgmt_cli show access-rulebase offset 0 limit 400 name "ASecurityPolicy Security" details-level "full" use-object-dictionary true --format json -s id.txt > RuleSet_`date +%y%m%d%H%M%S`.json
mgmt_cli show access-rulebase offset 400 limit 400 name "ASecurityPolicy Security" details-level "full" use-object-dictionary true --format json -s id.txt > RuleSet_`date +%y%m%d%H%M%S`.json
mgmt_cli show access-rulebase offset 800 limit 400 name "ASecurityPolicy Security" details-level "full" use-object-dictionary true --format json -s id.txt > RuleSet_`date +%y%m%d%H%M%S`.json
mgmt_cli show access-rulebase offset 1200 limit 400 name "ASecurityPolicy Security" details-level "full" use-object-dictionary true --format json -s id.txt > RuleSet_`date +%y%m%d%H%M%S`.json
mgmt_cli show access-rulebase offset 1600 limit 400 name "ASecurityPolicy Security" details-level "full" use-object-dictionary true --format json -s id.txt > RuleSet_`date +%y%m%d%H%M%S`.json
mgmt_cli show access-rulebase offset 2000 limit 400 name "ASecurityPolicy Security" details-level "full" use-object-dictionary true --format json -s id.txt > RuleSet_`date +%y%m%d%H%M%S`.json
mgmt_cli show access-rulebase offset 2400 limit 400 name "ASecurityPolicy Security" details-level "full" use-object-dictionary true --format json -s id.txt > RuleSet_`date +%y%m%d%H%M%S`.json

tail each result:

[Expert@gw-83919d:0]# tail RuleSet_181123160805.json
  "from" : 1,
  "to" : 400,
  "total" : 2569
}
[Expert@gw-83919d:0]# tail RuleSet_181123161507.json
  "from" : 401,
  "to" : 800,
  "total" : 2569
}
[Expert@gw-83919d:0]# tail RuleSet_181123180927.json
  "from" : 801,
  "to" : 1200,
  "total" : 2569
}
[Expert@gw-83919d:0]# tail RuleSet_181123181125.json
  "from" : 1201,
  "to" : 1600,
  "total" : 2569
}
[Expert@gw-83919d:0]# tail RuleSet_181123181317.json
  "from" : 1601,
  "to" : 2000,
  "total" : 2569
}
[Expert@gw-83919d:0]# tail RuleSet_181123181509.json
  "from" : 2001,
  "to" : 2400,
  "total" : 2569
}
[Expert@gw-83919d:0]# tail RuleSet_181123181635.json
  "from" : 2401,
  "to" : 2569,
  "total" : 2569
}

NOTE: See "total" in the end of file? 2569 is the total number of rules, so has a tip you will know if you have reached "to" the last rule when "total" and "to" matches.

The rest is just fine.

Hope this helps,

Regards

[AR00473455]

Hello All.

I am trying to run below command on checkpoint  management center for security policy contains around 1500 security policies. So I am runing the below command to fecth the policy in 400 offset limit  however i have not received output in the file and size 1KB size.

mgmt_cli show access-rulebase offset 0 limit 400 name "Internet Security" details-level "full" use-object-dictionary true -u username --format -json > RuleSet_0_400.json.

When I opend the 1 KB size file. The content shows timeout reached.

---

[alestevez]

Probably is Checkpoint API problem, try to reduce to 100 instead of 400 and see...

[cpsantos]

Hi AR00473455,

Check Point api can expire during your query hence the timeout, reducing the limit might help, but there are other issues to look at.

details-level "full" calls in new queries, like listing "hosts, networks, groups, services tcp/udp/other, and again service-groups, etc" unfortunately there are also limits to this new queries, and in some cases only running rule by rule will do, or even worse big rule can even crash the api, I noticed that a rule with a big list of hosts will crash the api, grouping them in a group however helps it get fixed, but other issues show up, like unexported objects within the show rulebase command, producing UID network objects with 1.1.1.1, I do have to say it's a nice trick from Albert's team at PaloAlto, but never the less this needs to get fixed by exporting smaller and smaller data through the api and compile this for Expedition.

Best regards, I hope reducing the limit like Albert says works for you, it's not working for me with some rules anyway, but I hope I'll get it fixed using the api and will share it here once I figure it out.

Regards,

Carlos Santos

[AR00473455]

Hi Albert/Carlos,

It worked,. Started to get output when I set limit to 100 so each file contains 100 rules. The same limit applied for NAT rules as well.  

However I would like to know that how to combine all files into one file to process on Expedition tool . As I have done the same way what mentioned in the above article but not worked.

After we created all the json files we will need to create a new file and we will name it "order" to put inside the filenames in the order they need to be read, like in this example:  

Edit "order" file

NATRuleSet_0_500.json
NATRuleSet_501_600.json

After that we have to ZIP it, to do it right all these 3 files needs to be under the same folder and from within the folder we can run the command from the CLI:

zip NatRules.zip *

[cpsantos]

@AR00473455 

Sorry for not replying, you cannot combine the files, just add them to a the zip, along with "order" file, the content of the order file should be the list of the rules by order top down, that's it.

[IvanMoore]

I wrote a script for exporting policies on our Provider-1 system.  It takes 2 variables, one is the CMA IP and the other is the Policy name.   It will export the files in chunks of 50 rules and create a zip file containing them all with the order file as well.  It will do this for both the access policy and the nat policy.  We had performance issues on our MDS for exports so we have it set to 50.  The big issue we still run into due to performance is if it fails to properly get the data back.  You never get a warning, you just end up with a mostly empty file.  So look for any files in your zips that are oddly small.

 #!/bin/bash
#
# This script exports the given access and nat policy for the CMA given and
# puts it into the proper format for importing into Expedition
#
#
VERSION=1.0.21
. /etc/profile.d/CP.sh
cd /var/log/tmp
ID="id-$$.txt"

USAGE='Usage:  export-policy.sh (CMA IP) (Policy)\t\n '

if [ $# == 0 ] ; then
    echo -e $USAGE
    exit 1;
fi

CMAIP="$1"
if [ $2 ] ; then
    POLICY="$2"
else
    echo -e $USAGE
    exit 1;
fi
if ! mdsenv $CMAIP >& /dev/null ;
then
  if [ ! $CMAIP = "Global" ] ; then
    echo "That is not a proper CMA IP"
    exit 1
  fi
fi

mgmt_cli login -r true -d $CMAIP > $ID

TOP_LIMIT=`mgmt_cli show access-rulebase -d $CMAIP offset 50000 limit 50 name "$POLICY Security" details-level "standard" use-object-dictionary true --format json -s $ID | grep total | awk -F " " '{print $3}'`
OFFSET="0"
FILENAME="0_50"

echo "Total Number of Rules: $TOP_LIMIT"
echo -n "Exporting Rules."
while [ "$OFFSET" -lt "$TOP_LIMIT" ] ;
do
  mgmt_cli show access-rulebase -d $CMAIP offset $OFFSET limit 50 name "$POLICY Security" details-level "full" use-object-dictionary true --format json --conn-timeout 3600 -s $ID >> RuleSet_$FILENAME.json
  OFFSET=$((OFFSET+50));
  FILENAME="$((OFFSET+1))_$((OFFSET+50))"
  echo -n ".$OFFSET"
done
echo " "
echo "Packing up files"
ls -rt RuleSet*.json >order
/opt/CPsuite-R80/fw1/Python/bin/python -m zipfile -c $POLICY-Rules.zip order RuleSet*.json >/dev/null 2>&1
rm RuleSet*.json
echo " "

if [ ! $CMAIP = "Global" ] ; then

TOP_LIMIT=`mgmt_cli show nat-rulebase -d $CMAIP offset 50000 limit 50 package $POLICY details-level "standard" use-object-dictionary true --format json -s $ID | grep total | awk -F " " '{print $3}'`
OFFSET="0"
FILENAME="0_500"

echo "Total Number of NAT Rules: $TOP_LIMIT"
echo -n "Exporting NAT Rules."

while [ "$OFFSET" -lt "$TOP_LIMIT" ] ;
do
 mgmt_cli show nat-rulebase -d $CMAIP offset $OFFSET limit 500 package "$POLICY" details-level "full" use-object-dictionary true --format json --conn-timeout 3600 -s $ID >> NATRuleSet_$FILENAME.json
  OFFSET=$((OFFSET+500));
  FILENAME="$((OFFSET+1))_$((OFFSET+500))"
  echo -n ".$OFFSET"
done
echo " " 
echo "Packing up files"
ls -rt NATRuleSet*.json >order
/opt/CPsuite-R80/fw1/Python/bin/python -m zipfile -c $POLICY-NatRules.zip order NATRuleSet*.json >/dev/null 2>&1
rm NATRuleSet*.json
echo " "

echo "Output found in $POLICY-Rules.zip and $POLICY-NatRules.zip."


else
  echo "Output found in $POLICY-Rules.zip"
fi
echo " "

mgmt_cli logout -s $ID
rm $ID >/dev/null 2>&1
echo "All Done.  Don't forget to grab a copy of the routing table from the gateway"

  (Updated to fix NAT export/import) 

[rkalinowski]

This thread has been helpful, however when I go to import the created zip files into expedition it hangs on "Reading config files" with no error or indication as to what's happening. Any insight into system generated logs or ways to determine what's causing this problem?

[cpsantos]

You can search for something /tmp/error, from my side it wasn't very helpful, maybe you get luck here.

[VijiSridhar]

Hi ..trying to migrate from R80.10 using Expedition tool version 1.1.35. When I use the 80.10 Beta it says "No Index File Found", if I try the R80 command it stays on Reading config file and does not seem to progress. Any help would be appreciated.Thanks.

[VijiSridhar]

Use this link to update the file size limit. I was able to upload Checkpoint 80.10 config both ways using json and R80.10 Beta zip file

https://live.paloaltonetworks.com/t5/Expedition-Discussions/How-to-Upload-configuration-files-bigger...