Rule merge all results

[aporue]

I am running Expedition 1.0.106 and have a question about merging rules. Once the analysis is done I am presented with cases that I look at individually. From there I can merge by highlighting the rules or by clicking on the 'merge by selection or all results'.  Either one of those ways works to merge a case. The question I have is the part that says '...or all results'. I have over 4000 cases and would like to choose to merge all results but it does not work. When I have no cases selected and click the button it says it has merged 2 rules (not sure what is up with that) but it does not merge all results. As an alternative, I check the box next to 'Duplicates' which should select all cases but it says you can only select 10 cases at a time. 

Thanks for any help.

[aporue]

Thanks for the update, 10 at a time is better than 1 so that will be helpful. I do want to ask about the 2nd part of your response as it relates to 'any'. When migrating from Cisco ASA (and other firewalls also) the application field will always be 'Any' as the ASA is not doing app-id. Does that mean we can still only do 1 rule merge at a time? Hopefully, that is not the case.

[dgildelaig]

No, that is not the case.

What we won't merge are rules where some have "any" (does not have a specfic value) and others have specific values. As the result would not be "any", but the specifics.

For instance, 
RULE1  Trust 10.0.0.0/24          -> DMZ 172.16.0.0/25   SSH   app-default ALLOW    (users)

RULE2  Trust 192.168.10.0/24 -> DMZ 172.16.0.0/25   any   app-default          ALLOW   (admins)

We do not want to merge them into
RULE3  Trust 10.0.0.0/24, 192.168.10.0/24 -> DMZ 172.16.0.0/25   SSH   app-default ALLOW

As admins lost access to other allowed apps.
The same would apply to users and to ports.

[rolinger]

I am dealing with something similar, we are importing thousands of rules from multipe FWs and needing to consolidate/merge rules.  Each 'filter' we apply is returning hundreds of cases.  We need an option that we can select that will simply merge ALL Cases into case specific rules.  The project I am on has over 250 FWs, from different vendors, being merged into a handful of 7000s.  Every two weeks we are migrating 10,000+ rules....we have a lot of it streamlined until we get to rule consolidation...6 or 7 different consolidation filters are spawning thousands of cases per migration.

[ ] Merge ALL Cases by Case

IE:

Case 1 (9 rules) 

Case 2 (24 rules)

Case 3 (19 rules)

...

Case 347 (17 rules)

= would out put 347 individual rules.

[rolinger]

My client is also on an older version of Expedition (1.0.105).  Working on getting them to upgrade.  However is the intent of Consolidating Rules "Merge Selected" to Merge Cases by Case - but only (currently) 10 at a time?

Case 1 (9 rules)

Case 2 (16 rules)

....

Case 10 (14 rules)

== outputs 10 individual rules???  Cause right now, in our version I think thats broken and we are having to do one case at a time.  This is painful.

[dgildelaig]

It is now supporting 10 cases (which it means more than 10 rules).

We will modify the merge behaviour to support multiple merges in the background, as merging security rules implies quite a number of check and calculations.

[aporue]

I just thought I would respond with an update on this thread. While the 10 case merge is working there are some anomalies while using it. The one that shows up most is that sometimes instead of merging the 10 cases it will only do some of the cases. It seems to only show up after doing a bunch in a row so I am guessing maybe the system is not keeping up. In any case, I just wanted to throw that out there. Would still be nice to do more than 10 at a time as I am currently working on a firewall with 3700 cases.

Thanks.

[dgildelaig]

Would it be possible to share the project with us a fwmigrate at paloaltonetworks dot com?

Next week I will work on transfering this process into a background process, so it can run to merge as many cases as you may want to select.

Best,

[rolinger]

We used the newest version of Expedition for our most recent migration.  We had an insane amount of problems.  Ultimately the problems corrupted the project(s) and we had to start over multiple times.  Some of this is due to lack of memory or system resources.   So after that project we increased the VMs memory and the up'd the php.ini memory to 512M.  We are going to be running tests with the same files this week to see if the memory increases help performance and stop corrupting the projects.  Corruption usually happened after a timeout/error...once that happened, 9 time out of 10 the project was useless.

Regarding the 10 cases at a time, if you click on all 10 cases it will populate the corresponding rules in the background, if you click 'merge' before they are fully populated then it will merge only the ones that made it into the view before you clicked merge.  We found that that if we highlighted 10 at a time, we had to wait a few seconds before doing the actual merge - if we clicked to soon, it would merge only a few of them.

[aporue]

Do you mean the project after it is in Expedition? Or just the ASA config file? Never had much luck with the export feature for projects that are this big from Expedition. I will need to check with the client to see if they are ok to send the file first but is there a way to send it encrypted?

Thanks.

[aporue]

For large projects like these is there a recommendation for the resources (processor, memory hdd space) the VM needs?

Thanks.

[sjanita]

recommended resources:

CPU or cores: 4

RAM: 16 GB

Storage for ML usage: 100 GB (minimum), recommended 1 TB (10 or more firewalls)

[rolinger]

yeah, my clients Expedition deployment was bare minimum.  1.5Gb memory and 1 cpu.  We were able to process about 10,000 objects and about 2000 rules - it was slow and occassionally would time out on specific filter/queries but ultimately we got through it.  We tried a project with 14,000 objects and over 5000 rules (from 33 merged FW configs) and Expedition lost its mind every time - we had to break it down into smaller projects.  As well, the php.ini was still set to max memory of 128M

We just upped the VM memory to 4Gb and upped the php.ini to 512M....unfortunately not able to give it anymore CPUs at the moment.  Will be doing a new project friday to see how all the memory improvements will help Expedition not puke.

Trying to get my client ti up the VM to 8GB and at least 2 CPUs....not an easy task though.

[aporue]

Any updates this issue? It seems that it has kind of gotten worse in the last release. I have almost no luck selecting 10 cases where all of the cases get merged using the method of selecting the first one, holding down the shift key and selecting the 10th one. On the other hand, if I hold down the control key and select 10 cases that seems to always work but of course it is slower. And with 1000's of cases it is already slow.

[dgildelaig]

Could you share the project to replicate it on our side?

fwmigrate at paloaltonetworks dot com

[rolinger]

@aporue  - have you tried upgrading the assinged Expedition resources?  The default resources Expedtion uses is quite low.  Once I upgraded the VM memory and increase the php memory buffer many annoying problems, timeouts and errors went away.

My configuration still has 1 cpu assigned, but it now has 4Gb of memory assigned and the php memory setting was changed from the default 128mb to 512mb and it makes a world of difference in performance.  if you assign more memory to the VM you will need to delete a certain file (you will need to look it up) that then recalculates assigned vm memory when expedition boots up.