topic Re: Rule Enrichment Error in Expedition Discussions

Rule Enrichment Error

Tim_Grossner — Thu, 14 Jun 2018 15:54:30 GMT

Anyone have a problem with, when you try to do rule enrichment on a rule(s) that is marked for RE, when you click on "Analyze Data" it says "no rules selected for learning"?

Re: Rule Enrichment Error

Tim_Grossner — Thu, 14 Jun 2018 17:38:56 GMT

Just found my problem. Wrong device group selected in the log connector. 😕

Re: Rule Enrichment Error

rohill — Tue, 09 Apr 2019 20:39:19 GMT

Im getting the same error message and my device group is correct in the log collector. Any other suggestions?

Re: Rule Enrichment Error

dgildelaig — Wed, 10 Apr 2019 14:29:42 GMT

The configuration that you are using for the RE needs to come from the device.

I mean, do not directly upload the XML configuration into a project, but attach a device into the project and use the device as the source for importing the configuration.

This may be the reason that provoked that you would get no results.

QUESTION: Why can't I bring the XML manually into the project via the Palo Alto Networks import field?
ANSWER: When doing Rule Enrichment or Machine Learning processes, we will have to go into the information we learnt from logs. We do need a way to map the security rules to logs that the firewall has generated. We do have a map between logs and devices, as the logs provide the serial number of the device, and we do need to have a mapping between the device and the configuration. This mapping is done by importing the configuration from the device itself.

QUESTION: Does this mean that we need to have connectivity to the firewall to download the config?
ANSWER: Until version 1.1.12, the answer was YES. In 1.1.12 we have provided a functionality to upload the XML config into the device (I refer to the device within Expedition), so you won't need to provide API Keys to do HTTPS connections to the FW and retrieve the XML config.

Re: Rule Enrichment Error

RMikalauskas — Tue, 09 Jul 2019 07:00:46 GMT

Hello,

Device imported. Running configuration imported.

All Policies are there. All Policies are tagged for ML and RE (Monitor>All Rules). But trying to use ML or RE (Analyze Data) gives the same error: "No rules selected for learning".

What I want to do is - grab seen Source and Destination addresses and Apps in the Traffic logs.

App ID Adoption > Retrieve Apps (Fast or Slow) is working - so Log Collector Plugin is functioning.

What mystery is this?

v. 1.1.23. All internal checks pass.

Re: Rule Enrichment Error

dgildelaig — Tue, 09 Jul 2019 07:06:36 GMT

Two things that you could check are:

- If your configuration is from a Panorama device, the log connector needs to refer to the Panorama device, selecting the desired device group and serial number that applies

- If you have imported multiple times the same configuration from a firewall (the sources will have the time-stamp in its name), you need to recreate the log connector to make sure that it is using the correct config (even they may have the same config content, their internal IDs are different). This is something that we need to improve in Expedition, so the step is not required.

Re: Rule Enrichment Error

RMikalauskas — Tue, 09 Jul 2019 09:02:24 GMT

Thank you. Must have been the second thing. Or somethign else, because I upgraded tool to the 1.1.27 and Imported Device/Config and recreated Project from sratch.

The error is gone, but there is another thing: after RE analysis is complete - I am presented with 0 results. The table is empty.

Is there a Time Frame limitation of logs? Mine are about two months old (using custom Time Frame in Log Collector).

Edit> nope, still no Enrichment or ML data even with fresh logs.

Re: Rule Enrichment Error

dgildelaig — Wed, 24 Jul 2019 18:55:17 GMT

There is no limitation software based.

The only limitations that we could face are:

- Bugs in the ML module

. The HW fails to process all the data while placing it in memory

- There is no disk space enouigh for hosting all the data while processing

Re: Rule Enrichment Error

ghalbedel — Wed, 14 Aug 2019 18:50:38 GMT

I also have the problem with "no data to display" or "no rules selected for learning" in rule enrichment. I have tried everything suggested in this thread including upgrading to the latest version of Expedition, 1.1.35.

I have rules tagged for RE. I recreated the connector, I only have one device and no panorama. No dice.

Re: Rule Enrichment Error

GNeyrinck — Mon, 09 Sep 2019 11:53:24 GMT

I've deleted all the different imports, unset base config and deleted it.

I re-loaded the content from the device again -> OK

Re-configured the log-connector, I'm only having one PA-220, with one VSYS

I've selected 3 rules on wich I wanted to test the rule Enrichment.

I can do an "analysis" but now it's still not showing any output.

Re: Rule Enrichment Error

dgildelaig — Mon, 09 Sep 2019 11:58:38 GMT

THe log connector specifies a device that has reported logs.

Have you imported those logs into Expedition and have you processed those logs first? And, are the rules that you flagged for RE reported traffic for the selected days in the log connector?

Re: Rule Enrichment Error

GNeyrinck — Mon, 09 Sep 2019 12:09:48 GMT

Hi,

The following logs have been processed:

root@Expedition:/home/expedition# ls -al /PALogs/

total 47612

drwxrwxrwx 5 www-data www-data 4096 Sep 9 06:25 .

drwxr-xr-x 25 root root 4096 Aug 29 09:16 ..

-rw-rw-r-- 1 www-data www-data 7555873 Aug 29 21:01 PA-220_traffic_2019_08_30_last_calendar_day.csv.gz

-rw-rw-r-- 1 www-data www-data 12578779 Aug 30 21:01 PA-220_traffic_2019_08_31_last_calendar_day.csv.gz

-rw-rw-r-- 1 www-data www-data 7259591 Aug 31 21:01 PA-220_traffic_2019_09_01_last_calendar_day.csv.gz

-rw-rw-r-- 1 www-data www-data 5756529 Sep 1 21:00 PA-220_traffic_2019_09_02_last_calendar_day.csv.gz

-rw-rw-r-- 1 www-data www-data 5795674 Sep 6 21:00 PA-220_traffic_2019_09_07_last_calendar_day.csv.gz

-rw-rw-r-- 1 www-data www-data 5976994 Sep 7 21:00 PA-220_traffic_2019_09_08_last_calendar_day.csv.gz

-rw-rw-r-- 1 www-data www-data 3169942 Sep 8 21:00 PA-220_traffic_2019_09_09_last_calendar_day.csv.gz

drwxr-xr-x 7 www-data www-data 4096 Sep 9 06:25 connections.parquet

-rw-r--r-- 1 www-data www-data 623918 Mar 1 2019 iron-skillet-90dev.zip

drwxr-xr-x 2 www-data www-data 4096 Sep 9 06:24 spark-warehouse

drwxr-xr-x 2 www-data www-data 4096 Sep 9 06:25 sparkLocalDir

-rw-rw-r-- 1 expedition expedition 17 Sep 6 09:48 ssh-export-test.txt

I've unset base config after the logs have been processed, but I think that shouldn't really matter?

Re: Rule Enrichment Error

GNeyrinck — Mon, 09 Sep 2019 12:09:16 GMT

I've enabled the RE on all rules, and now it's ok I think:

After setting the correct dates can now see this output.

Re: Rule Enrichment Error

ghalbedel — Mon, 09 Sep 2019 12:23:50 GMT

this is now working for me. not sure why. when I open the rule enrichment window in policies there are initially no selected rules showing (but I have selected rules). Previously, when I clicked on "analyze data", I got a pop up that said "no rules selected" or something like. that.

Now, when I click "analyze data" it does, and then all the selected rules show in the window.

Re: Rule Enrichment Error

GNeyrinck — Mon, 09 Sep 2019 12:25:39 GMT

did you have to change the dates in "Time Frame Overrirde"?

I had to and then analysis started

Re: Rule Enrichment Error

ghalbedel — Mon, 09 Sep 2019 12:41:07 GMT

I had tried that previously but nothing happened.