Rule Enrichment Error

[Tim_Grossner]

Anyone have a problem with, when you try to do rule enrichment on a rule(s) that is marked for RE, when you click on "Analyze Data" it says "no rules selected for learning"?

[Tim_Grossner · Accepted solution]

Just found my problem. Wrong device group selected in the log connector. 😕

[rohill]

Im getting the same error message and my device group is correct in the log collector.  Any other suggestions?

[dgildelaig]

The configuration that you are using for the RE needs to come from the device.

I mean, do not directly upload the XML configuration into a project, but attach a device into the project and use the device as the source for importing the configuration.

This may be the reason that provoked that you would get no results.

QUESTION: Why can't I bring the XML manually into the project via the Palo Alto Networks import field? 
ANSWER: When doing Rule Enrichment or Machine Learning processes, we will have to go into the information we learnt from logs. We do need a way to map the security rules to logs that the firewall has generated. We do have a map between logs and devices, as the logs provide the serial number of the device, and we do need to have a mapping between the device and the configuration. This mapping is done by importing the configuration from the device itself.

QUESTION: Does this mean that we need to have connectivity to the firewall to download the config?
ANSWER: Until version 1.1.12, the answer was YES. In 1.1.12 we have provided a functionality to upload the XML config into the device (I refer to the device within Expedition), so you won't need to provide API Keys to do HTTPS connections to the FW and retrieve the XML config.

[RMikalauskas]

Hello,

Device imported. Running configuration imported.

All Policies are there. All Policies are tagged for ML and RE (Monitor>All Rules).  But trying to use ML or RE (Analyze Data) gives the same error: "No rules selected for learning".

What I want to do is - grab seen Source and Destination addresses and Apps in the Traffic logs.

App ID Adoption > Retrieve Apps (Fast or Slow) is working - so Log Collector Plugin is functioning.

What mystery is this?

v. 1.1.23. All internal checks pass.

[dgildelaig]

Two things that you could check are:

- If your configuration is from a Panorama device, the log connector needs to refer to the Panorama device, selecting the desired device group and serial number that applies

- If you have imported multiple times the same configuration from a firewall (the sources will have the time-stamp in its name), you need to recreate the log connector to make sure that it is using the correct config (even they may have the same config content, their internal IDs are different). This is something that we need to improve in Expedition, so the step is not required. 

[RMikalauskas]

Thank you. Must have been the second thing. Or somethign else, because I upgraded tool to the 1.1.27 and Imported Device/Config and recreated Project from sratch.

The error is gone, but there is another thing: after RE analysis is complete - I am presented with 0 results. The table is empty.

Is there a Time Frame limitation of logs? Mine are about two months old (using custom Time Frame in Log Collector).

Edit> nope, still no Enrichment or ML data even with fresh logs.

[dgildelaig]

There is no limitation software based.

The only limitations that we could face are:

- Bugs in the ML module

. The HW fails to process all the data while placing it in memory

- There is no disk space enouigh for hosting all the data while processing

[ghalbedel]

I also have the problem with "no data to display" or "no rules selected for learning" in rule enrichment. I have tried everything suggested in this thread including upgrading to the latest version of Expedition, 1.1.35.

I have rules tagged for RE. I recreated the connector, I only have one device and no panorama.  No dice.

[GNeyrinck]

I've deleted all the different imports, unset base config and deleted it.

I re-loaded the content from the device again -> OK

Re-configured the log-connector, I'm only having one PA-220, with one VSYS

I've selected 3 rules on wich I wanted to test the rule Enrichment.

I can do an "analysis" but now it's still not showing any output.

[dgildelaig]

THe log connector specifies a device that has reported logs.

Have you imported those logs into Expedition and have you processed those logs first? And, are the rules that you flagged for RE reported traffic for the selected days in the log connector?

[GNeyrinck]

Hi,

The following logs have been processed:

root@Expedition:/home/expedition# ls -al /PALogs/

total 47612

drwxrwxrwx  5 www-data   www-data       4096 Sep  9 06:25 .

drwxr-xr-x 25 root       root           4096 Aug 29 09:16 ..

-rw-rw-r--  1 www-data   www-data    7555873 Aug 29 21:01 PA-220_traffic_2019_08_30_last_calendar_day.csv.gz

-rw-rw-r--  1 www-data   www-data   12578779 Aug 30 21:01 PA-220_traffic_2019_08_31_last_calendar_day.csv.gz

-rw-rw-r--  1 www-data   www-data    7259591 Aug 31 21:01 PA-220_traffic_2019_09_01_last_calendar_day.csv.gz

-rw-rw-r--  1 www-data   www-data    5756529 Sep  1 21:00 PA-220_traffic_2019_09_02_last_calendar_day.csv.gz

-rw-rw-r--  1 www-data   www-data    5795674 Sep  6 21:00 PA-220_traffic_2019_09_07_last_calendar_day.csv.gz

-rw-rw-r--  1 www-data   www-data    5976994 Sep  7 21:00 PA-220_traffic_2019_09_08_last_calendar_day.csv.gz

-rw-rw-r--  1 www-data   www-data    3169942 Sep  8 21:00 PA-220_traffic_2019_09_09_last_calendar_day.csv.gz

drwxr-xr-x  7 www-data   www-data       4096 Sep  9 06:25 connections.parquet

-rw-r--r--  1 www-data   www-data     623918 Mar  1  2019 iron-skillet-90dev.zip

drwxr-xr-x  2 www-data   www-data       4096 Sep  9 06:24 spark-warehouse

drwxr-xr-x  2 www-data   www-data       4096 Sep  9 06:25 sparkLocalDir

-rw-rw-r--  1 expedition expedition       17 Sep  6 09:48 ssh-export-test.txt

I've unset base config after the logs have been processed, but I think that shouldn't really matter?

[GNeyrinck]

I've enabled the RE on all rules, and now it's ok I think:

After setting the correct dates can now see this output.

[ghalbedel]

this is now working for me. not sure why. when I open the rule enrichment window in policies there are initially no selected rules showing (but I have selected rules). Previously, when I clicked on "analyze data", I got a pop up that said "no rules selected" or something like. that.

Now, when I click "analyze data" it does, and then all the selected rules show in the window.

[GNeyrinck]

did you have to change the dates in "Time Frame Overrirde"?

I had to and then analysis started