Searching through logs, where do I start?

by 2 weeks ago - last edited 2 weeks ago (1,876 Views)

It's important to log traffic flowing through the firewall but it can become a daunting task to find the proverbial needle in the haystack if the log volume is so high you're being buried under several haystacks before you even get a chance to take a look at the first one.

In come log filters to save the day and make your job a little easier.

 

But where do you start?

 

Is this the right filter for the job?Is this the right filter for the job?

Luckily you don't need to use @kiwi's tea strainer to find the logs you're looking for!

 

 

There are several methods to build filters, with the simplest route being that almost everything in the log viewer is clickable and will autopopulate the filter.

Click an entry you want to filter and the filter will automatically be created, click the green cursor or hit 'enter' when your cursor is in the search field to apply the filter

Automated filter generatorAutomated filter generator

If the logs currently displayed in the log viewer do not match what you are looking for, you can still quickly select all the relevant fields and then manually replace all the attributes you want to look for, you can even use subnet masks if you're looking for an entire subnet:

 

Searching for a subnetSearching for a subnet

In the example above, my view gets kind of cluttered by the .114 host. If I still want to see the whole subnet, except for .114, I can add a second filter by clicking on the source IP once more, which will add the second filter and will automatically set an inclusive AND relationship between the filters. I can then add a 'not' operator inside the second filter to not show any results for .114.

Show the entire subnet AND NOT 192.168.0.114Show the entire subnet AND NOT 192.168.0.114

If I now want to track .116 and .119 exclusively, I can delete the current filters and create 2 filters for both these IPs, but I will need to change the operator to 'OR' since there can't be 2 source IPs in the same log entry:

 

 OR operator to apply 2 filters at onceOR operator to apply 2 filters at once

You can now take this one step further: If I want to, for example, get all the logs for .116 and .119 for application SSL, I can add a filter for the application by clicking 'ssl' in the policy, but this will create an 'AND' operator and will change the relationships between the filters, so I need to add parentheses '( )' to isolate the source IP addresses from the application (omg we're doing math).

 

Parentheses or no parentheses, that's the questionParentheses or no parentheses, that's the question

Still with me? Great!

For the intermezzo, let's customize the log view a little bit. Some columns may be redundant or irrelevant while others are currently not being displayed in the default view. If you hover your cursor over one of the column headers, the header will highlight and a little drop-down arrow will appear. If you click the arrow you will see an option 'Columns >' and one 'Adjust Columns'. Let's first open the 'Columns' and check or uncheck any log fields you want to see or remove.

 

So, for example, I want to see the Ingress Interface and if a session was decrypted, so I tick the corresponding box and the columns appear right next to the 'Receive Time' column:

Adding and removing columnsAdding and removing columns

You may notice that by adding these columns, a few of my other columns now have fallen off the right end of the traffic log, which means i'd need to scroll to see the content of these fields. This is where the second option in the dropdown comes into play: If you click 'Adjust Columns', the columns will be automatically resized to fit the screen (better) in relation to your screen size:

Adjust Columns.png

 

Lastly, you can click and drag columns to a more suitable location.

Dragging columns to a better locationDragging columns to a better location

To make life a little easier and so you don't need to worry about getting the operators right, there is also a filter builder that lets you select the attributes you want with all the options neatly listed:

Filter BuilderFilter Builder

The filter builder let's you immediately choose if you want to add an 'AND' or 'OR' operated filter, you can then select the attribute you want to search for, select its own operator and set the value.

Depending on what attribute you want to search for, the operators will change to reflect the attribute, for example. an application will have operators 'is present', 'equal' and 'not equal'. The 'is present' operator simply requires an application to be there, a session dropped at the SYN will not have an application, for example. If a value needs to be entered that is an object or an action on the firewall, they will be listed or made available through a dropdown in the value column:

Filter values provided from objects or actionsFilter values provided from objects or actions

Other attributes can have different operators, like 'Destination Port' can allow you to define a port range by using the operators 'greater than or equal' end 'less than or equal' which could also apply to time or bytecount attributes.

 

 Greater than, less than or in a predefined time rangeGreater than, less than or in a predefined time range

There are also multiple ways to negate a filter: either by incorporating a 'not' function in the filter itself or by using the 'negate' option in the connector

 

And not, or 'neq'And not, or 'neq'

Once you've perfected your filter, you can also save it in case you need it later, there's a 'save filter' button on the right, with the load option right next to it:

Save a filter for laterSave a filter for laterLoad a filter when you need itLoad a filter when you need it

All the options we've just gone over are available in every single log view (traffic, threat, URL, WildFire, System, etc.) all following the same principles but each log with its own unique Attributes, so I would encourage you to play around with the filters to get acquainted with the different views and attributes, but also feel free to post follow-up questions or remarks below.

 

A quick cheatsheet for the connectors and operators:

 

Description Operators
 is equal to  eq
 is not equal to  neq
 object in a range  in
 object not in a range  not in
 less than or equal  leq
 greater than or equal  geq
 matches a string  contains

 log entry contains a certain flag

 (has a pcap, has NAT, ...)

 has
 is present/is not empty  neq ''
Description Connector
 combine filters in a single match  and
 either filter may math  or
 negate a filter  not ('or not'  or 'and not' in between filters)

 

 

Reaper out!

Comments
by DonohoeRobert
2 weeks ago

nice post, few questions based on this on the beta pcnse 8.0 :-) 

Ask Questions Get Answers Join the Live Community