Expedition Version 1.1.50 Log Analysis: CSV Still Pending after processing completes successfully

Reply
L1 Bithead

Expedition Version 1.1.50 Log Analysis: CSV Still Pending after processing completes successfully

After upgrading to Expedition 1.1.50 Devices show CSV Pending after log processing completes successfully.  Additionally, Apache Spark URL does not appear on the M. Learning tab Processing status bar when Spark is processing the CSV. Also the grey message bar always displays: The '/PALogs/NTXIRT00-IG0?-H_traffic_*' cannot be accessed. '

 

Finally, Once log processing completes the Traffic tab displays traffic for the log that was imported however Devices shows the CSV is still pending

 

PAN1.png.

 

PAN2.png

L1 Bithead

Re: Expedition Version 1.1.50 Log Analysis: CSV Still Pending after processing completes successfull

More detail for this issue. A fresh Expedition VM was built using New Expedition Installation Procedure.

 

  1. Added Device to Expedition using Active IP & Serial. CSV's processed correctly and shows as processed.
  2. Deleted Device in Expedition and Deleted Connections.parquet FILES.
  3. Added Device to Expedition using Passive IP & Serial + HA Serial (currently active). CSV's are processed but still show as pending in UI

Looking in the pandbRBAC.device_logs table the logs are listed as processed with the HA Serial. It seems the logic is not working correctly to flag logs sent from the HA Serial as processed.

L4 Transporter

Re: Expedition Version 1.1.50 Log Analysis: CSV Still Pending after processing completes successfull

Could you click on the Summary tab within the device?

If the CSV logs were processed, you should see entries for the dates that are compressed in those CSV files.

 

That could help us identifying where the issue could be. It is strange that the first time your file was processed without issues, but the second time, having deleted the connections.parquet, it would report as not processed when you actually processed it.

L1 Bithead

Re: Expedition Version 1.1.50 Log Analysis: CSV Still Pending after processing completes successfull

The difference between the first time and second time is which firewall in the HA pair is active vs configuration in Expedition. If the firewall configured as primary in Expedition generated the logs then everything works as expected. However, if the logs were generated by the device in HA Serial # field, then logs are process but listed as unprocessed.

 

2019-12-09_10-19-44.pngLogs already processed but still listed as new

L4 Transporter

Re: Expedition Version 1.1.50 Log Analysis: CSV Still Pending after processing completes successfull

Thanks a lot for the clarification.

I will take a look at it with this new understanding of the issue

L1 Bithead

Re: Expedition Version 1.1.50 Log Analysis: CSV Still Pending after processing completes successfull

An additional confirmation. We failed our HA pair back where Primary IP & Serial match Expedition primary and logs are processing correctly.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!