Expedition Version 1.1.50 Log Analysis: CSV Still Pending after processing completes successfully

[Krueger]

After upgrading to Expedition 1.1.50 Devices show CSV Pending after log processing completes successfully.  Additionally, Apache Spark URL does not appear on the M. Learning tab Processing status bar when Spark is processing the CSV. Also the grey message bar always displays: The '/PALogs/NTXIRT00-IG0?-H_traffic_*' cannot be accessed. '

Finally, Once log processing completes the Traffic tab displays traffic for the log that was imported however Devices shows the CSV is still pending

.

[Krueger]

More detail for this issue. A fresh Expedition VM was built using New Expedition Installation Procedure.

1. Added Device to Expedition using Active IP & Serial. CSV's processed correctly and shows as processed.
2. Deleted Device in Expedition and Deleted Connections.parquet FILES.
3. Added Device to Expedition using Passive IP & Serial + HA Serial (currently active). CSV's are processed but still show as pending in UI

Looking in the pandbRBAC.device_logs table the logs are listed as processed with the HA Serial. It seems the logic is not working correctly to flag logs sent from the HA Serial as processed.

[dgildelaig]

Could you click on the Summary tab within the device?

If the CSV logs were processed, you should see entries for the dates that are compressed in those CSV files.

That could help us identifying where the issue could be. It is strange that the first time your file was processed without issues, but the second time, having deleted the connections.parquet, it would report as not processed when you actually processed it.

[Krueger]

The difference between the first time and second time is which firewall in the HA pair is active vs configuration in Expedition. If the firewall configured as primary in Expedition generated the logs then everything works as expected. However, if the logs were generated by the device in HA Serial # field, then logs are process but listed as unprocessed.

Logs already processed but still listed as new

[dgildelaig]

Thanks a lot for the clarification.

I will take a look at it with this new understanding of the issue

[Krueger]

An additional confirmation. We failed our HA pair back where Primary IP & Serial match Expedition primary and logs are processing correctly.