Log Processing not working

[ehelm]

Greetings! My instance of Expedition (1.1.43) has stopped processing all logs. The devices detail shows the new logs. I've clicked process logs from there and it appears a job is added but the job includes zero tasks and never completes or fails. I've tried clicking the little green "play" button and get "Error processing log files". /tmp/PAN_LogCoCo.log is empty. Any suggestions of things I can look at or do to kick start this log processing issue? Thanks!

[GNeyrinck]

I have the exact same issue after the upgrade from 1.41 to this version.

I saw some errors in cli while upgrading.

Did you also update Expedition?

Regards

[dgildelaig · Accepted solution]

We are working on an issue related to CSV files processing.

I expect to release an update later today or tomorrow when this issue has been addressed.

[GNeyrinck]

I have now upgraded to the latest version 1.1.45 and it seems that the logs are being processed.

Even though I've only selected one log file to be processed, all pending (20) files seem to be processed in one time.

I'll check if the processing of logs ends ok now.

Regards,

[dgildelaig]

When you say that "only selected one file" are you referring to having the other files set as Ignored?

If so, I will open a bug ticket to check the issue.

Otherwise, remember that the way to state which files are going to be processed is by defining which files are ignored. By default, all files are set to be processed.

[GNeyrinck]

Edit: ok, all files are going to be processed except the ones you set to ignore.

I meant to say that I'm only selecting one file to test if the processing would work again.

Since it's taking that long I'm assuming that all logs are being processed:

root@Expedition:/home/expedition# tail -f /tmp/error_logCoCo

SLF4J: Class path contains multiple SLF4J bindings.

SLF4J: Found binding in [jar:file:/opt/Spark/extraLibraries/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: Found binding in [jar:file:/opt/Spark/spark-2.4.3-bin-hadoop2.7/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]

SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.

SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]

---- CREATING SPARK Session:

          warehouseLocation:/datastore/spark-warehouse

+------------+--------+--------------------+----+------------+

|    fwSerial|panosver|             csvpath|size|afterProcess|

+------------+--------+--------------------+----+------------+

|011901016881|   8.1.0|/PALogs/paloalto_...|4957|    Compress|

|011901016881|   8.1.0|/PALogs/paloalto_...|4332|    Compress|

|011901016881|   8.1.0|/PALogs/paloalto_...|4629|    Compress|

|011901016881|   8.1.0|/PALogs/paloalto_...|3861|    Compress|

|011901016881|   8.1.0|/PALogs/paloalto_...|4629|    Compress|

|011901016881|   8.1.0|/PALogs/paloalto_...|3840|    Compress|

|011901016881|   8.1.0|/PALogs/paloalto_...|5264|    Compress|

|011901016881|   8.1.0|/PALogs/paloalto_...|3892|    Compress|

|011901016881|   8.1.0|/PALogs/paloalto_...|3646|    Compress|

|011901016881|   8.1.0|/PALogs/paloalto_...|4783|    Compress|

|011901016881|   8.1.0|/PALogs/paloalto_...|4772|    Compress|

|011901016881|   8.1.0|/PALogs/paloalto_...|4619|    Compress|

|011901016881|   8.1.0|/PALogs/paloalto_...|5161|    Compress|

|011901016881|   8.1.0|/PALogs/paloalto_...|4752|    Compress|

|011901016881|   8.1.0|/PALogs/paloalto_...|4076|    Compress|

|011901016881|   8.1.0|/PALogs/paloalto_...|4895|    Compress|

|011901016881|   8.1.0|/PALogs/paloalto_...|3789|    Compress|

|011901016881|   8.1.0|/PALogs/paloalto_...|4742|    Compress|

|011901016881|   8.1.0|/PALogs/paloalto_...|5059|    Compress|

|011901016881|   8.1.0|/PALogs/paloalto_...|5407|    Compress|

+------------+--------+--------------------+----+------------+

Memory: 6373m

LogCollector&Compacter called with the following parameters:

     Parameters for execution

          Master[processes]:............  local[3]

          Available RAM (MB):...........  6525952

          User:.........................  admin

          debug:........................  false

     Parameters for Job Connections

          Task ID:......................  152

          My IP:........................  10.3.1.30

          Expedition IP:................  10.3.1.30:3306

          Time Zone:....................  Europe/Helsinki

          dbUser (dbPassword):..........  root (************)

          projectName:..................  demo

     Parameters for Data Sources

          App Categories (source):........ (Expedition)

          CSV Files Path:................./tmp/1572951103_traffic_files.csv

          Parquet output path:..........  file:///PALogs/connections.parquet

          Temporary folder:.............  /datastore

---- AppID DB LOAD:

          Application Categories loading...

     Application Categories loaded

+------------+--------+--------------------+----+------------+--------+---+---------------+

|    fwSerial|panosver|             csvpath|size|afterProcess| grouped|row|accumulatedSize|

+------------+--------+--------------------+----+------------+--------+---+---------------+

|011901016881|   8.1.0|/PALogs/paloalto_...|4957|    Compress|grouping|  1|         4957.0|

|011901016881|   8.1.0|/PALogs/paloalto_...|4332|    Compress|grouping|  2|         9289.0|

|011901016881|   8.1.0|/PALogs/paloalto_...|4629|    Compress|grouping|  3|        13918.0|

|011901016881|   8.1.0|/PALogs/paloalto_...|3861|    Compress|grouping|  4|        17779.0|

|011901016881|   8.1.0|/PALogs/paloalto_...|4629|    Compress|grouping|  5|        22408.0|

|011901016881|   8.1.0|/PALogs/paloalto_...|3840|    Compress|grouping|  6|        26248.0|

|011901016881|   8.1.0|/PALogs/paloalto_...|5264|    Compress|grouping|  7|        31512.0|

|011901016881|   8.1.0|/PALogs/paloalto_...|3892|    Compress|grouping|  8|        35404.0|

|011901016881|   8.1.0|/PALogs/paloalto_...|3646|    Compress|grouping|  9|        39050.0|

|011901016881|   8.1.0|/PALogs/paloalto_...|4783|    Compress|grouping| 10|        43833.0|

|011901016881|   8.1.0|/PALogs/paloalto_...|4772|    Compress|grouping| 11|        48605.0|

|011901016881|   8.1.0|/PALogs/paloalto_...|4619|    Compress|grouping| 12|        53224.0|

|011901016881|   8.1.0|/PALogs/paloalto_...|5161|    Compress|grouping| 13|        58385.0|

|011901016881|   8.1.0|/PALogs/paloalto_...|4752|    Compress|grouping| 14|        63137.0|

|011901016881|   8.1.0|/PALogs/paloalto_...|4076|    Compress|grouping| 15|        67213.0|

|011901016881|   8.1.0|/PALogs/paloalto_...|4895|    Compress|grouping| 16|        72108.0|

|011901016881|   8.1.0|/PALogs/paloalto_...|3789|    Compress|grouping| 17|        75897.0|

|011901016881|   8.1.0|/PALogs/paloalto_...|4742|    Compress|grouping| 18|        80639.0|

|011901016881|   8.1.0|/PALogs/paloalto_...|5059|    Compress|grouping| 19|        85698.0|

|011901016881|   8.1.0|/PALogs/paloalto_...|5407|    Compress|grouping| 20|        91105.0|

+------------+--------+--------------------+----+------------+--------+---+---------------+

  Selection criteria:   0 < accumulatedSize and accumulatedSize <= 6525952

     Processing from lowLimit:0 to highLimit:6525952 with StepLine:6525952

     Few logs can fit in this batch:20

8.1.0:/PALogs/paloalto_1_traffic_2019_10_25_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_11_01_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_11_02_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_31_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_19_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_17_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_11_03_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_28_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_11_05_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_27_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_30_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_22_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_11_04_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_18_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_26_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_21_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_29_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_20_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_23_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_24_last_calendar_day.csv

Logs of format 7.1.x NOT found

Logs of format 8.0.2 NOT found

Logs of format 8.1.0-beta17 NOT found

Logs of format 8.1.0 found

Logs of format 9.0.0 NOT found

Logs of format 9.1.0-beta NOT found

[GNeyrinck]

Another thing which isn't ok is the machine learning auto processing 

It should be hard coded, but it's showing as empty in the gui

[GNeyrinck]

I meant to say that I've only select one file.

But you explained that all will be processed except the ones that you set to ignore.

[GNeyrinck]

All files have been processed now.

The only thing I need to verify now is the automatic processing of the logs.

[dgildelaig]

I am modifying the interface to avoid this confusion. You are not the first one, neither the second or the tenth that though only the check-boxed would be processed.

[ehelm]

Unfortunately I'm not seeing the logs processed. I do not get an error now, and it appears that logs process as there are several java processes that spin up when I click the "process files" button. The jobs show failed but no clue as to why. Also a link to the error_logCoCo is included below. I haven't seen anything in there that indicates where the problem is. Any thoughts? Thanks! 

error_logCoCo 

[ehelm]

UPDATE: I noticed a low disk space error in the log. Took care of that (Moved location of DATA ANALYSIS STRUCTURES FOLDERs). Re-processed the files and things worked as expected. 

[GNeyrinck]

I encountered this error also, due to the longer time the logs weren't processed the foldersize got bigger.

I've added some additional space and relaunched the processing and all went ok.

I'm still seeing the (reduced) logfiles even if you select the "delete logs after processing"