ML gets stuck at "Pending"

[mbowling]

I started by running the command

scp export log traffic start-time equal 2018/07/30@00:00:00 end-time equal 2018/07/30@23:45:00 to expedition@172.30.200.117:/PALogs/mltest.csv

on my PA220. 

root@Expedition:/PALogs# ls -l
total 64296
-rw-rw-r-- 1 expedition expedition 65830760 Aug 1 17:35 mltest.csv
drwxr-xr-x 2 www-data www-data 4096 Aug 1 17:45 sparkLocalDir
drwxr-xr-x 2 www-data www-data 4096 Aug 1 17:37 spark-warehouse
root@Expedition:/PALogs#

What am I missing here?

[BenLassila]

I had this problem too on Expedition 1.0.7.

Similar to the solution below I hard-coded the real (non-localhost) address to the bind-address line of /etc/mysql/my.cnf.

Don't forget sudo service mysql restart.

[ederg]

For me its not working, but I get the pendig at the Dashboard.

Its not working with remarked : 127.0.01 nor with the real Host IP.

I get in the /tmp/error_logCoCo log:

(/opt/Spark/spark/bin/spark-submit --class com.paloaltonetworks.tbd.LogCollectorCompacter --deploy-mode client --supervise /var/www/html/OS/spark/packages/LogCoCo-1.3.0-SNAPSHOT.jar MLServer='172.30.104.33', master='local[3]', debug='false', taskID='8', user='admin', dbUser='root', dbPass='paloalto', dbServer='172.30.104.33:3306', timeZone='Europe/Helsinki', mode='Expedition', input=0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_24_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_26_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_12_03_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_27_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_28_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_29_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_23_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_22_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_25_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_21_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_12_02_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_30_last_calendar_day.csv, output='/home/www-data/connections.parquet', tempFolder='/home/www-data'; echo /data/SnifferPalo_traffic_2018_11_24_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_11_26_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_12_03_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_11_27_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_11_28_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_11_29_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_11_23_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_11_22_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_11_25_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_11_21_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_12_02_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_11_30_last_calendar_day.csv; )>> "/tmp/error_logCoCo" 2>>/tmp/error_logCoCo &
 ---- CREATING SPARK Session:
          warehouseLocation:/tmp/spark-warehouse
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/Spark/extraLibraries/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/Spark/spark-2.1.1-bin-hadoop2.7/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]
+--------------------+-----------+--------+--------------------+
|             rowLine|   fwSerial|panosver|             csvpath|
+--------------------+-----------+--------+--------------------+
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
+--------------------+-----------+--------+--------------------+

8.1.0:/data/SnifferPalo_traffic_2018_11_29_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_12_02_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_11_24_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_11_23_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_11_26_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_11_21_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_11_28_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_12_03_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_11_27_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_11_30_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_11_22_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_11_25_last_calendar_day.csv
LogCollector&Compacter called with the following parameters:
     Parameters for execution
          Master[processes]:............  local[3]
          User:.........................  admin
          debug:........................  false
     Parameters for Job Connections
          Task ID:......................  8
          My IP:........................  172.30.104.33
          Expedition IP:................  172.30.104.33:3306
          Time Zone:....................  Europe/Helsinki
          dbUser (dbPassword):..........  root (************)
          projectName:..................  demo
     Parameters for Data Sources
          App Categories (source):........ (Expedition)
          CSV Files Path:.................0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_24_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_26_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_12_03_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_27_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_28_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_29_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_23_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_22_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_25_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_21_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_12_02_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_30_last_calendar_day.csv
          Parquet output path:..........  file:///home/www-data/connections.parquet
          Temporary folder:.............  /home/www-data
 ---- AppID DB LOAD:
          Application Categories loading...
     DONE

Logs of format 7.1.x NOT found
Logs of format 8.0.2 NOT found
Logs of format 8.1.0-beta17 NOT found
Logs of format 8.1.0 found
Logs of format 8.1.0-beta17 NOT found
Size of trafficExtended: 50 MB
[Stage 54:>                                                       (0 + 3) / 246]Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 54.0 failed 1 times, most recent failure: Lost task 1.0 in stage 54.0 (TID 1148, localhost, executor driver): org.apache.spark.SparkException: Failed to execute user defined function(anonfun$18: (string) => bigint)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.agg_doAggregateWithKeys$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
        at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:126)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
        at org.apache.spark.scheduler.Task.run(Task.scala:99)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string: "fe80::340b:b764:f9ad:a59a"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Integer.parseInt(Integer.java:580)
        at java.lang.Integer.parseInt(Integer.java:615)
        at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:272)
        at scala.collection.immutable.StringOps.toInt(StringOps.scala:29)
        at com.paloaltonetworks.tbd.LogCollectorCompacter$.com$paloaltonetworks$tbd$LogCollectorCompacter$$IPv4ToLong$1(LogCollectorCompacter.scala:279)
        at com.paloaltonetworks.tbd.LogCollectorCompacter$$anonfun$18.apply(LogCollectorCompacter.scala:1002)
        at com.paloaltonetworks.tbd.LogCollectorCompacter$$anonfun$18.apply(LogCollectorCompacter.scala:1002)
        ... 13 more

Driver stacktrace:
        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
        at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1925)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1938)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1951)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1965)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:936)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
        at org.apache.spark.rdd.RDD.collect(RDD.scala:935)
        at com.paloaltonetworks.tbd.LogCollectorCompacter$.main(LogCollectorCompacter.scala:1156)
        at com.paloaltonetworks.tbd.LogCollectorCompacter.main(LogCollectorCompacter.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.spark.SparkException: Failed to execute user defined function(anonfun$18: (string) => bigint)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.agg_doAggregateWithKeys$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
        at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:126)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
        at org.apache.spark.scheduler.Task.run(Task.scala:99)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string: "fe80::340b:b764:f9ad:a59a"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Integer.parseInt(Integer.java:580)
        at java.lang.Integer.parseInt(Integer.java:615)
        at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:272)
        at scala.collection.immutable.StringOps.toInt(StringOps.scala:29)
        at com.paloaltonetworks.tbd.LogCollectorCompacter$.com$paloaltonetworks$tbd$LogCollectorCompacter$$IPv4ToLong$1(LogCollectorCompacter.scala:279)
        at com.paloaltonetworks.tbd.LogCollectorCompacter$$anonfun$18.apply(LogCollectorCompacter.scala:1002)
        at com.paloaltonetworks.tbd.LogCollectorCompacter$$anonfun$18.apply(LogCollectorCompacter.scala:1002)
        ... 13 more
/data/SnifferPalo_traffic_2018_11_24_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_11_26_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_12_03_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_11_27_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_11_28_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_11_29_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_11_23_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_11_22_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_11_25_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_11_21_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_12_02_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_11_30_last_calendar_day.csv

Any Ideas?

Gernot

[ederg]

For me its not working, but I get the pendig at the Dashboard.

Its not working with remarked : loopback-IP nor with the Host IP

I get in the /tmp/error_logCoCo log:

(/opt/Spark/spark/bin/spark-submit --class com.paloaltonetworks.tbd.LogCollectorCompacter --deploy-mode client --supervise /var/www/html/OS/spark/packages/LogCoCo-1.3.0-SNAPSHOT.jar MLServer='172.30.104.33', master='local[3]', debug='false', taskID='8', user='admin', dbUser='root', dbPass='paloalto', dbServer='172.30.104.33:3306', timeZone='Europe/Helsinki', mode='Expedition', input=0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_24_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_26_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_12_03_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_27_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_28_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_29_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_23_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_22_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_25_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_21_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_12_02_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_30_last_calendar_day.csv, output='/home/www-data/connections.parquet', tempFolder='/home/www-data'; echo /data/SnifferPalo_traffic_2018_11_24_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_11_26_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_12_03_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_11_27_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_11_28_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_11_29_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_11_23_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_11_22_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_11_25_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_11_21_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_12_02_last_calendar_day.csv;  echo /data/SnifferPalo_traffic_2018_11_30_last_calendar_day.csv; )>> "/tmp/error_logCoCo" 2>>/tmp/error_logCoCo &
 ---- CREATING SPARK Session:
          warehouseLocation:/tmp/spark-warehouse
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/Spark/extraLibraries/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/Spark/spark-2.1.1-bin-hadoop2.7/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]
+--------------------+-----------+--------+--------------------+
|             rowLine|   fwSerial|panosver|             csvpath|
+--------------------+-----------+--------+--------------------+
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
|0009C100289:8.1.0...|0009C100289|   8.1.0|/data/SnifferPalo...|
+--------------------+-----------+--------+--------------------+

8.1.0:/data/SnifferPalo_traffic_2018_11_29_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_12_02_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_11_24_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_11_23_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_11_26_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_11_21_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_11_28_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_12_03_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_11_27_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_11_30_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_11_22_last_calendar_day.csv,/data/SnifferPalo_traffic_2018_11_25_last_calendar_day.csv
LogCollector&Compacter called with the following parameters:
     Parameters for execution
          Master[processes]:............  local[3]
          User:.........................  admin
          debug:........................  false
     Parameters for Job Connections
          Task ID:......................  8
          My IP:........................  172.30.104.33
          Expedition IP:................  172.30.104.33:3306
          Time Zone:....................  Europe/Helsinki
          dbUser (dbPassword):..........  root (************)
          projectName:..................  demo
     Parameters for Data Sources
          App Categories (source):........ (Expedition)
          CSV Files Path:.................0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_24_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_26_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_12_03_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_27_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_28_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_29_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_23_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_22_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_25_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_21_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_12_02_last_calendar_day.csv,0009C100289:8.1.0:/data/SnifferPalo_traffic_2018_11_30_last_calendar_day.csv
          Parquet output path:..........  file:///home/www-data/connections.parquet
          Temporary folder:.............  /home/www-data
 ---- AppID DB LOAD:
          Application Categories loading...
     DONE

Logs of format 7.1.x NOT found
Logs of format 8.0.2 NOT found
Logs of format 8.1.0-beta17 NOT found
Logs of format 8.1.0 found
Logs of format 8.1.0-beta17 NOT found
Size of trafficExtended: 50 MB
[Stage 54:>                                                       (0 + 3) / 246]Exception in thread "main" org.apache.spark.SparkException: Job aborted due to stage failure: Task 1 in stage 54.0 failed 1 times, most recent failure: Lost task 1.0 in stage 54.0 (TID 1148, localhost, executor driver): org.apache.spark.SparkException: Failed to execute user defined function(anonfun$18: (string) => bigint)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.agg_doAggregateWithKeys$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
        at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:126)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
        at org.apache.spark.scheduler.Task.run(Task.scala:99)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string: "fe80::340b:b764:f9ad:a59a"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Integer.parseInt(Integer.java:580)
        at java.lang.Integer.parseInt(Integer.java:615)
        at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:272)
        at scala.collection.immutable.StringOps.toInt(StringOps.scala:29)
        at com.paloaltonetworks.tbd.LogCollectorCompacter$.com$paloaltonetworks$tbd$LogCollectorCompacter$$IPv4ToLong$1(LogCollectorCompacter.scala:279)
        at com.paloaltonetworks.tbd.LogCollectorCompacter$$anonfun$18.apply(LogCollectorCompacter.scala:1002)
        at com.paloaltonetworks.tbd.LogCollectorCompacter$$anonfun$18.apply(LogCollectorCompacter.scala:1002)
        ... 13 more

Driver stacktrace:
        at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1435)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1423)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1422)
        at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
        at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
        at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1422)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
        at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:802)
        at scala.Option.foreach(Option.scala:257)
        at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:802)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:1650)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1605)
        at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1594)
        at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)
        at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:628)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1925)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1938)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1951)
        at org.apache.spark.SparkContext.runJob(SparkContext.scala:1965)
        at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:936)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
        at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
        at org.apache.spark.rdd.RDD.withScope(RDD.scala:362)
        at org.apache.spark.rdd.RDD.collect(RDD.scala:935)
        at com.paloaltonetworks.tbd.LogCollectorCompacter$.main(LogCollectorCompacter.scala:1156)
        at com.paloaltonetworks.tbd.LogCollectorCompacter.main(LogCollectorCompacter.scala)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:743)
        at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:187)
        at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:212)
        at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:126)
        at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: org.apache.spark.SparkException: Failed to execute user defined function(anonfun$18: (string) => bigint)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.agg_doAggregateWithKeys$(Unknown Source)
        at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown Source)
        at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
        at org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
        at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
        at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:126)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:96)
        at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:53)
        at org.apache.spark.scheduler.Task.run(Task.scala:99)
        at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NumberFormatException: For input string: "fe80::340b:b764:f9ad:a59a"
        at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
        at java.lang.Integer.parseInt(Integer.java:580)
        at java.lang.Integer.parseInt(Integer.java:615)
        at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:272)
        at scala.collection.immutable.StringOps.toInt(StringOps.scala:29)
        at com.paloaltonetworks.tbd.LogCollectorCompacter$.com$paloaltonetworks$tbd$LogCollectorCompacter$$IPv4ToLong$1(LogCollectorCompacter.scala:279)
        at com.paloaltonetworks.tbd.LogCollectorCompacter$$anonfun$18.apply(LogCollectorCompacter.scala:1002)
        at com.paloaltonetworks.tbd.LogCollectorCompacter$$anonfun$18.apply(LogCollectorCompacter.scala:1002)
        ... 13 more
/data/SnifferPalo_traffic_2018_11_24_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_11_26_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_12_03_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_11_27_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_11_28_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_11_29_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_11_23_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_11_22_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_11_25_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_11_21_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_12_02_last_calendar_day.csv
/data/SnifferPalo_traffic_2018_11_30_last_calendar_day.csv

Any Ideas?

Gernot

[dgildelaig]

Thanks for the report.

The issue you are finding is that the current version of the ML package is not supporting IPv6.

We are working on the next version of the package to support IPv6 for loading and Rule Enrichment and Rule Learning.

[ederg]

Ah IPv6 is the Issue?

I will try to modify the Logs and remove the IPv6 Logging entries.

Many thanks for that very quick information.

Kind Regards

Gernot

[GNeyrinck]

This worked for me too!

All files being processed within a minute now.

[MartinMalek]

@dgildelaig 

Hello @dgildelaig,

i have the same issue with my Expedition. I did everything according the official guide and for some reason the ML is stucked on Pending. Could you please help me out, it really urgent.

[dgildelaig]

You are within a Panorama device. To be able to process the files, you need to specify it from the FW devices.

Notice that you can set up the auto-process so you do not need to get into the devices in the future for processing CSV log files

[MartinMalek]

I've setup LOG export on each FW separately.  I'm receiving the LOGs from the FW - that's not a problem. My problem with data processing.  The option "process" is not available. I've tried everything, even check the auto-process option , without any luck.

Any Idea?

[dgildelaig]

The process button is not available because you are trying to click the button within a Panorama device.

You can only click that button if you ara within a FW device in Expedition