cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 

Expedition csv logs stuck in pending

cancel
Turn on suggestions
Auto-suggest helps you quickly narrow down your search results by suggesting possible matches as you type.
Showing results for 
Search instead for 
Did you mean: 
Reply
Highlighted
simontaylor
L1 Bithead
‎09-08-2019 10:34 PM
‎09-08-2019 10:34 PM

spent a long time in "No supported files to process" until I read this post. I added Panorama as a device and retrieved all the managed devivces and even though the tool recognises the log format and the source shows the name of the firewalll I want, I had to delete the panorama entry and add the firewall instead under devices->imported devices. Shame it doesnt just work the other way around.

0 Likes
Reply
Highlighted
dgildelaig
L5 Sessionator
‎09-09-2019 01:55 AM
‎09-09-2019 01:55 AM

You should still be able to process the logs from within the devices that were connected in a Panorama.

I will take a note so we disable the button within the Panorama and notify in how to process the logs.

0 Likes
Reply
Highlighted
jrtuck
L2 Linker
‎01-29-2020 05:58 AM
‎01-29-2020 05:58 AM

Having the same problem and cannot figure why Expedition will not process the CSV files.  www-data is in the expedition group and has the permission to access the CSV files.  Please help!

 

jrtuck_0-1580306211183.pngjrtuck_1-1580306228685.png

 

 

ubuntu@ip-10-170-1-35:/$ tail -f /tmp/error_logCoCo
at org.apache.spark.util.Utils$.localCanonicalHostName(Utils.scala:996)
at org.apache.spark.internal.config.package$.<init>(package.scala:302)
at org.apache.spark.internal.config.package$.<clinit>(package.scala)
... 17 more
Caused by: java.net.UnknownHostException: ip-10-170-1-35: Name or service not known
at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
at java.net.InetAddress$2.lookupAllHostAddr(InetAddress.java:929)
at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1324)
at java.net.InetAddress.getLocalHost(InetAddress.java:1501)
... 26 more

0 Likes
Reply
Highlighted
dgildelaig
L5 Sessionator
‎01-29-2020 06:36 AM
‎01-29-2020 06:36 AM

I have the feeling that in your case, the issue is related to the name resolution of your Expedition instance.

Check that the /etc/hostname is valid; and also check, via your Expedition web browser, that the ML Settings are correct pointing to your Expedition ip address.

0 Likes
Reply
Highlighted
jrtuck
L2 Linker
‎01-29-2020 12:13 PM
‎01-29-2020 12:13 PM

/etc/hostname shows ip-10-170-1-35

jrtuck_3-1580328514699.png

etc/hosts - changed from "127.0.0.1 localhost" to :127.0.0.1 ip-10-170-1-35 localhost" 

 

jrtuck_4-1580328611088.png

 

Now I get a weird message stating http://10.170.1.135:4050

 

jrtuck_2-1580328487483.png

Then the process fails again.

jrtuck_5-1580328716813.png

My ML settings seem to look fine.  

jrtuck_6-1580328797331.png

Any idea what else is causing this?  

 

 

 

0 Likes
Reply
Highlighted
dgildelaig
L5 Sessionator
‎01-30-2020 12:18 AM
‎01-30-2020 12:18 AM

That URL that you called "weird message" is something expected and desired, as provides the link to see how the processing is going under Spark computation.

The fact that failed is not desired, tho. Check if you have files in /tmp with a name starting with error_, for instance /tmp/error_logCoCo, which would provide information regarding the execution of the CSV processing into parquet (internal format for later log analysis) and errors that may have occurred.

 

 

0 Likes
Reply
Highlighted
jrtuck
L2 Linker
‎01-30-2020 04:05 AM
‎01-30-2020 04:05 AM

That is good the weird message is expected behavior!  I do have the /tmp/error_logCoCo and the contents of that file is below.  In addition, we are running PAN-OS 9.04 on the firewalls, so I am not sure if Expedition can read the CSV files from that version of code.  However, I am running the latest version of Expedition.  

 

jrtuck_0-1580385870734.png

 

 

ubuntu@ip-10-170-1-35:/tmp$ more error_logCoCo
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/Spark/extraLibraries/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/Spark/spark-2.4.3-bin-hadoop2.7/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]
---- CREATING SPARK Session:
warehouseLocation:/data/spark-warehouse
+------------+--------+--------------------+----+------------+
| fwSerial|panosver| csvpath|size|afterProcess|
+------------+--------+--------------------+----+------------+
|012001008166| 9.0.0|/home/expedition/...| 972| null|
+------------+--------+--------------------+----+------------+

Memory: 5838m
LogCollector&Compacter called with the following parameters:
Parameters for execution
Master[processes]:............ local[3]
Available RAM (MB):........... 5978112
User:......................... admin
debug:........................ false
Parameters for Job Connections
Task ID:...................... 2152
My IP:........................ 10.170.1.35
Expedition IP:................ 10.170.1.35:3306
Time Zone:.................... Europe/Helsinki
dbUser (dbPassword):.......... root (************)
projectName:.................. demo
Parameters for Data Sources
App Categories (source):........ (Expedition)
CSV Files Path:................./tmp/1580321790_traffic_files.csv
Parquet output path:.......... file:///data/connections.parquet
Temporary folder:............. /data
---- AppID DB LOAD:
Application Categories loading...
Application Categories loaded

+------------+--------+--------------------+----+------------+--------+---+---------------+
| fwSerial|panosver| csvpath|size|afterProcess| grouped|row|accumulatedSize|
+------------+--------+--------------------+----+------------+--------+---+---------------+
|012001008166| 9.0.0|/home/expedition/...| 972| null|grouping| 1| 972.0|
+------------+--------+--------------------+----+------------+--------+---+---------------+

Selection criteria: 0 < accumulatedSize and accumulatedSize <= 5978112
Processing from lowLimit:0 to highLimit:5978112 with StepLine:5978112
Few logs can fit in this batch:1
9.0.0:/home/expedition/logs/ivc-ind-mdf1-fw1_traffic_2020_01_29_last_calendar_day.csv
Logs of format 7.1.x NOT found
Logs of format 8.0.2 NOT found
Logs of format 8.1.0-beta17 NOT found
Logs of format 8.1.0 NOT found
Logs of format 9.0.0 found
Logs of format 9.1.0-beta NOT found
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/Spark/extraLibraries/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/Spark/spark-2.4.3-bin-hadoop2.7/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]
---- CREATING SPARK Session:
warehouseLocation:/data/spark-warehouse
One loop done
+--------+--------+-------+----+------------+
|fwSerial|panosver|csvpath|size|afterProcess|
+--------+--------+-------+----+------------+
+--------+--------+-------+----+------------+

Memory: 5838m
LogCollector&Compacter called with the following parameters:
Parameters for execution
Master[processes]:............ local[3]
Available RAM (MB):........... 5978112
User:......................... admin
debug:........................ false
Parameters for Job Connections
Task ID:...................... 2153
My IP:........................ 10.170.1.35
Expedition IP:................ 10.170.1.35:3306
Time Zone:.................... Europe/Helsinki
dbUser (dbPassword):.......... root (************)
projectName:.................. demo
Parameters for Data Sources
App Categories (source):........ (Expedition)
CSV Files Path:................./tmp/1580321882_traffic_files.csv
Parquet output path:.......... file:///data/connections.parquet
Temporary folder:............. /data
---- AppID DB LOAD:
Application Categories loading...
Application Categories loaded

Exception in thread "main" java.util.NoSuchElementException: next on empty iterator
at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
at scala.collection.Iterator$$anon$2.next(Iterator.scala:37)
at scala.collection.IndexedSeqLike$Elements.next(IndexedSeqLike.scala:63)
at scala.collection.IterableLike$class.head(IterableLike.scala:107)
at scala.collection.mutable.ArrayOps$ofInt.scala$collection$IndexedSeqOptimized$$super$head(ArrayOps.scala:234)
at scala.collection.IndexedSeqOptimized$class.head(IndexedSeqOptimized.scala:126)
at scala.collection.mutable.ArrayOps$ofInt.head(ArrayOps.scala:234)
at com.paloaltonetworks.tbd.LogCollectorCompacter$.main(LogCollectorCompacter.scala:441)
at com.paloaltonetworks.tbd.LogCollectorCompacter.main(LogCollectorCompacter.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/Spark/extraLibraries/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/Spark/spark-2.4.3-bin-hadoop2.7/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]
---- CREATING SPARK Session:
warehouseLocation:/data/spark-warehouse
+--------+--------+-------+----+------------+
|fwSerial|panosver|csvpath|size|afterProcess|
+--------+--------+-------+----+------------+
+--------+--------+-------+----+------------+

Memory: 5838m
LogCollector&Compacter called with the following parameters:
Parameters for execution
Master[processes]:............ local[3]
Available RAM (MB):........... 5978112
User:......................... admin
debug:........................ false
Parameters for Job Connections
Task ID:...................... 2154
My IP:........................ 10.170.1.35
Expedition IP:................ 10.170.1.35:3306
Time Zone:.................... Europe/Helsinki
dbUser (dbPassword):.......... root (************)
projectName:.................. demo
Parameters for Data Sources
App Categories (source):........ (Expedition)
CSV Files Path:................./tmp/1580322005_traffic_files.csv
Parquet output path:.......... file:///data/connections.parquet
Temporary folder:............. /data
---- AppID DB LOAD:
Application Categories loading...
Application Categories loaded

Exception in thread "main" java.util.NoSuchElementException: next on empty iterator
at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
at scala.collection.Iterator$$anon$2.next(Iterator.scala:37)
at scala.collection.IndexedSeqLike$Elements.next(IndexedSeqLike.scala:63)
at scala.collection.IterableLike$class.head(IterableLike.scala:107)
at scala.collection.mutable.ArrayOps$ofInt.scala$collection$IndexedSeqOptimized$$super$head(ArrayOps.scala:234)
at scala.collection.IndexedSeqOptimized$class.head(IndexedSeqOptimized.scala:126)
at scala.collection.mutable.ArrayOps$ofInt.head(ArrayOps.scala:234)
at com.paloaltonetworks.tbd.LogCollectorCompacter$.main(LogCollectorCompacter.scala:441)
at com.paloaltonetworks.tbd.LogCollectorCompacter.main(LogCollectorCompacter.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/Spark/extraLibraries/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/Spark/spark-2.4.3-bin-hadoop2.7/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]
---- CREATING SPARK Session:
warehouseLocation:/data/spark-warehouse
+--------+--------+-------+----+------------+
|fwSerial|panosver|csvpath|size|afterProcess|
+--------+--------+-------+----+------------+
+--------+--------+-------+----+------------+

Memory: 5838m
LogCollector&Compacter called with the following parameters:
Parameters for execution
Master[processes]:............ local[3]
Available RAM (MB):........... 5978112
User:......................... admin
debug:........................ false
Parameters for Job Connections
Task ID:...................... 2157
My IP:........................ 10.170.1.35
Expedition IP:................ 10.170.1.35:3306
Time Zone:.................... Europe/Helsinki
dbUser (dbPassword):.......... root (************)
projectName:.................. demo
Parameters for Data Sources
App Categories (source):........ (Expedition)
CSV Files Path:................./tmp/1580328172_traffic_files.csv
Parquet output path:.......... file:///data/connections.parquet
Temporary folder:............. /data
---- AppID DB LOAD:
Application Categories loading...
Application Categories loaded

Exception in thread "main" java.util.NoSuchElementException: next on empty iterator
at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
at scala.collection.Iterator$$anon$2.next(Iterator.scala:37)
at scala.collection.IndexedSeqLike$Elements.next(IndexedSeqLike.scala:63)
at scala.collection.IterableLike$class.head(IterableLike.scala:107)
at scala.collection.mutable.ArrayOps$ofInt.scala$collection$IndexedSeqOptimized$$super$head(ArrayOps.scala:234)
at scala.collection.IndexedSeqOptimized$class.head(IndexedSeqOptimized.scala:126)
at scala.collection.mutable.ArrayOps$ofInt.head(ArrayOps.scala:234)
at com.paloaltonetworks.tbd.LogCollectorCompacter$.main(LogCollectorCompacter.scala:441)
at com.paloaltonetworks.tbd.LogCollectorCompacter.main(LogCollectorCompacter.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/Spark/extraLibraries/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/Spark/spark-2.4.3-bin-hadoop2.7/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory]
---- CREATING SPARK Session:
warehouseLocation:/data/spark-warehouse
+--------+--------+-------+----+------------+
|fwSerial|panosver|csvpath|size|afterProcess|
+--------+--------+-------+----+------------+
+--------+--------+-------+----+------------+

Memory: 5838m
LogCollector&Compacter called with the following parameters:
Parameters for execution
Master[processes]:............ local[3]
Available RAM (MB):........... 5978112
User:......................... admin
debug:........................ false
Parameters for Job Connections
Task ID:...................... 2158
My IP:........................ 10.170.1.35
Expedition IP:................ 10.170.1.35:3306
Time Zone:.................... Europe/Helsinki
dbUser (dbPassword):.......... root (************)
projectName:.................. demo
Parameters for Data Sources
App Categories (source):........ (Expedition)
CSV Files Path:................./tmp/1580328212_traffic_files.csv
Parquet output path:.......... file:///data/connections.parquet
Temporary folder:............. /data
---- AppID DB LOAD:
Application Categories loading...
Application Categories loaded

Exception in thread "main" java.util.NoSuchElementException: next on empty iterator
at scala.collection.Iterator$$anon$2.next(Iterator.scala:39)
at scala.collection.Iterator$$anon$2.next(Iterator.scala:37)
at scala.collection.IndexedSeqLike$Elements.next(IndexedSeqLike.scala:63)
at scala.collection.IterableLike$class.head(IterableLike.scala:107)
at scala.collection.mutable.ArrayOps$ofInt.scala$collection$IndexedSeqOptimized$$super$head(ArrayOps.scala:234)
at scala.collection.IndexedSeqOptimized$class.head(IndexedSeqOptimized.scala:126)
at scala.collection.mutable.ArrayOps$ofInt.head(ArrayOps.scala:234)
at com.paloaltonetworks.tbd.LogCollectorCompacter$.main(LogCollectorCompacter.scala:441)
at com.paloaltonetworks.tbd.LogCollectorCompacter.main(LogCollectorCompacter.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:849)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:195)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
ubuntu@ip-10-170-1-35:/tmp$

  

0 Likes
Reply
Highlighted
dgildelaig
L5 Sessionator
‎01-30-2020 05:48 AM
‎01-30-2020 05:48 AM

Which version of Expedition are you running? 1.1.53?

 

For some reason, Expedition is trying to process "no files". For instance, you will see that the /tmp/1580328212_traffic_files.csv file may be empty.

This was an issue we fixed back in 1.1.50 (if I remember correctly), so I just want to confirm that you are running 1.1.53 and it is then a new bug.

 

0 Likes
Reply
Highlighted
jrtuck
L2 Linker
‎01-30-2020 07:05 AM
‎01-30-2020 07:05 AM

Yes, I am running version 1.1.53.

 

jrtuck_0-1580396686299.png

 

0 Likes
Reply
Highlighted
dgildelaig
L5 Sessionator
‎01-30-2020 07:06 AM
‎01-30-2020 07:06 AM

Could you contact us to fwmigrate at paloaltonetworks dot com to do a Zoom session and check live the issue?

 

Best,

0 Likes
Reply
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!

Latest Blogs
Latest Posts
Privacy Policy Terms of Use
Copyright 2007 - 2020 - Palo Alto Networks