About GNeyrinck

Hi,   Yes, I was able to do so. I had to install the correct version of the Ubuntu server (download from: http://releases.ubuntu.com/16.04/) Pick version: http://releases.ubuntu.com/16.04/ubuntu-16.04.6-server-amd64.iso And afterwards I could do a clean install. Didac, the person who develops this tool told me you had to use the AMD version: Yes, it has been tested with AMD64 packages. Several packages require 64bit computations. Haven't had the issue since then.   Off course I don't know if you already did install the right version. ... View more

I encountered this error also, due to the longer time the logs weren't processed the foldersize got bigger. I've added some additional space and relaunched the processing and all went ok. I'm still seeing the (reduced) logfiles even if you select the "delete logs after processing"   ... View more

All files have been processed now. The only thing I need to verify now is the automatic processing of the logs.     ... View more

I meant to say that I've only select one file. But you explained that all will be processed except the ones that you set to ignore. ... View more

Another thing which isn't ok is the machine learning auto processing  It should be hard coded, but it's showing as empty in the gui ... View more

Edit: ok, all files are going to be processed except the ones you set to ignore.       I meant to say that I'm only selecting one file to test if the processing would work again.     Since it's taking that long I'm assuming that all logs are being processed: root@Expedition:/home/expedition# tail -f /tmp/error_logCoCo SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/opt/Spark/extraLibraries/slf4j-nop-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/opt/Spark/spark-2.4.3-bin-hadoop2.7/jars/slf4j-log4j12-1.7.16.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.helpers.NOPLoggerFactory] ---- CREATING SPARK Session:           warehouseLocation:/datastore/spark-warehouse +------------+--------+--------------------+----+------------+ |     fwSerial|panosver|             csvpath|size|afterProcess| +------------+--------+--------------------+----+------------+ |011901016881|   8.1.0|/PALogs/paloalto_...|4957|     Compress| |011901016881|   8.1.0|/PALogs/paloalto_...|4332|     Compress| |011901016881|   8.1.0|/PALogs/paloalto_...|4629|     Compress| |011901016881|   8.1.0|/PALogs/paloalto_...|3861|     Compress| |011901016881|   8.1.0|/PALogs/paloalto_...|4629|     Compress| |011901016881|   8.1.0|/PALogs/paloalto_...|3840|     Compress| |011901016881|   8.1.0|/PALogs/paloalto_...|5264|     Compress| |011901016881|   8.1.0|/PALogs/paloalto_...|3892|     Compress| |011901016881|   8.1.0|/PALogs/paloalto_...|3646|     Compress| |011901016881|   8.1.0|/PALogs/paloalto_...|4783|     Compress| |011901016881|   8.1.0|/PALogs/paloalto_...|4772|     Compress| |011901016881|   8.1.0|/PALogs/paloalto_...|4619|     Compress| |011901016881|   8.1.0|/PALogs/paloalto_...|5161|     Compress| |011901016881|   8.1.0|/PALogs/paloalto_...|4752|     Compress| |011901016881|   8.1.0|/PALogs/paloalto_...|4076|     Compress| |011901016881|   8.1.0|/PALogs/paloalto_...|4895|     Compress| |011901016881|   8.1.0|/PALogs/paloalto_...|3789|     Compress| |011901016881|   8.1.0|/PALogs/paloalto_...|4742|     Compress| |011901016881|   8.1.0|/PALogs/paloalto_...|5059|     Compress| |011901016881|   8.1.0|/PALogs/paloalto_...|5407|     Compress| +------------+--------+--------------------+----+------------+   Memory: 6373m LogCollector&Compacter called with the following parameters:      Parameters for execution           Master[processes]:............   local[3]           Available RAM (MB):...........   6525952           User:.........................   admin           debug:........................   false      Parameters for Job Connections           Task ID:......................   152           My IP:........................   10.3.1.30           Expedition IP:................   10.3.1.30:3306           Time Zone:....................   Europe/Helsinki           dbUser (dbPassword):..........   root (************)           projectName:..................   demo      Parameters for Data Sources           App Categories (source):........ (Expedition)           CSV Files Path:................./tmp/1572951103_traffic_files.csv           Parquet output path:..........   file:///PALogs/connections.parquet           Temporary folder:.............   /datastore ---- AppID DB LOAD:           Application Categories loading...      Application Categories loaded   +------------+--------+--------------------+----+------------+--------+---+---------------+ |     fwSerial|panosver|             csvpath|size|afterProcess| grouped|row|accumulatedSize| +------------+--------+--------------------+----+------------+--------+---+---------------+ |011901016881|   8.1.0|/PALogs/paloalto_...|4957|     Compress|grouping|   1|         4957.0| |011901016881|   8.1.0|/PALogs/paloalto_...|4332|     Compress|grouping|   2|         9289.0| |011901016881|   8.1.0|/PALogs/paloalto_...|4629|     Compress|grouping|   3|         13918.0| |011901016881|   8.1.0|/PALogs/paloalto_...|3861|     Compress|grouping|   4|         17779.0| |011901016881|   8.1.0|/PALogs/paloalto_...|4629|     Compress|grouping|   5|         22408.0| |011901016881|   8.1.0|/PALogs/paloalto_...|3840|     Compress|grouping|   6|         26248.0| |011901016881|   8.1.0|/PALogs/paloalto_...|5264|     Compress|grouping|   7|         31512.0| |011901016881|   8.1.0|/PALogs/paloalto_...|3892|     Compress|grouping|   8|         35404.0| |011901016881|   8.1.0|/PALogs/paloalto_...|3646|     Compress|grouping|   9|         39050.0| |011901016881|   8.1.0|/PALogs/paloalto_...|4783|     Compress|grouping| 10|         43833.0| |011901016881|   8.1.0|/PALogs/paloalto_...|4772|     Compress|grouping| 11|         48605.0| |011901016881|   8.1.0|/PALogs/paloalto_...|4619|     Compress|grouping| 12|         53224.0| |011901016881|   8.1.0|/PALogs/paloalto_...|5161|     Compress|grouping| 13|         58385.0| |011901016881|   8.1.0|/PALogs/paloalto_...|4752|     Compress|grouping| 14|         63137.0| |011901016881|   8.1.0|/PALogs/paloalto_...|4076|     Compress|grouping| 15|         67213.0| |011901016881|   8.1.0|/PALogs/paloalto_...|4895|     Compress|grouping| 16|         72108.0| |011901016881|   8.1.0|/PALogs/paloalto_...|3789|     Compress|grouping| 17|         75897.0| |011901016881|   8.1.0|/PALogs/paloalto_...|4742|     Compress|grouping| 18|         80639.0| |011901016881|   8.1.0|/PALogs/paloalto_...|5059|     Compress|grouping| 19|         85698.0| |011901016881|   8.1.0|/PALogs/paloalto_...|5407|     Compress|grouping| 20|         91105.0| +------------+--------+--------------------+----+------------+--------+---+---------------+     Selection criteria:   0 < accumulatedSize and accumulatedSize <= 6525952      Processing from lowLimit:0 to highLimit:6525952 with StepLine:6525952      Few logs can fit in this batch:20 8.1.0:/PALogs/paloalto_1_traffic_2019_10_25_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_11_01_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_11_02_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_31_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_19_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_17_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_11_03_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_28_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_11_05_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_27_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_30_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_22_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_11_04_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_18_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_26_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_21_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_29_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_20_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_23_last_calendar_day.csv,/PALogs/paloalto_1_traffic_2019_10_24_last_calendar_day.csv Logs of format 7.1.x NOT found Logs of format 8.0.2 NOT found Logs of format 8.1.0-beta17 NOT found Logs of format 8.1.0 found Logs of format 9.0.0 NOT found Logs of format 9.1.0-beta NOT found     ... View more

I have now upgraded to the latest version 1.1.45 and it seems that the logs are being processed. Even though I've only selected one log file to be processed, all pending (20) files seem to be processed in one time. I'll check if the processing of logs ends ok now.   Regards, ... View more

I have the exact same issue after the upgrade from 1.41 to this version. I saw some errors in cli while upgrading. Did you also update Expedition?   Regards ... View more

Hi,   GP is not active. It resolves correctly I think: @expedition-ubuntu : ~ $ dig conversionupdates.paloaltonetworks.com   ; <<>> DiG 9.10.3-P4-Ubuntu <<>> conversionupdates.paloaltonetworks.com ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 46632 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1   ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 4096 ;; QUESTION SECTION: ;conversionupdates.paloaltonetworks.com. IN A   ;; ANSWER SECTION: conversionupdates.paloaltonetworks.com. 173 IN A 199.167.52.71   ;; Query time: 0 msec ;; SERVER: 192.168.60.100#53(192.168.60.100) ;; WHEN: Tue Sep 10 12:14:44 CEST 2019 ;; MSG SIZE   rcvd: 83   Sent an email as you suggested.   thanks   ... View more

It did complain about the release files: expedition-ubuntu : ~ $ sudo apt-get update Hit:1 http://ppa.launchpad.net/adiscon/v8-stable/ubuntu xenial InRelease Hit:2 http://be.archive.ubuntu.com/ubuntu xenial InRelease                                                                                                                         Hit:3 http://be.archive.ubuntu.com/ubuntu xenial-updates InRelease                                                                                 Hit:5 http://be.archive.ubuntu.com/ubuntu xenial-backports InRelease                                                                               Hit:6 http://sgp1.mirrors.digitalocean.com/mariadb/repo/10.1/ubuntu xenial InRelease                                         Hit:7 http://security.ubuntu.com/ubuntu xenial-security InRelease                                                             Hit:4 https://www.rabbitmq.com/debian testing InRelease Ign:8 https://conversionupdates.paloaltonetworks.com expedition-updates/ InRelease Ign:9 https://conversionupdates.paloaltonetworks.com expedition-updates/ Release Ign:10 https://conversionupdates.paloaltonetworks.com expedition-updates/ Packages.diff/Index Ign:11 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en Ign:12 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en_GB Ign:13 https://conversionupdates.paloaltonetworks.com expedition-updates/ Packages Ign:11 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en Ign:12 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en_GB Ign:13 https://conversionupdates.paloaltonetworks.com expedition-updates/ Packages Ign:11 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en Ign:12 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en_GB Ign:13 https://conversionupdates.paloaltonetworks.com expedition-updates/ Packages Ign:11 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en Ign:12 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en_GB Hit:13 https://conversionupdates.paloaltonetworks.com expedition-updates/ Packages Ign:11 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en Ign:12 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en_GB Ign:11 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en Ign:12 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en_GB Reading package lists... Done W: The repository 'https://conversionupdates.paloaltonetworks.com expedition-updates/ Release' does not have a Release file. N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use. N: See apt-secure(8) manpage for repository creation and user configuration details.   If I try to install expedition-beta: @expedition-ubuntu : ~ $ sudo apt-get install expedition-beta Reading package lists... Done Building dependency tree         Reading state information... Done E: Unable to locate package expedition-beta   expedition-ubuntu : ~ $ cat /etc/apt/sources.list.d/ex-repo.list deb https://conversionupdates.paloaltonetworks.com/ expedition-updates/   Still stuck at this ... View more

I'm guessing that's ok. Still not resolved though:   @expedition-ubuntu : ~ $ sudo apt-get update Hit:1 http://be.archive.ubuntu.com/ubuntu xenial InRelease Hit:2 http://be.archive.ubuntu.com/ubuntu xenial-updates InRelease                         Hit:3 http://ppa.launchpad.net/adiscon/v8-stable/ubuntu xenial InRelease                   Hit:5 http://be.archive.ubuntu.com/ubuntu xenial-backports InRelease                       Hit:6 http://security.ubuntu.com/ubuntu xenial-security InRelease     Hit:7 http://sgp1.mirrors.digitalocean.com/mariadb/repo/10.1/ubuntu xenial InRelease Hit:4 https://www.rabbitmq.com/debian testing InRelease               Ign:8 https://conversionupdates.paloaltonetworks.com expedition-updates/ InRelease Ign:9 https://conversionupdates.paloaltonetworks.com expedition-updates/ Release Ign:10 https://conversionupdates.paloaltonetworks.com expedition-updates/ Packages.diff/Index Ign:11 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en Ign:12 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en_GB Ign:13 https://conversionupdates.paloaltonetworks.com expedition-updates/ Packages Ign:11 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en Ign:12 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en_GB Ign:13 https://conversionupdates.paloaltonetworks.com expedition-updates/ Packages Ign:11 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en Ign:12 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en_GB Ign:13 https://conversionupdates.paloaltonetworks.com expedition-updates/ Packages Ign:11 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en Ign:12 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en_GB Hit:13 https://conversionupdates.paloaltonetworks.com expedition-updates/ Packages Ign:11 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en Ign:12 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en_GB Ign:11 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en Ign:12 https://conversionupdates.paloaltonetworks.com expedition-updates/ Translation-en_GB Reading package lists... Done W: The repository 'https://conversionupdates.paloaltonetworks.com expedition-updates/ Release' does not have a Release file. N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use. N: See apt-secure(8) manpage for repository creation and user configuration details. ... View more