Problem 1 in 1,000,000? I hope not

[S.Fernandez413037]

hello community! I hope you are doing well. I'm here to ask a question that PAN support didn't know how to deal with and maybe some of you have already gone through the same thing. It was Saturday morning and xsoar and its functions were completely frozen, absolutely nothing was running. As XSOAR administrator, this had already happened so I restarted the service and to my surprise, not only the service was not up but it had failed, something that had never happened to me before. Checking the logs we found very strange things like the following:

runtime error: invalid memory address or nil pointer dereference
fatal error: stack overflow
/builds/GOPATH/src/gitlab.xdr.pan.local/xdr/xsoar/server/util/goutils.go:31
/builds/GOPATH/src/gitlab.xdr.pan.local/xdr/xsoar/server/repoDB/complexRepo/statistics.go:93
/builds/GOPATH/src/gitlab. xdr.pan.local/xdr/xsoar/server/repoDB/complexRepo/wrapper_bolt.go:73
/builds/GOPATH/src/gitlab.xdr.pan.local/xdr/xsoar/server/vendor/go.etcd.io/bbolt/db.go:741
2025-06-04 03:22:00.6888 error runtime.errorString runtime error: invalid memory address or nil pointer dereference.

On the other hand, we saw this commit that was made and that appears to have been corrupted. Has anyone experienced something similar? *Here's the interesting part*
The solution was to take an uncorrupted snapshot from the previous day, but when it was taken at 3:22 a.m. the next day, boom, same error, same failure. This repeated itself for two more days while we waited for a response from Palo Alto support. Has anyone encountered a similar error that they can help us with? It would be greatly appreciated. Keep it up!

[Ivetto]

This sounds like an issue at the kernel level with the memory management, source: https://stackoverflow.com/questions/16280176/go-panic-runtime-error-invalid-memory-address-or-nil-po...

"Can be caused by multiple reasons that all of them are eventually related to memory allocations." 

Given the fact that XSOAR is such a big program and you also have several auxiliary programs and other packages installed, I'd recommend moving the data to a backup location and wiping the server and reinstalling everything fresh. And by everything I do mean everything from the server OS to Docker and any other needed packages.

Article on how to install from scratch: https://docs-cortex.paloaltonetworks.com/r/Cortex-XSOAR/6.12/Cortex-XSOAR-Administrator-Guide/Single...