Cdb process not running on PA firewall

[tamilvanan]

Hi Folks,

Auto-commit on our passive firewall is failing. When checking the logs we could the see the commit failure reason as below:

PA-3220 not started, auto commit failed:

Details:
Management server failed to send phase 1 to client cord
Commit failed
Failed to commit policy to device

When checking the logs we could see the CDB process is not running as given below:

Process authd running (pid: 7957)
Process cdb running (pid: -1) - Exit Signal: SIGABRT
Process chasd running (pid: 4721)
Process contentd running (pid: 4455)
Process cord running (pid: 7419)
Process crypto running (pid: 4908)

Also under system logs we could see the following logs:

2022-03-08 01:33:44 Issue Warning MP System Commit Commit Job failed and also the cdb process is restarting multiple times.

Under Cordlog we could see the following logs:

mp        cord.log                           2022-03-07 00:49:56   2022-03-07 00:49:56.055 +0300 Error:  pan_corr_db_conn_connect(pan_corr_cdb.c:72): pan_mongo_connect() failed
mp        cord.log                           2022-03-07 00:49:58   2022-03-07 00:49:58.056 +0300 Error:  pan_mongo_connect(pan_mongo.c:101): mongo_client() failed:2
mp        cord.log                           2022-03-07 00:49:58   2022-03-07 00:49:58.056 +0300 Error:  pan_corr_db_conn_connect(pan_corr_cdb.c:72): pan_mongo_connect() failed
mp        cord.log                           2022-03-07 00:50:00   2022-03-07 00:50:00.056 +0300 Error:  pan_mongo_connect(pan_mongo.c:101): mongo_client() failed:2
mp        cord.log                           2022-03-07 00:50:00   2022-03-07 00:50:00.056 +0300 Error:  pan_corr_db_conn_connect(pan_corr_cdb.c:72): pan_mongo_connect() failed
mp        cord.log                           2022-03-07 00:50:02   2022-03-07 00:50:02.057 +0300 Error:  pan_mongo_connect(pan_mongo.c:101): mongo_client() failed:2
mp        cord.log                           2022-03-07 00:50:02   2022-03-07 00:50:02.057 +0300 Error:  pan_corr_db_conn_connect(pan_corr_cdb.c:72): pan_mongo_connect() failed
mp        cord.log                           2022-03-07 00:50:04   2022-03-07 00:50:04.057 +0300 Error:  pan_mongo_connect(pan_mongo.c:101): mongo_client() failed:2
mp        cord.log                           2022-03-07 00:50:04   2022-03-07 00:50:04.057 +0300 Error:  pan_corr_db_conn_connect(pan_corr_cdb.c:72): pan_mongo_connect() failed
mp        cord.log                           2022-03-07 00:50:06   2022-03-07 00:50:06.058 +0300 Error:  pan_mongo_connect(pan_mongo.c:101): mongo_client() failed:2
mp        cord.log                           2022-03-07 00:50:06   2022-03-07 00:50:06.058 +0300 Error:  pan_corr_db_conn_connect(pan_corr_cdb.c:72): pan_mongo_connect() failed
mp        cord.log                           2022-03-07 00:50:08   2022-03-07 00:50:08.058 +0300 Error:  pan_mongo_connect(pan_mongo.c:101): mongo_client() failed:2
mp        cord.log                           2022-03-07 00:50:08   2022-03-07 00:50:08.058 +0300 Error:  pan_corr_db_conn_connect(pan_corr_cdb.c:72): pan_mongo_connect() failed
mp        cord.log                           2022-03-07 00:50:10   2022-03-07 00:50:10.059 +0300 Error:  pan_mongo_connect(pan_mongo.c:101): mongo_client() failed:2
mp        cord.log                           2022-03-07 00:50:10   2022-03-07 00:50:10.059 +0300 Error:  pan_corr_db_conn_connect(pan_corr_cdb.c:72): pan_mongo_connect() failed
mp        cord.log                           2022-03-07 00:50:12   2022-03-07 00:50:12.059 +0300 Error:  pan_mongo_connect(pan_mongo.c:101): mongo_client() failed:2
mp        cord.log                           2022-03-07 00:50:12   2022-03-07 00:50:12.059 +0300 Error:  pan_corr_db_conn_connect(pan_corr_cdb.c:72): pan_mongo_connect() failed
mp        cord.log                           2022-03-07 00:50:14   2022-03-07 00:50:14.060 +0300 Error:  pan_mongo_connect(pan_mongo.c:101): mongo_client() failed:2
mp        cord.log                           2022-03-07 00:50:14   2022-03-07 00:50:14.060 +0300 Error:  pan_corr_db_conn_connect(pan_corr_cdb.c:72): pan_mongo_connect() failed
mp        cord.log                           2022-03-07 00:50:16   2022-03-07 00:50:16.060 +0300 Error:  pan_mongo_connect(pan_mongo.c:101): mongo_client() failed:2
mp        cord.log                           2022-03-07 00:50:16   2022-03-07 00:50:16.060 +0300 Error:  pan_corr_db_conn_connect(pan_corr_cdb.c:72): pan_mongo_connect() failed
mp        cord.log                           2022-03-07 00:50:18   2022-03-07 00:50:18.061 +0300 Error:  pan_mongo_connect(pan_mongo.c:101): mongo_client() failed:2
mp        cord.log                           2022-03-07 00:50:18   2022-03-07 00:50:18.061 +0300 Error:  pan_corr_db_conn_connect(pan_corr_cdb.c:72): pan_mongo_connect() failed
mp        cord.log                           2022-03-07 00:50:20   2022-03-07 00:50:20.061 +0300 Error:  pan_mongo_connect(pan_mongo.c:101): mongo_client() failed:2
mp        cord.log                           2022-03-07 00:50:20   2022-03-07 00:50:20.061 +0300 Error:  pan_corr_db_conn_connect(pan_corr_cdb.c:72): pan_mongo_connect() failed
mp        cord.log                           2022-03-07 00:50:22   2022-03-07 00:50:22.062 +0300 Error:  pan_mongo_connect(pan_mongo.c:101): mongo_client() failed:2

The firewall is running on PAN-OS-10.0.8.

We are not able to find any known issues also.

Any one faced the same issue before.

Thanks in advance

Regards,

Tamilvanan

[kiwi]

Hi @tamilvanan ,

I've seen this before.  Try the following:

#commit force
Then check the status 
>show system software status

If the process cdb is still stopped you might have to get in touch with support to root into the device to clear some corrupt files in mongo and restart the cdb.

Cheers,

-Kiwi.

[reaper]

i'd try upgrading to 10.0.9 😉

else you'll want to open a support case so TAC can look into this issue

[tamilvanan]

Hi @kiwi  We had tried to do force commit but not successful. Still the CDB process is not running

admin@pa> configure
Entering configuration mode
[edit]
admin@pa# commit force

Server error : Commit job was not queued. All daemons are not available.

Group    services             running
Group    third_party          running
Process  authd                running  (pid: 7907)
Process  cdb                  stopped  (pid: -1) - Exit Signal: SIGABRT
Process  chasd                running  (pid: 4648)
Process  contentd             running  (pid: 4392)

[tamilvanan · Accepted solution]

We had raised the case with PA TAC. The Engineer had taken root access to the firewall and made the changes in db. After that the cdb process started to work again without any issues.

[s.debus]

Hi

A Restart of the device don't helps. The only chance is support from TAC. He can create a Challenge Code and with this higher administrativ level he can restart the process.

Regards

Sascha