Next-Gen VM-Series and Panorama generates "Invalid Opcode" VSCSI messages on VMware 6.0

[kielecm]

We had an issue on our ESXi server and in looking through the logs found a large number of "Invalid Opcode" log messages related to the Panorama VM and Next Gen FW VM trying to access features of the VSCSIFs made available by VMware 6.0. These do not appear to affect the performance of either product, but it does indicate a disconnect between Palo Alto's VMs and services available in VMware's ESXi hypervisor. They generally occur at a rate of around 10 per day with some days generating over 200 of these log messages.

Sample messages:

–cpu11:37401)VSCSIFs: 2235: handle 8194(vscsi0:0):Invalid Opcode (0x4d) from (vmm0:xxxxxxxxxxxxxFWH)

–cpu11:37401)VSCSIFs: 2235: handle 8194(vscsi0:0):Invalid Opcode (0x4d) from (vmm0:xxxxxxxxxxxxxFWH)

–cpu11:37401)VSCSIFs: 2235: handle 8194(vscsi0:0):Invalid Opcode (0xb7) from (vmm0:xxxxxxxxxxxxxFWH)

–cpu11:37401)VSCSIFs: 2235: handle 8194(vscsi0:0):Invalid Opcode (0x37) from (vmm0:xxxxxxxxxxxxxFWH)

–cpu11:37401)VSCSIFs: 2235: handle 8194(vscsi0:0):Invalid Opcode (0xb7) from (vmm0:xxxxxxxxxxxxxFWH)

We have also seen the Panorama VM generating Invalid Opcode (0x4d) errors in the ESX vmkernel.log files.

We have alerted VMware to investigate on their end.

[jdelio]

I searched a little and found this info on vmware's site about invalid opcode errors:

https://kb.vmware.com/s/article/1003278

I hope this helps. Doesn't seem like it is a PAN issue directly.. but I cannot find anything about it internally.

[kielecm]

Thank you for the response Joe. This is a ESX6.0 server with the latest patches which clears the ESX mention of using version 4.1 or earlier. The server is hosting a number of other Linux and Windows VMs which are not generating these Invalid Opcode messages which I assume to mean that these general OS images and installed applications are not issuing these SCSI commands to the host server. My assumption is that PanOS is based on some Linux/UNIX derivative with additional packages loaded to improve packet processing, disk and memory performance in order to get as close to line rate processing as possible. The guess then is that one of those packages is issuing these additional commands in order to optimize it's parameters.

The goal of this post is to:

1) Identify whether any other Palo Alto customers are seeing similar behavior (we're using PanOS version 8.1.10)

2) Ask Palo Alto to look through their testing/debug logs and see if they can identify which package within PanOS is issuing these SCSI commands and to work with VMware and their package vendor to resolve the mismatch

It's not everyday you have someone combing through the ESXi logs and since this doesn't seem to impact functionality of the appliance, I'll bet most users would not detect this issue. It would be in Palo Alto's best interest to grep for the string across their test labs and see if/where this is occurring as well as identify the root cause and any impact to functional performance on customer installations.