Next-Gen VM-Series and Panorama generates "Invalid Opcode" VSCSI messages on VMware 6.0

Reply
Highlighted
L0 Member

Next-Gen VM-Series and Panorama generates "Invalid Opcode" VSCSI messages on VMware 6.0

We had an issue on our ESXi server and in looking through the logs found a large number of "Invalid Opcode" log messages related to the Panorama VM and Next Gen FW VM trying to access features of the VSCSIFs made available by VMware 6.0. These do not appear to affect the performance of either product, but it does indicate a disconnect between Palo Alto's VMs and services available in VMware's ESXi hypervisor. They generally occur at a rate of around 10 per day with some days generating over 200 of these log messages.

 

Sample messages:

–cpu11:37401)VSCSIFs: 2235: handle 8194(vscsi0:0):Invalid Opcode (0x4d) from (vmm0:xxxxxxxxxxxxxFWH)

–cpu11:37401)VSCSIFs: 2235: handle 8194(vscsi0:0):Invalid Opcode (0x4d) from (vmm0:xxxxxxxxxxxxxFWH)

–cpu11:37401)VSCSIFs: 2235: handle 8194(vscsi0:0):Invalid Opcode (0xb7) from (vmm0:xxxxxxxxxxxxxFWH)

–cpu11:37401)VSCSIFs: 2235: handle 8194(vscsi0:0):Invalid Opcode (0x37) from (vmm0:xxxxxxxxxxxxxFWH)

–cpu11:37401)VSCSIFs: 2235: handle 8194(vscsi0:0):Invalid Opcode (0xb7) from (vmm0:xxxxxxxxxxxxxFWH)

 

We have also seen the Panorama VM generating Invalid Opcode (0x4d) errors in the ESX vmkernel.log files.

 

We have alerted VMware to investigate on their end.

Highlighted
Community Team Member

I searched a little and found this info on vmware's site about invalid opcode errors:

https://kb.vmware.com/s/article/1003278

 

I hope this helps. Doesn't seem like it is a PAN issue directly.. but I cannot find anything about it internally.

Stay Secure,
Joe
End of line
Highlighted
L0 Member

Thank you for the response Joe. This is a ESX6.0 server with the latest patches which clears the ESX mention of using version 4.1 or earlier. The server is hosting a number of other Linux and Windows VMs which are not generating these Invalid Opcode messages which I assume to mean that these general OS images and installed applications are not issuing these SCSI commands to the host server. My assumption is that PanOS is based on some Linux/UNIX derivative with additional packages loaded to improve packet processing, disk and memory performance in order to get as close to line rate processing as possible. The guess then is that one of those packages is issuing these additional commands in order to optimize it's parameters.

 

The goal of this post is to:

1) Identify whether any other Palo Alto customers are seeing similar behavior (we're using PanOS version 8.1.10)

2) Ask Palo Alto to look through their testing/debug logs and see if they can identify which package within PanOS is issuing these SCSI commands and to work with VMware and their package vendor to resolve the mismatch

 

It's not everyday you have someone combing through the ESXi logs and since this doesn't seem to impact functionality of the appliance, I'll bet most users would not detect this issue. It would be in Palo Alto's best interest to grep for the string across their test labs and see if/where this is occurring as well as identify the root cause and any impact to functional performance on customer installations.

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!