Troubleshooting VMware NSX/ESXi Deployment

Issues related to VMware NSX can fall broadly into the following categories:

• Panorama unable to register or out-of-sync to NSX Manager
• PA-VM not passing traffic
• DAGs are not updated on Panorama
• Miscellaneous

1.  Collect answers for following questions:
     Questions (related to PAN).
     • Which PAN product for VMware?
            - VM-Series for vSphere
            - VM-Series for NSX
            - VM-Series for NSX with Cross vCenter NSX (Multi NSX Manager)
     • What version of Panorama?
     • What version of VMware NSX Integration Plugin (if applicable)?
     • What version of VM-Series?
     • How many VM IP addresses are registered in Panorama?
     • What changes were recently made on Panorama and/or VM-Series?

    Questions (related to VMware).
     • Is this a Virtual Desktop Infrastructure (VDI) or Virtual Server Infrastructure (VSI) deployment?
     • How many Clusters are targeted for deployment?
     • How many ESXi Hosts in each Cluster(s)?
     • What version of vCSA/vCenter?
     • What version of ESXi Hosts?
     • How many VM IP addresses are registered in Panorama?
     • What changes were recently made on vSphere or NSX?

    Questions (specific to VM-Series for vSphere).
     • Which firewall mode?
            - vwire
            - L2/L3
            - L3
            - L3 (Use Hypervisor Assigned MAC Addresses)
     • Which virtual switch?
            - Virtual Standard Switch (vSS)
            - Virtual Distributed Switch(vDS)

    Questions (specific to VM-Series for NSX).
    • What version of NSX?
    • Is Panorama connected to one NSX Manager or multiple NSX Managers?
    • If Cross vCenter NSX, how many vCenters?
    • What is the Panorama VMware Service Manager Status?
    • What OVF version of VM-Series for NSX is configured in Panorama?
    • How many Panorama Service Definitions?
    • How many NSX Security Groups are used?
    • How many NSX Security Policies are used?
    • Are VM IPv6 addresses deployed?
    • Is VM traffic being punted from the NSX DFW to VM-Series?
    • Is the VM-Series receiving VM traffic (e.g. pcaps)?
    • What changes were recently made on NSX?

2.  Validate the configuration on Panorama-VM:

      Register VM-Series Firewall as a Service with NSX Manager.
      • Panorama -> VMware NSX -> Service Managers -> Add (Edit)
            ø NSX Manager URL: "https://<NSX Manager IP Address>" or "https://<NSX Manager FQDN>"
            ø NSX Manager Login: <Username must be same used to login NSX Manager>
            ø NSX Manager Password: <Password must be same as used to login NSX Manager>
            ø Confirm NSX Manager Password: <Confirm the password>
     • Verify by login into NSX Manager URL and enter same credentials.
     • Select "Commit" and Commit Type "Commit to Panorama"
     • Verify: Panorama
            ø Verify Status "Registered"

3. Troubleshoot: Panorama status as “Error! Trigger NSX-Config-Sync!” or "Connection Error" etc..:

    The unsuccessful status messages are:

• Not connected: Unable to reach/establish a network connection to the NSX Manager.
• Not authorized: The access credentials (username and/or password) are incorrect.
• Not registered: The service, service manager, or service profile is unavailable or was deleted on the NSX Manager.
• Out of sync: The configuration settings defined on Panorama are different from what is defined on the NSX Manager. Check whether any changes done on NSX Manager. Validate the config changes and try manually syncing the config on Panorama.
• No service/ No service profile: Indicates an incomplete configuration on the NSX Manager.

      • For PANOS 8.0.x, check if plugin version installed on Panorama is 2.0.0/2.0.2 or TLSv1.0 is enabled on NSX Manager  under:

      NSX Manager > Manage > Settings > General > FIPS Mode and TLS Settings. [Jira Issue: PLUG-252]

      • For new NSX Manager 6.4.0 deployments, TLS 1.0 is disabled by default.
            1) If using Panorama NSX Plugin 2.0.1 or lower, NSX Manager TLS 1.0 has to be enabled for Panorama to communicate, or
            2) If using Panorama NSX Plugin 2.0.2 or greater, Panorama will use TLS 1.2 by default (then fall back to TLS 1.1 is necessary)

     • Examine plugins or NSX errors under system logs:

            > show log system subtype equal plugin direction equal backward

4. Troubleshoot: Issues related to DAGs missing on Panorama, out-of-sync to NSX Manager etc..

    Debug Logs: Might need to enable debug for more detailed information:

       • Check NSX Status via CLI:
              > show plugins vmware_nsx status
       • Enable debug level log for configd
             > debug management-server on debug
              > request plugins debug plugin-name vmware_nsx level high
       • Web Browser PHP debug logs:
               https://<panorama_ip>/debug (check box "Debug")
               Enable debug level log for php.debug.log

Manually sync config on Panorama by selecting 'NSX Config Sync' in the Operations section to synchronize the changes on the NSX Manager.

     OR
     For DAGs issues, click 'Synchronize Dynamic Objects'

        > tail follow yes mp-log plugin_vmware_nsx.log
        > tail follow yes mp-log php.debug.log

     Debug DAU updates to the managed firewalls:
        > tail follow yes mp-log configd.log

     Disable debugging post replication:
        > debug management-server off
        > request plugins debug plugin-name vmware_nsx level off

5. Troubleshoot: NSX Manager and Panorama connectivity:
    •  Management Port Captures : How To Packet Capture (tcpdump) On Management Interface

6. Troubleshoot: PA-VM is not passing traffic:

• If no sessions are generated on firewall, verify if traffic is punted to PA-VM from NSX Manager. Navigate to 'NSX > Networking & Security > Firewall > Partner Security Services' tab to confirm. 
• Use 'NSX Manager > Tools > Traceflow' tool to identify whether packet drops are happening at the NSX or the firewall.
• Verify if Security Groups and Steering Rules[only valid for PANOS 8.0.x or above] are configured on Panorama. Security groups created in NSX Manager must be associated with dynamic address groups on Panorama.
• Note: The default policy on the VM-Series firewall is set to deny all traffic, which means that all traffic redirected to the VM-Series firewall will be dropped.
• Make sure Panorama pushed the security policies to allow the interested traffic through PA-VM.
• If packets are received and but dropped on PA-VM, troubleshoot by collecting global-counters, flow-basic, pacaket-captures etc.. Reference: Packet Capture, Debug Flow-basic and Counter Commands