Expanding a PANOS Firewall VM log file on KVM

[William_Law]

There's plenty of discussions here and a couple of KB articles on adding disks to PAN VMs for extra logging space, but the only articles around for extending/increasing the size of the logging disk say to delete it and add a new one in its place. Downside to that is that you lose the logs that were on the disk at the time.

I raised this with TAC and the response was that "this is the way". When I queried what would happen if we were to resize the disk and filesystem offline and boot up the VM with the different disk all ready to go, there was an essential shrugging of shoulders. I find it surprising in this day that the OS isn't capable of recognising that a disk has grown, and/or offer a way of being able to resize a filesystem. What is more is that it is all done with static partitions and not LVM (yeah yeah, dependencies and security considerations).

Result? I span up a VM, attached a smaller logging disk to it and let it soak for an hour or so to generate reference logs. In the meantime a little googling found a couple of articles talking about growing qcow2 images including the filesystem inside. **NOTE** this requires that you have enough spare space as the current size of your logging disk image. In this example the original is 64GB and the target 96GB.

Pre-work disk and partition info:

PAN-VM> show system disk-space

Filesystem      Size  Used Avail Use% Mounted on
/dev/root       6.9G  5.2G  1.4G  80% /
none            5.9G   88K  5.9G   1% /dev
/dev/vda5        16G  3.2G   12G  21% /opt/pancfg
/dev/vda6       7.9G  2.2G  5.3G  30% /opt/panrepo
tmpfs           2.5G  1.9G  539M  79% /dev/shm
cgroup_root     5.9G     0  5.9G   0% /cgroup
/dev/vdb1        63G  188M   60G   1% /opt/panlogs

PAN-VM> debug system disk-partition-info

Disk /dev/vda: 60 GiB, 64424509440 bytes, 125829120 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7bdebb67

Device     Boot    Start       End  Sectors  Size Id Type
/dev/vda1  *        2176   2056319  2054144 1003M 83 Linux
/dev/vda2        2097280  16777216 14679937    7G 83 Linux
/dev/vda3       16777344  31457280 14679937    7G 83 Linux
/dev/vda4       31457344 125829119 94371776   45G  f W95 Ext'd (LBA)
/dev/vda5       31457408  65011712 33554305   16G 83 Linux
/dev/vda6       65011840  81788928 16777089    8G 83 Linux
/dev/vda7       81789056  81824256    35201 17.2M 83 Linux
/dev/vda8       81824384 125829119 44004736   21G 83 Linux


Disk /dev/vdb: 64 GiB, 68719476736 bytes, 134217728 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x140d924c

Device     Boot Start       End   Sectors Size Id Type
/dev/vdb1        2048 134217727 134215680  64G 83 Linux


Disk /dev/zram0: 3.9 GiB, 4194304000 bytes, 1024000 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

PAN-VM>

An now how to do it.  Log onto your KVM host (or a host with qemu disk tool installed that can access shared storage) and change to the directory holding the VMs files.  The below was done on a Debian 12.2 system.  virt-resize came from the guestfs-tools package.

1) Create a new logging disk:

kvmhost# qemu-img create -f qcow2 -o preallocation=metadata pan-logging2.qcow2 96G
Formatting 'pan-logging2.qcow2', fmt=qcow2 cluster_size=65536 extended_l2=off compression_type=zlib size=103079215104 lazy_refcounts=off refcount_bits=16
kvmhost# qemu-img info pan-logging2.qcow2
image: pan-logging2.qcow2
file format: qcow2
virtual size: 96 GiB (103079215104 bytes)
disk size: 196 KiB
cluster_size: 65536
Format specific information:
compat: 1.1
compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: false
extended l2: false
kvmhost#

2) shut down your VM

PAN-VM> request shutdown system

3) Once shut down, check which partition is needing to be resized

kvmhost# qemu-img info pan-logging.qcow2
image: pan-logging.qcow2
file format: qcow2
virtual size: 64 GiB (68719476736 bytes)
disk size: 446 MiB
cluster_size: 65536
Format specific information:
compat: 1.1
compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: false
extended l2: false
kvmhost# virt-filesystems --long --parts --blkdevs -h -a pan-logging.qcow2
Name Type MBR Size Parent
/dev/sda1 partition 83 64G /dev/sda
/dev/sda device - 64G -
kvmhost#

4) copy the data to the new image and expand the listed partition

kvmhost# virt-resize --expand /dev/sda1 pan-logging.qcow2 pan-logging2.qcow2
[ 0.0] Examining pan-logging.qcow2
**********

Summary of changes:

virt-resize: /dev/sda1: This partition will be resized from 64.0G to 96.0G.
The filesystem ext3 on /dev/sda1 will be expanded using the
‘resize2fs’ method.

**********
[ 3.8] Setting up initial partition table on pan-logging2.qcow2
[ 4.7] Copying /dev/sda1
100% ⟦▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒▒⟧ 00:00
[ 57.6] Expanding /dev/sda1 using the ‘resize2fs’ method

virt-resize: Resize operation completed with no errors. Before deleting
the old disk, carefully check that the resized disk boots and works
correctly.

kvmhost#

At this point virt-resize will use the target image type as a template. If you create a raw image it will end up as that and less joy will ensue.

5) Once completed, either rename the old image file to something else then rename the new file back to what the original (and the VM config) want, or remove the old disk from the config and attach the new one

kvmhost# ls -l
total 21918220
-rw-r--r-- 1 root root 92602368 Nov 9 14:48 pan-logging2.qcow2
-rw-r--r-- 1 root root 466944000 Nov 9 14:42 pan-logging.qcow2
-rwxr--r-- 1 root root 21883256832 Nov 9 14:42 PA-VM-KVM-10.2.3.qcow2
kvmhost#

Then either rename:

kvmhost# mv pan-logging.qcow2 pan-logging.qcow2-orig
kvmhost# mv pan-logging2.qcow2 pan-logging.qcow2
kvmhost#

or detach the old and attach the new:

kvmhost# virsh detach-disk pan-test vdb
kvmhost# virsh attach-disk pan-test /path/to/pandisks/pan-logging2.qcow2 vdb --cache writethrough --persistent --driver=qemu --subdriver=qcow2

6) Boot the VM and wait for it to get itself together.

7) Once you can log in, check your disk stats to ensure that the greater size is showing.

PAN-VM> show system disk-space

Filesystem Size Used Avail Use% Mounted on
/dev/root 6.9G 5.2G 1.4G 80% /
none 5.9G 88K 5.9G 1% /dev
/dev/vda5 16G 3.2G 12G 21% /opt/pancfg
/dev/vda6 7.9G 2.2G 5.3G 30% /opt/panrepo
tmpfs 2.5G 1.9G 539M 79% /dev/shm
cgroup_root 5.9G 0 5.9G 0% /cgroup
/dev/vdb1 95G 201M 90G 1% /opt/panlogs

PAN-VM> debug system disk-partition-info

Disk /dev/vda: 60 GiB, 64424509440 bytes, 125829120 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x7bdebb67

Device Boot Start End Sectors Size Id Type
/dev/vda1 * 2176 2056319 2054144 1003M 83 Linux
/dev/vda2 2097280 16777216 14679937 7G 83 Linux
/dev/vda3 16777344 31457280 14679937 7G 83 Linux
/dev/vda4 31457344 125829119 94371776 45G f W95 Ext'd (LBA)
/dev/vda5 31457408 65011712 33554305 16G 83 Linux
/dev/vda6 65011840 81788928 16777089 8G 83 Linux
/dev/vda7 81789056 81824256 35201 17.2M 83 Linux
/dev/vda8 81824384 125829119 44004736 21G 83 Linux


Disk /dev/vdb: 96 GiB, 103079215104 bytes, 201326592 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xbb0f256d

Device Boot Start End Sectors Size Id Type
/dev/vdb1 2048 201324287 201322240 96G 83 Linux


Disk /dev/zram0: 3.9 GiB, 4194304000 bytes, 1024000 sectors
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes

PAN-VM>

Check your logs from before the shutdown and you should still see your existing logs in place and accessible.