Fix VMware VCSA /storage/log filesystem out of disk space

  • VMware

This morning I ran into an issue where users were reporting the production VCSA 6.0 was not allowing them to connect into the web or thick client. Another administrator rebooted the VCSA which seemed to work only briefly. I then logged into the VCSA web management (https://<VCENTER_IP>:5480) and noticed the following health status right away:

The /storage/log filesystem is out of disk space or inodes

vcsa /storage/log full

So I opened up PuTTY and ran a df -h command and confirmed the issue:

VMware VCSA storage log VMDK full

How to fix VCSA /storage/log filesystem out of disk space

Luckily the fix is rather easy, thanks to a blog post by @lamw that I found while looking for a solution, he mentions in VCSA 6.x you can now expand VMDK’s on the fly since the VCSA takes advantage of LVM.

  1. So open the vSphere client (web or thick) and expand the VMDK (see below table for which VMDK to expand)
    VCSA /storage/log expand vmdk
  2. Next open PuTTY (or other terminal window) and run the following command on your VCSA
    vpxd_servicecfg storage lvm autogrow

    VCSA /storage/log autogrow vmdk

And that is all there is to it, we now have a healthy VCSA again!

VMware VCSA healthy

VMWare VCSA 6.0 VMDK list and purpose

VMDK Disk Size Mount Point Purpose
VMDK1 12 GB / & /boot Boot
VMDK2 1.3 GB /tmp/mount Temp mount
VMDK3 25 GB SWAP Swap space
VMDK4 25 GB /storage/core Core dumps
VMDK5 10 GB /storage/log System logs
VMDK6 10 GB /storage/db Postgres DB location
VMDK7 5 GB /storage/dblog Postgres DB logs
VMDK8 10 GB /storage/seat Stats, events, and tasks (SEAT) for Postgres
VMDK9 1 GB /storage/netdump Netdump collector
VMDK10 10 GB /storage/autodeploy Auto Deploy repository
VMDK11 5 GB storage/invsvc Inventory service bootstrap and tomcat config

So in the issue above, VMDK disk 5 was expanded and then the autogrow command ran which resolved our issue.

  • Chip Zoller

    Hey Mike, you might also want to investigate the cause of that full disk scenario. One such known condition I’ve seen multiple times is caused by incorrect log rotation settings in the sso filesystem. Have a look here to see if you might be suffering the same fate:

    • Thanks @chipzoller:disqus for the related KB, I’ll certainly take a look at those suggested steps this evening after hours. Much appreciated!