Fix VMware VCSA /storage/log filesystem out of disk space
This morning I ran into an issue where users were reporting the production VCSA 6.0 was not allowing them to connect into the web or thick client. Another administrator rebooted the VCSA which seemed to work only briefly. I then logged into the VCSA web management (https://<VCENTER_IP>:5480) and noticed the following health status right away:
The /storage/log filesystem is out of disk space or inodes
So I opened up PuTTY and ran a df -h command and confirmed the issue:
How to fix VCSA /storage/log filesystem out of disk space
Luckily the fix is rather easy, thanks to a blog post by @lamw that I found while looking for a solution, he mentions in VCSA 6.x you can now expand VMDK’s on the fly since the VCSA takes advantage of LVM.
- So open the vSphere client (web or thick) and expand the VMDK (see below table for which VMDK to expand)
- Next open PuTTY (or other terminal window) and run the following command on your VCSA
vpxd_servicecfg storage lvm autogrow
And that is all there is to it, we now have a healthy VCSA again!
VMWare VCSA 6.0 VMDK list and purpose
VMDK Disk | Size | Mount Point | Purpose |
---|---|---|---|
VMDK1 | 12 GB | / & /boot | Boot |
VMDK2 | 1.3 GB | /tmp/mount | Temp mount |
VMDK3 | 25 GB | SWAP | Swap space |
VMDK4 | 25 GB | /storage/core | Core dumps |
VMDK5 | 10 GB | /storage/log | System logs |
VMDK6 | 10 GB | /storage/db | Postgres DB location |
VMDK7 | 5 GB | /storage/dblog | Postgres DB logs |
VMDK8 | 10 GB | /storage/seat | Stats, events, and tasks (SEAT) for Postgres |
VMDK9 | 1 GB | /storage/netdump | Netdump collector |
VMDK10 | 10 GB | /storage/autodeploy | Auto Deploy repository |
VMDK11 | 5 GB | storage/invsvc | Inventory service bootstrap and tomcat config |
So in the issue above, VMDK disk 5 was expanded and then the autogrow command ran which resolved our issue.