Fix VMware VCSA /storage/log filesystem out of disk space

VMware

This morning I ran into an issue where users were reporting the production VCSA 6.0 was not allowing them to connect into the web or thick client. Another administrator rebooted the VCSA which seemed to work only briefly. I then logged into the VCSA web management (https://<VCENTER_IP>:5480) and noticed the following health status right away:

The /storage/log filesystem is out of disk space or inodes

vcsa /storage/log full

So I opened up PuTTY and ran a df -h command and confirmed the issue:

VMware VCSA storage log VMDK full

How to fix VCSA /storage/log filesystem out of disk space

Luckily the fix is rather easy, thanks to a blog post by @lamw that I found while looking for a solution, he mentions in VCSA 6.x you can now expand VMDK’s on the fly since the VCSA takes advantage of LVM.

  1. So open the vSphere client (web or thick) and expand the VMDK (see below table for which VMDK to expand)
    VCSA /storage/log expand vmdk
  2. Next open PuTTY (or other terminal window) and run the following command on your VCSA 
    vpxd_servicecfg storage lvm autogrow

    VCSA /storage/log autogrow vmdk

And that is all there is to it, we now have a healthy VCSA again!

VMware VCSA healthy

VMWare VCSA 6.0 VMDK list and purpose

VMDK Disk Size Mount Point Purpose
VMDK1 12 GB / & /boot Boot
VMDK2 1.3 GB /tmp/mount Temp mount
VMDK3 25 GB SWAP Swap space
VMDK4 25 GB /storage/core Core dumps
VMDK5 10 GB /storage/log System logs
VMDK6 10 GB /storage/db Postgres DB location
VMDK7 5 GB /storage/dblog Postgres DB logs
VMDK8 10 GB /storage/seat Stats, events, and tasks (SEAT) for Postgres
VMDK9 1 GB /storage/netdump Netdump collector
VMDK10 10 GB /storage/autodeploy Auto Deploy repository
VMDK11 5 GB storage/invsvc Inventory service bootstrap and tomcat config

So in the issue above, VMDK disk 5 was expanded and then the autogrow command ran which resolved our issue.

Similar Posts

  • VMware Horizon View 5.3 has been announced

    VMware
    1 Comment

    Horizon View 5.2 is receiving an update to 5.3! Doesn’t sound like much, but in that tiny dot number increase VMware has in fact brought a number improvements and features to Horizon View.

    In a nut shell, some features Horizon View 5.3 brings is the ability to connect to a virtual desktop from any HTML5 capable web browser (yes, even Chromebook!), ability to bring full workstation class user experience with 3D graphics, deploy Windows Server 2008 as a desktop, as well as completely rewritten the Windows client with support for Windows 8.1 and a new iOS7 client too.

    In more detail, Horizon View 5.3 brings…

    Read More “VMware Horizon View 5.3 has been announced”

  • How to update VMware Windows VM’s DNS using PowerCLI

    VMware
    4 Comments

    This post will show you how to update a VMware Windows virtual machine’s DNS IP addresses using PowerCLI.

    You’ll also see how I went from a starting point to a fully working script – all thanks to the awesome #vCommunity!

    Here at work we’ve been doing a number of networking, AD, and DNS changes. In doing so, our DNS IP address have changed. Changing the DNS settings for our workstations was easy, simply edit the DHCP scope.

    The servers on the other hand all had static IP’s and static DNS IP’s entered, each of which needed to be updated.

    Read More “How to update VMware Windows VM’s DNS using PowerCLI”

  • VMware Workstation 8.0.4 released

    VMware
    0 Comments

    VMware has just released it’s forth minor update for VMware Workstation 8, bringing it up to 8.0.4. The looks to contain mostly a few bug and security fixes.

    General Issues

    • Linux guests running the Linux kernel version 2.6.34 or later could not be pinged from the host via an IPv6 address.
    • On rare occasions, Linux guests would suddenly fail to Autofit or enter Unity.
    • Unity mode would exit if the title bar of an application contained certain non UTF-8 encoded extended ASCII characters.
    • On Windows hosts, the VMware Workstation user interface sometimes became unresponsive when minimized from full-screen mode if the suggestion balloon was being displayed.
    • On Windows hosts, the user interface sometimes became unresponsive if the application was rendered on an extended display that was abruptly disconnected.

    Read More “VMware Workstation 8.0.4 released”

  • Increase incremental backup speed by enabling CBT

    VMware
    2 Comments

    In a recent VMware environment I was working in, we noticed that our incremental backups was taking a long time (like up to 7-8 hours) to complete. After digging around it was found that VMware actually has a feature built-in since version 4 called Change Block Tracking (CBT). CBT will track any blocks that were changed since the last backup and tag them and stores the information in a -CTK file. The obvious benefit is that now the third party software only backs up the changed block and not the entire VM each time, reducing the amount of data being backed thus speeding up backups and even lowers the CPU utilization on the VM host.

    CBT is disabled by default though there are some backup tools that will enable it automatically or you can enable it manually which can easily be done following these steps:

    Read More “Increase incremental backup speed by enabling CBT”

Leave a Reply

Your email address will not be published. Required fields are marked *

14 Comments

  1. Hey Mike, you might also want to investigate the cause of that full disk scenario. One such known condition I’ve seen multiple times is caused by incorrect log rotation settings in the sso filesystem. Have a look here to see if you might be suffering the same fate: https://kb.vmware.com/kb/2143565

    Reply

  7. Total gold! Thanks for an easy explanation. I found that this error was preventing me from upgrading/patching VCSA .

    Reply