Fix VMware VCSA /storage/log filesystem out of disk space

VMware

This morning I ran into an issue where users were reporting the production VCSA 6.0 was not allowing them to connect into the web or thick client. Another administrator rebooted the VCSA which seemed to work only briefly. I then logged into the VCSA web management (https://<VCENTER_IP>:5480) and noticed the following health status right away:

The /storage/log filesystem is out of disk space or inodes

vcsa /storage/log full

So I opened up PuTTY and ran a df -h command and confirmed the issue:

VMware VCSA storage log VMDK full

How to fix VCSA /storage/log filesystem out of disk space

Luckily the fix is rather easy, thanks to a blog post by @lamw that I found while looking for a solution, he mentions in VCSA 6.x you can now expand VMDK’s on the fly since the VCSA takes advantage of LVM.

  1. So open the vSphere client (web or thick) and expand the VMDK (see below table for which VMDK to expand)
    VCSA /storage/log expand vmdk
  2. Next open PuTTY (or other terminal window) and run the following command on your VCSA 
    vpxd_servicecfg storage lvm autogrow

    VCSA /storage/log autogrow vmdk

And that is all there is to it, we now have a healthy VCSA again!

VMware VCSA healthy

VMWare VCSA 6.0 VMDK list and purpose

VMDK Disk Size Mount Point Purpose
VMDK1 12 GB / & /boot Boot
VMDK2 1.3 GB /tmp/mount Temp mount
VMDK3 25 GB SWAP Swap space
VMDK4 25 GB /storage/core Core dumps
VMDK5 10 GB /storage/log System logs
VMDK6 10 GB /storage/db Postgres DB location
VMDK7 5 GB /storage/dblog Postgres DB logs
VMDK8 10 GB /storage/seat Stats, events, and tasks (SEAT) for Postgres
VMDK9 1 GB /storage/netdump Netdump collector
VMDK10 10 GB /storage/autodeploy Auto Deploy repository
VMDK11 5 GB storage/invsvc Inventory service bootstrap and tomcat config

So in the issue above, VMDK disk 5 was expanded and then the autogrow command ran which resolved our issue.

Similar Posts

  • Thank you VMware Community!

    VMware
    0 Comments

    VMware vExpert 2014

    So far, 2014 has been a very rewarding year for a number of reasons, two of which has happened in just a week or two span. First, Eric Siebert (@ericsiebert) announced on March 27th, this years results of the 2014 Top VMware & Virtualization Blog voting. My first year entered into voting and made it to 71st place! A huge thanks goes out to not only Eric but just as much so to everyone who voted for me!

    To top it off, yesterday VMware announced 2014’s first quarter VMware vExpert list. While vExpert isn’t a technical certification or even a general measure of VMware expertise. The VMware judges selected people who were engaged with their community and who had developed a substantial personal platform of influence in those communities. There were a lot of very smart, very accomplished people, even VCDXs, that weren’t named as vExpert this year. VMware awarded this title to 754 people this year and on that list of many impressive names you’ll find yours truly, Michael Tabor!

    I’m both honored and humbled by both lists. It’s a great feeling to be recognized by not only my peers through the voting in the Top vBlog but also by VMware themselves through the vExpert title.

    So again THANK YOU very much to the entire VMware community, a spectacular community indeed, and congratulations to everyone else that made the Top vBlog and vExpert lists!

  • VMware Workstation updated to 9.0.2

    VMware
    0 Comments

    VMware is released a maintenance update for VMware Workstation to version 9.0.2. The update is available for free for all VMware Workstation 9.x customers.

    Mostly just bug fixes and a few improvements, here are a few highlights:

    • Fedora 18 supported as a guest operating system
    • Windows 8 host permissions issues fixed
    • VMware Tool updated to support the latest Linux kernels
    • Improved general host resource management
    • OVF Tool has been updated to version 3.0.2
    • Virtual machine sharing improvements
    • Memory leak which impacted Windows hosts has been fixed

    Read More “VMware Workstation updated to 9.0.2”

  • VMware Workstation 8.0.4 released

    VMware
    0 Comments

    VMware has just released it’s forth minor update for VMware Workstation 8, bringing it up to 8.0.4. The looks to contain mostly a few bug and security fixes.

    General Issues

    • Linux guests running the Linux kernel version 2.6.34 or later could not be pinged from the host via an IPv6 address.
    • On rare occasions, Linux guests would suddenly fail to Autofit or enter Unity.
    • Unity mode would exit if the title bar of an application contained certain non UTF-8 encoded extended ASCII characters.
    • On Windows hosts, the VMware Workstation user interface sometimes became unresponsive when minimized from full-screen mode if the suggestion balloon was being displayed.
    • On Windows hosts, the user interface sometimes became unresponsive if the application was rendered on an extended display that was abruptly disconnected.

    Read More “VMware Workstation 8.0.4 released”

  • Another CBT bug found in VMware ESXi 6.0

    VMware
    0 Comments

    VMware

    VMware just published KB 2136854 regarding a new bug found in ESXi 6.0 that causes virtual machine backups, which use Changed Block Tracking (CBT), to be inconsistent. VMware says the cause of the issue is this:

    This issue occurs due to an issue with CBT in the disklib area, this causes the change tracking information of I/Os that occur during snapshot consolidation to be lost. The main backup payload data is never lost and it is always written to the backend device. However, the corresponding change tracking information entries which occur during the consolidation task are missed. Subsequent QueryDiskChangedAreas() calls do not include these missed blocks, hence a backup based on this CBT data is inconsistent.

    Read More “Another CBT bug found in VMware ESXi 6.0”

  • VCP5 Exam: Passed

    VMware
    0 Comments

    Pass

    This morning I took the VCP510 exam and passed it on my first go at it!

    Overall I felt the exam was a nice challenge, while I can’t go into details of the exam, per the NDA. I will say I was surprised at the number of multiple answer questions and the VPC510 was certainly more “real-world” based and not so much memorizing config maximums.

    That said, I’ll do as others have done and list what I used in my preparation for the VCP510 exam…

    Read More “VCP5 Exam: Passed”

  • How to unlock vRealize Orchestrator default account

    VMware
    0 Comments

    vrealize orchestrator account locked out

    Earlier this month I posted how on how to fix the weak ephemeral Diffie-Hellman key issue in VMware vRealize Orchestrator 6.0.2 appliance. Another issue I’ve ran into is how to unlock vRealize Orchestrator default account. It seems you can lock the account after to many failed attempts but once locked the account does not seem to unlock… or at least for me after waiting more than a couple hours.

    Luckily unlocking the vRealize Orchestrator default account is pretty straight forward.

    Read More “How to unlock vRealize Orchestrator default account”

Leave a Reply to hJensen Cancel reply

Your email address will not be published. Required fields are marked *

14 Comments

  1. Hey Mike, you might also want to investigate the cause of that full disk scenario. One such known condition I’ve seen multiple times is caused by incorrect log rotation settings in the sso filesystem. Have a look here to see if you might be suffering the same fate: https://kb.vmware.com/kb/2143565

    Reply

  7. Total gold! Thanks for an easy explanation. I found that this error was preventing me from upgrading/patching VCSA .

    Reply