After I decided to have VSAN on my small home lab, I looked up some best practices, and this is what I've come up with.
1. The Host Syslog should not be on the VSAN Datastore
2. The ScratchConfig should not be on the VSAN Datastore, or on SD card.
3. Ideally there would be some local storage for these, but if VSAN claims all my local disks, what to do?
My proposed solutions are.
1. Sacrifice a HDD from each Host for Syslog and ScratchConfig. (defiantly not a solution for homelab)
2. Buy some smaller HDD's that are slower and cheaper for each host.
3. Run without a local storage (in this case VMware uses a ramdisk and data is not saved between boots).
In my case I choose third solution:
I had to reclaim a disk from VSAN, format it for local use, create a filesystem, then re-point Syslog and ScratchConfig to use this new space.
First I moved all of the Vms off of the machine that I wanted to reclaim a disk from and put the host in maintenance mode. Next, I went to the disk management portion of VSAN configuration, and made a note of the UUID of the disk that I wanted to remove. Then I clicked "remove". All is well, nothing exploded.
The issue I experienced is not able to delete storage because storage is busy.
Quick research and I found reason. It is scratch file.
Just create folder:
mkdir /tmp/scratch
Next, copy the mount location and paste it into the Configuration>Advanced>ScratchConfig.ConfiguredScratchLocation.
Next, enter the value
[] /tmp/scratch/log
in Advanced->Syslog->Syslog.global.logdir
