I don't believe there is a remove adapter for the vswitch. What you can do is set the switch for the adapters you want it it to have as opposed to what to remove. IE, the below will set your virtual switch to have only Vmnic1 if you wanted to remove vmnic0.
Get-VirtualSwitch -VMhost $VMHost -name "vSwitch0" | Set-VirtualSwitch -Nic "vmnic1" -Confirm:$false
If you have multiple and need to keep say 2 but not 3 it would be:
Get-VirtualSwitch -VMHost $VMHost -name "vSwitch0" | Set-VirtualSwitch -Nic "vmnic0","vmnic1" -Confirm:$false