# URGENT HELP NEEDED : Not able to start or create a vm inside any host

[arpanbht]

Hey guys we are facing a problem in our production cloudstack setup, one of our vm was somehow became read only so we went to restart the vm but it failed to restart and now i cannot restart or start any vm into any specific host. Also cannot create new vm in any host it says this error

2026-07-13 05:41:56,141 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-15:[ctx-fe4148f6, job-30103]) (logid:bc46d542) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin com.cloud.utils.exception.CloudRuntimeException: Unable to start a VM [ff06ec9d-644d-4033-a66c-7c6cb80b7ddc] due to [Unable to create a deployment for VM instance {"id":867,"instanceName":"i-2-867-VM","state":"Starting","type":"User","uuid":"ff06ec9d-644d-4033-a66c-7c6cb80b7ddc"}].
        at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:935)
        at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:245)
        at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:214)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5822)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5674)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5275)
        at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:5264)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:109)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:52)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at jdk.proxy3/jdk.proxy3.$Proxy219.startVirtualMachine(Unknown Source)
        at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:97)
        at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:173)
        at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:110)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:698)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:646)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)
Caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":867,"instanceName":"i-2-867-VM","state":"Starting","type":"User","uuid":"ff06ec9d-644d-4033-a66c-7c6cb80b7ddc"}Scope=interface com.cloud.dc.DataCenter; id=3
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1454)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5951)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
        at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:6075)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)

Also error is coming when I start any vm in any host machine

0111,"initMsid":163763490546348,"instanceId":null,"instanceType":null,"lastPolled":null,"lastUpdated":null,"processStatus":0,"removed":null,"result":null,"resultCode":0,"status":"IN_PROGRESS","userId":2,"uuid":"874ce4e6-f75f-4051-9760-3e1d1d53e7fc"}, job origin: 30110 com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM instance {"id":856,"instanceName":"i-2-856-VM","state":"Starting","type":"User","uuid":"5849b9aa-648d-4684-9a61-8f45de46741c"}Scope=interface com.cloud.dc.DataCenter; id=3
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1454)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5951)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:569)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:102)
        at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:6075)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:698)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:646)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:840)

It says insufficient server capacity but cpu and ram is available in that host.

Is this any agent.properties issue?
we have two host of intel xeon cpu and one host of amd epyc cpu

if you need i can share the agent.properties file for your reference.

Also I can ssh into the vms but the file system is read-only, I cannot edit or create anything

Please help guys!!!

[herrickP]

Good news, your logs actually point to a pretty specific cause!
What's happening
InsufficientServerCapacityException while CPU/RAM are free, combined with read-only filesystem inside the VM, points hard at shared primary storage, not compute capacity.
Likely chain: your primary storage (probably NFS) disconnected, filled up, or got remounted read-only. That makes the guest filesystem inside the VM go read-only too (why you can SSH in but can't write). And when CloudStack checks host + storage together for deployment planning, if the storage pool is down/disconnected, it throws out every host as invalid, even with free CPU/RAM sitting right there.
Quick checks

1. Run mount on each host, confirm NFS primary storage is mounted and not ro.
2. Check the NFS server itself is up and the export isn't full.
3. In CloudStack UI, check Infrastructure > Primary Storage, is it "Down" or in "Maintenance"?
4. Check dmesg for NFS timeout/stale handle errors around when this started.
5. Also worth checking for a storage tag mismatch since you've got mixed Xeon/EPYC hosts.

Fix the storage connectivity and the capacity errors usually clear up on their own, no need to touch host capacity settings.
We've run into this exact "storage goes read-only and kills capacity checks" scenario a few times managing CloudStack setups at AccuWeb.Cloud, happy to help troubleshoot further if you share mount | grep nfs output!

[arpanbht]

@herrickP
Hey man, I have managed to solve the issue.

I tried everything from restarting the vms, system vms, virtual router.
But it looked like the problem was elsewhere, I updated and upgraded in all the physical machines i.e. management server, hosts and nfs and restarted everything physically and after that I restarted management server and agent and then inside tue management ui dashboard restsrted the system vms and virtual and everything is up and running now.

I don't know what was the issue but if I somehow face the same issue I will apply your steps and will try to solve the issue 😅