Repository navigation
VM and Volume stats cleaner threads gets killed after any exception #13648
Description
problem
Upon any exception during VmStatsCleaner and VolumeStatsCleaner threads the scheduled task is not going to run again until the management servers are restarted.
Following is the related snippet to handle
Observed the below error after which the threads are not running again
2026-07-19 12:44:43,622 ERROR [c.c.v.d.VmStatsDaoImpl] (StatsCollector-5:[ctx-cb54c457]) (logid:494590e1) DB Exception on: HikariProxyPreparedStatement@311375727 wrapping com.mysql.cj.jdbc.ServerPreparedStatement[28]: DELETE FROM vm_stats WHERE vm_stats.timestamp < '2026-06-18 03:59:03' com.mysql.cj.jdbc.exceptions.CommunicationsException: Communications link failure
at com.mysql.cj.jdbc.exceptions.SQLError.createCommunicationsException(SQLError.java:175)
at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:64)
at com.mysql.cj.jdbc.ServerPreparedStatement.serverExecute(ServerPreparedStatement.java:555)
at com.mysql.cj.jdbc.ServerPreparedStatement.executeInternal(ServerPreparedStatement.java:339)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1061)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdateInternal(ClientPreparedStatement.java:1009)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeLargeUpdate(ClientPreparedStatement.java:1320)
at com.mysql.cj.jdbc.ClientPreparedStatement.executeUpdate(ClientPreparedStatement.java:994)
at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1305)
at com.cloud.utils.db.GenericDaoBase.batchExpunge(GenericDaoBase.java:1329)
at com.cloud.vm.dao.VmStatsDaoImpl.removeAllByTimestampLessThan(VmStatsDaoImpl.java:126)
at jdk.internal.reflect.GeneratedMethodAccessor566.invoke(Unknown Source)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
versions
Observed in 4.20.2
The steps to reproduce the bug
- When MS is running, try to either lock the tables "vm_stats" and "volume_stats" or kill the mysql communication
- Observe that the table vm_stats and volume_stats piles up
What to do about it?
No response
Reacted by Davi Torres
Activity
harikrishna-patnala commentedon Aug 17, 2026
fixed by #12695