Skip to content

Unable to delete User Data #8703

Closed
Closed

Description

@adidiborg
ISSUE TYPE
Bug Report
COMPONENT NAME
User Data
CLOUDSTACK VERSION
4.19
CONFIGURATION
N/A
OS / ENVIRONMENT
Oracle Linux Server 8.9 / KVM
SUMMARY

Unable to delete created User Data after registering with a Template and creating Instance from that template. After deleting the Instance and then the Template that are using User Data, unable to delete that specific user data. Attached screenshot error for reference.

STEPS TO REPRODUCE
Create a User Data. Register a Template with this user data. 
Create a Instance from the template. 
Delete both Instance and Template. 
Try deleting User Data
EXPECTED RESULTS
User data is deleted.
ACTUAL RESULTS
User data not deleted with below error.

image

Activity

  1. boring-cyborg commented on Feb 26, 2024

    @boring-cyborg

    Thanks for opening your first issue here! Be sure to follow the issue template!

  2. adidiborg commented on Feb 26, 2024

    @adidiborg
    Author

    Below is error log from management-server.log

    2024-02-26 04:52:07,978 DEBUG [c.c.u.d.T.Transaction] (qtp139928643-20033:ctx-38ba6222 ctx-f48be2d1) (logid:e2eac635) Rolling back the transaction: Time = 0 Name =  qtp139928643-20033; called by -TransactionLegacy.rollback:889-TransactionLegacy.removeUpTo:832-TransactionLegacy.close:656-TransactionContextInterceptor.invoke:36-ReflectiveMethodInvocation.proceed:175-ExposeInvocationInterceptor.invoke:97-ReflectiveMethodInvocation.proceed:186-JdkDynamicAopProxy.invoke:215-$Proxy268.remove:-1-ManagementServerImpl.deleteUserData:4700-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:62
    2024-02-26 04:52:07,978 ERROR [c.c.a.ApiServer] (qtp139928643-20033:ctx-38ba6222 ctx-f48be2d1) (logid:e2eac635) unhandled exception executing api command: [Ljava.lang.String;@610c1627
    com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.cj.jdbc.ClientPreparedStatement: DELETE FROM user_data WHERE user_data.id= 1
            at com.cloud.utils.db.GenericDaoBase.expunge(GenericDaoBase.java:1207)
            at com.cloud.utils.db.GenericDaoBase.remove(GenericDaoBase.java:1899)
            at jdk.internal.reflect.GeneratedMethodAccessor377.invoke(Unknown Source)
            at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.base/java.lang.reflect.Method.invoke(Method.java:566)
            at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
            at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
            at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
            at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
            at com.sun.proxy.$Proxy268.remove(Unknown Source)
            at com.cloud.server.ManagementServerImpl.deleteUserData(ManagementServerImpl.java:4700)
            at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
            at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
            at java.base/java.lang.reflect.Method.invoke(Method.java:566)
            at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
            at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
            at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
            at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
            at com.sun.proxy.$Proxy227.deleteUserData(Unknown Source)
            at org.apache.cloudstack.api.command.user.userdata.DeleteUserDataCmd.execute(DeleteUserDataCmd.java:90)
            at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:172)
            at com.cloud.api.ApiServer.queueCommand(ApiServer.java:782)
            at com.cloud.api.ApiServer.handleRequest(ApiServer.java:603)
            at com.cloud.api.ApiServlet.processRequestInContext(ApiServlet.java:347)
            at com.cloud.api.ApiServlet$1.run(ApiServlet.java:154)
            at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
            at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
            at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
            at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:151)
            at com.cloud.api.ApiServlet.doGet(ApiServlet.java:105)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:645)
            at javax.servlet.http.HttpServlet.service(HttpServlet.java:750)
            at org.eclipse.jetty.servlet.ServletHolder$NotAsync.service(ServletHolder.java:1450)
            at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:799)
            at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:554)
            at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
            at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:600)
            at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
            at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
            at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624)
            at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
            at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1440)
            at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
            at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:505)
            at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594)
            at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
            at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1355)
            at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
            at org.eclipse.jetty.server.handler.gzip.GzipHandler.handle(GzipHandler.java:772)
            at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:146)
            at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
            at org.eclipse.jetty.server.Server.handle(Server.java:516)
            at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:487)
            at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:732)
            at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:479)
            at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:277)
            at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
            at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
            at org.eclipse.jetty.io.ssl.SslConnection$DecryptedEndPoint.onFillable(SslConnection.java:555)
            at org.eclipse.jetty.io.ssl.SslConnection.onFillable(SslConnection.java:410)
            at org.eclipse.jetty.io.ssl.SslConnection$2.succeeded(SslConnection.java:164)
            at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105)
            at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
            at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:338)
            at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:315)
            at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:173)
            at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:131)
            at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:409)
            at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)
            at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)
            at java.base/java.lang.Thread.run(Thread.java:887)
    Caused by: java.sql.SQLIntegrityConstraintViolationException: Cannot delete or update a parent row: a foreign key constraint fails (`cloud`.`vm_template`, CONSTRAINT `fk_vm_template__user_data_id` FOREIGN KEY (`user_data_id`) REFERENCES `user_data` (`id`))
            at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:117)
            at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
            at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
            ... 85 more
    2024-02-26 04:52:07,979 DEBUG [c.c.a.ApiServlet] (qtp139928643-20033:ctx-38ba6222 ctx-f48be2d1) (logid:e2eac635) ===END===  10.119.206.194 -- GET  id=b4926eb5-d4db-48b1-b86a-879c517322b5&account=admin&domainid=849cfd03-d21c-11ee-84ad-246e96679ca0&command=deleteUserData&response=json
    
  3. kiranchavala commented on Feb 26, 2024

    @kiranchavala
    Member

    thanks @adidiborg i am also able to reproduce the issue with the same exception

    cc @harikrishna-patnala

    Caused by: java.sql.SQLIntegrityConstraintViolationException: Cannot delete or update a parent row: a foreign key constraint fails (`cloud`.`vm_template`, CONSTRAINT `fk_vm_template__user_data_id` FOREIGN KEY (`user_data_id`) REFERENCES `user_data` (`id`))
    	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:117)
    	at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
    	at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
    	... 81 more
    
  4. adidiborg commented on Feb 26, 2024

    @adidiborg
    Author

    @kiranchavala ok, thanks for checking. We can delete User data by just creating and then deleting (i.e without attaching to template / instance). But once attached, get this error.

  5. weizhouapache commented on Feb 26, 2024

    @weizhouapache
    Member

    @kiranchavala @adidiborg
    can you share the result of mysql command ?

    show create table vm_template;
    
  6. kiranchavala commented on Feb 26, 2024

    @kiranchavala
    Member

    @weizhouapache

    mysql> show create table vm_template \G;
    *************************** 1. row ***************************
           Table: vm_template
    Create Table: CREATE TABLE `vm_template` (
      `id` bigint unsigned NOT NULL AUTO_INCREMENT,
      `unique_name` varchar(255) NOT NULL,
      `name` varchar(255) NOT NULL,
      `uuid` varchar(40) DEFAULT NULL,
      `public` int unsigned NOT NULL,
      `featured` int unsigned NOT NULL,
      `type` varchar(32) DEFAULT NULL,
      `hvm` int unsigned NOT NULL COMMENT 'requires HVM',
      `bits` int unsigned NOT NULL COMMENT '32 bit or 64 bit',
      `url` varchar(255) DEFAULT NULL COMMENT 'the url where the template exists externally',
      `format` varchar(32) NOT NULL COMMENT 'format for the template',
      `created` datetime NOT NULL COMMENT 'Date created',
      `removed` datetime DEFAULT NULL COMMENT 'Date removed if not null',
      `account_id` bigint unsigned NOT NULL COMMENT 'id of the account that created this template',
      `checksum` varchar(255) DEFAULT NULL COMMENT 'checksum for the template root disk',
      `display_text` varchar(4096) DEFAULT NULL COMMENT 'Description text set by the admin for display purpose only',
      `enable_password` int unsigned NOT NULL DEFAULT '1' COMMENT 'true if this template supports password reset',
      `enable_sshkey` int unsigned NOT NULL DEFAULT '0' COMMENT 'true if this template supports sshkey reset',
      `guest_os_id` bigint unsigned NOT NULL COMMENT 'the OS of the template',
      `bootable` int unsigned NOT NULL DEFAULT '1' COMMENT 'true if this template represents a bootable ISO',
      `prepopulate` int unsigned NOT NULL DEFAULT '0' COMMENT 'prepopulate this template to primary storage',
      `cross_zones` int unsigned NOT NULL DEFAULT '0' COMMENT 'Make this template available in all zones',
      `extractable` int unsigned NOT NULL DEFAULT '0' COMMENT 'Is this template extractable',
      `hypervisor_type` varchar(32) DEFAULT NULL COMMENT 'hypervisor that the template belongs to',
      `source_template_id` bigint unsigned DEFAULT NULL COMMENT 'Id of the original template, if this template is created from snapshot',
      `template_tag` varchar(255) DEFAULT NULL COMMENT 'template tag',
      `sort_key` int NOT NULL DEFAULT '0' COMMENT 'sort key used for customising sort method',
      `size` bigint unsigned DEFAULT NULL,
      `state` varchar(255) DEFAULT NULL,
      `update_count` bigint unsigned DEFAULT NULL,
      `updated` datetime DEFAULT NULL,
      `dynamically_scalable` tinyint unsigned NOT NULL DEFAULT '0' COMMENT 'true if template contains XS/VMWare tools inorder to support dynamic scaling of VM cpu/memory',
      `parent_template_id` bigint unsigned DEFAULT NULL COMMENT 'If datadisk template, then id of the root template this template belongs to',
      `direct_download` tinyint(1) DEFAULT '0' COMMENT 'Indicates if Secondary Storage is bypassed and template is downloaded to Primary Storage',
      `deploy_as_is` tinyint(1) NOT NULL DEFAULT '0' COMMENT 'True if the template should be deployed with disks and networks as defined by OVF',
      `user_data_id` bigint unsigned DEFAULT NULL COMMENT 'id of the user data',
      `user_data_link_policy` varchar(255) DEFAULT NULL COMMENT 'user data link policy with template',
      PRIMARY KEY (`id`),
      UNIQUE KEY `uc_vm_template__uuid` (`uuid`),
      KEY `i_vm_template__removed` (`removed`),
      KEY `i_vm_template__public` (`public`),
      KEY `fk_vm_template__user_data_id` (`user_data_id`),
      CONSTRAINT `fk_vm_template__user_data_id` FOREIGN KEY (`user_data_id`) REFERENCES `user_data` (`id`)
    ) ENGINE=InnoDB AUTO_INCREMENT=211 DEFAULT CHARSET=utf8mb3
    
  7. weizhouapache commented on Feb 26, 2024

    @weizhouapache
    Member

    @weizhouapache

    mysql> show create table vm_template \G;
    *************************** 1. row ***************************
           Table: vm_template
    Create Table: CREATE TABLE `vm_template` (
      `id` bigint unsigned NOT NULL AUTO_INCREMENT,
      `unique_name` varchar(255) NOT NULL,
      `name` varchar(255) NOT NULL,
      `uuid` varchar(40) DEFAULT NULL,
      `public` int unsigned NOT NULL,
      `featured` int unsigned NOT NULL,
      `type` varchar(32) DEFAULT NULL,
      `hvm` int unsigned NOT NULL COMMENT 'requires HVM',
      `bits` int unsigned NOT NULL COMMENT '32 bit or 64 bit',
      `url` varchar(255) DEFAULT NULL COMMENT 'the url where the template exists externally',
      `format` varchar(32) NOT NULL COMMENT 'format for the template',
      `created` datetime NOT NULL COMMENT 'Date created',
      `removed` datetime DEFAULT NULL COMMENT 'Date removed if not null',
      `account_id` bigint unsigned NOT NULL COMMENT 'id of the account that created this template',
      `checksum` varchar(255) DEFAULT NULL COMMENT 'checksum for the template root disk',
      `display_text` varchar(4096) DEFAULT NULL COMMENT 'Description text set by the admin for display purpose only',
      `enable_password` int unsigned NOT NULL DEFAULT '1' COMMENT 'true if this template supports password reset',
      `enable_sshkey` int unsigned NOT NULL DEFAULT '0' COMMENT 'true if this template supports sshkey reset',
      `guest_os_id` bigint unsigned NOT NULL COMMENT 'the OS of the template',
      `bootable` int unsigned NOT NULL DEFAULT '1' COMMENT 'true if this template represents a bootable ISO',
      `prepopulate` int unsigned NOT NULL DEFAULT '0' COMMENT 'prepopulate this template to primary storage',
      `cross_zones` int unsigned NOT NULL DEFAULT '0' COMMENT 'Make this template available in all zones',
      `extractable` int unsigned NOT NULL DEFAULT '0' COMMENT 'Is this template extractable',
      `hypervisor_type` varchar(32) DEFAULT NULL COMMENT 'hypervisor that the template belongs to',
      `source_template_id` bigint unsigned DEFAULT NULL COMMENT 'Id of the original template, if this template is created from snapshot',
      `template_tag` varchar(255) DEFAULT NULL COMMENT 'template tag',
      `sort_key` int NOT NULL DEFAULT '0' COMMENT 'sort key used for customising sort method',
      `size` bigint unsigned DEFAULT NULL,
      `state` varchar(255) DEFAULT NULL,
      `update_count` bigint unsigned DEFAULT NULL,
      `updated` datetime DEFAULT NULL,
      `dynamically_scalable` tinyint unsigned NOT NULL DEFAULT '0' COMMENT 'true if template contains XS/VMWare tools inorder to support dynamic scaling of VM cpu/memory',
      `parent_template_id` bigint unsigned DEFAULT NULL COMMENT 'If datadisk template, then id of the root template this template belongs to',
      `direct_download` tinyint(1) DEFAULT '0' COMMENT 'Indicates if Secondary Storage is bypassed and template is downloaded to Primary Storage',
      `deploy_as_is` tinyint(1) NOT NULL DEFAULT '0' COMMENT 'True if the template should be deployed with disks and networks as defined by OVF',
      `user_data_id` bigint unsigned DEFAULT NULL COMMENT 'id of the user data',
      `user_data_link_policy` varchar(255) DEFAULT NULL COMMENT 'user data link policy with template',
      PRIMARY KEY (`id`),
      UNIQUE KEY `uc_vm_template__uuid` (`uuid`),
      KEY `i_vm_template__removed` (`removed`),
      KEY `i_vm_template__public` (`public`),
      KEY `fk_vm_template__user_data_id` (`user_data_id`),
      CONSTRAINT `fk_vm_template__user_data_id` FOREIGN KEY (`user_data_id`) REFERENCES `user_data` (`id`)
    ) ENGINE=InnoDB AUTO_INCREMENT=211 DEFAULT CHARSET=utf8mb3
    

    thanks @kiranchavala

    It looks like relates to the non-existing removed column in user_data.
    Once userdata is removed, the record is gone from user_data table.
    cc @harikrishna-patnala

  8. DaanHoogland commented on Feb 26, 2024

    @DaanHoogland
    Contributor

    oh, and as user-data is a first class citizen now we should have a column like that.
    @adidiborg adding that column could solve your issue, Did you try that?

  9. weizhouapache commented on Feb 26, 2024

    @weizhouapache
    Member

    oh, and as user-data is a first class citizen now we should have a column like that. @adidiborg adding that column could solve your issue, Did you try that?

    VO needs to be updated as well

  10. harikrishna-patnala commented on Feb 27, 2024

    @harikrishna-patnala
    Member

    agreed to the solution to have removed column instead of removing the entry itself. I'll create a PR for that.

  11. adidiborg commented on Feb 27, 2024

    @adidiborg
    Author

    oh, and as user-data is a first class citizen now we should have a column like that. @adidiborg adding that column could solve your issue, Did you try that?

    @DaanHoogland / @weizhouapache , no I haven't. I'm new to using Cloudstack and haven't touched its mysql DB entries yet. Can you please share SQL commands that would allow to delete User data? Thanks

  12. sureshanaparti commented on Feb 27, 2024

    @sureshanaparti
    Contributor

    agreed to the solution to have removed column instead of removing the entry itself. I'll create a PR for that.

    Also, I think, it's better to have a background thread / any API to purge these removed records later (some point of time after removed, can be controlled through global setting).

  13. added this to the 4.19.1.0 milestone on Feb 27, 2024
  14. weizhouapache commented on Feb 27, 2024

    @weizhouapache
    Member

    agreed to the solution to have removed column instead of removing the entry itself. I'll create a PR for that.

    @harikrishna-patnala
    an alternative solution without db changes is

    • set user_data_id to null for the vm_template
    • then remove user_data record (if removed column does not exist, records are expunged)

    this looks very simple, and no need to clean up userdata. @sureshanaparti

  15. sureshanaparti commented on Feb 27, 2024

    @sureshanaparti
    Contributor

    oh, and as user-data is a first class citizen now we should have a column like that. @adidiborg adding that column could solve your issue, Did you try that?

    @DaanHoogland / @weizhouapache , no I haven't. I'm new to using Cloudstack and haven't touched its mysql DB entries yet. Can you please share SQL commands that would allow to delete User data? Thanks

    @adidiborg Follow the steps listed (assuming user data id = 1, based on the screenshot added)

    • Check/Confirm there are no templates with the user data - empty result for the sql below.
      SELECT id, name, user_data_id, user_data_link_policy FROM cloud.vm_template WHERE user_data_id = 1 AND removed IS NULL;
    • If no templates, update user_data_id to NULL, in vm_template table - use update sql below.
      UPDATE cloud.vm_template SET user_data_id = NULL, user_data_link_policy = NULL WHERE user_data_id = 1 AND removed IS NOT NULL;
    • Now, Delete the user data from UI, or use delete sql below
      DELETE FROM cloud.user_data WHERE id = 1;

    Hope this helps.

  16. self-assigned this
    on May 24, 2024
  17. added a commit that references this issue on May 24, 2024
    f02ab50
  18. moved this from Todo to In Progress in Apache CloudStack 4.19.1on May 24, 2024
  19. shwstppr commented on May 28, 2024

    @shwstppr
    Contributor

    Fixed with #9120

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Type

No type

Projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions