Error tying to register ISO/Template with DirectDownload over HTTPS with Let's Encrypt #12878

problem

When using the "Direct Download" feature for an ISO or Template (bypassing Secondary Storage), the Agent fails to verify Let's Encrypt certificates due to the absence of Let's Encrypt’s main CA certificate.

Note: Let's Encrypt is widely used on the internet (>50% of all certificates).

ACS currently loads and trusts certificates exclusively from /etc/cloudstack/agent/cloud.jks and does not fall back to Java (/usr/lib/jvm/java-17-openjdk-amd64/lib/security/cacerts) or the system store (/etc/ssl/certs/ca-certificates.crt). Both of these contain the missing certificate (ISRG Root X1), which has been in use since 2015.

See: https://letsencrypt.org/certificates/

ISRG Root X1 is the current root of the trust chain (valid until 2030), after which it will be replaced by ISRG Root X2.

Recommendation: Add a fallback to Java’s trust store to avoid maintaining an ever-changing list of certificates.

Alternative: As a short-term fix, include the missing CA certificate (https://letsencrypt.org/certs/isrgrootx1.pem) in /etc/cloudstack/agent/cloud.jks for the next release, while a more sustainable solution is developed.

versions

We are running ACS 4.20.2 on Ubuntu 24.04. However, this issue likely affects all versions starting from 4.19, when the feature to bypass Secondary Storage was introduced.

Related issues and PRs:

• Failed to register direct-download template in 4.18.1.0-RC1 #7929
• https://github.com/apache/cloudstack/pull/7693/changes
• https://github.com/apache/cloudstack/pull/7923/changes
• https://github.com/apache/cloudstack/pull/7932/changes
• https://github.com/apache/cloudstack/pull/11113/changes

The steps to reproduce the bug

1. When registering an ISO or Template for Direct Download, use any HTTPS URL whose TLS certificate is issued by Let's Encrypt.

What to do about it?

As a workaround, the following command can be run for each Zone to add the missing certificate. Note that this introduces additional manual steps for platform maintenance:

cmk upload templatedirectdownloadcertificate hypervisor="KVM" name="isrg-root-x1-2" certific
ate="$(curl -s https://letsencrypt.org/certs/isrgrootx1.pem)" zoneid="00000000-0000-0000-00000-000000000000"