Let's Encrypt Revokes 3,000,000 Certs
Bottom line - if your certificates are affected and you will not renew and deploy new certs within hours, you will have effective downtimes - certificates will be revoked and invalid. The estimated total is 3 million, of which 1 million are duplicates.
Let's Encrypt celebrated its success last week only to revoke 2.5% of all its valid certificates in the next few days. Check the Let's Encrypt announcement for advice to check your certificates and how to react, as there seems to be some confusion in determining which certificates will be revoked.
The reason is a bug in their validation process. It caused insufficient validation of certificates with 2 or more domain names. While the chance of this being actively exploited is small, they have to revoke all affected certificates within 1-5 days according to their security compliance procedures.
Revocation of Let's Encrypt certificates will case many websites showing this message.
It is a massive issue as renewals require "force" flag, which may need manual renewals. The usual automation ignores renewals of certificates that have more than 30 days of validity left.
The Technical Cause
One of the steps in certificate request validations is a check of DNS records (DNS CAA) that may contain information about approved CAs. You can use your DNS records to prohibit certain CAs issuing certificates for your domain names.
You may wonder why such a technical detail can cause revocation of more than 3,000,000 certificates. The answer lies strictly in the compliance of CAs with rules for issuance of web certificates. Revocations should be done within 24 hours and it has to be completed within 5 days. Very rarely, however, you need to revoke such a large volume of certificates.
In 2018, Symantec had to revoke 23,000 of certificates after private key compromise and the case caused a lot of problems as well as press coverage.
This time, the issue is much, much bigger. The official announcement is at the Let's Encrypt Community forums.
Due to the 2020.02.29 CAA Rechecking Bug 4.3k, we unfortunately need to revoke many Let’s Encrypt TLS/SSL certificates. We’re e-mailing affected subscribers for whom we have contact information.
One of the problems is that many users don't share their email addresses with Let's Encrypt, or the email addresses are not valid any more. For example, when a sysadmin left a company.
Automation v Reliability
This is not the first case when Let's Encrypt users were affected by changes to Let's Encrypt "configuration". In 2017, Let's Encrypt started using IPv6 when available. It lead to issues with many users who had IPv6 entries in the DNS but they have not been correctly configured, while managed by ISPs in some cases.
Let's Encrypt also implemented changes to its API last year and is in the process of switching off the previous version. This change is "breaking", which means that users have to change the clients and integration they have for Let's Encrypt certificate renewals.
These situations are not unusual, although the size of the latest incident is unprecedented. It shows that "just install and it works" is not quite the appropriate approach if you the reliability is important. Questions focused on "what if X breaks" are suddenly more important than business-as-usual.
You’ve got better things to do than worry about web certificates. Try KEYCHEST.