KeyChest blog | KeyChest

Why HTTPS Matters for Busy Folks

This is my second blog post explaining the concepts of HTTPS. I will focus on the importance of HTTPS and how it affects the internet.

This is my second blog post explaining the concepts of HTTPS. I will focus on the importance of HTTPS and how it affects the internet.My previous text <a href="https://keychest.net/stories/understanding-pki-for-busy-folks" target="_blank">PKI for busy folks</a> looked at the concepts of certificates and PKI. While it attempts to simplify, it still contains loads of technical aspects that you should not need to know. Here I will focus on the role of different parties and how they can change our online behavior.<h2>The Role of Web Browsers</h2>If we want to use the internet, we need a web browser. There are a relatively <a href="https://en.wikipedia.org/wiki/Usage_share_of_web_browsers" target="_blank">small number of web browsers</a> that have the power to force changes in our behavior. <ul><li>65% - Chrome / Google</li><li>16% - Safari / Apple</li><li>4% - Firefox / Mozilla </li><li>3% - Samsung Internet / Samsung</li><li>3% - UC Browser / UCWeb</li><li>4% - Edge, Explorer / Microsoft</li><li>2% - Opera</li></ul>It is ultimately the web browser that will decide whether a given website is safe and secure or whether it will show a danger page. <div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Mozilla browser detected an insecure website." src="/storage/storage/wink/images/4FLznbhXLXHf1rJSZ5ewAIKsKYevCOXPXP5mp8Q1.png">Mozilla browser detected an insecure website.</div>If a web browser does not recognize an HTTPS certificate or when the certificate is invalid (e.g., expired), it will prevent you from accessing the website.<h2>What About Mobile or Internet Apps </h2>The purpose of many mobile apps is to simplify the use of the internet. They depend heavily on the operating system (iOS, Android, etc.) to help with the internet connectivity. In my experience, most apps don't expect invalid HTTPS and can't handle it. If that happens, e.g., because of an expired certificate, the app fails to work correctly. I wrote about one of <a href="https://magicofsecurity.com/dash-cashless-design-and-operation-https/" target="_blank">my experiences with cashless parking</a> a couple of years back.<h2>How does web browser recognize valid encryption (HTTPS)</h2>Web browsers depend on lists of "trusted third parties" - also called Certification Authorities. It is a list of certificates that are trusted implicitly by the web browser and/or by the operating system.<a href="https://gs.statcounter.com/os-market-share" target="_blank">Market share of OS</a>: 38% Android (Google), 36% Windows (Microsoft), 15% iOS (Apple), 9% OSX (Apple), 1% Linux.These lists are controlled by the vendors of MS Windows / OS X, Linux, Android - the platform of your phone or computer. This can be further restricted by the browser.If, for example, Google decides that one of the trusted parties is not good anymore, it will be included in the next upgrade. As a result, all websites using that third party will be shown as insecure in all Chrome browsers. That happened to <a href="https://security.googleblog.com/2018/03/distrust-of-symantec-pki-immediate.html" target="_blank">Symantec back in 2018</a>.<h2>The Role of Search Engines - SEO</h2>Major search engines have recently started using the presence of HTTPS on web sites as an indicator in their ranking algorithms. This includes Google search.It has a fundamental impact on all of us with an internet presence. If we don't use HTTPS, our websites are ranked below our competitors. That means less traffic, less business, and lower revenue. <h2>"Insecure Content" Message </h2>HTTPS is a must for websites that combine their own content with content provided by other websites or web applications. If you have an online chat button, contact form, or a button to a payment service, all of them have to have HTTPS. If any of this third-party content is insecure, the whole website may become inaccessible. The most popular browser Chrome <a href="https://blog.chromium.org/2019/10/no-more-mixed-messages-about-https.html" target="_blank">will start blocking such websites possibly from July 2020</a>.<div class="inline_html" contenteditable="false"><div style="padding: 56.25% 0 0 0; position: relative"><div style="height:100%;left:0;position:absolute;top:0;width:100%"><iframe height="100%" width="100%;" src="https://embed.wave.video/b388cf5e8d32d5da80a00eb4" frameborder="0" allow="autoplay; fullscreen"></iframe></div></div></div> <h2>Does HTTPS Impact Resilience of Internet?</h2>The short answer is: yes it does. While there are millions of websites, and thousands of internet providers, there are only a handful of web browsers and scores of Certification Authorities. I believe that HTTPS has become the most important aspect, or risk, in the internet resiliency. To demonstrate, back in February, <a href="https://keychest.net/stories/lets-encrypt-revokes-3000000-certs" target="_blank">Let's Encrypt revoked 3,000,000 certificates</a>. While the actual impact was much smaller - only thanks to the timing of events, this incident had the potential to take that many websites and internet services down with less than 24 hours notice.<ul><li>If a web browser stops accepting a particular Certification Authority, it can impact millions of websites using that Certification Authority.</li><li>If a Certification Authority revokes your certificate, your website becomes instantly inaccessible.</li><li>If a Certification Authority revokes its own certificate, all websites using that certificate will become instantly inaccessible.</li></ul>In terms of the market share, "Let's Encrypt" is a company that provides HTTPS for 50-60% of the internet. It has <a href="https://en.wikipedia.org/wiki/Let%27s_Encrypt" target="_blank">13 staff</a> and its budget in 2019 was around $3 million. In other words, this small not-for-profit company has the power to take down 60% of the internet by revoking their certificates.

Publication Date: 2020-06-02 16:37:00

Understanding PKI for busy folks

Public-key infrastructure (PKI) is a term for everything that has to do with web encryption beyond. This is a list of main terms to understand what it is and how it works.

Public-key infrastructure (PKI) is a term for everything that has to do with web encryption beyond. This is a list of main terms to understand what it is and how it works.<h2>Public-key cryptography</h2>Public-key cryptography uses keys of two parts: public key and private key. You need your own pair to "do" things. You only need public keys of others to "validate" things. The main problem is to widely share public keys while ensuring they are linked to their owners.Public keys can do 2 things:<ul><li>Encrypt a message so you can send it securely to the owner of the private key.</li><li>Verify signature of a message with the private key.</li></ul>Two most common algorithms are <a href="https://en.wikipedia.org/wiki/RSA_%28cryptosystem%29" target="_blank">RSA</a> (used for both) and <a href="https://en.wikipedia.org/wiki/Elliptic_Curve_Digital_Signature_Algorithm" target="_blank">ECDSA</a> (only for signatures).Public-key cryptography is slow and it also increases the size of the data. For these two reasons, nearly all protocols (such as <a href="https://en.wikipedia.org/wiki/Transport_Layer_Security" target="_blank">TLS</a> or <a href="https://en.wikipedia.org/wiki/Secure_Shell" target="_blank">SSH</a>) only use it to establish a secure channel with a "handshake". This creates a shared symmetric key that encrypts the actual data (e.g., with the <a href="https://en.wikipedia.org/wiki/Advanced_Encryption_Standard" target="_blank">AES</a> algorithm). <h2>Hashing</h2>Hashing algorithms (such as <a href="https://en.wikipedia.org/wiki/Secure_Hash_Algorithms" target="_blank">SHA</a>) are one-way functions that take any input and compute a unique fixed-size output. The output is called a hash (or sometimes digest). Hashing provides 2 main properties:<ol><li>it is impossible to re-create data from a hash</li><li>it is hard to find another data with the same hash. </li></ol><h2>Signatures</h2>Signatures authenticate messages (or people and computers). To sign data you would:<ul><li>Take a message, and if it's too long calculate a hash of the message.</li><li>Create a random "padding" that you add to the message/hash to make it random.</li><li>Use the private key to calculate a signature.</li></ul>To verify a message, you only need the signature and the public key.As using hash is almost universal, that's why you can see SSL/TLS versions with names that contain 2 or 3 algorithms: "ECDSA with SHA-512."<h2>Certificates</h2>The problem of public keys is to validate that they belong to a particular entity, that's why we use "certificates". Certificate contains a name and a public key and these are signed by a "trusted party". It also contains time validity that makes certificates such a pain.In the internet, the trusted party is called a certificate authority (CA). The CA is run by a company, like GeoTrust or Let's Encrypt, that has to comply with security and financial requirements of the <a href="https://cabforum.org/" target="_blank">CA/Browser Forum</a> . With internal PKI, you are free to create your own CAs and define mechanisms to share their public keys.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="KeyChest renewal management and monitoring - https://keychest.net" src="/storage/storage/wink/images/t8pUQFGVnsCoJzTzc3x5RbVEPP0WXufMXIn0eH3F.png">KeyChest renewal management and monitoring - https://keychest.net</div>A CA's certificate can be signed by another CA, and so on -this creates a certificate chain. The last certificate in the chain is called a root certificate. Root certificates are trusted and stored locally. For publicly trusted CAs, browsers and operating systems include public keys of root CAs that have been audited by the CA/Browser Forum. That's why we can trust websites that we visit for the very first time.<h3>Formats</h3>Most often when people talk about certificates, they're referring to <a href="https://en.wikipedia.org/wiki/X.509" target="_blank">X.509</a>. It's a name of a family of standards that has been followed-up by a number of internet standards - RFCs.The native X.509 certificate encoding is binary and described with the <a href="https://en.wikipedia.org/wiki/Abstract_Syntax_Notation_One" target="_blank">ASN.1</a> syntax. Windows usually work with the <a href="https://en.wikipedia.org/wiki/X.690#DER_encoding" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">DER</a> encoding of ASN.1. However, for practical purposes, ASN.1 can be wrapped with a number of trivial encodings - e.g., PEM encoding used in linux systems (base64 with a header and footer line).<h3>Verification</h3>Certificate verification consists of making sure the certificate chain is valid and leads to a trusted root certificate. The validation is a rather complex procedure that checks:<ol><li>signatures in the chain all the way to the root certificate;</li><li>time validity;</li><li>critical extensions included in certificates;</li><li>online lists of revoked certificates - CRL;</li><li>online, real-time validation responders - OCSP.</li></ol><h3>Bundling</h3>Since verification requires the complete chain, certificates are often distributed as a bundle. In the case of the TLS protocol, the chain is sent during the <a href="https://tools.ietf.org/html/rfc8446#section-4.4.2" target="_blank">handshake</a>.<ol><li>DER/BER (i.e., binary version of certs) are wrapped in a special PKCS7 or even PKCS12 files.</li><li>PEM formatted certs are simply concatenated into one text file.</li></ol>To use new certificates, you also need a private key. You can store certificate bundles and the private key in a single  <a href="https://en.wikipedia.org/wiki/PKCS_12" target="_blank">PKCS #12</a> (aka. PFX) file. PEM certificates will be accompanied with a separate file that will contain the private key.<h3>Issuance</h3>When applying for a certificate:<ol><li>you need to create your public and private keys;</li><li>create a <a href="https://en.wikipedia.org/wiki/Certificate_signing_request" target="_blank">certificate signing request</a> (CSR), which contains your name and public key and is signed by your private key (a self-signed data);</li><li>send the CSR to the CA</li><li>the CA will validate that the name inside the CSR is yours (e.g., by sending an email to your domain name address, checking DNS, checking Yellow Pages, giving you a phone call);</li><li>if everything looks good, the CA generates a certificate from the CSR.</li></ol>Depending on the thoroughness of step 4, you can get a "domain validated" (DV) certificate, "organization validated" (OV), or "extended validated" (EV) certificate. The more thorough validation, the more expensive is the cert.EC certs used to be shown with special green bars and company names in browsers. Since last year, there is no visual distinction between OV and EV certificates.For internal PKI, you can do step 4 whatever way you want. <h3>Issuance Let's Encrypt</h3>Let's Encrypt defined its own protocol for automated issuance of certificates - <a href="https://en.wikipedia.org/wiki/Automated_Certificate_Management_Environment" target="_blank">ACME</a>. Its flow is different from steps above:<ol><li>client sends domain names for which it requires a certificate;</li><li>the CA will send back "validation secrets" that the client has to either add to DNS records, or make visible via an HTTP server;</li><li>the CA, once it check the validation secret has been published, will send an authentication secret that is valid for up to 4 weeks;</li><li>the client generates a CSR, adds to it the authentication secret and sends it to the CA;</li><li>the CA checks the names match the list from step 1 and issues a certificate. It sends a download link back to the client.</li></ol>Let's Encrypt starts being used by companies and so we cover this particular certification authority in more details:<ul><li><a href="https://keychest.net/stories/what-are-disadvantages-of-lets-encrypt" target="_blank" style="color: rgb(51, 85, 187);">What are disadvantages of Let's Encrypt</a></li><li><a href="https://keychest.net/stories/lets-encrypt-revokes-3000000-certs" target="_blank">Let's Encrypt Revokes 3,000,000 Certs</a></li><li><a href="https://keychest.net/stories/lets-encrypt-is-free-are-you-the-product" target="_blank" style="color: rgb(51, 85, 187);">What does Let's Encrypt sell? It's not certs.</a></li><li><a href="https://keychest.net/stories/lets-encrypt-for-companies-with-keychest" target="_blank" style="color: rgb(64, 64, 200);">Let’s Encrypt for Companies with KeyChest</a></li></ul><h3>Revocation</h3>As certificates can be valid for up to 3 years (and it's not unusual for CA certificates to be valid 20 or more years), we need a mechanism to revoke certificates in case the private key is compromised.There are 2 ways to advertise revocations: <ol><li><a href="https://en.wikipedia.org/wiki/Certificate_revocation_list" target="_blank">certificate revocation lists</a> (CRLs) - a list of certificates revoked and not yet expired;</li><li>online certificate validation (<a href="https://en.wikipedia.org/wiki/Online_Certificate_Status_Protocol" target="_blank">OCSP</a>) - a protocol that allows inquiring about a specific certificate. In many cases, OCSP is simply and API using CRLs as its source data.</li></ol>OCSP can be more accurate but needs the 24x7 availability of OCSP servers. It also adds to the latency of website loading and so web browsers can skip these validations.<h2>Summary</h2><ul><li>Server's public key allows us to verify authenticity of the server.</li><li>Browsers use certificates to verify owners of public keys that we see for the first time.</li><li>The internet trust is defined and audited by CA/Browser forum (CA/B).</li><li>We can trust internet certificates only as we trust Google, Apple, Mozilla, Microsoft, etc.</li></ul><div class="inline_html" contenteditable="false"><a href="https://keychest.net/auditnow?url=teams.microsoft.com"> <img alt="Instant domain expiration audit" src="/storage/storage/wink/images/4FThBvWAw7KgmINIzlR1AJQimP9pQYXn5we2yecB.png"> </a></div><blockquote><a href="https://keychest.net" target="_blank">KeyChest is a cloud service to manage expirations and integrate Let's Encrypt, Internal, and commercial certificate management.</a></blockquote>

Publication Date: 2020-03-14 20:08:00

How to Keep Covid-19 From Killing Remote Access

The next 3-6 months will see large numbers of people off work. We can already see a huge uptake of remote working but that depends on the IT infrastructure and that depends on valid SSL certificates.

The Coronavirus can't be stopped and the implications are quite clear: the next 3-6 months will see large numbers of people off work, and we can already see a huge increase in remote working—which depends entirely on the IT infrastructure working. As a recent Let's Encrypt incident showed, HTTPS represents the ultimate risk to remote working.Governments are engaged in a balancing act of shifting the peak infection rate, investing in vaccine research, and protecting the most vulnerable people. IT managers start to face critical business dilemmas. Are we going to survive when 30% of our workforce is out sick for 4 weeks or more; when the system administrators can’t come to data centers? Are you sure you have everything under control and systems can run for 4-6 weeks without physical access?<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Remote access, just like everything else today, depends on certificate renewals." src="/storage/storage/wink/images/eugrnM0qsbVISb90p20uvuRZrKuQ1De6YPelOobK.png">Remote access, just like everything else today, depends on certificate renewals.</div><h2>Remote access, just like everything else today, depends on certificate renewals</h2>Most of the things can wait - vulnerability patching, upgrades - but several recent incidents showed that certificate management will not wait. When a single certificate expires, it can take down the whole of cellular / mobile networks, remote access systems, or our revenue-generating business applications.<h3>Incident 1: The U.S. Government Shutdown</h3>Last year, the US government shut down for 35 days. Online systems were switched off to show messages like “due to the government shutdown this service is not available”. However, in a couple of weeks' time, we couldn’t even read the message as online portals didn’t renew their certificates and browsers stopped trusting them. <h3>Incident 2: Let’s Encrypt Revokes Three Million Certificates</h3>Last week, the Let’s Encrypt certification authority had to renew up to 3,000,000 certificates due to a bug in its validation process. All that because of about 500 certificates that made use of the feature with the bug. They eventually stalled revocations because they feared a global impact on the internet. You may remember that the largest such “recall” impacted 23,000 certificates and everyone talked about it.<h3>Incident 3: 30 Million People Lost Cell Phone Access for a Day</h3>In December 2018, 30 million people lost their cell phone connectivity. <a href="https://www.theregister.co.uk/2018/12/06/ericsson_o2_telefonica_uk_outage/" target="_blank" style="color: rgb(0, 82, 204);">Ericsson said</a> an expired software certificate caused the outage that left <a href="https://www.theregister.co.uk/2018/12/06/o2_fries_burning_data_for_32_million_brits/" target="_blank" style="color: rgb(0, 82, 204);">tens of millions in the UK</a> unable to call or text from their mobile phones, nor use 4G connections, on Thursday.  <blockquote>... downtime was due to an expired certificate in a version of its management software used by European telcos to provide services to subscribers.</blockquote><h2>Improving Security Means Total Dependence on Certificates</h2>What it shows is that the whole of the internet is now absolutely dependent on something that "improves security". <blockquote>Certificates are like security badges to get into the gate at work. If yours is expired, you won’t be getting in. You have to come to me every 3 months to get a new one. Now imagine the gateway is access to all of the internet. Uptime for business apps, remote desktops, e-commerce sites, all precariously balanced on whether your certificates are properly managed.</blockquote>The main difference from internet certificates is that there are many companies who sell certificates. All companies, however, face three big issues:<ul><li>visibility</li><li>skills shortage</li><li>the complexity of HTTPS and network encryption</li></ul>Which brings us back to this current pandemic of Covid-19 and its huge impact on the skills shortage, which in turn will significantly lower the visibility of certificate expiration. How many of you have always-available, up-to-date information about your certificates and their expirations? How many of you depend on your sysadmins and what they have in their heads or what chances to land in their personal mailboxes? How many of you know what certificates are needed for your core business applications? And for your remote access and collaboration tools?<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Source: AppViewX survey of BlackHat 2019 participants" src="/storage/storage/wink/images/MyoCtAR5l7T2FCGub235Uy2m7AA5Yo8xZLVgIwV6.png">Source: AppViewX survey of BlackHat 2019 participants</div>These questions are important at all times but the current situation exacerbates the situation and we all need to focus on the visibility of issues that will have to be dealt within the next 4 weeks or so. Whether we like it or not, SSL certificates have hard expiration dates and they often silently kill IT applications without warning, at any time of the day.<h2>Get Started</h2>Keep your certificates from sabotaging your remote work environment. Try our instant domain audit.<div class="inline_html" contenteditable="false"><a href="https://keychest.net/auditnow/?url=teams.microsoft.com" style="text-decoration-line: none; color: rgb(64, 64, 200);"><img alt="" src="https://keychest.net/storage/storage/wink/images/EuU5VPUNsk1ncax3qB4PMHXEAFgbknC4Yu70MjHO.png"></a></div>Or you <a href="https://keychest.net/register" target="_blank" style="color: rgb(0, 82, 204); background-color: rgb(255, 255, 255);">can sign up</a> and manage all of your certificates with KeyChest. Hopefully, this pandemic will end up milder than we forecast but we all should be ready for a reasonable worst-case scenario.

Publication Date: 2020-03-13 08:23:00

Fighting the 'Good' Internet War

I occasionally re-read a paper I wrote with George Danezis in 2008. It is still true and in some sense, the way we fight cybercriminals has moved towards the approach we suggested then. In particular towards "active defense".

We propose strategies for defenders to regain the initiative and push security solutions far beyond the reach of current security tools – yet those strategies start mirroring the actions and technologies of the bad guys, and confront us with important technical, legal and moral dilemmas.<blockquote>“He who fights with monsters should be careful lest he thereby become a monster” -- Friedrich Nietzsche</blockquote>I occasionally re-read a paper I wrote with George Danezis in 2008. It is still true and in some sense, the way we fight cyber criminals has moved towards the approach we suggested then. One particular aspect as an "active defence" that was not really actively considered back then. A PDF is available <a href="https://www.researchgate.net/publication/221291282_Fighting_the_'Good'_Internet_War" target="_blank">here</a> and here's a couple of sections as a teaser.<h2>Tactics</h2>We believe that tactics used to combat malicious parties on the Internet are always a step behind and a strategy does not really exist. However, Jomini stated several centuries back that the key to warfare is strategy. Let us start with several notes from history.<ul><li>There is an asymmetrical relationship between attack and defense. One should try to reverse the asymmetry whenever possible. Attacks have several decisive advantages of attack: surprise, the benefit of terrain, concentric attack, popular support, and moral factors.</li><li>Divide and conquer, a particular tactic that was further developed e.g. by Matyas Rakosi for Hungarian Communist Party in the late 1940s as a salami tactic uses alliances to increase political power.</li><li>If you use common sense, you are inevitably doing a bad strategy choice.</li><li>The inner line of operations, i.e. operations inside the enemy’s army will allow for fighting separate parts of the enemy’s forces.</li><li>Mobilization of the whole nation forces the enemy to “defend the area” and we can pick the right time and the right place to fight battles.</li></ul><div class="embedded_image" contenteditable="false" data-layout="default"><img alt="" src="/storage/storage/wink/images/yToRsX5ECYmHMFJYtKwqwweDIuNLtOKBMgk0n1St.jpeg"></div><blockquote>Yet the most neglected advice when it comes to Internet conflict, is the focus on offensive. Defenders make no attempt to ‘reverse the asymmetry’ and instead believe that security shall come by digging deeper trenches around the few secured hosts. This gives the adversary full strategic advantage to attack when, where and how she wishes. </blockquote><h2>The Power of Information</h2>Many commanders have quickly realized the power of information and careful planning. Interestingly, there are more rules related to concealment of own actions and strengths. <ul><li>Hiding the real purpose of actions is an important element in the strategy. </li><li>Conceal your plans, or plan for several steps ahead.</li><li>One should not engage the enemy in combat or show their strength before learning the enemy’s intentions. </li><li>One should prevent hostile reconnaissance and thereby conceal the second line of their forces. </li></ul><blockquote>Internet: There is warfare already taking place and so we can learn a bit about our enemy and study their behavior. We can design new tools and use them in the war but no-one is doing any plans. We are fighting isolated battles and losing the war. </blockquote>The plans are very easy to read if anyone bothered to do that, as the threat posed by them is very small. The organization of the enemy consists of a head, support groups (organized for specific crimes), and working units. The working units will cover the following activities: vulnerability discovery, exploit design, spam management, managing DNS records, coding, web site building and managing, managing botnets, sales agents. <blockquote>The most activities are offline and the enemy goes on-line only to manage the botnets and web sites, and sales agents. It is also possible to detect on-line malicious activities – the actual attacks. We can learn from the way the bot-nets are commanded and organised, but our tools must be equally stealthy so they cannot become easy targets in the wars of bot-nets. </blockquote> <blockquote><a href="https://keychest.net/monitoring" target="_blank" style="color: rgb(51, 85, 187); background-color: rgb(255, 255, 255);">You have better things to do, Let KEYCHEST look after your certs. KeyChest with its global database of web certificates can instantly create an initial "big picture" so you can start analyzing your exposure to cyber attacks and adjust it according to your risk appetite.</a></blockquote>

Publication Date: 2020-03-05 23:06:00

Browser Updates To Kill 850,000 Web Sites

All major web browsers - Safari, Mozilla, Chrome, and Edge - will disable support of TLS 1.0 and TLS 1.1. The old and insecure versions of SSL protocols.

The days of old TLS versions are nigh. All major web browsers - Safari, Mozilla, Chrome, and Edge - will disable support of TLS 1.0 and TLS 1.1. The old and insecure versions of SSL protocols.Browsers will phase out the old versions this and next months. According to <a href="https://news.netcraft.com/archives/2020/03/03/browsers-on-track-to-block-850000-tls-1-0-sites.html" target="_blank">Netcraft</a> and <a href="https://www.zdnet.com/article/browsers-to-block-access-to-https-sites-using-tls-1-0-and-1-1-starting-this-month/" target="_blank">ZD Net</a>, this change will affect websites for major banks, governments, news organizations, telecoms, e-commerce stores, and internet communities.This step is a planned change that was <a href="https://www.entrustdatacard.com/blog/2018/november/deprecating-tls" target="_blank">announced back in November 2018</a>. At the time the statistics of the SSL versions were as follows:<ul><li><a href="https://webkit.org/blog/8462/deprecation-of-legacy-tls-1-0-and-1-1-versions/" target="_blank">Safari</a> - TLS 1.2 was used on 99.6% of connections. TLS 1.0 and 1.1 planned for March 2020.</li><li><a href="https://security.googleblog.com/2018/10/modernizing-transport-security.html" target="_blank">Chrome</a> - 99.5% of connections with TLS1.2 or TLS1.3. Warnings from version 72, disabling planned for January 2020.</li><li><a href="https://blogs.windows.com/msedgedev/2018/10/15/modernizing-tls-edge-ie11/" target="_blank">Edge and Explorer 11</a> - planned to disable TLS 1.0 and 1.1 in the first half of 2020.</li><li><a href="https://blog.mozilla.org/security/2018/10/15/removing-old-versions-of-tls/" target="_blank">Firefox</a> - planned to disable support of TLS1.0 and TLS1.1 in March 2020.</li></ul>Upgrade from TLS1.1 to TLS1.2 allows using new encryption algorithms and deprecation of weak functions MD5 and SHA1. The latter have practical attacks where the reward is sufficiently high and where the attacked data has long-term validity (like SSL certificates). Having said that big companies are still using SHA-1 on their TLS-based servers, including Google who invested in SHA-1 attacks to encourage the use of new encryption methods.TLS1.0 contains a vulnerability allowing <a href="https://en.wikipedia.org/wiki/Transport_Layer_Security#TLS" target="_blank">downgrade of the protocol</a> to the insecure SSL3.0 (that was first implemented in 1996 still as a Mozilla "standard"). As a result, attackers can force the use of keys that can be broken in hours or less.What will practically happen? Browsers will start showing full-page warnings - similar to expired certificates and prohibiting access to servers with old versions of TLS. This means that around 5,000 of the top 1,000,0000 web sites still using old protocols will become inaccessible unless they complete an upgrade in the coming days.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Full page warning in the new Mozilla browser." src="/storage/storage/wink/images/drQjXmlwizkGOA0UzWJUBrp14fkU5yO0CLkQCWch.png">Full page warning in the new Mozilla browser.</div>The most recent update clarifies the dates of complete deprecation of TLS1.0 and TLS1.1, which will happen in the next upgrades of Chrome (version 81) and Firefox (version 74). Releases of these versions are scheduled for later this month. Safari will follow suit this month as well, although no recent update has been given. Microsoft will join other browsers with its browser Edge (version 82) at the end of April.<blockquote><a href="https://keychest.net/monitoring" target="_blank" style="color: rgb(51, 85, 187); background-color: rgb(255, 255, 255);">You have better things to do, Let KEYCHEST look after your certs. KeyChest with its global database of web certificates can instantly create an initial "big picture" so you can start analyzing your exposure to cyber attacks and adjust it according to your risk appetite.</a><a href="https://keychest.net/monitoring" target="_blank"> </a></blockquote>

Publication Date: 2020-03-05 11:38:00

Let's Encrypt Revokes 3,000,000 Certs

Bottom line - if your certificates are affected and you will not renew and deploy new within hours, you will have effective downtimes - certificates will be revoked and invalid.

Bottom line - if your certificates are affected and you will not renew and deploy new certs within hours, you will have effective downtimes - certificates will be revoked and invalid. The estimated total is 3 million, of which 1 million are duplicates.Let's Encrypt celebrated its success last week only to revoke 2.5% of all its valid certificates in the next few days. <a href="https://community.letsencrypt.org/t/revoking-certain-certificates-on-march-4/114864" target="_blank">Check the Let's Encrypt announcement</a> for advice to check your certificates and how to react, as there seems to be some confusion in determining which certificates will be revoked.The reason is a bug in their validation process. It caused insufficient validation of certificates with 2 or more domain names. While the chance of this being actively exploited is small, they have to revoke all affected certificates within 1-5 days according to their security compliance procedures.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Revocation of Let's Encrypt certificates will case many websites showing this message." src="/storage/storage/wink/images/2UjXDWP29EeCUMz7VHQBFLqTDRRnm7rN7m4eFoAq.png">Revocation of Let's Encrypt certificates will case many websites showing this message.</div>It is a massive issue as renewals require "force" flag, which may need manual renewals. The usual automation ignores renewals of certificates that have more than 30 days of validity left.<h2>The Technical Cause</h2>One of the steps in certificate request validations is a check of DNS records (DNS CAA) that may contain information about approved CAs. You can use your DNS records to prohibit certain CAs issuing certificates for your domain names.You may wonder why such a technical detail can cause revocation of more than 3,000,000 certificates. The answer lies strictly in the compliance of CAs with rules for issuance of web certificates. Revocations should be done within 24 hours and it has to be completed within 5 days. Very rarely, however, you need to revoke such a large volume of certificates.In 2018, <a href="https://www.theregister.co.uk/2018/03/01/trustico_digicert_symantec_spat/" target="_blank">Symantec had to revoke </a>23,000 of certificates after private key compromise and the case caused a lot of problems as well as press coverage.This time, the issue is much, much bigger. The <a href="https://community.letsencrypt.org/t/revoking-certain-certificates-on-march-4/114864" target="_blank">official announcement</a> is at the Let's Encrypt Community forums. <blockquote>Due to the <a href="https://community.letsencrypt.org/t/2020-02-29-caa-rechecking-bug/114591" target="_blank" style="color: rgb(0, 136, 204);">2020.02.29 CAA Rechecking Bug </a><a href="https://community.letsencrypt.org/t/2020-02-29-caa-rechecking-bug/114591" target="_blank" style="color: rgb(145, 145, 145); background-color: rgb(233, 233, 233);">4.3k</a>, we unfortunately need to revoke many Let’s Encrypt TLS/SSL certificates. We’re e-mailing affected subscribers for whom we have contact information.</blockquote>One of the problems is that many users don't share their email addresses with Let's Encrypt, or the email addresses are not valid any more. For example, when a sysadmin left a company.<h2>Automation v Reliability</h2>This is not the first case when Let's Encrypt users were affected by changes to Let's Encrypt "configuration". In 2017, Let's Encrypt started using IPv6 when available. It lead to issues with many users who had IPv6 entries in the DNS but they have not been correctly configured, while managed by ISPs in some cases.Let's Encrypt also implemented changes to its API last year and is in the process of switching off the previous version. This change is "breaking", which means that users have to change the clients and integration they have for Let's Encrypt certificate renewals.These situations are not unusual, although the size of the latest incident is unprecedented. It shows that "just install and it works" is not quite the appropriate approach if you the reliability is important. Questions focused on "what if X breaks" are suddenly more important than business-as-usual.<blockquote><a href="https://keychest.net" target="_blank">You’ve got better things to do than worry about web certificates. Try KEYCHEST.</a></blockquote>

Publication Date: 2020-03-03 23:14:00

What does Let's Encrypt sell? It's not certs.

Last week saw huge coverage and celebration of Let's Encrypt, a free certificate authority that issued its billionth free certificate on Feb 27. But "if you are not paying for it, you're not the customer" ... are you the product being sold?

Last week saw huge coverage and celebration of Let's Encrypt, a free certificate authority that issued its billionth free certificate on Feb 27. But "if you are not paying for it, you're not the customer" ... are you the product being sold?This is a quote from the official Let's Encrypt announcement:<blockquote>One thing that’s different now is that the Web is much more encrypted than it was. In June of 2017, approximately 58% of page loads used HTTPS globally, 64% in the United States. Today 81% of page loads use HTTPS globally, and we’re at 91% in the United States! This is an incredible achievement. That’s a lot more privacy and security for everybody.</blockquote> Let's Encrypt now protects almost 200 million domain names. As there are around 360 million registered domain names, Let's Encrypt has become a monopoly for HTTPS, with its certificates installed on more than 60% of all internet websites. As it is a not-for-profit organization, very few ask the question of its business model. <h2>Who Is Let's Encrypt</h2>Let's Encrypt is controlled by <a href="https://www.abetterinternet.org/" target="_blank" style="color: rgb(14, 16, 26);">ISRG</a> aka "A better internet". An organization founded by Mozilla, EFF (Electronic Frontier Foundation), the University of Michigan, Cisco, and Akamai. Its budget is today made up of donations from a large number of sponsors but 40% of all the funding comes from "platinum and gold level sponsors" - a much smaller group that consists of Mozilla, CISCO, OVHcloud, EFF, Chrome, Facebook, and Internet Society, IdenTrust, and The Ford Foundation. These top sponsors also control the board of directors as well as the technical advisory board. The time and travel costs of the board members will be paid for from the internal budgets of their employers.As much as I believe in charitable causes, no company gives money for nothing - there is always some kind of return on investment. It could be improving the brand value, but I haven't come across adverts that would say "proudly funding Let's Encrypt". Some want to use certificates for free - but really, only OVH does so. Google has its own CAs, CISCO doesn't really need them, Mozilla either and its website is Digicert protected, just like Facebook. Let's have a look at another name from this list of companies - Akamai. Akamai is one of the largest CDN providers. If you think about what CDN does, you start wondering about the role of HTTPS. HTTPS is end-to-end encryption so if I encrypt end-to-end connections to my bank, how can Akamai cache and speed-up delivery of the bank's website across the globe? The answer is simple - Akamai is man-in-the-middle. It decrypts the bank website data and sends its copy of the data pretending to be the bank. The trouble is that to provide this speed and global coverage, Akamai needs to have private HTTPS keys of all its customers (e.g. the bank in this example) readily available. The way they do it is to keep all the private keys in plaintext files that are copied wherever they are needed. CDN providers represent 15-30% of internet traffic, which means that up to 30% of end-to-end encryption is not quite end-to-end. <h2>What does HTTPS actually do?</h2>When I worked for the University of Cambridge as a researcher, the security of the Internet was a frequent topic. I don't recall significant calls for "let's encrypt the whole of the internet". The debates were more around the effects of HTTPS. Do people notice anything like the color of the address bar (or even whether there's a small padlock) - frankly no one cares even today. Any such discussion today is obsolete as all cybercriminals have learned how to use and exploit HTTPS."Let's encrypt the whole of the internet" has some good justification but that was not enough to "sell the idea". There is a lot of misconception around what HTTPS can do. It's a little bit like "blockchain" that can allegedly solve almost all the problems of the world.Let's try an example - we need HTTPS for secure use of APIs and embedding of third-party content on websites. So if I have my own company website: <a href="https://keychest.net" target="_blank" style="color: rgb(14, 16, 26);">https://keychest.net</a>, I can add a payment button - let's say Stripe - securely. Indeed, browsers will not allow the use of insecure HTTP from HTTPS websites - great. However, as any criminal can create an HTTPS service the added value is close to zero - encryption didn't solve the problem, we need something else! So how did we solve this "secure embedding" problem? Web servers can now provide a list of domains that can be embedded in their websites (<a href="https://en.wikipedia.org/wiki/Content_Security_Policy" target="_blank" style="color: rgb(14, 16, 26);">content security policy - CSP</a>). Nothing to do with HTTPS or encryption. It is a simple access control mechanism.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="A single Let's Encrypt certificate" src="/storage/storage/wink/images/gw0eOBYQ4uaJcxrcXtKtAiTb2Hr5Xwy2KtuVzxgr.png">A single Let's Encrypt certificate</div>The other day, I put a question to the sysadmin subreddit. I basically said that HTTPS was a universal pain for small companies while it was really useful only in particular situations, such as when we need to exchange data items that are sensitive: credit card numbers, personal data, work documents, etc. I was "told off", of course. It was interesting, however, what examples were given to justify the HTTPS. They all involved big internet companies. I mentioned the payment data as one of the most sensitive data that needs protection. Interestingly, anyone who processes payments is subject to PCI standards - something completely unrelated to HTTPS. PCI data processing requirements are pretty strict, mandate regular penetration testing by certified vendors and include pretty harsh penalties as well. As such, the global enforcement of HTTPS does not really have much of an impact on the security of this data.I'm not saying that HTTPS is a bad idea, what I am saying is that it empowers a few companies that control our access to the internet through their web browsers and it's not quite clear to me whether this authority is justified.<blockquote>Check the status of your web domain with our instant online domain audit tool.</blockquote><div class="inline_html" contenteditable="false"><a href="https://keychest.net/auditnow"><img alt="" src="/storage/storage/wink/images/EuU5VPUNsk1ncax3qB4PMHXEAFgbknC4Yu70MjHO.png"></a></div> <h2> What or Who Is The Product?</h2>Just to remind you - the product of the two of the biggest sponsors of Let's Encrypt is our user data. So one could stretch the argument and say that they use HTTPS to strengthen their exclusive access to our personal and behavioral data. But maybe the exploitation of the HTTPS could go even further. Chris Matyszczyk wrote a <a href="https://www.zdnet.com/article/google-says-microsoft-edge-isnt-secure-i-asked-why/" target="_blank" style="color: rgb(14, 16, 26);">post for ZDNet</a> "Google says Microsoft Edge isn't secure. I asked Google why". The core argument of this article is that Google warns users that using non-Google add-ons or applications is not secure. I don't think it makes much difference to many a user but the impact is different if you are a company building those applications. How much risk related to a browser warning your users about the security are you willing to take. Is the cost of using a particular certificate / app store too high? You can get what the answer would be. In a sense, HTTPS and security in general start being exploited for a wide range of commercial advantages. I wonder what is the place of Let's Encrypt in these online wars. I don't know the answer but I'm sure it's a long-term strategy game and only time will show. <h2>Do We Need - Encrypt The Whole Internet?</h2>It's interesting that mandatory HTTPS only really started once Let's Encrypt was up and running. As if Google and others were not so sure until people had to pay for certificates. Once Let's Encrypt was here, they thought it's fair to ask everyone to use HTTPS as certificates were "free". They are not really free - a hidden annual cost of a certificate is at least $40. If you're a geek at home with plenty of free "free" time on your hands you will disagree. For companies it is different - the cost is significant in their IT security budget. On top of that, any incident due to expired HTTPS costs revenue, real money. You can see a recent study of AppViewX, which we analyzed here, which confirms an earlier study of Venafi - 80% of companies experienced a related business outage. One significant impact I noticed with Google implementing HTTPS (2010) on their server was the loss of search details for my websites. You can now find this via Google tools but it's only available with 1-2 days delay. One might say that HTTPS allows Google to determine who's the real owner of this data.Not very many remember but the main <a href="https://www.wired.com/2010/05/google-https-search/" target="_blank" style="color: rgb(14, 16, 26);">Google argument</a> against HTTPS was its cost. "Google says it’s been working for several months on the service, and that it’s not simple". We talking about the company that was worth around $200 million then. In short, web giants must be aware of the cost that HTTPS puts on small companies and yet, one has to notice the lack of justification. HTTPS is given credit for a lot of things it simply does not do and many keep quiet about problems it introduced. The Hacker News just <a href="https://thehackernews.com/2020/02/lets-encrypt-ssl-certificate.html?m=1" target="_blank">published an article</a> that says:<blockquote>Capping certificate lifetimes improves website security, not least because it reduces the possibility of criminals stealing neglected certificates to mount phishing and malware attacks.</blockquote>Well - a typical phishing attack lasts hours or single days. That means that the phishers can even get a refund for their certificates as most providers give you a "one-month money-back guarantee"!Is there a downside to a monopoly created by a not-for-profit company? When you try to find economic studies there are not many relevant to the situation of Let's Encrypt. Firstly - the market of HTTPS certificates is a commodity market - hugely competitive with a large number of vendors working with small margins … but only if you buy in bulk. For example, one certificate can cost you $50 if you buy one and <$1 if you buy 10,000 of them. <h2>Can We Break Let's Encrypt Dominance?</h2>It is hard, like really hard. The main reasons are business reasons. ISRG decided to completely disrupt the certificate market. While <a href="https://letsencrypt.org/2020/02/27/one-billion-certs.html" target="_blank" style="color: rgb(14, 16, 26);">Let's Encrypt says</a> its annual budget is just over $3.3 million, it took around 5 years to its first certificate. The time is mostly down to regulatory hoops. It's harder and longer than creating a new bank!By disrupting the market, ISRG created a monopoly that is here to stay. Creating a new root certificate authority is expensive and takes 3-5 years. Who would be willing to invest into a 5-year project to compete with Let's Encrypt when the "product" is free? Right, so we have an excellent source of certificates, right? Well, if it's so good, why are none (ok, there's one exception - OVH) of the main sponsors using Let's Encrypt? On top of that, I have started hearing about cases where Google doesn't trust Let's Encrypt certificates - why would they do it? Is the budget and activities of Chromium really completely independent of the group providing G-suite?Having said that, Let's Encrypt is a bit tricky to use for companies. Let's Encrypt's justification of a short certificate life-cycle is not that persuasive and its main line is "automation allows us to shorten the validity and it makes the misuse of certificates harder" - an argument I queried above. The risk of expirations is simply too high and there are also additional technical <a href="https://keychest.net/content/letsencrypt_numbers_to_know" target="_blank" style="color: rgb(14, 16, 26);">constraints</a>. Although these can be relaxed on request.The bottom line is that we now have a monopoly for HTTPS that is not-for-profit but sponsored by big internet players. As such, it is difficult for me to get a clarity in terms of the real purpose of Let's Encrypt. As a result, I see the use of Let's Encrypt as a higher risk than using other commercial certificates. Especially if we can provide the latter for $5-6/yr.I have written a post about the <a href="https://keychest.net/stories/what-are-disadvantages-of-lets-encrypt" target="_blank" style="color: rgb(14, 16, 26);">disadvantages of Let's Encrypt</a>. I suppose the business risk of using a product of such a dominant vendor that doesn't have a clear business justification is good enough to be added to the list.<blockquote><a href="https://keychest.net" target="_blank">You’ve got better things to do than worry about web certificates. KeyChest is a certificate expiry management service with instant discovery of internet certificates - no software installation needed.</a></blockquote>

Publication Date: 2020-03-01 20:54:00

Let's Encrypt - 1bln, Time To Beat Others On Uptime

So Let's Encrypt issued a billionth certificate yesterday. It is an absolutely amazing number and I'm pretty sure no-one would have thought 5 years ago that any single CA can ever achieve this number.

So Let's Encrypt issued a billionth certificate yesterday. It is an absolutely amazing number and I'm pretty sure no-one would have thought 5 years ago that any single CA can ever achieve this number.<hr>It reminds me of YouTube - who would have thought that it's possible for a single video to be viewed more than 4 billion times - why would we need a counter bigger than 32 bits. Well, here we go. Well done to all at Let's Encrypt as well as their technical advisors who made sure that this will keep running and remain secure!The press release also mentions the size of Let's Encrypt and its fairly modest budget of less than $3 mln. My take from that is that you can run a company with a core business based on an API very efficiently indeed.<h2>UpTime Against Infrastructure</h2>But it's not been all roses and there is one thing that Let's Encrypt can get even better at - reliability. It's not so much about the actual software, but the infrastructure around that ensures that its service is reliably delivered to users all around the globe.Issues related to the infrastructure have been the main cause of downtimes and for example a switch to a new CDN provider in 2019 didn't go quite to the plan. <h2>Disruptions - First Number</h2><a href="https://keychest.net/stories/lets-encrypt-uptime-2-years-on" target="_blank">I used the same source</a> as the first time - <a href="https://letsencrypt.status.io" target="_blank">https://letsencrypt.status.io</a>. The records seem to be fairly reliable. However, it appears that on several occasions the time of an incident starts when a report is received rather than the time potentially provided by the reporting party.The history of the Let's Encrypt status over the last 12 months (20 Nov 2018 to 20 Nov 2019) contains 95 records, from which 77 relate to the production services. I ignore non-core services, e.g., web site.<h3>Planned Upgrades</h3>When you look at the Let's Encrypt status history, you will immediately notice a fairly large number of upgrades. There is in fact 46 Boulder upgrades, which sounds like an incredibly high number.<div class="inline_html" contenteditable="false"><div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Planned upgrades of Let's Encrypt production services. Multiple upgrades per month are stacked." src="/storage/storage/wink/images/kihG3ywBhQWZtu3L1OVdk2epH4UtjC7oW0SPbYFB.png">Planned upgrades of Let's Encrypt production services. Multiple upgrades per month are stacked.</div></div>As far as I could see, none of them caused any downtime and my main observation is that while the total number is surprisingly high, the team has certainly mastered the upgrade process and is able to quickly react to potential incidents. Below is a similar graph from 2017. While the number of upgrades is similar, it is clear that the time required for each upgrade as stabilised. Which means much better reliability of upgrades.<h3>Full Disruptions</h3>The most important aspect, from users' point of view, is the probability of full service disruptions. This directly impacts the overall reliability of certification services and it is one of the main selling points for commercial CAs. Many of them like to quote their 99.99% or higher uptime.Let's Encrypt has significantly improved over the last 2 years in terms of the number of incidents. Although it's still quite some distance behind commercial CAs (although they don't publish their incident data). You can see that the length of downtimes is relatively short. These include at least 3 instances of downtime related to maintenance - one infrastructure and two on the application level. The overall uptime in 2019 is 99.92%, which is better than the figure I found in 2017 (99.86 - with 99.9% over the second part of the interval analyzed then) - see below.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Full Disruptions in minutes (logarithmic scale)." src="/storage/storage/wink/images/bliH3OHIKQIdmsJ9eDyxSxg8XuFYn3T4y0ypTwBR.png">Full Disruptions in minutes (logarithmic scale).</div>The 2019 incidents include a couple of issues with OCSP responder. This is a fairly critical part of the LE services as it may result in unavailability of websites if browsers require OCSP validations.<h3>Partial Disruptions</h3>Partial disruptions occur when most of the Let's Encrypt services work but some components are unavailable. Data for these incidents are less transparent as the comparison is very blunt and lacks the detail of the impact of particular incidents. Nevertheless, I was somewhat surprised that the downtime was still relatively high compared to 2017.<div class="inline_html" contenteditable="false"><div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Partial Disruptions of Let's Encrypt in 2019. Multiple incidents per month are stacked." src="/storage/storage/wink/images/nlkIHJcM3mVd2ST2gG6REbRJJV3yIe82sPNOE4sG.png">Partial Disruptions of Let's Encrypt in 2019. Multiple incidents per month are stacked.</div></div>The combined uptime figure (all services fully functional) is only 96.4%, which is lower than what I found in 2017, when the figure stood at 98% (again, the 2017 chart is below).It's worth mentioning that the 2019 results are significantly affected by a couple of incidents that lasted several days. One of them was caused by a <a href="https://community.letsencrypt.org/t/lets-encrypt-uptime-comparing-2019-with-2016-17/107114/8?u=dancvrcek" target="_blank">migration to another CDN</a> (details provided by "_az"). If we exclude one of both of these partial outages, the uptime is 97.9% and 99.7% respectively.<h3>OCSP</h3>I have not charted OCSP outages separately for two reasons. Firstly, I can see only two of them (one full, one partial). Secondly, these outages where short - 5 and 7 minutes and it's not clear to me whether they cover the whole outage or whether the measured interval starts only when the LE team started investigating it.If we assume the logged times are accurate, the overall uptime of OCSP responders would be >99.99%.<h2>It's All Downhill From Here ... Well Done!</h2>LE's main feature is automation. While we generally don't mind if a certificate is renewed today or a week later, the relatively low uptime is a concern. Especially as there are no official service level guarantees.This "exercise" made me also wonder whether it makes sense to implement an uptime monitoring system that would provide an independent uptime data. LE issues around 1.3 million certificates a day so every one minute downtime impacts over 900 certificate requests. Similarly, 1 minute downtime of OCSP responders is significant as 50-60% of websites use Let's Encrypt certificates. Arguably, these websites will not include any of the large one but still.<blockquote><a href="https://keychest.net" target="_blank">This analysis was created by Dan Cvrcek, a founder of KeyChest - TLS expiry monitoring service with a global database of all certificates and automation of certificate issuance.</a></blockquote><div class="inline_html" contenteditable="false"><a href="https://keychest.net/auditnow"><img alt="" src="/storage/storage/wink/images/EuU5VPUNsk1ncax3qB4PMHXEAFgbknC4Yu70MjHO.png"></a></div>

Publication Date: 2020-02-28 08:16:00

25 Years of Internet Hijacking Nears Its End

The first step of securing BGP is to find out which BGP routes are valid. In the Asia Pacific & Africa region, this happened towards the end of last year, and internet operators validated 79% of routes in January 2020 (a big jump from 29% in November 2019). The process was not smooth though.

If you make an internet call from Sydney to Texas, a technology called Border Gateway Protocol, or BGP for short, will ensure that your computer will find your friend in Texas. Just like people have to go through a border gate (or customs if you are at the airport) to enter another country, internet users have to go through a gate to access anything on the internet in another country. Internet users have unknowingly used BGP for over 30 years. The problem is that this old technology leaves the border gate wide open for a special kind of cyberattack.<hr>The attack is a kind of hijacking because it pretends to be the border gate into one country, but users first have to go through another country to get there—without ever knowing it. So if you try to call your friend in Texas from your internet in Sydney, you may be directed through China first without ever knowing it!<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="" src="/storage/storage/wink/images/oMs5kuTStSVzNcDlJMKTYBUibGSx2jMD2uxDCBlq.png"></div><ul><li>The attacker then hijacks ALL of the information that passes through: things like usernames and passwords, bank card information, pictures, videos, and everything in between. BGP connects the networks of entire countries so any attack will impact millions of users.<a href="https://keychest.net/auditnow/?url=teams.microsoft.com" target="_blank"> </a></li></ul><div class="inline_html" contenteditable="false"><a href="https://keychest.net/auditnow"><img alt="" src="/storage/storage/wink/images/EuU5VPUNsk1ncax3qB4PMHXEAFgbknC4Yu70MjHO.png"></a></div><ul><li>In 2014, a Canadian internet provider executed 22 separate attacks, each lasting 30 seconds. They targeted BitCoin mining and the attacks were successful;</li><li>April 2017 - Russian telco Rostelecom hijacked 37 of autonomous domains, including Visa and MasterCard. The attack lasted 7 minutes but no-one was ever able to find out if any data or what data was compromised;</li><li>July 2018 - Iran hijacked traffic of Telegram Messenger; </li><li>November 2018 - China Telecom site in the US redirected Google for more than 1 hour and 20 minutes.</li></ul><h2>A Problem of True Identity</h2>How hard is it to hijack the traffic? It is relatively simple as there is no authentication of who announces the best route from Sydney to Texas. So if someone in Norway starts loudly shouting - "this is the quickest route now". The traffic will go through Norway. Someone in Brazil will start saying the same 10 minutes later, all the traffic will start going through Brazil.What BGP needs is authentication of those who “shout”… A sort of identity check at the gate. This new technology is call RPKI. "<a href="https://blog.cloudflare.com/rpki/" target="_blank">RPKI, uses a certificate system</a> that's akin to secure web browsing (or at-least its early days). While secure web browsing has moved on and is far more secure and is somewhat the default these days, the state of BGP route validation has not moved forward."The first step of securing BGP is to find out which BGP routes are valid. In the Asia Pacific & Africa region, this happened towards the end of last year, and internet operators validated 79% of routes in January 2020 (a big jump from 29% in November 2019). The process was not smooth with some big internet users.The stakes are high as those who do not cooperate will lose access to the Internet. Mapping valid routes is important because the rest will soon stop working. Big US ISPs like Google, Cloudflare and others will start dropping invalid connections in coming months. Some regions are behind including Australia and its biggest operator Telstra.What does the hijack mean? An analogy would be a diversion on a motorway  that would suddenly direct all the traffic to a military posts for inspection. What it means is that a "random" entity anywhere in the world (geographic location is not a barrier so it can be happening in China, France, or Mexico) can filter, analyze and extract everything that you send over the internet - be it backups of customer information, IP, personal data between your servers, or telephone calls from your laptop, or even from your mobile phone - all that data goes via the Internet.This can impact big companies as well as small ones. The best way to protect yourself is to use web encryption (SSL / HTTPS). It is an end-to-end encryption between computers. If someone hijacks this traffic, they may slow you down but the data will remain secure. The downside is that web encryption expires. This can impact your revenue as well as create a dangerous backdoor to your data.KeyChest manages your SSL for you. Set up our cloud service and then forget it. Restore confidence in your IT and make encryption a valuable business asset instead of a time-sucking drag. Check the status of your web domain with our instant online domain audit tool.<div class="inline_html" contenteditable="false"><a href="https://keychest.net/auditnow"><img alt="" src="/storage/storage/wink/images/EuU5VPUNsk1ncax3qB4PMHXEAFgbknC4Yu70MjHO.png"></a></div>

Publication Date: 2020-02-25 19:49:00

Apple Safari Not Trusting Long Certs from Sept 1

Apple believes that SSL/HTTPS certificates valid for more than a year are not secure enough. Change comes on September 1. What does it mean?

Apple believes that SSL/HTTPS certificates valid for more than a year are not secure enough. As such the Safari browser will not be trusting certs valid for more than 13 months. Change comes on September 1. What does it mean?<hr>You probably never heard of the CA/Browser Forum as it is one of those deeply technical groups that rarely gets to the front pages. This time it may be different as Apple has announced during its last meeting that their browser Safari will not trust HTTPS certificates valid more than 12 months. While it may sound like a technical detail, it will certainly make life of many small companies even harder.<div class="inline_html" contenteditable="false"><script type="application/ld+json"> { "@context": "https://schema.org", "@type": "NewsArticle", "mainEntityOfPage": { "@type": "WebPage", "@id": "https://keychest.net/stories/apple-drops-sslhttps-bomb-forget-long-certificates" }, "headline": "Apple Drops SSL/HTTPS Bomb - Forget Long Certificates", "image": [ "https://keychest.net/storage/storage/wink/images/OGmHESVOXUmHkcMibdERLLrHm3LT5Ry4dnDg6eOs.png" ], "datePublished": "2020-02-23T21:00:00+00:00", "dateModified": "2020-02-23T21:05:00+00:00", "author": { "@type": "Person", "name": "Dan Cvrcek" }, "publisher": { "@type": "Organization", "name": "KeyChest", "logo": { "@type": "ImageObject", "url": "https://keychest.net/storage/storage/wink/images/wRyJcM4Tfkw6I3KEJxXP0cLns1H5w9xargV0LlJI.png" } }, "description": "... from September 1, any new website cert valid for more than 398 days will not be trusted by the Safari browser and instead rejected. Older certs, issued prior to the deadline, are unaffected by this rul" } </script></div> CA/Browser Forum is a technical group that defines requirements on certificate authorities (CAs) that issue certs for your websites. The group is also responsible for security audits of CAs. As a result, it has a direct impact on the list of trusted certificates for your computer and web browser.The policy change as announced by Apple during the meeting in Slovakia and reported by <a href="https://www.theregister.co.uk/2020/02/20/apple_shorter_cert_lifetime/" target="_blank">The Reg</a> is as follows.<blockquote>... from September 1, any new website cert valid for more than 398 days will not be trusted by the Safari browser and instead rejected. Older certs, issued prior to the deadline, are unaffected by this rule.</blockquote> Let's Encrypt was heavily criticised when it decided to issue its free certificates with the validity of only 90 days as it significantly increases the risk of outages due to expired certificates. Let's Encrypt argued that as its operation is fully automated, users can automate renewals to avoid these issues. At the same time the argued that the short lifetime will mitigate security concerns rising from the full automation of the issuance process. Does it sounds a bit like Catch 22? There is a logic behind it thought and the success of Let's Encrypt justifies their decision.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Operation report of KeyChest (https://keychest.net) - you certainly want to keep it green." src="/storage/storage/wink/images/OGmHESVOXUmHkcMibdERLLrHm3LT5Ry4dnDg6eOs.png">Operation report of KeyChest (https://keychest.net) - you certainly want to keep it green.</div>However, IT managers well free to purchase a long-term certificate with the validity of 2 or even 3 years so they wouldn't have to think about renewals and keep their companies online and safe. This will now change as the new Apple policy means that you will not be able to purchase 2 year certs from September 1 without losing over 10% of internet users with Safari browsers. The impact can increase though as others may join. Cutting certificate lifetimes has been mulled by others for some time with the aim to improve security. This justification, that attackers will only be able to exploit bugs in SSL for 12 months instead of 24 months, and its real-life impact is open to discussion.One thing is for sure - the pain of web expiration due to expired security (i.e.,. SSL/HTTPS) will grow even further this year and the need for reliable monitoring and renewal automation will grow.<blockquote><a href="https://keychest.net/" target="_blank" style="color: rgb(51, 85, 187); background-color: rgb(255, 255, 255);">You have better things to do. KeyChest with its global database of web certificates can instantly create an initial "big picture" so you can start analyzing your exposure to cyber attacks and adjust it according to your risk appetite.</a> </blockquote>

Publication Date: 2020-02-23 20:27:00

FBI for passphrases - Cambridge Uni disagrees

This week, in its weekly tech advice column known as Tech Tuesday, the FBI Portland office positioned itself on the side of longer passwords. Would that really make a difference?

This week, in its weekly tech advice column known as Tech Tuesday, the FBI Portland office positioned itself on the side of longer passwords. Would that really make a difference?<hr>Let me start with the <a href="https://www.fbi.gov/contact-us/field-offices/portland/news/press-releases/oregon-fbi-tech-tuesday-building-a-digital-defense-with-passwords" target="_blank">statement of the FBI Portland office</a> as presented in its weekly tech advice column - Tech Tuesday :<blockquote>"Instead of using a short, complex password that is hard to remember, consider using a longer passphrase,"</blockquote><blockquote>"This involves combining multiple words into a long string of at least 15 characters," it added. "The extra length of a passphrase makes it harder to crack while also making it easier for you to remember."</blockquote>I remember Joe Bonneau analysing several password datasets at around 2013. They also looked at the strength of the <a href="https://nakedsecurity.sophos.com/2012/03/19/multi-word-passphrases/" target="_blank">passphrases in Amazon PayPhrase system</a>. They found around 8,000 phrases using a 20,000 phrase dictionary. When they added a bit of statistical processing, they could translated that into a bit more readable result. It said the complexity of 20bits for the attacker trying to compromise 1% of all user accounts. This compares to the complexity of 10bits in systems using passwords without any "password policy".Now 20bits equals 1,000,000 tests, compared to 1,000 for 10 bits. But as I mentioned the 10bits number if when no password policy is in place, i.e., users can choose whatever password they want. When we enforce minimum length, prohibit most common passwords and add a need for a special character, the 10 bits is likely to turn into more than 20bits.Passphrases are hard however, and there are three reasons for that:<ol><li>They are much harder to remember as we are in general not used to them - they also take longer to type - especially on mobile devices - and there's hardly to be any typo-correction in place.</li><li>There are no commonly used rules that would detect "weak passphrase" - you can't enforce special characters any more as the whole point of passphrases would be defeated. And the length will always be loooong enough.</li><li>Many systems can't deal with passwords longer than 20-30 characters.</li></ol>(Believe me, it's really hard to come up with a system of secure passwords that could be used across all the websites any one of us uses.)While the <a href="https://pages.nist.gov/800-63-3/" target="_blank" style="font-size: 1.1rem;">NIST standard 800-63</a> from 2017 urged websites to allow use of passwords of up to 64 characters, I'm pretty sure majority of companies never heard of that.What's the bottomline? Be reasonable - when you can, use the support offered by operating systems and browsers. Avoid really weak passwords- see below for some examples. A good approach is to combine an easy to remember word with something personal (like part or your address, pet names, ... something you don't use online too much). All major web browsers will generate secure passwords and store them securely on your computer - Google and Apple now even sync passwords between your devices. So long as you remember the username, you can always request an email with a recovery password or link.<blockquote><a href="https://keychest.net/" target="_blank" style="color: rgb(51, 85, 187); background-color: rgb(255, 255, 255);">You have better things to do. KeyChest with its global database of web certificates can instantly create an initial "big picture" so you can start analyzing your exposure to cyber attacks and adjust it according to your risk appetite.</a></blockquote><h2>Password Experiments</h2>I think that the best way to learn is to experience. If you have a Wordpress site, it's really easy to run experiments to detect passwords that are tried by hackers. WP is so common platform that whole bot-nets specialised in this particular system. I did a small experiment myself and here are my own results - which very much agree with a general advice.<h2>Which Passwords To Avoid</h2>It seems to be a very bad idea to use password consisting of only digits. I have logged passwords of 1 digit to passwords of 12 digits. As such, even a long number does not help. 22% of all guesses used number passwords.Another bad idea is to use a name as your password, be it the name of your girlfriend or son. The number of names being tested is very high indeed.The common often used passwords is another thing to avoid. Here is a selection of “password” variations we found: Password!, P@ssw0rd1, P@$w0rd, pa$$w0rd, password12345, pass1234, pa$$word, Pa55word, pass12, p4ssw0rd, p@55w0rd.Finally, if you believe that qwezxczasda is a good password, think again. Passwords made form keys that are close to each other are not so often but I was still surprised by some of them. Here is again a small selection: q1q1q1, qwertuiop, ytngfh, k,jdm, qweasd123, 123asd, qazwsxedcrfv.The biggest surprise however was when we identified passwords that used names of post authors as well as the website’s URL. There were more than 10 variations of one of the author’s name and even more passwords made from the website’s name and “padding” (like 11111, 12345, pass, …).<h2>Most Often Tested Passwords</h2>While in general the results are similar to what you can read in annual surveys, I found "new" passwords even in the 10% of most frequent attacks. Here is our top 33.<ul><li>admin1234</li><li> Password!</li><li> internet:)</li><li> 23946587</li><li>123456789</li><li>P@ssw0rd1</li><li>qwerty1</li><li>nursing</li><li>naked</li><li>admin_id</li><li>michelle1</li><li> tuborg</li><li> pass1</li><li> P@ssword123</li><li> nemesis</li><li>ibanez</li><li>eugene</li><li>56789q</li><li>danny</li><li>connect</li><li>(website name)</li><li>chantal</li><li>buttons</li><li>baseball1</li><li>qweqweqwe</li><li>liliana</li><li>control</li><li>administrator1</li><li>987654321</li><li>12345</li><li>101012474</li><li>stacey</li><li>Passw0rd1 </li></ul>

Publication Date: 2020-02-23 17:57:00

Microsoft HTTPS and DNS Hijacking = Big Mess

Two stories in as many weeks have flushed out some of the management problems Microsoft has with the management of its vast IT inventory - DNS and SSL.

Two stories in as many weeks have flushed out some of the management problems Microsoft has with the management of its vast IT inventory - DNS and SSL.<hr>The first story was a downtime of Microsoft Teams. The reason - an expired certificate. A bit of background from one of the manager's LinkedIn headhunting post:<blockquote>Microsoft Teams is the most exciting product Microsoft has produced in the last decade, and has been an incredible growth story. Unstoppable user growth also means challenges ... Arunachalam Thenappan - Principal Group Engineering Manager.</blockquote><div class="embedded_image" contenteditable="false" data-layout="default"><img alt=""... challenges to solve in our middle tier/backend services layers ..."" src="/storage/storage/wink/images/m76sRqrbmGkARerzq0YyZ23Q5uGxwIKy4qe6CvTk.png">"... challenges to solve in our middle tier/backend services layers ..."</div><h2>Keeping Up with Product Growth</h2>We did a quick report on MS teams subdomain and at the time it showed 22 certificates either expired or expiring within 7 days. We did the same today (as you <a href="https://keychest.net/auditnow?url=teams.microsoft.com" target="_blank">can do yourself</a>) and the number went up to 39. A large part of that number will be certificates for test/dev or legacy domains that are not in use any more.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Rapid growth of a cloud service means massive changes with security implications." src="/storage/storage/wink/images/ElTTk08fHSCC2W5VukMkdBpZRyS5OEHMdLLUhQgQ.png">Rapid growth of a cloud service means massive changes with security implications.</div>Two weeks ago and only with the outside view, I'd say that MS Teams may need a better way to manage legacy certs. But, maybe they use a simple approach -wipe unused virtual servers with everything on them, including private keys for those certificates. Still, I would expect them to revoke unused certificates If they had an IT security admin, but they will have to balance an additional cost, priorities, etc. You can see from the quote above that they've been massively growing, which makes it much harder to keep on top of things that are not directly impacting the core service.But let's have a look at the bigger picture. In terms of certificate expirations across the Microsoft domains, the free KeyChest domain audit is way too restricted (around 3,000 subdomains and only one domain at a time) but even with that, you can see that the problem is not limited to the MS Teams product. <div class="embedded_image" contenteditable="false" data-layout="default"><img alt="msn.com is one of many domains owned by Microsoft, and it's mentioned in the DNS hijacking context." src="/storage/storage/wink/images/zKI8Cd0WSJDHEH1mH537QcbkV3dcolyCG05Xsfur.png">msn.com is one of many domains owned by Microsoft, and it's mentioned in the DNS hijacking context.</div>What would be more worrying though is if there was an intersection of domains that could be subject to a potential compromise of their private keys and other vulnerabilities - like DNS domain hijacking.<h2>Certificate Management and DNS Hijacking</h2>Having read several articles about the domain hijacking within the <a href="http://microsoft.com" target="_blank">microsoft.com</a> and other Microsoft-owned domains, the certificate problem appears to be more significant. Microsoft is a massive company so there is inevitably going to be mistakes being made. They even had issues with the security of code signing keys in the past. We can assume that there is a distinct possibility that attackers can combine several vulnerabilities to launch powerful attacks.The researcher - Michel Gaschet of NIC.gp - quoted in the ZDNet article "<a href="https://www.zdnet.com/article/microsoft-has-a-subdomain-hijacking-problem/" target="_blank">Microsoft has a subdomain hijacking problem</a>" is quoted:<blockquote>Microsoft has a problem in managing its thousands of subdomains, many of which can be hijacked and used for attacks against users, its employees, or for showing spammy content". </blockquote>Michel was reporting issues to Microsoft over the last 3 years but only 5-10% were fixed - the reports included 142 domains in 2019 and 117 in 2020 (we are still in February).<h2>Validity of Cursory Checks</h2>You may say that this is a pretty far-fetched theory and only based on a pretty high-level "circumstantial evidence". That is true, but my experience tells me that if you can see issues in cursory checks, there is a good chance that a proper audit will confirm some and discover more. Whenever I, or my colleagues looked properly at a potential issue, we were almost always rewarded (whether it was vulnerabilities discovered recently by the team at Brno University - ROCA - insecure key generation in 25% of all TPMs, Minerva - insecure elliptic curve signing in open source libraries, or some previous work like Unwrapping Chrysalis - hidden API in HSMs, or sensor network security).The risk of hijacking a Microsoft subdomain and getting a valid HTTPS certificate and its private key is certainly there. The ZDNet article mentions that one of the issues is that this kind of integration and configuration bugs is "not part of Microsoft's bug bounty program" which lowers its priority. This is something that I find a big issue across all kinds of companies - from high-tech startups, major banks, to big technology companies. <h2>Product Bugs v Integration Gaps</h2>It is generally much easier to fix bugs in "products" as there is an easy to identify the vendor. Fixes are localized and companies have processes to test changes and deploy them. Management and integration issues have to be identified, analyzed, and fixed by the companies themselves. They are likely to include vulnerabilities (or simply incorrect configuration options) of several products as well as IT system "glue" developed either by the integrators or the companies themselves. While it is easier to read about bugs in products than gaps and weaknesses in business system designs, the threat of the latter is increasing with the use of public APIs and opening-up of legacy systems - e.g., banking systems. All that is needed is a sufficiently high reward to motivate attacks to focus on a particular company, rather than products used across the globe. <blockquote><a href="https://keychest.net" target="_blank" style="color: rgb(51, 85, 187); background-color: rgb(255, 255, 255);">You have better things to do. KeyChest with its global database of web certificates can instantly create an initial "big picture" so you can start analyzing your exposure to cyber attacks and adjust it according to your risk appetite.</a></blockquote>

Publication Date: 2020-02-20 10:32:00

Let's Encrypt certificate into Java JKS

If you have Java applications you need to convert Linux PEM files created by Let's Encrypt clients into JKS. It's just a few steps, if you know which ones.

If you have Java applications you need to convert Linux PEM files created by Let's Encrypt clients into JKS. It's just a few steps, if you know which ones.<hr>Because I was doing it only a few times a year I always forgot. If you have the same problem, here's a short step-by-step. It should work for any applications run by a recent Java version (Java 6, Java 7, Java 8, Java 11).What you need: cerbot (or other Let's Encrypt client), openssl, keytool (a part of Java distributions).STEP 0: I sometimes have to find the path for certbot if it doesn’t get set for the “root” user (assuming certbot is installed):<pre class="ql-syntax" spellcheck="false">which certbot sudo su # or switch to whatever user you intend to use export PATH=<certbotpath>:$PATH </pre>STEP 1: assuming you’re root (or other account with necessary privileges):<pre class="ql-syntax" spellcheck="false">certbot renew </pre>Note: you can use any other Let's Encrypt client, the only difference is possibly a different location of keys&certificates.STEP 2: if the renewal was successful, we can move on to the “tricky part”. Change the working directory to a folder with the new certificate:<pre class="ql-syntax" spellcheck="false">cd /etc/letsencrypt/live/<domain name> </pre>STEP 3: create a PKCS12/PFX file containing the new private key and certificates (intermediary password is “password”):<pre class="ql-syntax" spellcheck="false">openssl pkcs12 -export -in fullchain.pem -inkey privkey.pem -out server.p12 -name tomcat </pre>Note: you will be asked for a password, you have to use the same in the following step - replacing password.Note: initially, we tried “-in cert.pem -CAfile chain.pem ….”, which doesn’t include the chain to the P12 file, so Keep It Simple Stupid (KISS). :)STEP 4: convert the PKCS12 file into a JKS Java keystore. There are three variables in the following command: "password" - which you used in STEP3, "/tmp/le_keystore.jks" - the location and name of the JKS file, and "alias" - a key name in the JKS file. You need to make sure that your Java configuration file matches your values!<pre class="ql-syntax" spellcheck="false">keytool -importkeystore -deststorepass password -destkeypass password -destkeystore /tmp/le_keystore.jks -srckeystore server.p12 -srcstoretype PKCS12 -srcstorepass password -alias tomcat </pre>STEP 5: We used it for a Tomcat app, which required a restart to pick-up the new SSL certificate.STEP 6: clean up - delete the temporary P12 file and possibly JKS so you don't keep unnecessary copies of the private key.A note of caution - it may be the case that configuration files are wrapped in a jar file. (Just in case you can’t find any and have been banging your head against your deck for the last 2 days :)<hr><blockquote><a href="https://keychest.net/" target="_blank" style="color: rgb(51, 85, 187); background-color: rgb(255, 255, 255);">Certificate expiration causes costly downtimes and complete breakdowns of encryption. Our expert service automatically checks and renews your certificates, on time, and correctly, so you can start every day with confidence.</a></blockquote>

Publication Date: 2020-02-19 15:44:00

HTTPS Certificates - Keys and Issuers

Let's have a look at the quality of keys in internet certificates and who are the main certificate issuers.

Let's have a look at the quality of keys in internet certificates and who are the main certificate issuers.<hr>This time, I have analyzed a sample of 500,000 certificates from the database of <a href="https://keychest.net" target="_blank" style="color: rgb(43, 43, 43);">KeyChest</a>. I have looked at the numbers of all certificates that KeyChest pulls from CT (certificate transparency) logs and also at data of server audits. The two datasets differ as we request more certificate than we actually use – especially when they are free. When I query our CT (certificate transparency) database directly, I can see large numbers of duplicates, but it’s non-trivial to eliminate the effect of CDNs (content delivery networks). The data from actual audits of web services show how important this aspect is.Anyway, I had worked for large enterprises and I was curious. An enterprise is still unlikely to use Let’s Encrypt but how successful are “enterprise trusted” names elsewhere. And it’s pretty bad for some names I believed to be market leaders.Let me start with some simple but nevertheless interesting stats around algorithms and key length. The data is for domains audited by KeyChest.Algorithms and key length<ul><li>12% RSA 4,096 bits</li><li>68% RSA 2,048 bits</li><li>2% RSA 1,024 bits</li><li>15% ECC 256bits</li><li>0.8% ECC 384bits</li></ul>Personally, I’m surprised by the large use of elliptic curve (ECC) certificates, which I’d expect to be less than 5%. So well done to all the admins who optimize their servers for faster encryption.Types of certificates<ul><li>1.2% - self-signed certificates</li><li>4.8% - extended validation certificates (EV)</li><li>20.5% - organization validation certificates (OV)</li><li>73% - domain validation certificates (DV)</li></ul>Another two statistics I was interested in was CDN and wildcard certificates:<ul><li>Cloudflare certificates: 15%</li><li>Wildcard certificates: 22%</li></ul>What you can start seeing from this is the role of CDNs, and especially Cloudflare among KeyChest users. Cloudflare would have OV certificates, which means that when we take those away, only organizations are more likely to use the most expensive EV certificates than OV certificates.<blockquote><a href="https://keychest.net" target="_blank">You’ve got better things to do than worry about web certificates. Leave them to KeyChest.</a></blockquote>I’m finally getting to the results of the main excercise – how big is Let’s Encrypt. It is not 80% but it’s still majority of all certificates that KeyChest pulled from CT logs.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Distribution of certificates per issuer" src="/storage/storage/wink/images/Hi8ZYoxUN9eTpmfCSYOTotIWPvSR0xkuy2lP64B5.png">Distribution of certificates per issuer</div>There are some unexpected entries there (cPanel) but in general – there are basically 2 leaders in the market – Let’s Encrypt for free, short term certificates that we are using on our web servers. COMODO used by main CDNs. GlobalSign and Digicert are sharing the market of “end user organizations” with COMODO.Let’s have a look at the data from actual audits of servers. I have done two samples, one for audits over the KeyChest lifetime (since Q3 2017) and a sample of audits executed in Q1/2019.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Distribution of certificates from server audits." src="/storage/storage/wink/images/ihHgVqsGGugwUAUH9yWVMJW9GSqJIrReAJuHArlr.png">Distribution of certificates from server audits.</div>The main number here is the increase of Let’s Encrypt – from 33.1% to 38.8%. Another interesting point is that COMODO is back in line with other commercial issuers – something that most likely reflects how CDNs use certificates. My experience is that the number of certificates issued for CDNs is much higher than what can be see on CDN servers (although we are limited by the use of only a couple of DNS servers we use to resolve domain names to IP addresses).<blockquote><a href="https://keychest.net" target="_blank" style="color: rgb(23, 43, 77);">You are overwhelmed with certificates to manage, websites are regularly going down, and your stress spills over into your personal life. Let KeyChest get you back on top.</a></blockquote>Some interesting numbers. Certainly well done to Let’s Encrypt, which seems to keep getting new customers as the long-validity certificates expire. I was surprise by the amount of certs issued by COMODO. It seems they benefit from their targeting of CDNs, as well as low cost for small users.

Publication Date: 2020-02-18 20:20:00

The State of PKI by AppViewX

AppViewX has conducted a research during the 2019 BlackHat conference asking cybersecurity professionals about their experience with PKI. I will give you an alternative exec summary.

AppViewX has conducted a research during the 2019 BlackHat conference asking cybersecurity professionals about their experience with PKI. I will give you an alternative exec summary.<hr>You can find the original report at <a href="https://pages.appviewx.com/rs/249-TWN-899/images/Black%20Hat%20Survey%20Report_1.1.pdf" target="_blank">pages.appviewx.com</a>. Their exec summary highlighted the following three numbers:<ol><li>40% use encryption to protect private keys - now this is something of a problem and you are likely to really encrypt only high-value keys, e.g., private keys of internal certificate authorities. The problems is that many servers only work with text files and can't do decryption.</li><li>30% of companies use spreadsheets to track their certificates.</li><li>5% of companies have at least 5 expiration-related incidents every year.</li></ol><h2>Other Facts</h2>Let's have a look at other information that the report provides, section by section:<ol><li>Frequency of outages due to certificates</li><li>Use of best practices</li><li>Business impacts of certificate expiration outages</li><li>Challenges of deploying and managing PKI</li><li>Need of PKI for new disruptive technologies</li></ol><h2>Certificate Expiration Outages</h2>This is a bit confusing as it doesn't quite support the figure from the exec summary so let's have a look at the detail.65% experienced 1 to 10 outages - application or device - and an additional 25% of respondents had more than 10 outages. It is in line with a previous study of Venafi from 2017. It is actually 10% more as Venafi found that only 79% of business experience an outages due to an expired certificate.While not conclusive, it certainly suggests that the problem is getting worse. This is something we can see when talking to companies and we know that the number of certificates grows by 5-10% every year. That means that the number of "near misses" grows and it will eventually lead to downtimes as the statistics can't be fooled forever.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="" src="/storage/storage/wink/images/pdRwM6hnO053c4k4pYxalCzBxTaITAkmcN9bZtXr.jpeg"></div>The report further looks at data breaches caused by expired certificates, which stands at 6%. I believe that is a little consolation for companies as they are more concerned about their revenue than a risk of security incidents. Although the general data protection act (GDPR) and the California Consumer Privacy Act (CCPA) have caused internal governance changes in companies.<h2>Best Practices</h2>This section focuses on three aspects: certificate management, private key protection, and monitoring.I am not quite sure what the private key protection means. The mainstream format to protect private keys is PKCS#12, which is a password-protected file. Linux systems would not use any password once the private key is installed and Windows will allow web servers (IIS) access private keys without passwords as well.However, protecting keys of certificate authorities is important as their compromise would impact large numbers of servers and services and possibly cause a complete breakdown of the security of internal PKIs and all systems that depend on them.Certificate management tools are more interesting:<ul><li>20% use certificate lifecycle management tool</li><li>16% use spreadsheets</li><li>33% have custom solutions</li><li>30% have an issuer-provided tool</li></ul>The two middle numbers are interesting, and especially the high number of companies that built their own custom solution is surprising. The reason being that these tools are unlikely to scale. They also often depend on people who built them and there is a relatively small chance of having a well-documented system that will keep running when the team changes.Only 39% of companies do monitoring as well as audits of certificates. Let's look at this number and the previous breakdown of management tools. Let's assume that lifecycle management tools provide audit and monitoring.<a href="https://keychest.net" target="_blank">KEYCHEST is a certificate management service that provides audit, monitoring, and issuance of certificates for confident management of certificates. It's designed for business with any budget.</a>That leaves 19% of 63% (custom solutions and isssuer-provided tools) include audit and monitoring. That is only one fifth of companies. The rest will do some monitoring or audit, but not both. That is probably the most interesting number so far.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="A simple view showing your risk in the next 7 days is the first step for effective certificate management." src="/storage/storage/wink/images/DRRjOgmxuTvj0CMfhAI4hYVHWmtfBJkNWDWF6fVL.png">A simple view showing your risk in the next 7 days is the first step for effective certificate management.</div><h2>Business Impact of Certificate Expiration</h2>The numbers in this part of the report are focused on concerns, not necessarily on the actual history and experience of companies.A quite non-European number is 40% worried about a potential legal action and fines. It is interesting, because certificates would usually cause downtimes and I'd expect the main impact to be a breach of SLAs or similar.The two main areas of concern are, however the loss of revenue - 62% - and the loss of productivity - 51%. A quarter of companies also recognized remediation costs as significant. What this means is a tough life for IT managers and system administrators. While they may not necessarily be in a position to improve the management of certificates and internal SSL certificate monitoring, they will be required to restore systems in the shortest possible time 24x7.The aim of best practices is to minimize the number of near misses and eradicate downtimes. Use of a simple SSL monitor can only get you that far and effective certificate management requires an efficient management of the whole lifecycle.<blockquote>HTTPS Certificate Cost is a combination of several significant items. The purchase cost of certificates - this item can be lowered by volume purchases or using <a href="https://keychest.net/certificate" target="_blank">systems that pass the volume savings</a> to customers. Monitoring and audit cost that would comprise the internal labor costs or the cost of a third-party service, which can be fixed unmetered, or calculated per certificate. The third is the cost of certificate installation, which can be planned and calculated for well-managed certificate inventories. The last one is the hardest to measure as it's based on the probability of incidents. It would involve high levels of stress, a loss of revenue and productivity, brand damage, etc.</blockquote><h2>Challenges of PKI</h2>Let's start with numbers:<ul><li>74% - lack of visibility</li><li>46/48% - lack of skills and understanding</li><li>20% - technology limitations</li></ul>What's interesting is that there are several free / open-source tools that can help detect certificates and improve the visibility. The matter of fact is that the tools will not provide the regular and up-to-date information you need for confident certificate management.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Main challenges of certificate management" src="/storage/storage/wink/images/IflVTRmDPNKT59qEUXqS1YG1lvOKxSdU7hNpm8Rc.png">Main challenges of certificate management</div><h2>New Technologies Require PKI</h2>We can easily skip the "disruptive" from the section title. Where ever you look - from RESTful APIs to mobile web (AMP) to building information-rich online presentations, you need SSL/TLS and you need certificates.Even small businesses have to tackle the PKI management problems that were experienced only by corporations 5 or 10 years ago. Even when you just want to run a more traditional online business, you need to include the cost of HTTPS and certificates in your budget. It can be a direct cost or increases in the service cost of your managed service providers. While 60% of companies said that the next big trend was IoT, 50% mentioned "cloud". And the cloud is what any business with online presence needs today. When one certificate expires, there would be 10, 20 or more certificates that "almost expired" and there may be a hundred others that may expire in the next week or two. The problem of certificate management is its scale. The scale that only grows.

Publication Date: 2020-02-18 14:00:00

Scalable Certificate Monitoring

The enforcement of HTTPS by web browsers has introduced the pain of certificate management to small and medium businesses. My rules of thumb to make your life much easier.

The enforcement of HTTPS by web browsers has introduced the pain of certificate management to small and medium businesses. My rules of thumb to make your life much easier.<hr>Corporations have substantial budgets for the IT security management. Certificate management is one of the largest items. What has happened in the last five years is that small and medium businesses have started experiencing the same pain. Everyone who wants to keep their internet presence, but implement web encryption - HTTPS. Preventing encryption expiration has become an integral part of "keeping the business running".As far as I can see, you can easily manage your certificates the way you want. IF you own a handful of servers, you're educated in IT management. You possibly remember the expiration of all your certificates by heart as well as you can name all your servers. I am pretty sure you have a strong feeling about someone like me telling you how to do things. This piece is not for you.I wrote this for people who are less lucky and have to deal with growing numbers of certificates without getting more people, bigger budget, or a raise. Most of us have better things to do than firefight missed expired certificates. What we want is a simple approach that will make your life easier. An approach that can turn the hassle of certificate management into an opportunity to use it as one of the most powerful intelligence tools about your IT. The silver lining is that there are a few simple rules that significantly help if you stick with them. <h2>Wildcard Certificates</h2>Wildcard certificates are a powerful tool. They can authenticate an unlimited number of domain names and services. The trouble with using such powerful tools is that they can easily turn into powerfully dangerous.<blockquote>The long term wild-card certificates are also much more expensive than certificates where you have to name every name with which they can be used. This is another incentive to use them to the limit.</blockquote>Danger: the incorrect way of using wildcard certificates is to copy them (and their private keys) on many different locations. In a small company, you are likely to have a number of small servers or virtual machines that you've added over time for different purposes. You would need to copy your wildcard certificate to every single location. This will cause a problem a year later when you need to renew them. Getting a new certificate is easy. The hard bit is to install them. The success of the installation will depend on your spreadsheet with all the locations where you need to install the new one. This is prone to errors.<blockquote>"Wildcard certificates are certificates that contain a domain name that starts with "*", e.g., "*.keychest.net". It is beautiful in a sense that you can use that one certificates for all subdomains: "a.keychest.net", "mywork.dev.keychest.net", etc. It will, however, not cover "keychest.net" as this main domain doesn't contain the leading ".".</blockquote>Rule of thumb: A good use of wildcard certificates is on load balancers and servers that host a number of subdomains. This ensures that there is (ideally) just one location where the wildcard certificate is installed. Yet, you get all the benefits. It requires a good design of your networks. If you're not sure, stick with "SAN" certificates that contain a list of all domain names where they can be used.<h2>Unique Names</h2>We can sometimes see a group of certificates all with the same name and similar expiration date. You often need multiple certificates when you have a failover or load-balanced service. Each of these certificates will be installed on a different server. The problem is that it is impossible to instantly find out which certificate is where. It's a good idea to add markers that would help you quickly identify the source of potential incidents. As many of us use Let's Encrypt, and we should have as few rules as possible, the scope for "tags" is quite limited and we are basically left with an additional domain name.Rule of thumb: create alias DNS records that you can add to your certificates simply to identify services with which they are used. They can all be on the same server, i.e., the "functional" domain name is the same and you add a new one: "mq.domain.com", "mx.keychest.net", "web.keychest.net". That way you can instantly see if a potential problem is related to your Rabbit MQ, Postfix, or Apache.<h2>Define Business Renewal Deadlines</h2>The main criticism of certificates is that their expiration, while purely a technical "detail", has a direct impact on services that use them. What it means is that a low-cost or even free technical detail can cause tens of millions of dollar losses. Without any warning.What we have implemented in <a href="https://keychest.net" target="_blank">KeyChest</a> early on, was to de-couple the technical expiry from "business expiry" (e.g., Let's Encrypt certificates has this difference set to 2 weeks). This de-coupling, while a simple measure, has an immense business impact. You get rid of the cliff-edge caused by the technical expiry of certificates and get a gradual devaluation of certificates that can last weeks to months, or even years. This works well with usual business processes based on incident escalation used for any other security incidents.More importantly - when you get an alert of an end-of-life certificate on Saturday night, you can finish your pint, go home, enjoy a family Sunday and fix things on Monday. You get the confidence that certificates will become a part of your life, but will not take it over.Rule of thumb: define business deadlines for all your certificate renewals. Take into account your availability on the worst day of the year, internal policies and escalation processes. When your boss's key indicators (KPI) show RED, you know you have enough time to prevent any business impact.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Define your own deadlines for raising critical incidents. This will give you time to fix issues without business downtimes." src="/storage/storage/wink/images/nHHpavkB5NxtrLETEEemIde3nLpC631T6xErjwmY.png">Define your own deadlines for raising critical incidents. This will give you time to fix issues without business downtimes.</div><h2>Use DNS CAA Records and CT Logs</h2>One of the biggest issues with free Let's Encrypt certificates is unauthorized internal use. In most cases, it will be justified as a "quick fix to get things working again". Sometimes it will be misused to bypass security policies by diverting traffic or allowing personal devices pretend to be part of your company - to keep things up and running. Sometimes it can be, however, down to downright malicious actions.There are two main technological solutions. The first one is to use your DNS records to limit the CAs that can issue certificates for your domains. <a href="https://support.dnsimple.com/articles/caa-record/" target="_blank">DNS CAA records</a> have to be followed by all internet CAs. You can specify which CAs are allowed to issue certificates for your domain. When a CA gets an  unauthorized request, it has to send you a notification.The second tool is the use of CT logs. Those are global logs of all issued certificates and if you search "CT log search", you can get several services that will let you search for certificates for your domain(s). Or you can use <a href="https://crt.sh" target="_blank">crt.sh</a> to run quick searches.Rule of thumb: Control and limit CAs that can issue certificates for your internet domains and set up mechanisms to receive notifications. Regularly check that there are no rogue certificates issued for your domains.<h2> Extended Validation (EV) Certificates</h2>You can now easily demonstrate that it is virtually impossible to say whether a website is protected with an EV certificate or OV (organization validated) certificate. A year or two back, web browsers showed a big green bar with the company name, when you opened a website with an EV certificate. Not any more.Rule of thumb: save on EV certificates and use the leftover budget for useful gadgets that will make you look forward to getting to the office every morning.<hr>I will regularly revisit this post and add new items as they become relevant. Meantime, you can try our instant domain expiry audit - completely free at <a href="https://keychest.net/auditnow" target="_blank">https://keychest.net/auditnow</a>. (You can <a href="https://keychest.net/auditnow?url=teams.microsoft.com" target="_blank">check how Microsoft Teams</a> are getting with improving their own certificate management after a recent <a href="https://techcrunch.com/2020/02/03/microsoft-teams-has-been-down-this-morning/" target="_blank">downtime</a>. Or <a href="https://keychest.net/register" target="_blank">create a KeyChest account </a>to get most of the "policies" above out-of-the-box.

Publication Date: 2020-02-14 09:52:00

C-level Cyber Security Report with Surprises

Thales regularly publishes a Data Threat Report. It is created from responses provided by high-level execs so one wouldn't expect to find anything much of interest. But I was wrong, this time.

Thales regularly publishes a Data Threat Report. It is created from responses provided by high-level execs so one wouldn't expect to find anything much of interest. But I was wrong, this time.The 2019 Data Threat Report by Thales is available online and you can download in exchange for your email from <a href="https://www.thalesesecurity.com/2019/data-threat-report" target="_blank">https://www.thalesesecurity.com/)</a> or directly from their <a href="https://go.thalesesecurity.com/rs/480-LWA-970/images/2019-Thales-Data-Threat-Report-European-Edition-A4-ar.pdf" target="_blank">file server as a PDF</a>. It is a high-level threat intelligence report collected from C-level execs from around the world. I didn't expect too many surprises, but there are several things that are worth mentioning.<h2>This Is What I Expected</h2>My first impression was that of a general inconclusiveness of many of the report's sections. Most charts are based on multi-choice questions with very little difference between the least likely and most likely. Let's take concerns around using Infrastructure as a Service (IaaS). The question has 9 options with the most concerning on 47% and least concerning on 42%. I am not sure what can this section of the report say except that IaaS has a number of significant issues - from SLA and liability for data encryption incidents. At the same time, all the questioned organizations use at least one instance of IaaS so clearly security concerns are outweighed by the perceived cost saving on managing own private cloud / data centers.<h2>Unexpected In Expected - Low Use of FDE</h2>Another example is a part about data security technologies. The most common is "file encryption" on 53% with the "data loss prevention" at the opposite end with 43% (if we ignore payment-specific items). Now, I believe it is impossible to use Windows without disk encryption inside an organization's domain. When we look at major public cloud platforms, they all integrate disk encryption in such a way that it's for me hard to imagine a corporation that wouldn't do it. Still, the survey shows that only 46% of companies use full-disk encryption. Is it because C-level execs don't know or because it is being used without an enterprise-wide project, or is it real?<h2>Surprise 1 - EU Execs Are Relaxed About Cybersecurity</h2>There are, however, two specific aspects of the survey that surprised me. The first one is how European companies are relaxed about cloud security. This question had topics that included failure of the provider, attacks on the cloud platform, compliance, key management, privacy. Across all those questions, only 35% of European respondents were concerned compared to the 45% of the global respondents. <div class="embedded_image" contenteditable="false" data-layout="default"><img alt="" src="/storage/storage/wink/images/OxWTFzOHkApJSHMRDfPmW9M8qH3qUYFEvq0qZJFd.png"></div>That is a massive difference and quite surprising. If only for the fact that we have a fresh data protection regulation (GDPR) that allows penalties up to 4% of global turnover. It just can't be true, or can it.<h2>Surprise 2 - Proliferation of Blockchain</h2>The second massive surprise for me was that more than 30% of companies are worried about sensitive / regulated data being submitted to public ledgers and compromise of cryptocurrency. I find this somewhat surprising as pushing sensitive data is much harder than posting it to social networks. The responses suggest that 30% of companies use bitcoin or other public ledgers, which sounds like a high number to me, given the regulatory issues around using cryptocurrencies in business. <div class="embedded_image" contenteditable="false" data-layout="default"><img alt="" src="/storage/storage/wink/images/Y183nGZA64Z4NBda4IUHmeht8DNepUm8XPCRhMLG.png"></div>Another explanation could be that senior execs don't understand the technology and simply accept related risks as relevant til they are properly identified and assessed. Alternatively, they simply answered the survey and marked threats (e.g., compromise of exchange provider or redirecting cryptocurrency destination) they see as relevant regardless of whether they actually have any possible impact on their business. Still, a possibility of such a large number of companies to actively use cryptocurrencies is surprising.<h2>Surprise 3 - IoT Security by Design May Actually Be Really Broken </h2>My last note is about the security of the IoT (internet or insecurity of things - depends on where you stand). Interestingly, the level of security concerns is lower than it is with using public clouds. On average, less than 30% of companies worry about the data security there. I found quite interesting that the most concerning are attacks on low-cost devices:<ul><li>attacks on IoT devices will impact critical operations (33%);</li><li>lack of frameworks and control;</li><li>loss or theft of IoT devices;</li><li>privacy violations (30%);</li><li>discovery of sensitive data created by IoT devices;</li><li>and so on</li></ul>I was not sure how good is my <a href="https://keychest.net/stories/secure-by-design-will-not-work-the-economics-stupid-" target="_blank">empirical experience in my previous post</a> where I talked about a likely failure of the "security by design". My premise there is that we focus on securing low-cost devices that are used to protect and manage high-value assets. I believe this will not work from the economic point of view. And here we go, the biggest concern among large companies is the impact of $5 a piece devices on "critical operations".<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Instant audit of the teams.microsoft.com domain that experienced downtime due to an authorization certificate." src="/storage/storage/wink/images/bcjjmWigP6Y984GYIfulX0OFnT95DdDL1ZdO7911.png">Instant audit of the teams.microsoft.com domain that experienced downtime due to an authorization certificate.</div>As the key management in the IoT world is currently almost a synonym of the "IoT security", we can only start managing cyber threats when we can create an accurate picture of whole IoT deployments. Once we know what the status is, we can start enforcing the baseline security policies and gradually improve the overall resilience against cyber attacks.<blockquote><a href="https://keychest.net" target="_blank">KeyChest with its global database of PKI certificates can instantly create an initial "big picture" so you can start analyzing your exposure to cyber attacks and adjust it according to your risk appetite.</a></blockquote>

Publication Date: 2020-02-11 15:08:00

Secure By Design Will Not Work - The economics, stupid

Secure by design has been touted by governments as the way to solve the threat from insecure IoT devices. Here is a thought - it will never work because the focus is wrong.

Secure by design has been touted by governments as the way to solve the threat from insecure IoT devices. Here is a thought - it will never work because the focus is wrong.<hr>The UK government has made "security by design" one of a handful of priorities in cyber-security. And they have claimed a significant progress admired across the globe. In October 2019, ARM, the processor design company, has announced a $50 million project to design a secure processor. A processor that is unlikely to ever go into production. The justification was "The government wants the UK to be the safest place to be online and the best place to start and grow a digital business"<h2>The Economics In IoT</h2>Here's why it will never scale-up to solve the problem of insecure IoT devices - the economics. IoT is the technology to manage high-value global assets, but IoT devices could cost less than $5 a piece in sales price. As far as I can see, the money for the security is on the user side, rather than IoT device manufacturers operating with much smaller margins.Also, I am still to meet someone who is truly worried about the cybersecurity in the IoT world - I mean a user, not a vendor. There are many vendors writing about the importance of security in IoT. When you look at the "buyer" side, these voices are much harder to find.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Internet of Things - (source: Open Educational Resource - OER)" src="/storage/storage/wink/images/beUiQyBvHdPnbooQRALDK4qb4yBu08nAgGUBe9LR.jpeg">Internet of Things - (source: Open Educational Resource - OER)</div>I had many discussions about IoT use cases and those with sensitive data are small and hardly scalable enough to make a vendor business plan viable for equity funding, i.e., likely to scale. IoT simply doesn't cater for sensitive data that would justify a proper security design, key management, and well, stop black hats laughing. You can argue that key management - PKI - is fundamental for IoT. Yes, it is, but not necessarily for security reasons. What you need in an IoT network is to be able to identify devices and when you look at technology options, PKI as something fairly universal that can do the job. That's why you can find some kind of digital certificates in many devices. That's also why these default certificates are never updated or renewed.<h2>Securing IoT Devices</h2>There are initial drafts of standards - Software Updates for Internet of Things (SUIT) - <a href="https://datatracker.ietf.org/group/suit/about/" target="_blank">https://datatracker.ietf.org/group/suit/about/</a> - that focus on the device security. To be more accurate, just and only secure updates. When I read the documents, my overall impression was that of a lot of technical details replicating secure boot principles from mobile devices. I did not find anything about how it should work in a wider IT ecosystem. For example, how would a secure upgrade work when device vendors are cut off from access to devices. Today, companies will not allow pushing OS upgrades into their networks without first testing them with their laptop/server images.What are the chances of these standards having an impact that would scale globally? Not very good in my opinion as they go deep into the software stack of IoT devices, possibly all the way to the CPU level. It means an incredibly high cost of designing changes, implementing and testing. At the same time, it is not clear if IoT users have bought into this. It is much more likely that there will be "secure" versions of IoT products that would cost 5-10x more than "common" devices.There is also a question of trust - who should hold the keys needed for firmware upgrades - these seem to be prone to leaks. We are talking about low-cost devices potentially protecting high-value assets. But the value is different for the IoT device manufacturer (low value) and the IoT device users (impact on high-value assets). In this respect, the SUIT documents may be getting assumptions for their architecture wrong. You can legislate but if there's no/weak business justification for proper security management, it may be seen as too heavy a price.<h2>IoT Security = PKI, Not Really</h2>It's possible you've never heard of "PKI" but it's a concept behind HTTPS - i.e., web encryption we all use now. It has become a commercial technology during the dotcom boom. At the time it had no practical applications. There was simply no viable use case. This has been a problem for a long time and it only has changed with an HTTPS proliferation in the last 5 years.However, as IoT has become a buzzword for venture capital as one of the "future technologies", PKI has been rebranded as IoT security. And IoT security has been reduced to PKI. The trouble is that the PKI technology is incredibly complex and IoT devices are not able to implement it reliably.PKI in IoT has to be significantly simplified to have a desirable security impact. We need to move as much functionality from devices to management systems to build the confidence in the IoT security required by businesses. Systems like KeyChest replace the focus on endpoints with a focus on the ecosystem to create and enforce a baseline security preventing attacks exploding weakest links.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="KeyChest provides a big picture of PKI status with its global database." src="/storage/storage/wink/images/bak2eiZ2x1LjjclsnKvVuPtrYz9xUATSCy4GxSbC.png">KeyChest provides a big picture of PKI status with its global database.</div> Despite all that, PKI is probably our best bet to implement a reliable device identification system and to bootstrap secure data transfers. We also need to move the difficult tasks from endpoints to high-security, possibly distributed, management systems. Recent research shows that the entropy of encryption keys generated by IoT devices is as little as 16 bits - remember that the minimum length for secure encryption is 128bits.Key generation on devices should be limited to initial communication keys with their management systems. Simplify PKI operations to a core set that is absolutely necessary for any secure communication. <h2>Encryption Fit for IoT = Simplicity</h2>The Achille's heel of PKI is expiry. The first thing you can do is to "ignore" key/cert expiry. What I mean is that you replace "system breaking expiry" with business-managed expiry. IoT devices today suffer from a complete compromise of their security when their certificates expire. What you can do is to align the validity of certificates with the expected lifetime of IoT devices. You can then start managing risks from the business point of view and focus on the important aspects - incidents.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Expired certificates in IoT can cause downtimes, or a complete breakdown of encryption when security checks are bypassed." src="/storage/storage/wink/images/aCr0uuyGYCEodEkNYvB9F5tWM1bXPgjtFFTeoVml.png">Expired certificates in IoT can cause downtimes, or a complete breakdown of encryption when security checks are bypassed.</div> It is absolutely incredible from the business point of view that a certificate that is perfectly secure at 11:59:59 is deemed useless, insecure, worthless at 12:00:00.To simplify the encryption, my starting point would be that IoT is a mesh of devices that have a very small intersection of capabilities (much smaller than Windows, Android, iOS, ... have). In terms of the PKI, what you end up with is a signature on a name-key relationship. I would also assume that reliability is the primary concern, so you want to offload as much as you can from nodes to a management system - another reason to simplify certificates ... and another one is the size of certificates. (Did you know that the size of certificates breaks the binary encoding of TLS packets? They are just too long.)For IoT devices, what we need are simple certificates with length limits on names to prevent memory issues. You also need a simplified chain validation algorithm. We want certificates to bootstrap trust not to define it. We can remove the risk of a root key compromise by deleting private key as soon as a batch of devices is personalized - whether it's done by the device vendor or a device user. You also need to create flexible, easy-to-recover mechanisms to validate root keys/certs - simplify incident recovery. Hacking these doesn't compromise root keys as they have been deleted long time ago => attacks can be fully contained with zero residual risk.<h2>Inventory of IoT</h2>A concept I really like is to mirror an IoT network with keys/certs just for the management purposes as it replaces a complex, heterogeneous system with an abstract inventory of keys that you can adjust to fit your management requirements. This can be a completely parallel inventory of what is actually installed on devices, or it can be tightly linked/identical with what is installed on devices. The expiry can then be used for an efficient management of data/device life-cycle.The main questions still remain though - incident response. How do I expel a device from my network, how do I rollover issuing/root CA. Expelling devices is an issue only if you allow IoT devices to directly communicate with each other. If you have the internet access, you can as well proxy all traffic via an IoT management system. This way you can whitelist and/or blacklist devices. Direct communication between devices could use whitelisting, i.e., trust on first sight with blacklisting provided by the management system. While the direct trust can technically work here, it's still not clear if it makes the management way too complicated. <h2>Encryption Is Not Security</h2>Let me finish off with the main point. The IoT security is not defined by the strength of the encryption. It was said before that if you think the encryption can solve your problems, you don't know what your problems are. While working for Deloitte I was gobsmacked when I joined a discussion where the topic was whether an enterprise can achieve more than 90% data accuracy. It just completely changed the way I looked at the security of business applications.There are use cases when each piece of data is important - like payment instructions from your credit card to your bank. But these are transactional data and we still need dispute resolution mechanisms and being able to identify fraudulent transactions and reverse them. When you look at business data in general, you create meaningful information from millions of data items. This is how you get the high-value, strategic data about businesses.Here's a practical example. You certainly want your account to be charged $19.99 when you purchase something for $19.99. At the end of the month, you will look at the aggregate of your purchases and compare it with your pay packet. Do you really worry if saved $200.50 or $205.00? Very few of us will go back and check all the transactions. That we got taxed / or tax refunds to the penny, etc. We are touching the world of statistics and the larger the budgets and spending, the higher is the chance that we made a mistake in particular transactions.Why is this important? Simply because when you scale your IoT systems, the problem will quickly become one of ensuring a baseline of security, reliability, accuracy. You will never be able to do that on the device level.That's why I believe that "Secure By Design" will not scale enough to solve the problem of IoT security. It will be management systems to give users confidence in IoT, that will ensure that devices will reliably work and that encryption will enhance the use of IoT, not hinder it. That's what we are doing with KeyChest. <blockquote><a href="https://keychest.net" target="_blank">Certificate expiration causes costly downtimes and complete breakdowns of encryption. Our expert service automatically checks and renews your certificates, on time, and correctly, so you can start every day with confidence.</a></blockquote>

Publication Date: 2020-02-09 12:57:00

HashiCorp Vault and PKI

I started playing with HashiCorp Vault about 2 years ago and I really struggled to start with. I didn't expect the simplicity. Here are some of my notes that may help you touch the ground running.

I started playing with HashiCorp Vault about 2 years ago and I really struggled to start with. I didn't expect the simplicity. Here are some of my notes that may help you touch the ground running.<hr>Note 1: I have updated the steps for Vault version 1.3+ (tested with v1.3.2) as the syntax has changed since my first deployments.Note 2: There is an official how-to page <a href="https://www.vaultproject.io/docs/secrets/pki/index.html" target="_blank">https://www.vaultproject.io/docs/secrets/pki/index.html</a> which I used for this description and it's worth having a look at. I have added a description of basic concepts to help you understand the Vault more quickly and also added some details that took me a little to figure out.<a href="https://github.com/hashicorp/vault" target="_blank">Vault</a> is an open source key management system by <a href="https://www.hashicorp.com/" target="_blank">HashiCorp</a>. You can use it with <a href="https://www.hashicorp.com/products/consul" target="_blank">Consul</a> and other CI/CD tools to securely manage passwords, keys, and certificates - basically any sensitive data items for your software configurations. Vault is a universal tool to manage all kinds of secrets - API keys, passwords, symmetric keys, certificates. It is open source and free if you can learn and use its CLI or RESTful API, if you want a graphic interface, you can go for a paid license.In a sense, the overall concept of the Vault is simple. There is a data storage - called Backends. On top of that are Secrets Engines. Each provides access and a logic for a particular type of secrets. On top of that are Auth Methods, which provide access control according to your access control policy. The use of the Vault is logged in JSON files and there's an "audit" command to inspect those.I didn't mention the "Seal" concept yet. When you start a Vault server, you need to "unseal" the backend storage - it basically gives the server an encryption key to access the backend storage. The Vault will keep the secret in memory so long as it's running. It is not unusual to require 2 or more people (so called dual control) to provide their seal secret to start a new instance of the Vault server. If you don't do it, you expose yourself to an internal threat of a rogue employee. As you can run many instance of the server against any backend, it's easy to launch a new server and pull secrets from it.If I sum up the "sealing/unsealing" - the backend is encrypted with a key that has to be reconstructed from "key shares". The server will keep this key in memory while it runs. If it fails, you need to unseal again. This is one of the main reason why you need several servers in the production environment to prevent denial of service.Let's get started with the PKI Secret Engine using the Vault's CLI. All commands can be replicated with a RESTful API calls.<h2>Download & Installation</h2><a href="https://www.vaultproject.io/downloads.html" target="_blank" style="color: rgb(0, 82, 204);">https://www.vaultproject.io/downloads.html</a> - here you can find binaries for quick get-started, no installation needed, just one file of about 50MB that will unzip to 140MB or so. The supported platforms are OSX, Windows, Linux, FreeBSD, NetBSD, OpenBSD, and Solaris.As the Vault provides a service, you need to run it as a server, and you need a client to run commands. The same binary file works for both if you use the CLI. <a href="https://www.postman.com/" target="_blank">Postman</a> is my preferred option for testing RESTful APIs.Before you start the server, you need an initial configuration file that will define:<ul><li>a path to data; and</li><li>the address and port for the server</li></ul>You can use '#' to comment out lines you don't need. Here's my simple test.hcl file:<code class="ql-font-monospace" style="color: rgb(23, 43, 77);">storage "file" {</code><code style="color: rgb(23, 43, 77);" class="ql-font-monospace"> path = "/Users/dcvrcek/Downloads/vault.cfg/data"</code><code class="ql-font-monospace" style="color: rgb(23, 43, 77);">}</code><code class="ql-font-monospace" style="color: rgb(23, 43, 77);">listener "tcp" {</code><code class="ql-font-monospace" style="color: rgb(23, 43, 77);">   address = "127.0.0.1:8200" tls_disable = 1</code><code class="ql-font-monospace" style="color: rgb(23, 43, 77);">}</code><code class="ql-font-monospace" style="color: rgb(101, 84, 192);">#listener "tcp" {</code><code class="ql-font-monospace" style="color: rgb(101, 84, 192);"># address = "172.16.8.29:8200"</code><code class="ql-font-monospace" style="color: rgb(101, 84, 192);"># tls_disable = 1</code><code class="ql-font-monospace" style="color: rgb(101, 84, 192);">#}</code>Note: I have played with a free UI <a href="https://github.com/djenriquez/vault-ui" target="_blank" style="color: rgb(0, 82, 204); background-color: rgb(255, 255, 255);">https://github.com/djenriquez/vault-ui</a>, which I didn't find that useful, really. However, it needs a non-localhost address to work.OK, now we can start the server:<pre class="ql-syntax" spellcheck="false">./vault server --config test.hcl </pre>You should see something like this:          <code> Cgo: disabled</code><code>       Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")</code><code>        Log Level: info</code><code>          Mlock: supported: false, enabled: false</code><code>      Recovery Mode: false</code><code>         Storage: file</code><code>         Version: Vault v1.3.2</code>Besides a few technical messages, you can see that we have successfully set up a basic configuration. The only mandatory items are the listener and the backend/storage, but you can add the following items later on:<ul><li>listener - where we can reach the server.</li><li>storage - the selection of a Backend - in our case it's "file", once you stop testing, you can choose from a wide selection from Zookeeper to MySQL, PostgreSQL, S3, to in-memory.</li><li>seal - it's an additional protection of the Vault - you can use an HSM, one of the cloud key vaults (AliCloud, Google, AWS, Azure), and a couple more.</li><li>telemetry - i.e., performance data - here you can specify an upstream server to collect the Vault's performance data.</li><li>entropy - an additional source of entropy for secrets generation.</li></ul><h2>Vault Client</h2>Now we have a server running in one terminal window, we move to another terminal window and set an env. variable VAULT_ADDR that should reflect the listener configuration.<pre class="ql-syntax" spellcheck="false">export VAULT_ADDR=http://127.0.0.1:8200 </pre>The important bit is to define the use of "http". The client would otherwise complain about an HTTPS/HTTP mismatch. We have a server running and we have an environment set up for the client. What we need to do first is to initialize the storage - create a "seal".<h2>Initialize the Vault instance</h2>First you need to initialize the "store backend. You can skip the parameters key-shares and key-threshold. The default values are 5 and 3, i.e., you need three to enter 3 shares out of 5 to unseal the backend storage.<pre class="ql-syntax" spellcheck="false">./vault operator init -key-shares=1 -key-threshold=1 </pre>The client will print the share and "root token" into the terminal:<code>Unseal Key 1: pvoVBIL9i+mcvU3iGXaToEAuocpACuO++IVZ0nGqPqY=</code><code>Initial Root Token: s.pST0K6ecmyK0eCXbACrtohjZ</code>Now comes the really hard bit - at least for long-term use of the Vault. What to do with these values. You need to keep them somewhere. The unseal keys should be distributed to different persons AND no person should have access to more than one key (unless it's part of your security policy).The root token is needed to load the initial security policy. You can then create tokens to manage particular parts of the policy and access particular Secrets Engines and secrets within. This initial token should be revoked as soon as practically possible.Note: when you stop the server (don't forget to start it again) you can see that it created a data folder "vault.cfg" - as set in the configuration file, with an initial structure.<h2>The First Unseal</h2>We have created an encrypted backend storage. The next step to do the first unseal - so you need the "threshold" number of keys. It's worth mentioning that you can use different clients to send an unseal key to the server. So if you need your team cooperation, each of them can use a client on their laptop ... so long as they can access the server's API. <pre class="ql-syntax" spellcheck="false">./vault operator unseal pvoVBIL9i+mcvU3iGXaToEAuocpACuO++IVZ0nGqPqY= </pre>When enough keys are entered, the server can calculate the Master Key, which will be in memory while the server is running. The client will also print the server's config:<ul><li>Seal Type - shamir</li><li>Initialized - true</li><li>Sealed - false</li><li>Total Shares - 1</li><li>Threshold - 1</li><li>Version - 1.3.2</li><li>Cluster Name  - vault-cluster-35aa9a47</li><li>Cluster ID - a5e043be-f857-2f8a-bd92-c048acdcdda1</li><li>HA Enabled - false</li></ul>The next step is to authenticate using the root token - it's basically an API key of sorts.<pre class="ql-syntax" spellcheck="false">./vault login token=s.pST0K6ecmyK0eCXbACrtohjZ </pre>You should see a "Success!" message with some important details:<ul><li>token - value of the token you used;</li><li>token_accessor - it's a token handle that you can use to query properties of the token - quite useful for auditing existing tokens;</li><li>token_duration  - how long till it expires, the default validity is forever</li><li>token_renewable - some tokens require renewals and they will expire if you don't do that. If you have a long running process, it can renew its token after 5 mins. If it dies the access automatically expires.</li><li>token_policies    ["root"] - the scope of the token</li><li>...</li></ul>Alright, you should be set up and we can start playing with the PKI Secrets Engine. Just a quick recap. We have:<ol><li>started a server with a simple default configuration;</li><li>initialized the data storage;</li><li>unsealed the storage; and</li><li>authenticated ourselves so we now have full access to the Vault server and its backend.</li></ol><h2>PKI Initialization</h2>Each Secrets Engine has to be enabled/mounted before its first use. It creates an instance of the selected Secrets Engine. For the PKI engine, you need a new mount for each CA you want to use. If you need a root and 2 issuing CAs, you will need to mount the pki three times. We will start with a root <pre class="ql-syntax" spellcheck="false">./vault secrets enable -path=rootca pki </pre>and one issuing CA.<pre class="ql-syntax" spellcheck="false">./vault secrets enable -path=ca pki </pre>That's it - we have created two certification authorities! Now comes the tuning. Let's start with the validity of the root CA cert. In the Vault's language it's "max-lease-ttl". Let's try 10 years.<pre class="ql-syntax" spellcheck="false">./vault secrets tune -max-lease-ttl=87600h rootca </pre>And the issuing the CA will have a cert valid 1 year.<pre class="ql-syntax" spellcheck="false">./vault secrets tune -max-lease-ttl=8760h ca </pre>That's the basic setup. Let's generate the first certificate for the root CA. The default name would be something like "myvault.com" but we can change it with a "common_name" parameter, and we can, e.g., specify the validity of certs a CA will be creating.<pre class="ql-syntax" spellcheck="false">./vault write rootca/root/generate/internal \ common_name=myrootca.com \ ttl=87600h </pre>You will get a new certificate printed on the screen. The private key is stored in the backend. Note: notice that the path in the command contains a keyword "root". This tells the Vault that it is a root CA and it will automatically create a self signed certificate. The next command is for "intermediate" CA. As a result, you will only get a CSR that will have to be signed by another CA.<pre class="ql-syntax" spellcheck="false">./vault write ca/intermediate/generate/internal \ common_name="Issuing CA" \ ttl=8760h </pre>Note: mind the space before "\".We can now store the CSR in a file and get it signed. Let's create, pki_int.csr , e.g. with a vim or any other editor - copy&paste the PEM string of the CSR into this new file.We can now call the rootca CA to sign the certificate request. We want the result as a PEM data with the whole chain = "pem_bundle".<pre class="ql-syntax" spellcheck="false">./vault write rootca/root/sign-intermediate csr=@pki_int.csr format=pem_bundle </pre>You need to copy&paste the result into a file or redirect stdout when you call this "write" command.Hooray - we have a certificate for the issuing CA. Let's import it. now you can import the signed certificate to the issuing CA<pre class="ql-syntax" spellcheck="false">vault write ca/intermediate/set-signed certificate=@signed_certificate.pem </pre>You should get back a "Data written" message.<h2>Policy/user and first certificate</h2>The next step is to create a role for clients to request certificates - lets' say we have one agent requesting certificates for many clients in the <a href="http://example.com/" target="_blank" style="color: rgb(0, 82, 204);">example.com</a> domain. Let's restrict the issuing CA to this domain. We can also set the certificate validity.<pre class="ql-syntax" spellcheck="false">./vault write ca/roles/example-dot-com \ allowed_domains=example.com \ allow_subdomains=true \ max_ttl=72h </pre>This command created a policy "example-dot-com". It has also created an entry in the policy file that you can assign a new token to. Whoever shows this new token will only be able to issue certificates under this "example-dot-com" policy.Let's create a first end-user certificate.<pre class="ql-syntax" spellcheck="false">./vault write ca/issue/example-dot-com \ common_name=blah.example.com </pre>The command will generate a private key and create a certificate for it. You get back:<ul><li>ca_chain - [ root CA in PEM, CA in PEM ]</li><li>certificate - PEM data</li><li>expiration - timestamp</li><li>issuing_ca - PEM data</li><li>private_key - PEM / PKCS1</li><li>private_key_type - rsa</li><li>serial_number = string</li></ul>You may need a bit more structured output for automation. I like JSON so I would just add "-format=json" to the command. Note: the private key is printed out only once so you need to carefully capture the output so you can install it. This tutorial loosely follows instructions from here - <a href="https://www.vaultproject.io/docs/secrets/pki/index.html" target="_blank">https://www.vaultproject.io/docs/secrets/pki/index.html</a> . Hopefully the text above should help you avoid some difficult bits that took me a little to get right. <blockquote>We really like vault and we work on integrating Vault with <a href="https://keychest.net" target="_blank">KeyChest</a> and with our hardware root CA service. The goal is to offer a very, very secure storage of root keys with a convenient remote access to generate keys for intermediate CAs or even end-points. If you'd be interested in this integration, drop us a line at <a href="support@keychest.net" target="_blank">support@keychest.net</a> .</blockquote>While the CLI is convenient for first testing, I would certainly recommend switching to the RESTful API as soon as you get to grips with the PKI secrets engine. Here's a query to get the configuration of our Issuing CA so you can see the flexibility you have:<code>curl   --header "X-Vault-Token: s.pST0K6ecmyK0eCXbACrtohjZ" http://127.0.0.1:8200/v1/ca/roles/example-dot-com</code>{<ul><li>"request_id":"97dfbae4-0a68-5e88-df0c-3c6db990d70d",</li><li>"lease_id":"",</li><li>"renewable":false,</li><li>"lease_duration":0,</li><li>"data":{</li><li class="ql-indent-1">"allow_any_name":false,</li><li class="ql-indent-1">"allow_bare_domains":false,</li><li class="ql-indent-1">"allow_glob_domains":false,</li><li class="ql-indent-1">"allow_ip_sans":true,</li><li class="ql-indent-1">"allow_localhost":true,</li><li class="ql-indent-1">"allow_subdomains":true,</li><li class="ql-indent-1">"allow_token_displayname":false,</li><li>"allowed_domains":[</li></ul><ol><li class="ql-indent-2">"example.com"</li></ol><ul><li class="ql-indent-1">],</li><li class="ql-indent-1">"allowed_other_sans":null,</li><li class="ql-indent-1">"allowed_serial_numbers":[],</li><li class="ql-indent-1">"allowed_uri_sans":[],</li><li class="ql-indent-1">"basic_constraints_valid_for_non_ca":false,</li><li class="ql-indent-1">"client_flag":true,</li><li class="ql-indent-1">"code_signing_flag":false,</li><li class="ql-indent-1">"country":[],</li><li class="ql-indent-1">"email_protection_flag":false,</li><li class="ql-indent-1">"enforce_hostnames":true,</li><li class="ql-indent-1">"ext_key_usage":[],</li><li class="ql-indent-1">"ext_key_usage_oids":[],</li><li class="ql-indent-1">"generate_lease":false,</li><li class="ql-indent-1">"key_bits":2048,</li><li class="ql-indent-1">"key_type":"rsa",</li><li>"key_usage":[</li></ul><ol><li class="ql-indent-2">"DigitalSignature",</li><li class="ql-indent-2">"KeyAgreement",</li><li class="ql-indent-2">"KeyEncipherment"</li></ol><ul><li class="ql-indent-1">],</li><li class="ql-indent-1">"locality":[],</li><li class="ql-indent-1">"max_ttl":259200,</li><li class="ql-indent-1">"no_store":false,</li><li class="ql-indent-1">"not_before_duration":30,</li><li class="ql-indent-1">"organization":[],</li><li class="ql-indent-1">"ou":[],</li><li class="ql-indent-1">"policy_identifiers":[],</li><li class="ql-indent-1">"postal_code":[],</li><li class="ql-indent-1">"province":[],</li><li class="ql-indent-1">"require_cn":true,</li><li class="ql-indent-1">"server_flag":true,</li><li class="ql-indent-1">"street_address":[],</li><li class="ql-indent-1">"ttl":0,</li><li class="ql-indent-1">"use_csr_common_name":true,</li><li class="ql-indent-1">"use_csr_sans":true</li><li>},</li><li>"wrap_info":null,</li><li>"warnings":null,</li><li>"auth":null</li></ul>}

Publication Date: 2020-02-06 14:57:00

KEYCHEST - Confidence In Your Online Business

While KEYCHEST as a brand started as a straightforward expiry management service for Let's Encrypt, it has become a service with a rich set of features and technologies to give you confidence in your online business uptime.

While KEYCHEST as a brand started as a straightforward expiry management service for Let's Encrypt, it has become a service with a rich set of features and there is still several technologies that wait for production deployment.<hr>The ultimate goal of KEYCHEST is to give you the confidence that your online business keeps running without interruptions. While you can create a new website in minutes, its online visibility now depends on a huge number of technologies depending on automations. If any of them fails, it has a direct impact on your business.KEYCHEST will give you advance warnings when your automated processes fail well before they can impact your business. It will let you know when you reach the limits of technology. The limits that you should never worry about nor understand. Whether its rate limits of Let's Encrypt certification authority, temporary network failures, changes in web browsers, data caching. We use a range of technologies to provide this confidence in a form of an unmetered service for a fixed monthly or annual cost. Some call it PKI (public key infrastructure), what we offer is much more - it's automated management of your web security and PKI is just a small part of what powers KEYCHEST. We believe web security should be a simple service that just works - like a network to connect you with your customers, a database to store your data, or a payment service that securely collects payments for your goods.<blockquote>4% of businesses experienced more than 100 outages in a year due to expired web or internal services, while 79% of businesses experience at least one outage in a year. (Venafi market research).</blockquote>The KEYCHEST architecture integrates several cutting-edge technological innovations developed by a team of academic researchers and experienced security professionals. These technologies tackle the core issues that determine a successful automation of key management. This enables KEYCHEST to integrate existing key management products into a scalable service. But ultimately, to give you the confidence that the web security keeps powering your business, not taking it down.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="KeyChest can be the gatekeeper for all your certificate management" src="/storage/storage/wink/images/ObeHcZYBQzZ5bDuydZW2k0EnjctUaW0PvBKqqO6Y.png">KeyChest can be the gatekeeper for all your certificate management</div>KEYCHEST own technologies include:<ul><li>CloudFoxy - secure hardware “virtualization” platform providing access to large numbers of high-security chips. There is a specific level achieved in independent security evaluations - FIPS140-2 Level 4 for physical security. This level of security is quite unique, but we can offer this in the form of a cloud service;</li><li>PKI and encryption service for high-security chips - it's a bit of a mouthful, but this provides one crucial service - cloud-friendly encryption management with enrollment automation;</li><li>Instant PKI – root CAs and PKI instances with FIPS140-2 L3 compliant key storage – legacy (PrimeKey/EJBCA), as well as modern products (e.g., HashiCorp Vault), we try to diverge from legacy PKI systems that are clunky, difficult to setup and manage but they will be used for quite some time due to internal dependencies;</li><li>KEYCHEST Expiry Management Service – a cloud management service with a global database of all public certificates with instant search. It is also the integration framework for all other technologies that your business may need. From an ssl expiry check, a web domain expiration audit, to real-time updates.</li></ul><h2>Technology Means Little - What You Really Need is The Confidence in Your Business Uptime</h2>This post is an overview of our technology that provides everything you may need: renewal of certificates, one view with all incidents, management support for products you may use to manage your certificates, a detail information for a given endpoint (ssl expiry check, configuration reports, continuous monitoring) as the big picture of your whole business.All that will only help your business if it can ensure that it keeps an eye on every part of your online business. And that's what KeyChest does - it finds any new parts of your secure business configuration on its own. You simply can't forget to manage your new service - we add it automatically. The reason is because we want to give you the confidence in your computers, networks, and services. <h2>Global Certificate Database</h2>We built our own global database of internet certificates issued over the last several years. It currently contains almost 10,000,000,000 entries (early Feb 2020) and provides instant search for domains and subdomains. The database itself will add new certificates in as little as 20 seconds of their creation. This provides KEYCHEST with a global view of internet encryption that can be used e.g. to identify phishing attacks using similar domains, growth of company infrastructures, quality of encryption management, and so on.<h2>CloudFoxy</h2> While a hardware platform, it truly virtualizes secure chips for high-security applications. It can provide up to 600 secure chips for high-value keys in one server, accessible with modern web APIs. It’s versatile design can be used for multi-party signing or encryption, which we demonstrated with <a href="https://backdoortolerance.org/" target="_blank">UCL London</a> at the BlackHat and DEFCON in 2017.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="CloudFoxy is a versatile security platform suitable for root CAs as well as high-security (e.g., eIDAS) document signing. Timing figures for for eIDAS signing, including PIN validation)." src="/storage/storage/wink/images/dtA2fgLNfkQFqTN3CfiHtwDkogrxJfKATscwB9T8.png">CloudFoxy is a versatile security platform suitable for root CAs as well as high-security (e.g., eIDAS) document signing. Timing figures for for eIDAS signing, including PIN validation).</div>“Full-stack” PKI integrated service – demonstrated on Amazon AWS, allows launching new PKI systems with FIPS140-2 Level 3 hardware in a fully automated fashion.<h2>KeyChest service</h2> It's the main integration service for all other technologies. Its automatic, continuous discovery of new certificates makes it an ideal tool for internal and external audits in small enterprise environments. You don't have to install a single piece of software to manage your web security and we offer lightweight software agents with a granular access control you can enforce internally.KeyChest now integrates renewal automation with third-party trust providers Thawte, Symantec, GeoTrust, COMODO and SectiGo via APIs.

Publication Date: 2020-02-06 10:09:00

Quick Inspection of Web Endpoints (incl.SSL Expiry Check)

KeyChest is about keeping your business up and running by preventing the expiry of important web services - this is our goal. While it may be prudent to reach A+ rating in specialized audit tools (like SSL Labs), it will not prevent your business downtime 3 months later when your super-secure ordering service expires.

KeyChest is about keeping your business up and running by preventing the expiry of important web services - this is our goal. While it may be prudent to reach A+ rating in specialised audit tools (like SSL Labs), it will not prevent your business downtime 3 months later when your super secure ordering service expires.<hr>KeyChest gives you all the information you need to keep your online business in good shape. It allows you to plan certificate renewals and tells you when something is going to break and needs a closer look. This protects you from downtimes as you can plan certificate renewals with enough to resolve any potential problems. Instant audits of KeyChest also help you set up your servers so that your users, customers, and clients can use them 24x7. We detect issues that may cause random unexpected problems to access your web services and we also provide a baseline security information.In this short post, I want to focus on these instant audits. KeyChest now offers two main types of instant audits - domain and endpoint. While pcviewx sounds too technical as a name, this KeyChest SSL audit certainly gives you an extended - view of your servers, maybe kcviewx could work. This quick and easy to use tool integrates several vital audits: ssl expiry check - not only at present but it will also show any gaps in renewals over the last 2 years - you can also add the given certificate to our continuous SSL expiry monitoring. SSL configuration audit - correct SSL bundle configuration, SSL server configuration, latency. A baseline security audit: that shows all algorithms supported by the given server.<h2>Endpoint Audit - Step 1</h2>The first step is simply to select the "Instant Audit" item in the menu - or click <a href="https://keychest.net/home/audit/run" target="_blank">here to go there directly</a> (it requires a registration, which is FREE and only asks for an email, and name - you can use one of support social networks login).All you now have to do is type in a domain name you want to audit - let's say we want to have a look at <a href="https://keychest.net/home/audit/run?url=teams.microsoft.com" target="_blank">teams.microsoft.com</a>. Once you click the button it only takes a few seconds to get a summary of the server. It looks like this.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Instant audit of an endpoint" src="/storage/storage/wink/images/BkAht7cgoM6MXjMLeI5TwlgVD2EqdHk9mNYEKKh6.png">Instant audit of an endpoint</div>This server has a very good configuration so you can't see any error or warning message at the top of the results. The audit will only show what you need to see once it checked:<ul><li>DNS configuration - resolving IP addresses from your server name;</li><li>Server configuration - warning if there is no server at all listening at the given server and port;</li><li>SSL configuration - if your server uses insecure version SSL2 or SSL3, TLS1.0, TLS1.1, it will be displayed (see errors below);</li><li>certificate expiration - how many days till the certificate expires;</li><li>downtime - downtime during the last 2 years; CT logs data amended with server checks if this data if available;</li><li>trust chain - whether the server provides a complete chain of certificates needed  for validation;</li><li>certificate issuer - it shows the name of the certificate issuer (if set);</li><li>list of neighbours - the list of all names in the certificate;</li><li>hostname match - whether the name(s) in the certificate contain the server's name;</li><li>HSTS - if the HSTS (HTTP Strict Server Security) is enabled;</li><li>HTTP redirection - an active redirection, which sends web browsers to another server;</li><li>IP addresses - a list of all IP addresses available in the KeyChest's geographic region.</li></ul>I want to mention that at the top of the results, you can see a list of alternative addresses - if there are any. There is one additional IP address for the domain name in the picture above. If you want to audit that, simply click on it and you will get a new set of results.We have recently expanded the audit information with certificate details, and security-related results. You can now see all the algorithms supported by the server and their strength - we use 4 categories: STRONG, OK, WEAK, BAD. If you can see "BAD" you should certainly review the configuration of the server. The "WEAK" algorithms use SHA-1, which should be replaced with a secure version of SHA-2.<ul><li>STRONG - TLSv1.3 algorithms;</li><li>OK - strong algorithms of TLSv1.2 with forward secrecy;</li><li>WEAK - older algorithms using week ciphers - they are still secure for the purpose of HTTPS but you may want to consider their removal; and </li><li>BAD - algorithms that can be successfully attacked.</li></ul><h2>Step 2 - Check The Certificate</h2>The second step is optional and it requires you to click on the magnifying glass in the results - if present.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Certificate detail of your endpoint" src="/storage/storage/wink/images/wVr5UEaBo2gAEVhBUFFXqbneaoMkPY9xJv6s917m.png">Certificate detail of your endpoint</div>The report here can get rather long - especially for users on the SCALE and higher price plan, where it will show the whole history of certificates for a given domain name. The picture above shows the main information from the certificate and also the status of its renewal - in this case, it only shows that everything is good for another 2 years.However, you can see all the IP addresses where we have detected the certificate, the type of the certificate, and other domain names for which it is valid.So here we go - a pcviewx, or kcviewx, or simply a KeyChest audit in a few button clicks. Oh, I forgot - we store the results so you can open your history tab and share your previous audit results with your colleagues.<blockquote><a href="https://keychest.net" target="_blank">KEYCHEST - web expiry management. Website certificate expiration is easily forgotten—causing costly downtime. Our expert service automatically checks and renews your certificates, on time, and correctly, so you can start every day with confidence.</a></blockquote>

Publication Date: 2020-02-05 11:19:00

Microsoft Teams - It's Not Just One Certificate

Microsoft Teams, Slack competitor, went down for nearly three hours after Microsoft forgot to renew a critical security certificate. ... It's not just one certificate you have to keep an eye on, they are many of them.

A friend tagged me yesterday on LinkedIn with an update that Microsoft Teams - a team communication service, something like Slack - had gone down due to an expired certificate. How can this even happen?<hr>Update Feb 5: I received some feedback that pointed to more MS certificates that were expired - <a href="https://markets.books.microsoft.com/" target="_blank" style="background-color: rgba(255, 255, 255, 0.8);">markets.books.microsoft.com</a> (e-books in MS app store - not supported any more) or <a href="https://vlscppe.microsoft.com/" target="_blank" style="background-color: rgba(255, 255, 255, 0.8);">vlscppe.microsoft.com</a> (volume licensing) - it showed expired last night but has a new cert installed now (issued 5 days ago) - both are API endpoints. I am sure there will be a post-mortem, pointing its finger to a number of "regular" near misses that grouped together and let a certificate slip through the net. I was curious though what users think. The best place for immediate feedback is Twitter so I spent a little bit of time just scrolling through hundreds of tweets.Interestingly, a lot of people were forgiving and simply said that it could happen to anyone - and they were right. DEFCON is the synonym for the most hard-core geeks, hackers, and experts on security. Still, their website fell victim of the expired certificate as well in mid-2017. Interestingly, they did renew the certificate (as you can find from public CT logs), they just didn't manage to install it properly.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="DefCon website down due to expired certificate. " src="/storage/storage/wink/images/7oMGMHPj0r2wLEoM0zBREXS0cvF5JoM3GUW8qcCG.png">DefCon website down due to expired certificate. </div> A large portion of people on Twitter tried to suggest "a simple solution" - calendar reminders, TO-DO tasks. The trouble is that these would have their deadline in a year or two - enough time to cause trouble again when the time comes. With the DEFCON incident in mind, it's actually much harder than it seems.Let's have a look at the <a href="https://keychest.net/auditnow?uuid=d9d97b40-4760-11ea-9576-1f869d48871c&url=teams.microsoft.com" target="_blank">teams.microsoft.com</a> domain using our quick domain audit tool (<a href="https://keychest.net/auditnow" target="_blank">https://keychest.net/auditnow</a>). Here's the screenshot of a summary.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Status of "teams.microsoft.com" domain as shown in our domain audit tool." src="/storage/storage/wink/images/tzaI5qz94WDvLkX2JKkVGn62dOli9SvGPYgvq1Df.png">Status of "teams.microsoft.com" domain as shown in our domain audit tool.</div>You can see that it has 19 Critical items (expired, or expiring within a week), and 4 imminent (expiring within 7-14 days). You can also see that there are 512 services, i.e., domain names within the "teams.microsoft.com" with either a valid certificate or a certificate expired in the last 4 weeks.500 is a huge number from the management point of view. Many of them will be abandoned - domains created for a particular project that has finished, temporary domains, or simply services not relevant anymore. It means that someone has to be able to say whether a particular domain is still "live" or whether it can be ignored. In my opinion, domains with certificates should be only abandoned once explicitly announced as such.Let's quickly have a look at the certificates that expired (or probably expired - this is a quick audit and change of issuer may cause inaccuracies) in the last 4 weeks.<ul><li> auditservice-staging.teams.microsoft.com - 09 Jan 2020, 05:49</li><li> auditservice.teams.microsoft.com - 17 Jan 2020, 03:29</li><li> auditservice-int.teams.microsoft.com  - 18 Jan 2020, 15:51</li><li> *.urlp.gcc.teams.microsoft.com  - 25 Jan 2020, 12:00</li><li> urlp.gcc.teams.microsoft.com - 25 Jan 2020, 12:00</li><li> stage.urlp.gcc.teams.microsoft.com - 25 Jan 2020, 12:00</li><li> *.stage.urlp.gcc.teams.microsoft.com - 25 Jan 2020, 12:00</li><li> eastus2.fabric.int.teams.microsoft.com - 28 Jan 2020, 12:56</li><li> emailactions.teams.microsoft.com - 29 Jan 2020, 16:20</li><li> emailactions-test.teams.microsoft.com - 29 Jan 2020, 16:20</li><li> emailactions-int.teams.microsoft.com - 29 Jan 2020, 16:20</li><li> retentionhook-int.teams.microsoft.com - 01 Feb 2020, 16:23</li><li> retentionhook-test.teams.microsoft.com - 01 Feb 2020, 16:23</li><li> retentionhook.teams.microsoft.com - 01 Feb 2020, 16:24</li><li> *.smba.gcc.teams.microsoft.com - 02 Feb 2020, 12:00</li><li> smba.gcc.teams.microsoft.com - 02 Feb 2020, 12:00</li><li> cachewriter-int.teams.microsoft.com - 02 Feb 2020, 18:20</li></ul>You can see that many of them are for test or staging systems - i.e., possibly safe to ignore. Some domains are not active (e.g., emailactions-int.teams.microsoft.com). Some, however, are hard to judge as they are publicly available and still expired (e.g., auditservice.teams.microsoft.com).This is only a "domain name" view. Each domain can have a number of separate services - https, imap, message queues, databases, etc. This makes management of expiry even more complicated. Further, the number of certificates never goes down. In my experience, having a 10% annual growth is not so bad place to be at.If you now think that managing certificates is hard then this is only the tip of the iceberg. Technology companies would have 10-100x more internal certificates providing internal infrastructure security). If we can see 500 domains within "teams.microsoft.com", it's very likely that there are another 50,000 certificates securing databases, APIs, inter-server communication, message queues, and so on.The bottomline is that things can get very messy once the number of certificates gets beyond a couple of dozen (with Let's Encrypt) and something like 100 for long-term certificates. Still, one would think that Microsoft is so close to technology to have mastered this side of its business by now.There are many services that try to be too clever with a complicated management and workflow approach. I have had my own experience and decided that the best way to do it is to copycat Google. Do you remember how it started? They wisely figured out that any data classification is hard and error-prone and it's much better to simply work with full-text data and search and build logic centrally. That's what we are doing with KeyChest for companies of any size and any budget. <blockquote><a href="https://keychest.net" target="_blank">KEYCHEST Web Expiry Management - website certificate expiration is easily forgotten—causing costly downtime. Our expert service automatically checks and renews your certificates, on time, and correctly, so you can start every day with confidence.</a></blockquote>

Publication Date: 2020-02-04 15:14:00

How Let's Encrypt Works

There is relatively little information about how Let's Encrypt actually works. Read on how is your HTTPS certificate created and how the magic works.

You may well know that Let's Encrypt is a not-for-profit organization that provides SSL certificates for free. You may also know there is a huge number of "clients" - small software packages that you need to install on your server to start using Let's Encrypt. There is relatively little information about how it actually works.<hr>I think it's important to bear in mind that Let's Encrypt is funded and managed by  <a href="https://www.abetterinternet.org/" target="_blank" style="background-color: rgb(253, 253, 253); color: rgb(23, 86, 169);">Internet Security Research Group (ISRG), aka "abetterinternet.com</a> . Its funding comes from many sponsors but  <a href="https://www.abetterinternet.org/documents/2019-ISRG-Annual-Report-Mobile.pdf" target="_blank">40% is provided by "platinum" and "gold" level</a> (Mozilla, CISCO, OVHcloud, EFF, Chrome, Facebook, and Internet Society, IdenTrust, and Ford Foundation).The Let's Encrypt's initial goal was to automate SSL management and provide it for free as a vehicle to push for 100% web encryption. I believe that the availability of free certificates was a significant aspect when Google, Apple, Microsoft, ... started discouraging users from visiting insecure web sites (i.e., web sites without HTTPS). Let's Encrypt does a great job when you have a few websites or a couple of administrators who can build in-house tools to ensure that all is running smoothly and resolve failures. I have summarised some of the <a href="https://keychest.net/stories/what-are-disadvantages-of-lets-encrypt" target="_blank">issues to consider before you start using Let's Encrypt in a previous blog post</a>.<h2>Let's Encrypt and Automation</h2>Let's Encrypt has developed its standard defining the API and operational concepts. They called it Automatic Certificate Management Environment (ACME). They built the certificate authority using the initial version but they also created a working group to develop an open standard. This has been <a href="https://tools.ietf.org/html/rfc8555" target="_blank">finalized</a> (currently RFC-8555 in the "proposed standard" state) in March 2019. Because it has introduced some breaking changes, Let's Encrypt started calling it ACMEv2, possibly because they implemented a new API compliant with the RFC-8555 standard and named it "acme-v02". Although it's simply "ACME" for the rest of the world.The automation is based on a RESTful API that requires users to create an account identified by a public key. This key is then used to protect all messages sent from clients to an ACME server (= Let's Encrypt).The protocol is defined around the following terms:<ul><li>account - an asymmetric key that identifies a client. It is also used to enforce some of the limits on the API usage, so-called "rate limits". Some of the rate limits are also measured per registered domain or the IP address of the client.</li><li>order - this is an actual request for a new certificate.</li><li>authorization / authz - it's an object with the validation status of an order. Each order has one validation object.</li><li>challenge - the challenge is an object that defines a method and parameters to complete authorizations. Each validation has several challenges - one for each validation method (Let's Encrypt offers HTTP, DNS, and TLS-ALPN).</li><li>certificate - it's an "address" from which you can eventually download a new certificate. This address is pretty much permanent.</li></ul><h2>Protocol</h2>The picture below shows the three basic steps of the ACME protocol. A client (e.g., acme.sh, certbot) will initiate an order and obtain back authentication data – step 1. At this point, the only specific information sent by the client is a list of domain names (i.e., no CSR).This authentication is absolutely crucial as it ensures that only you can request certificates for your servers. In other words, via the authentication, you prove the control over your internet domain name and servers. You can prove this control either by adding the authorization data to your web server (so it can be accessed with the HTTP protocol by the Let's Encrypt service) or by adding the authorization data to your DNS records.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="ACME Issuance Protocol" src="/storage/storage/wink/images/RgwMc3yKEwEx8qcPvcDxXPkbVIeJUwDiRmzChRXJ.png">ACME Issuance Protocol</div>Step 2 is the actual validation of your domain control. It is also the most difficult step as you need to integrate the certificate renewal with either your web server or your DNS management.Note: I’m sure you noticed that in the diagram, the data for Step 2 magically appear where it is expected. As no magic, it has to be a sophisticated technology.Let’s Encrypt will try to collect the authorization data it provides in step 1 using one of the available methods. Once the authorization is completed, Letsencrypt will store a completed autherization for 7 days. This means you have 7 days to request a certificate for the given domain names without a need to repeat the authentication for the domains you specified in step 1. As you can see all the client provided to the ACME server (i.e., Lets' Encrypt) is its account ID (registered with the server) and a list of domain that is sent in step 1. The key for a new certificate or the CSR hasn't been needed yet. While it simplifies the API, it also gives us huge flexibility in how we can implement the final step - the actual certificate request.What we need for Step 3 is:<ol><li>the address for the "finalize" step (it looks like api.letsencrypt.org/acme/finalize/51135296/308109706)</li><li>the account key</li><li>and of course, the domain names that should be in the certificate.</li></ol>This final step is a simple API request - the client sends a CSR and downloads the new certificate. This means that while the validation has to have access to your DND or web server, the Step 3 can be done from anywhere on the internet.<h2>KeyChest and Let's Encrypt</h2>This flexibility of the ACME protocol is something that KeyChest is exploiting to provide a managed service for companies or those of you who need to manage more than a handful of certificates.<ul><li>KeyChest Amp - our open source proxy was the first step that allows collecting logs from all your clients in one place. You can use it as a data source for your own audit systems. KeyChest will add a new web dashboard in the next couple of weeks and you will be able to see the last activity of clients to detect failures and rate limit information. You can download it from GitLab - <a href="https://gitlab.com/keychest/keychestamp/wikis" target="_blank">https://gitlab.com/keychest/keychestamp/wikis</a></li><li>KeyChest AmpX - is another proxy that fully uses the flexibility of the ACME protocol. It creates its own Let's Encrypt accounts to manage certificate renewals while you still keep control over the operations. But it also unifies Let's Encrypt with long-validity certificates. It can decide whether to request a new certificate from Let's Encrypt DigiCert, Comodo, GeoTrust, etc.</li></ul><blockquote><a href="https://keychest.net" target="_blank">KEYCHEST - HTTPS Exiry Management - website certificate expiration is easily forgotten—causing costly downtime, our expert service automatically checks and renews your certificates, on time, and correctly, so you can start every day with confidence.</a></blockquote>

Publication Date: 2020-01-24 08:32:00

"Unbreakable" Pen&Paper Encryption

We used one-time pad - the only encryption scheme that is unconditionally secure (no-one can break it - with a small caveat). The real challenge was to create an encryption table that would make it easy for anyone to encrypt and decrypt a message.

A friend came over to our office one day (some years ago) and started talking about the possibility of giving people a chance to encrypt messages without computers, just with a pen and paper. They would write a message, encrypt it by hand, burn/eat/melt the encryption tool (i.e., a sheet of paper), and send the message.<h3><a href="https://www.smartarchitects.co.uk/penpaper/script.py" target="_blank">CREATE ENCRYPTION SHEET NOW</a></h3>We thought it would be fun to create something useful and so we, eventually, created a procedure and a tool. It was quite easy from the cryptography point of view as we used a one-time pad - the only encryption scheme that is unconditionally secure (no-one can break it - with a small caveat). The real challenge was to create an encryption table that would make it easy for anyone to encrypt and decrypt a message.Due to our limitations to read small letters, our encryption sheets only allow to encrypt 50 characters. Longer messages will need several encryption sheets.<ul><li>The code is at GitLab - <a href="https://gitlab.com/cyberfun/penpaper/" target="_blank">https://gitlab.com/cyberfun/penpaper/</a></li><li>To print a sheet (it may download a PDF when clicked) - <a href="https://www.smartarchitects.co.uk/penpaper/script.py" target="_blank">https://www.smartarchitects.co.uk/penpaper/script.py</a></li></ul><div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Encryption sheet for one time pad" src="/storage/storage/wink/images/yyaDuE7Yj6QrZo7qqcd3Cymv5z4pqXeddExEro9X.png">Encryption sheet for one time pad</div><h2>How To</h2>To send a secure message, you need a hardcopy of an encryption sheet - that's what this repository is about.However, like with any other security, the difference between the smart and silly is how to use the tools you have.Here's a "secure procedure" that ensures that noone will be ever able to decipher encrypted messages.<h2>Basic Rules</h2><ol><li>You need a set of encryption sheets;</li><li>Each sheet can be used only once. If you need to send 10 messages, you need 10 different sheets (unless you are just testing to see whether it works);</li><li>One sheet allows encryption of a message of up to 50 characters;</li><li>Each sheet contains a code number that needs to be sent with the message - 5 letters that are in large font should be enough - so that the recipient can find the right decoding sheet; and</li><li>Sheets MUST NOT be disclosed to a third party, as that would compromise the message.</li></ol><h2>Basic Use</h2><ol><li>Print as many encryption sheets;</li><li>Fill in the names of people who share an encryption sheet. (1) the person how keeps the decrypting part of the sheet, (2) who will be encrypting/sending.</li><li>Cut the sheet into two - the upper part goes to the sender, the lower part goes to the recipient.</li></ol><h3>Sending</h3><ol><li>Write the message in the top row of empty boxes - maximum is 50 characters - skip spaces or replace them with 'X's;</li><li>For each character, find it in the column beneath it on the left, and write the character next to it into the box at the bottom of the column;</li><li>Once you've done all the characters, destroy everything above the line "CUT HERE - DESTROY UPPER PART"; and</li><li>Send the remaining piece to the recipient directly or type it into an SMS or email</li><li>You have to include the 5 letters of the sheet code - that will help the receiver find the decryption sheet.</li></ol><h3>Receiving</h3>Find the sheet with the code as received with a message. Write the received message in the top row of boxes; For each box, find its letter in the left column beneath and write the character next to it in the box at the bottom of the column; You can read the message now (spaces are missing or replaced with 'X's). You should now destroy the whole sheet.<h2>How to Use Sheets - Professional</h2>The 'professional' means that there is more than 2 people and you use this to send messages between any two of them. You now have to create your key management / sheet management so that encryption sheets don't get compromised.<ol><li class="ql-indent-1">Make a list of people who want to encrypt messages with our sheets and note who will want to talk to who.</li><li class="ql-indent-1">Estimate how many messages each pair will send in a month (or other suitable time interval) and write it all down.</li></ol>Note: each message requires one sheet and it is a good idea to have a small stock of sheets (too large a stock increases the danger of someone else making a copy and decrypting your messages).<ul><li>If you use a web browser to print sheets - switch your web browser to privacy mode so that no copies of sheets remains on your computer.</li><li>Print as many sheets as you need and close the browser when finished.</li><li>On each sheet, name who will keep the decrypting part of the sheet (1) and who will be encrypting (2).</li><li>Write the date after which the sheet should not be used (0).</li><li>Cut the sheet into two - the upper part goes to the sender, the lower part goes to the receiver (as you've just written on the sheet). Ideally, all steps up to now happen when senders and receivers are present.</li></ul><h3>Sending</h3><ol><li>Write the message in the top row of empty boxes - maximum is 50 characters - skip spaces or replace them with 'X's;</li><li>For each character, find it in the column beneath it on the left, and write the character next to it into the box at the bottom of the column;</li><li>Destroy everything above the line "CUT HERE - DESTROY UPPER PART"; and</li><li>Send the remaining piece to the recipient directly or type it into an SMS or email together with 5 letters of the sheet code.</li></ol><h3>Receiving</h3><ol><li>Find the sheet with the sheet code as received with the message.</li><li>Write received message in the top line of boxes;</li><li>For each box, find its letter in the left-hand column beneath it and write the character next to it in the box at the bottom;</li><li>When you've read the message (spaces are missing or replaced with 'X's) destroy the whole sheet.</li></ol><h2>Compromise Response - "Disaster Recovery (DR) Plan"</h2><ol><li>If you have a suspicion that some sheets have been compromised, senders and recipients must destroy all of their unused copies.</li></ol><h3>Management of sheets</h3><ol><li>Define maximum time for keeping printed sheets and destroy them if you do not use them by then.</li><li>You can define further rules for keeping sheets safe and for their destruction in particular circumstances.</li></ol><h2>Security Caveat</h2>The unconditional security depends on true randomness of encryption sheets. At the moment, this is ensured by the fact that the generation algorithm is on our server and we make sure that everything is erased when a new encryption sheet is downloaded.This however can be improved by:<ul><li>using secure hardware, moving server abroad, ... Let us know if you are interested in a hardened version; or</li><li>generate sheets in Javascript on your computer instead on our server - we may do a Javascript version if we feel idle.</li></ul> -----<blockquote>Website certificate expiration is easily forgotten—causing costly downtime, our expert service monitors your certificates automatically and renews them on time, and correctly, so you can start every day with confidence.</blockquote><blockquote>While <a href="https://keychest.net/auditnow" target="_blank" style="color: rgb(51, 85, 187);">you're here, why don't you have a look at our KeyChest HTTPS expiry management and test your domain for free?</a></blockquote><div class="embedded_image" contenteditable="false" data-layout="default"><img alt="" src="https://keychest.net/storage/storage/wink/images/aGOrZzRS6GvAVl5EWpGJddawB1YrCFuiBsSFdktU.png"></div>

Publication Date: 2020-01-16 09:56:00

Massive MS Windows bug - by NSA - how it works (maybe)

Breaking TLS by sending the same thing twice and MS CryptoAPI not checking that both versions are the same.

I have just skimmed a looong discussion at Hacker News - <a href="https://news.ycombinator.com/item?id=22047573" target="_blank">https://news.ycombinator.com/item?id=22047573</a> - about a vulnerability so big that <a href="https://www.us-cert.gov/ncas/alerts/aa20-014a" target="_blank">NSA was happy to be accredited</a>. (I only wonder whether they spotted someone else using it.)What does the bug do? It basically allows a random, malicious, server to pretend to be a legitimate server so long as the the latter uses ECC certificate. That completely breaks all the protection provided by HTTPS / TLS. Imagine there is no internet encryption. ... got it? Now imagine what you can do with DNS poisoning, MITM, etc.There's a lot of maths and complicated equations, but not really a hint of how the attacker can make MS Crypto API fall for the trap. Update 16 Jan 10:20am UTC: while the general principle below remains the same, it appears that the bug is related to a validation of certificates in MS Crypto API - rather than TLS handshake. You still need to de-couple the signature and EC parameters and the public key itself. The attack includes tweaking the root CA certificate. There was also an initial hint that CT logs can catch the attack but that is not the case.Saleem Rashid did a PoC of the attack and it works on all browsers except Firefox that uses own crypto library.Initial PoC results - <a href="https://twitter.com/saleemrash1d/status/1217495681230954506" target="_blank">https://twitter.com/saleemrash1d/status/1217495681230954506</a>A Chrome follow-up - <a href="https://twitter.com/saleemrash1d/status/1217519809732259840" target="_blank">https://twitter.com/saleemrash1d/status/1217519809732259840</a><h3>Preps</h3>What is clear - the attacker has to create a new ECC key that has the same public key as in a certificate of the server they want to impersonate. They do it by adjusting the parameters of the elliptic curve. In other words, the public key is the same, but the private key and the elliptic curve is different. What does it mean to change the parameters of an ECC? The school maths works with numbers as if they were on a straight line - multiplying by 2 = moving along the line twice the distance from zero. Changing ECC is like bending that line to get numbers next to each other, instead of calculating equations properly.OK - so the attacker has:<ol><li>a valid certificate from a legitimate server</li><li>own key pair - its public key is the same as the public key in the certificate</li><li>ECC parameters used to calculate the own key pair (private key)</li></ol><h3>Launch</h3>All the preps are useless, if we can't make clients (our prey) to cooperate. There aren't many options for the server to force a remote computer to co-operate. It will only "talk" TLS, so the leverage has to be found there.One of the TLS algorithms is a combination of ECDHE (Diffie-Hellman) and ECDSA. It should provide "perfect - forward - security". The crypto standards say that both algorithms have to use the same ECC parameters. But each of the algorithms is defined in a different standard, which caused redundancy - the same information - the ECC parameters are sent twice within each TLS/HTTPS handshake.ECDHE parameters are sent as part of the ServerKeyExchange message, ECDSA parameters are in the certificate. Validation order should be:<ol><li>validate the certificate - verified against a list of trusted root CAs </li><li>extract ECC parameters from the certificate - verified by the cert </li><li>check the signature on TLS message with ECDHE - using the public key of the server</li><li>check the params in the message are the same as ECC parameters</li><li>use the ECDHE with its ECC params</li></ol>It seems that MS Crypto API has done it differently: <ol><li>validate the certificate - verified against a list of trusted root CAs </li><li>get ECDHE and ECC params - NOT verified </li><li>check the signature on TLS message with ECDHE - using ECC params from (2) and the public key from the certificate</li><li>use the ECDH with its ECC params</li></ol>The problem is step (2), which breaks the chain of trust and allows the server to force clients into trusting malicious servers. It's like skipping a few transactions in a blockchain. Anyway, that's my take. It looks feasible but maybe the bug was somewhere completely different :)-----<blockquote><a href="https://keychest.net/auditnow" target="_blank">As you're here, why don't you have a look at our KeyChest HTTPS expiry management and test your domain for free?</a></blockquote><div class="embedded_image" contenteditable="false" data-layout="default"><img alt="" src="/storage/storage/wink/images/aGOrZzRS6GvAVl5EWpGJddawB1YrCFuiBsSFdktU.png"></div>

Publication Date: 2020-01-14 22:32:00

ACMEv2 Clients

A list of ACMEv2 (Let's Encrypt) clients with summaries and popularity (if available, e.g., GitHub stars).

As we have analyzed the existing Let's Encrypt clients for internal use, we realized that there is no public list that would also provide basic description with main features. We used the list of clients from the Let's Encrypt website as a starting point. Added notes of features that we found of interest and re-classified some of the entries to better reflect the nature of each software.We will occasionally review the list but if you find an incorrect info, would like to add a missing project, or change anything, do let us know at support@keychest.net. <h2>Outliers - projects I like for being different</h2>There are obvious, popular, clients that you can see mentioned everywhere. Here's a short list of projects - in no particular order - that tried to approach the integration differently. Maybe I also like them as they share something with our <a href="https://keychest.net/letsencrypt" target="_blank">KeyChest</a>.<ul><li><a href="https://github.com/leandromoreira/tls_certificate_generation" target="_blank">tls_certificate_generation</a> - a Docker project for renewing certificates for domain that are resolved by internal DNS. It launches a temporary AWS/DO VMs. ... the project looks dead (doesn't seem to support RFC8555 / ACMEv2).</li><li><a href="https://github.com/haproxytech/haproxy-lua-acme" target="_blank">haproxy-lua-acme </a>- an HAproxy client. When you need to renew a certificate, you send a POST request to your HAproxy ... instead of to the LE API directly.</li><li>Greenlock - just because the author has built a whole set of tools written in JavaScript for various use cases - see below. </li><li><a href="https://github.com/myfreeweb/freshcerts" target="_blank">freshcerts</a> - a Ruby project (also not supporting ACMEv2) that creates a centralized service allowing lightweight scripts on endpoints. I suspect that it fails with ACMEv2 as it doesn't allow changing the server address.</li><li><a href="https://man.openbsd.org/acme-client.1" target="_blank">acme-client</a> - the BSD version, well because it is part of the distro.</li><li><a href="https://github.com/hlandau/acmetool" target="_blank">acmetool</a> - the Go project - because it doesn't have any dependency and works as a server listening on port 402.</li></ul><h2>Clients</h2><h3>Bash</h3>acme.sh - <a href="https://github.com/Neilpang/acme.sh" target="_blank">https://github.com/Neilpang/acme.sh</a> - stars: 15,600 - a simple script, tests for a number of platforms. Standalone, nginx, and apache modes.dehydrated - <a href="https://github.com/lukas2511/dehydrated" target="_blank">https://github.com/lukas2511/dehydrated</a> - stars: 4,600 - a strong support of hooks for automation, allows use of custom CSRsGetSSL - <a href="https://github.com/srvrco/getssl/tree/APIv2" target="_blank">https://github.com/srvrco/getssl/tree/APIv2</a> - stars: 1,200 - includes ftp/sftp/ssh capability to copy challenges and certificates to remote servers. Configuration files for multiple servers are stored in subfolders.ght-acme.sh - <a href="https://github.com/bruncsak/ght-acme.sh" target="_blank">https://github.com/bruncsak/ght-acme.sh</a>, stars: 9 - key generation requires a separate use of openssl, computes private key thumbprints to pass to nginx for validation.<h3>C</h3>acme-client-portable - <a href="https://github.com/graywolf/acme-client-portable" target="_blank">https://github.com/graywolf/acme-client-portable </a>- stars: 10 - Maintained port of openBDS's acme-clientApache mod_md - <a href="https://httpd.apache.org/docs/trunk/mod/mod_md.html" target="_blank">https://httpd.apache.org/docs/trunk/mod/mod_md.html</a> - manages domain properties for a) renewal of HTTPS certs via ACME and b) offer OCSP stapling.Icing mod_md - <a href="https://github.com/icing/mod_md" target="_blank">https://github.com/icing/mod_md</a> - stars: 255 - an alternate implementation of Apache mod_md with more frequent updates. It provides a) renewal of HTTPS certs via ACME and b) offer OCSP stapling.uacme - <a href="https://github.com/ndilieto/uacme/" target="_blank">https://github.com/ndilieto/uacme/</a> - stars: 59 - lightweight client with only dependencies libcurl and GnuTLS/OpenSSL/mbedTLS. It checks OCSP as part of the 'renewal' command and supports OCSP stapling certs. It also includes a transparent proxying tls-alpn-01 responder. User has to setup and clean up challenges via hooks.<h3>C++</h3>acme-lw - <a href="https://github.com/jmccl/acme-lw" target="_blank">https://github.com/jmccl/acme-lw</a> - stars: 7 - only supports HTTP challenges. Dependencies are cmake, openssl, and curl. Staging / Prod is decided during build.esp32-acme-client - <a href="https://esp32-acme-client.sourceforge.io" target="_blank">https://esp32-acme-client.sourceforge.io</a> - (no downloads in 30 days) - work in progress - renewals not supported, tested as part of an author's app.<h3>Clojure</h3>certificaat - <a href="https://github.com/danielsz/certificaat" target="_blank">https://github.com/danielsz/certificaat</a> - stars: 77 - optimized for "get a cert, setup cron, forget", config files are in the EDN format.<h3>D</h3>amce-lw-d - <a href="https://github.com/cschlote/acme-lw-d" target="_blank">https://github.com/cschlote/acme-lw-d</a> - stars: 3 - install via apt/yum/dub, account creation via an openssl command.<h3>Docker</h3>tls_certificate_generation - <a href="https://github.com/leandromoreira/tls_certificate_generation" target="_blank">https://github.com/leandromoreira/tls_certificate_generation</a> - stars: 26 - ACME client for domains that are either resolved with internal DNS or temporary, the use case in the manual uses a temporary AWS/DO machine launched purely for the purpose of issuing a certificate.<h3>Go</h3>acmetool - <a href="https://github.com/hlandau/acmetool " target="_blank">https://github.com/hlandau/acmetool </a>- stars: 1,700 - binary releases, dependency-free, listens on port 402 to minimise web server configuration changes.autocert - <a href="https://godoc.org/golang.org/x/crypto/acme/autocert" target="_blank">https://godoc.org/golang.org/x/crypto/acme/autocert</a> - (part of golang crypto) - work in progress, library.Lego - <a href="https://go-acme.github.io/lego/" target="_blank">https://go-acme.github.io/lego/</a> - stars: 3,888 -full implementation of ACMEv2, use of custom CSRs.ngxpkg - <a href="https://github.com/webpkg/ngxpkg" target="_blank">https://github.com/webpkg/ngxpkg</a> - stars: 22 - a CLI client for nginx, configuration via env variables.openshift-acme - <a href="https://github.com/tnozicka/openshift-acme" target="_blank">https://github.com/tnozicka/openshift-acme</a> - client for OpenShift, upgrade to rfc8555 in progress (Jan/2020), client for OpenShift and Cubernates clusters.<h3>HAProxy</h3>haproxy-lua-acme - <a href="https://github.com/haproxytech/haproxy-lua-acme" target="_blank">https://github.com/haproxytech/haproxy-lua-acme </a>- stars: 34 - ACMEv2 operations done via POST requests to the proxy, HAproxy has to be manually restarted.<h3>Java</h3>ACME4J - <a href="https://github.com/shred/acme4j" target="_blank">https://github.com/shred/acme4j</a> - stars: 298 - JRE8 (101) or higher, maven, dependencies jose4j, Bouncy Castle (for generating keys and CSRs), slf4jacme-client - <a href="https://github.com/porunov/acme_client" target="_blank">https://github.com/porunov/acme_client</a> - stars: 53 - a CLI agent for automation tools or scripts, CLI commands for each ACME API function.<h3>Microsoft Azure</h3>appservice-acmebot - <a href="https://github.com/shibayan/appservice-acmebot" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://github.com/shibayan/appservice-acmebot</a> - stars: 124 - Win/Linux, Azure Web Apps, Functions, Zone Apex domain certs, wildcard, doesn't use KeyVault.AzureWebAppSSLManager - <a href="https://github.com/n3wt0n/AzureWebAppSSLManager" target="_blank">https://github.com/n3wt0n/AzureWebAppSSLManager</a> - stars: 15 - renewal of certs with Azure DNS challenges, certs are stored in Azure Blob Storage. Required Azure Service Principal and SendGrip API.GetSSL - Azure Automation - <a href="https://www.powershellgallery.com/packages/GetSSL-LetsEncrypt/" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://www.powershellgallery.com/packages/GetSSL-LetsEncrypt/</a> - PowerShell script that works with MS Azure KeyVault. A documentation for MS Azure Vault is at https://github.com/ebekker/foobar/tree/master/docs/platyps. Installation with "Install-Script".keyvault-acmebot - <a href="https://github.com/shibayan/keyvault-acmebot" target="_blank">https://github.com/shibayan/keyvault-acmebot </a>- stars: 100 - this is a "KeyVault" version of appservice-acmebot.<h3>Node.js</h3>acme2.js - <a href="https://git.cloudron.io/cloudron/box/blob/master/src/cert/acme2.js" target="_blank" style="color: rgb(64, 64, 200);">https://git.cloudron.io/cloudron/box/blob/master/src/cert/acme2.js</a> - requires 'openssl', one file, no documentation.greenlock.js - <a href="https://git.coolaj86.com/coolaj86/greenlock.js" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://git.coolaj86.com/coolaj86/greenlock.js</a> - stars: 19 - pure JS implementation, several DNS system integrations.greenlock-cli - <a href="https://git.coolaj86.com/coolaj86/greenlock-cli.js" target="_blank">https://git.coolaj86.com/coolaj86/greenlock-cli.js</a> - requires a webserver listening on port 80, default config is "staging", standalone and webroot modes. Greenlock Express - <a href="https://git.coolaj86.com/coolaj86/greenlock-express.js" target="_blank">https://git.coolaj86.com/coolaj86/greenlock-express.js</a> - stars: 31 - several DNS system integrations, examples for Express, Node's http2, websockets, socket.io.node-acme-client - <a href="https://github.com/publishlab/node-acme-client" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://github.com/publishlab/node-acme-client</a> - stars: 69 - account private key is set as a JS constant.node-acme-lambda -  <a href=" https://github.com/ocelotconsulting/node-acme-lambda" target="_blank">https://github.com/ocelotconsulting/node-acme-lambda</a> - stars: 93 - uses AWS Lambda, certs are stored in S3, environment configuration.<h3>Perl</h3>Crypt-LE - <a href="https://github.com/do-know/Crypt-LE" target="_blank">https://github.com/do-know/Crypt-LE</a> - stars: 202 - portable, with Windows binaries available, dependency openssl-devel/libssl-dev, export to PFX/p12.p5-Net-ACME2 - <a href="https://github.com/FGasper/p5-Net-ACME2" target="_blank">https://github.com/FGasper/p5-Net-ACME2</a> - stars: 3 - production-grade, own exception class, library with logic examples to build "application workflow".<h3>PHP</h3>acme - <a href="https://github.com/kelunik/acme" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://github.com/kelunik/acme</a> - stars: 96 - PHPDoc available, installation with composer, uses the Amp framework.acme2 -<a href="https://github.com/stonemax/acme2" target="_blank" style="color: rgb(64, 64, 200);"> https://github.com/stonemax/acme2</a> - Another PHP client for acme protocol.acmecert - <a href="https://github.com/skoerfgen/ACMECert" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://github.com/skoerfgen/ACMECert</a> - stars: 40 - renewals and revocations, cert parsing to extract expiry date, depends on OpenSSL and cURL.acmephp - <a href="https://github.com/acmephp/acmephp" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://github.com/acmephp/acmephp </a>- stars: 404 - CLI client, a single binary file, integration with Monolog for notifications, config formatters for nginx, nginx-proxy, haproxy. Configurable to copy certs to remote machines.acme_proxy.php - <a href="https://github.com/jpawlowski/acme_proxy.php" target="_blank">https://github.com/jpawlowski/acme_proxy.php</a> - stars: 1 - ACME challenge validation proxy using localhost DNS results. le-acme2-php - <a href="https://github.com/fbett/le-acme2-php" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://github.com/fbett/le-acme2-php</a> - stars: 16 - based on the yourivw/LEClient, rewritten and enhanced.leclient - <a href="https://github.com/yourivw/LEClient" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://github.com/yourivw/LEClient</a> - stars: 127 - install with composer, a clear documentation in the README.md file.lemanager - <a href="https://github.com/analogic/lemanager " target="_blank">https://github.com/analogic/lemanager </a>- stars: 91 - containerized app (docker container), has a GUI showing expiry of certs<h3>Python</h3>acertmgr - <a href="https://github.com/moepman/acertmgr" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://github.com/moepman/acertmgr</a> - stars: 8 - you can define own account key, or it can be generated, rich configuration with 4 "scopes" (global, domain, cmd line, defaults.acme-cert-tool - <a href="https://github.com/mk-fg/acme-cert-tool" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://github.com/mk-fg/acme-cert-tool</a> - stars: 2 - single script with 1 dependency (cryptography), python 3.7+.acme-nginx - <a href="https://github.com/kshcherban/acme-nginx" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://github.com/kshcherban/acme-nginx</a> - stars: 201 - ACME client for nginx. Includes DNS challenges for AWS route53 and Digital Ocean DNS.acme-nosudo - <a href="https://github.com/diafygi/acme-nosudo" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://github.com/diafygi/acme-nosudo</a> - stars: 1,200 - two scripts - sign_csr.py and revoke_crt.py to create/revoke a cert. acme-powerdns - <a href="https://github.com/adfinis-sygroup/acme-powerdns" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://github.com/adfinis-sygroup/acme-powerdns</a> - does NOT support rfc8555, but an interesting integration with PowerDNS.acme-tiny - <a href="https://github.com/diafygi/acme-tiny" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://github.com/diafygi/acme-tiny</a> - stars: 4,300 - tiny Python client, depends on openssl, account key has to be created separatelyacme-tiny-dns - <a href="https://acme-dns-tiny.adorsaz.ch/" target="_blank" style="color: rgb(64, 64, 200);">https://acme-dns-tiny.adorsaz.ch</a> - a fork of acme-tiny, only supports DNS challenges, ACME account and CSR have to be supplied.acmebot - <a href="https://github.com/plinss/acmebot" target="_blank">https://github.com/plinss/acmebot</a> - stars: 73 - domains for certificates are defined in a configuration file, default is DNS authorization, creates pinning header (HPKP).sewer - <a href="https://github.com/komuw/sewer" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://github.com/komuw/sewer</a> - stars: 102 - library and CLI app, supports several DNS providers (modular to add new ones).simp_le - <a href="https://github.com/zenhack/simp_le" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://github.com/zenhack/simp_le</a> - stars: 154 - a fork from kuba/simp_le, which is unmaintained, cmd line parameter setting minimum validity (for renewal to kick in), a docker container.txacme - <a href="https://github.com/mithrandi/txacme" target="_blank">https://github.com/mithrandi/txacme </a>- stars: 38 - an implementation of the ACME protocol for Twisted (event-driven net engine). It is not a client and under heavy development.<h3>Ruby</h3>acme-client - <a href="https://github.com/unixcharles/acme-client" target="_blank">https://github.com/unixcharles/acme-client</a> - stars: 380 - a Ruby client as a library, with code snippets to integrate into an application.acme-distributed - <a href="https://github.com/jannfis/acme-distributed" target="_blank">https://github.com/jannfis/acme-distributed</a> - stars: 1 - the main difference - it places challenges on remote servers. Incomplete implementation, outdated docs.acmesmith - <a href="https://github.com/sorah/acmesmith" target="_blank">https://github.com/sorah/acmesmith</a> - stars: 103 - it stores certs and keys in cloud services (e.g., S3). chef-acme - <a href="https://github.com/schubergphilis/chef-acme" target="_blank">https://github.com/schubergphilis/chef-acme</a> - stars: 102 - Chef cookbook for Let's Encrypt, examples of chef configurations.combine-acme - <a href="https://gitlab.com/6318613/combine-acme" target="_blank">https://gitlab.com/6318613/combine-acme</a> - stars: 1 - it not only generates certs but also uploads them got Google cloud (GCP) and Cloudflare.freshcerts - <a href="https://github.com/myfreeweb/freshcerts" target="_blank">https://github.com/myfreeweb/freshcerts</a> - stars: 53 - it is a centralized "server" that allows endpoints used lightweight scripts to request certificates.<h3>Rust</h3>acme-client - <a href="https://github.com/onur/acme-client" target="_blank">https://github.com/onur/acme-client</a> - stars: 178 - allows using own keys and CSRs.ACMEd - <a href="https://github.com/breard-r/acmed" target="_blank">https://github.com/breard-r/acmed</a> - stars: 24 - written as a daemon, fully customizable challenge validation, retries, key pair reuse, customizable HTTPS rate limits.le-opensrs - <a href="https://github.com/mhmd3bdo/le-opensrs" target="_blank">https://github.com/mhmd3bdo/le-opensrs</a> - stars: 2 - a client for opensrs DNS, only this DNS challenge validation.<h3>Windows/IIS</h3>acmesharpcore - <a href="https://github.com/PKISharp/ACMESharpCore" target="_blank">https://github.com/PKISharp/ACMESharpCore</a> - stars: 130 - .NET library.acmesharpcore-powershell - <a href="https://github.com/PKISharp/ACMESharpCore-PowerShell" target="_blank">https://github.com/PKISharp/ACMESharpCore-PowerShell</a> - stars: 27 - PowerShell libraryautoacme - <a href="https://github.com/ridercz/AutoACME" target="_blank">https://github.com/ridercz/AutoACME</a> - stars: 71 - .NET, an alternative to letsencrypt-win-simple, Windows&IIS, not requiring admin rights, supports web farmscertes - <a href="https://github.com/fszlin/certes" target="_blank">https://github.com/fszlin/certes</a> - stars: 246 - .NET client, CLI app and librarygetcert2 - <a href="https://github.com/GeorgeSchiro/GetCert2" target="_blank">https://github.com/GeorgeSchiro/GetCert2</a> - stars: 6 - CLI and GUI app, it copies certs to LAN storage, updates and restarts SSO acress server farm, creates new IIS websites for new domains in certificates, it is a front-end for "acme-ps".posh-acme - <a href="https://github.com/rmbolger/Posh-ACME" target="_blank">https://github.com/rmbolger/Posh-ACME</a> - stars: 206 - PowerShell, single commands for new/renewing certs, DNS challenge with many providers, pem/pfx output, no admin privilege requiredwin-acme - <a href="https://github.com/PKISharp/win-acme/" target="_blank">https://github.com/PKISharp/win-acme/</a> - stars: 3,300 - binaries available, cmd line application with "menus".wincertes - <a href="https://github.com/aloopkin/WinCertes/tree/master/WinCertes" target="_blank">https://github.com/aloopkin/WinCertes/tree/master/WinCertes</a> - stars: 34 - C# client, auto renewal with Scheduled Task, config in Registry, logs to stdout and files.<h2>Applications</h2>Caddy - <a href="https://caddyserver.com" target="_blank">https://caddyserver.com</a> - a web server with in-built certificate renewal.FreeSSL.tech auto - <a href="https://freessl.tech/" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://freessl.tech/</a> - Let's Encrypt app in PHP.haproxy-acme-validation-plugin - <a href="https://github.com/janeczku/haproxy-acme-validation-plugin" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://github.com/janeczku/haproxy-acme-validation-plugin</a> - stars: 268 - requires Certbot for renewals, the plugin manages challenge validation.Hiawatha webserver - <a href="https://www.hiawatha-webserver.org/letsencrypt" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://www.hiawatha-webserver.org/letsencrypt</a> - PHP package, no separate documentation is part of the web browser source code, although as a separate package.Key Manager Plus - <a href="https://www.manageengine.com/key-manager/" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://www.manageengine.com/key-manager/</a> - not open source nor free, key management for enterprises, binary downloads for Linux and Win.KeyChest AMP - <a href="https://gitlab.com/keychest/keychestamp/-/wikis/home" target="_blank">https://gitlab.com/keychest/keychestamp/-/wikis/home</a> - a proxy for ACMEv2 that creates JSON logs that can be be analyzed for malfunctions, downtimes, and rate-limits.lets-proxy2 - <a href="https://github.com/rekby/lets-proxy2" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://github.com/rekby/lets-proxy2</a> - stars: 14 - zero config (includes certs with main domain and www), proxy for shared hosting, Russian manual.lua-resty-auto-ssl - <a href="https://github.com/GUI/lua-resty-auto-ssl" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://github.com/GUI/lua-resty-auto-ssl </a>- stars: 1,500 - OpenResty plugin for Let's Encrypt registration and renewals.Mako server - <a href="https://makoserver.net/articles/Lets-Encrypt" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">https://makoserver.net/articles/Lets-Encrypt</a> - Lua web framework and non-blocking asynchronous sockets in a tiny ready to run application server package.ponzu-cms - <a href="https://ponzu-cms.org/" target="_blank" style="color: rgb(64, 64, 200);">https://ponzu-cms.org/</a> - CMS system with in-built certificate renewal.traefik - <a href="https://containo.us/traefik/" target="_blank">https://containo.us/traefik/</a> - reverse proxy with automatic certificate management.

Publication Date: 2020-01-04 01:45:00

Let’s Encrypt for Companies with KeyChest

Let’s Encrypt has a number of downsides when used on a large scale. It uses modern key management protocols, but the high-level of automation requires management. This is what KeyChest provides.

Let’s Encrypt has a number of downsides when used on a large scale. It uses modern key management protocols, but the high-level of automation requires management. This is what KeyChest provides.<h2>Unifying HTTPS Management</h2>The “zero” cost of Let’s Encrypt certificates is balanced-out with two main downsides: rate limits and short lifetime of certificates. While it’s great to get a free product, it becomes somewhat dangerous when you start relying on it without giving a proper thought to what should happen in 3 months’ time when you need to renew it.There is also an adoption barrier as Let’s Encrypt uses an automation for renewals (so called ACMEv2 standard) and it requires a complete change in the way you manage your certificates.Nevertheless, Let’s Encrypt is a viable option for HTTPS and other encrypted internet services.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="The US government has been using Let’s Encrypt on its gateways for some time now – a list of domains from just one such certificate." src="/storage/storage/wink/images/3XCtE4wpzgmEo51JrLotQrjrG6CGxNFGwX8QHgHu.png">The US government has been using Let’s Encrypt on its gateways for some time now – a list of domains from just one such certificate.</div> There are many instances where operational teams choose to deploy a Let’s Encrypt certificate when an important service went down due to an expired certificate or the time left is too short to follow usual purchasing. Now imagine you have hundreds such certificates – how can you keep on top of them all?KeyChest is now testing (Dec 2019) a new proxy service, which provides a valuable information about the status of Let’s Encrypt certificates. The use requires a small change to the way you use KeyChest agents – Certbot by setting an HTTPS_PROXY. If your existing command is “certbot renew”, the new version would be:<blockquote><code>“export HTTPS_PROXY=https://test.keychest.net:6443; certbot renew”</code></blockquote>KeyChest will start logging all your requests and create statistics of your usages of Let’s Encrypt. You will instantly get information about the activity of your Certbot agents and detect issues when they happen – not when your certificates start expiring.This also helps to comply with Let’s Encrypt rate limits, some of which are fairly strict:<ol><li>maximum of 5 new certificates per domain per week;</li><li>50 certificates per registered domain per week;</li><li>300 new “orders per account per 3 hours; or</li><li>5 failed validations per account per hour.</li></ol><h2>Deep Let’s Encrypt Proxy</h2>While the monitoring will get you on top of your Let’s Encrypt usage and reduce the risk of downtimes. Our ambition is to unify ACME and “legacy” certificate issuers.We achieve that with a deep proxy that does the actual communication with the Let’s Encrypt issuing servers and also validation challenges.This allows us to forward certificate requests to Let’s Encrypt, Comodo, Symantec, … or any other certificate issuer according to your requirements. To further simplify your IT operations, our proxy architecture allows endpoint agents to be much smaller with fewer dependencies and installation issues.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Keychest agent – one internal proxy per network, is a single gateway with an access control to minimize any new additional risk." src="/storage/storage/wink/images/Avk6zbPWHCEcf2AocOvUBB7Vao0OaUdbI5YbePhx.png">Keychest agent – one internal proxy per network, is a single gateway with an access control to minimize any new additional risk.</div>You can either forward (an HTTP reverse proxy) validation requests directly to the KeyChest service, or use our lightweight agents available to the Small Enterprise customers.Many companies are considering switching to Let’s Encrypt. What we offer is a management service to keep your commercial certificates while simplifying the configuration of your servers and preventing downtimes.KeyChest has a complete database of all internet certificates and can automatically start monitoring any new servers and services you create – providing you with ongoing 100% coverage of your registered domains.If you have any question, fire away: info@keychest.net, +1 512-696-1552 or check online: <a href="https://keychest.net" target="_blank">https://keychest.net</a>

Publication Date: 2019-12-03 13:53:00

Certificate Monitoring - HTTPS/TLS

There are so many tools to help you ensure that all your HTTPS and other secure services are up and running. You may have just one website or you create a number of web APIs every day. Here you can find a selection of tools we have found and that we see as competitors to our KEYCHEST.

As we continuously improve our own certificate management service, we keep an eye on other tools. There is a wide range of services and each of us has different requirements and preferences.When you use Let's Encrypt certificates, it’s perfectly alright to trust your cron and your favorite Let’s Encrypt client to renew your server’s certificate when it’s due. However, this doesn't really work when the number of servers reaches 10-20 or when your job depends on keeping all certificates up to date. Renewing a certificate involves many moving parts so it really is a good idea to continuously check that all keeps working perfectly.When the HTTPS really matters to you, the good good practice is to set up an end-to-end monitoring.The following text includes some multi-purpose cloud services, dedicate HTTPS cloud services, FOSS projects, as well as several commercial systems.<h2>Certificate Cloud Services - Free Usage</h2><a href="https://keychest.net/register" target="_blank">KEYCHEST</a> - Let me get our own KeyChest off my chest first. It currently offers a free plan for non-business users with up to 500 endpoints. It uses its own global database of all certificates so it can find all your subdomains automatically (set up and forget service). You will get weekly email reports, an integration API, and a web dashboard. You can also use our automated cert issuance. Paid plans include custom root certificates, additional reports (phishing threat, security report), and internal cert management.<a href="https://certificatemonitor.org" target="_blank">CertificateMonitor.org </a>- it's an alternative to the default Let's Encrypt email reminders. It does not have any dashboard, you simply give it a domain name and the service will start sending reminder emails.<a href="https://letsmonitor.org" target="_blank">LetsMonitor.org</a> - it's a free service for the first few domains. You need to enter each domain separately. It has a clear dashboard and email reminders.<h2>Standalone Applications - Free</h2>There is a large number of standalone applications (GitHub or other FOSS projects) for monitoring:<a href="https://github.com/drtoful/certinel" target="_blank">Certinel</a> (GitHub, lust update 2017) - It has been created, because Let's Encrypt certificates are only valid for 90 days and there's no automation or monitoring currently available to check. You can do automation with some cronjobs, but this is probably unreliable so it's better you monitor the status of your certificates. Certinel also provides a simple one-page monitoring page were you can add, remove and check the status of your domains.<a href="https://github.com/srvrco/checkssl" target="_blank">Checkssl</a> (GitHub, last commit March 2019) - With the good work by "Let’s Encrypt" in providing free SSL certs for users, I wanted a quick way to check all the domains I look after to determine which ones have correct SSL certs, and which ones are in need of updating etc.  This bash file is the first draft of a program to do that. It can either be run against a list of file names, from the directories in your Lets Encrypt live directory or on a single server with the aim of getting all the domain names from the server.<a href="https://github.com/sahsanu/lectl" target="_blank">lectl</a> (GitHub, last commit Aug 2018) - Script to check issued certificates by Let's Encrypt on CTL (Certificate Transparency Log) using https://crt.sh . It is directly dependent on the https://crt.sh service, which is managed by SectiGo.<a href="https://github.com/Matty9191/ssl-cert-check" target="_blank">SSL-cert-check</a> (GitHub, last commit Apr 2019) - a Bourne shell script that can be used to report on expiring SSL certificates. The script was designed to be run from cron and can e-mail warnings or log alerts through nagios.<h2>Certificate Only - Commercial Solutions</h2><a href="https://www.globalsign.com/en/blog/new-certificate-inventory-tool/" target="_blank">GlobalSign Inventory</a><a href="https://www.digicert.com/discovery-tool.htm" target="_blank">DigiCert certificate inspector</a><a href="https://sslmate.com/certspotter/pricing" target="_blank">SSLMate - Cert Spotter</a><a href="https://isitworking.info/" target="_blank" style="color: rgb(64, 64, 200); background-color: rgb(255, 255, 255);">Is It Working</a><a href="https://www.appviewx.com/products/cert/" target="_blank">AppViewX</a><h2>Multi-Purpose with SSL - Commercial Solutions</h2><a href="https://www.statuscake.com/" target="_blank" style="color: rgb(64, 64, 200);">StatusCake</a> has a simple certificate monitoring feature that sends reminders one week before expiration. You have to add each URL separately so it will work best if you have a few web services that are important for you.<a href="https://sucuri.net/website-security-platform/signup/" target="_blank" style="color: rgb(64, 64, 200);">Sucuri</a> - this service provides annual subscription plans with SSL included from the "Pro". However, this only includes one website. If you have more websites, you'd need to ask about the "Enterprise" option. The price reflects a range of tools offered (malware scan, SQL injection, blacklist monitor, etc.)<a href="https://www.dotcom-monitor.com/pricing/" target="_blank" style="color: rgb(64, 64, 200);">Dotcom-Monitor</a> is primarily uptime&performance monitoring service. SSL checks include basic info about the certificate (the issuer, name in the cert, if revoked, ...). You need "web services view", which is free for 5 URLs. The maximum number of domains is 100 which goes for $100/month. <a href="https://www.manageengine.com/products/applications_manager/ssl-certificate-monitoring.html" target="_blank">ManageEngine SSL cert monitoring</a><a href="https://www.solarwinds.com/server-application-monitor/use-cases/ssl-certificate-monitor" target="_blank">Solarwinds SSL cert monitor</a><a href="https://www.monitis.com/support/uptime-monitoring/ssl-certificate-monitoring" target="_blank">monitis SSL certificate monitoring</a><a href="https://www.uptrends.com/products/ssl-certificate-monitoring" target="_blank">Uptrends SSL cert monitoring</a><a href="https://www.site24x7.com/ssl-certificate-monitoring.html" target="_blank">site24x7 SSL expiry monitoring</a><a href="https://www.poweradmin.com/help/pa-server-monitor-8-0/howto_monitor_ssl_expiration.aspx" target="_blank">PA Server Monitor</a><a href="https://www.rapidspike.com/pricing/" target="_blank">RapidSpike</a><a href="https://redkestrel.co.uk/products/certalert/" target="_blank">RedKestrel CertAlert (Windows)</a>

Publication Date: 2019-11-28 11:35:00

Let's Encrypt Uptime - 2 years on

We look at the uptime and reliability of Let's Encrypt services. This is a follow-up activity to our 2017 analysis of Let's Encrypt downtimes from it's beginning.

I have looked at the service disruptions of Let's Encrypt back at the end of 2017. Two years on, I had another look - and compared twelve months periods.Let's Encrypt has been a new service that launched in mid-2016. One would naturally expect that first 12+ months were a bit bumpy as the technology, including the infrastructure, settles in. So I was curious how things changed since.I used the same source as the first time - <a href="https://letsencrypt.status.io " target="_blank" style="background-color: rgb(255, 255, 255);">https://letsencrypt.status.io</a>. The records seem to be fairly reliable. However, it appears that on several occasions the time of an incident starts when a report is received rather than the time potentially provided by the reporting party. I would not expect that to have a large impact but I wonder if there's a need for an independent uptime monitoring.A number of incidents are labeled as "partial" without any further information about the scale of the impact - does it mean longer latency but the provided services are still fully available, does it mean that a certain fraction of API calls fail or is the impact limited, e.g., geographically?The history of the Let's Encrypt status over the last 12 months (20 Nov 2018 to 20 Nov 2019) contains 95 records, from which 77 relate to the production services. I ignore non-core services, e.g., web site.<h2>Planned Upgrades</h2>When you look at the Let's Encrypt status history, you will immediately notice a fairly large number of upgrades. There is in fact 46 Boulder upgrades, which sounds like an incredibly high number.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Planned upgrades of Let's Encrypt production services. Multiple upgrades per month are stacked." src="/storage/storage/wink/images/kihG3ywBhQWZtu3L1OVdk2epH4UtjC7oW0SPbYFB.png">Planned upgrades of Let's Encrypt production services. Multiple upgrades per month are stacked.</div>As far as I could see, none of them caused any downtime and my main observation is that while the total number is surprisingly high, the team has certainly mastered the upgrade process and is able to quickly react to potential incidents. Below is a similar graph from 2017. While the number of upgrades is similar, it is clear that the time required for each upgrade as stabilised. Which means much better reliability of upgrades.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Let's Encrypt upgrades in 2017. Multiple upgrades per month are stacked." src="/storage/storage/wink/images/N0PjBzwgN99w8bbrc6EpH4TQkLQ0iT9pHXslNv2x.png">Let's Encrypt upgrades in 2017. Multiple upgrades per month are stacked.</div>Note: the chart above is the only in with linear Y-axis. I used logarithmic axis in the following charts to improve the visibility of short outages in the presence of large outliers.<h2>Full Disruptions</h2>The most important aspect, from users' point of view, is the probability of full service disruptions. This directly impacts the overall reliability of certification services and it is one of the main selling points for commercial CAs. Many of them like to quote their 99.99% or higher uptime. Let's Encrypt has significantly improved over the last 2 years in terms of the number of incidents. Although it's still quite some distance behind commercial CAs (although they don't publish their incident data).<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Full Disruptions in minutes (logarithmic scale)." src="/storage/storage/wink/images/VYQnIKjRkeZWLlrrXoFnKCW5k9FrA2PUv2cOMOnN.png">Full Disruptions in minutes (logarithmic scale).</div>You can see that the length of downtimes is relatively short. These include at least 3 instances of downtime related to maintenance - one infrastructure and two on the application level. The overall uptime in 2019 is 99.92%, which is better than the figure I found in 2017 (99.86 - with 99.9% over the second part of the interval analyzed then) - see below. <div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Full service disruptions in 2017. Logarithmic scale for the durations." src="/storage/storage/wink/images/ZNat4hyWDzp3XJuWfuHo0mINdkgiB6GajSPCBxol.png">Full service disruptions in 2017. Logarithmic scale for the durations.</div>The 2019 incidents include a couple of issues with OCSP responder. This is a fairly critical part of the LE services as it may result in unavailability of websites if browsers require OCSP validations.<h2>Partial Disruptions</h2>The final part of my comparison is partial disruptions. It is less transparent as the comparison is very blunt and lacks the detail of the impact of particular incidents. Nevertheless, I was somewhat surprised that the downtime was still relatively high.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Partial Disruptions of Let's Encrypt in 2019. Multiple incidents per month are stacked." src="/storage/storage/wink/images/nlkIHJcM3mVd2ST2gG6REbRJJV3yIe82sPNOE4sG.png">Partial Disruptions of Let's Encrypt in 2019. Multiple incidents per month are stacked.</div>The combined uptime figure (all services fully functional) is only 96.4%, which is lower than what I found in 2017, when the figure stood at 98% (again, the 2017 chart is below).<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Partial disruptions of Let's Encrypt in 2017. Y axis is logarithmic." src="/storage/storage/wink/images/EblYk325BIGRcvoJQob44VuRKh6avnrH1BPcB2qj.png">Partial disruptions of Let's Encrypt in 2017. Y axis is logarithmic.</div>It's worth mentioning that the 2019 results are significantly affected by a couple of incidents that lasted several days. One of them was caused by a <a href="https://community.letsencrypt.org/t/lets-encrypt-uptime-comparing-2019-with-2016-17/107114/8?u=dancvrcek" target="_blank">migration to another CDN</a> (details provided by "_az"). If we exclude one of both of these partial outages, the uptime is 97.9% and 99.7% respectively. <h2>OCSP</h2>I have not charted OCSP outages separately for two reasons. Firstly, I can see only two of them (one full, one partial). Secondly, these outages where short - 5 and 7 minutes and it's not clear to me whether they cover the whole outage or whether the measured interval starts only when the LE team started investigating it.If we assume the logged times are accurate, the overall uptime of OCSP responders would be >99.99%.<h2>A Final Note</h2>Looking at the data and charts above, the main question of mine is whether Let's Encrypt can ever reach the reliability of commercial CAs nor whether LE has such an aspiration. LE's main feature is automation. While we generally don't mind if a certificate is renewed today or a week later, the relatively low uptime is a concern. Especially as there are no official service level guarantees.This "exercise" made me also wonder whether it makes sense to implement an uptime monitoring system that would provide an independent uptime data. LE issues around 1.3 million certificates a day so every one minute downtime impacts over 900 certificate requests. Similarly, 1 minute downtime of OCSP responders is significant as 50-60% of websites use Let's Encrypt certificates. Arguably, these websites will not include any of the large one but still. This analysis was created by Dan Cvrcek, a founder of <a href="https://keychest.net" target="_blank">KeyChest</a> - TLS expiry monitoring service with a global database of all certificates and automation of certificate issuance.

Publication Date: 2019-11-24 17:37:00

Minerva Attack and Humble Beginnings

ROCA Attack has impacted more than 25% of TPM chips - chips that provide secure boot in laptops, PC, servers of all types. That was in 2017. This year, the same team discovered a new vulnerability that impacts a number of software libraries widely used in open source and commercial projects, using Elliptic Curve encryption.

Do you remember ROCA attack - the most devastating attack in 2017 that extracted secret keys from 25% of TPM module? It has a kind of a sibling - Minerva. While ROCA was about the RSA encryption, MINERVA is about Elliptic Curve (ECC) signing.<hr>TLDR: a bunch of open source encryption libraries leak Elliptic Curve signing key. Quite a few widely used libraries were bitten (libgcrypt, wolfSSL, MatrixSSL, SunEC/OpenJDK, Crypto++). Check your projects and upgrade if at all possible.<h3>First Signs of Problem</h3>It seems that the Minerva attack came to light when the research team in Brno were playing with old IDProtect Athena cards. They discovered a correlation between key lengths and timing. This allowed them to eventually recover the whole private key with relatively small number of signatures.<blockquote>The details of the attack are available from <a href="https://minerva.crocs.fi.muni.cz/" target="_blank" style="color: rgb(0, 123, 255);">https://minerva.crocs.fi.muni.cz/</a></blockquote>This first version of the attack was against specialist security chips. It took around 30 minutes due to slow communication speeds of the chips that would not allow more than 10 signatures per second. Indeed, the number of operations required was only 11,000.Athena SCS IDProtect that was successfully attacked was approved for use in the <a href="https://www.securetechalliance.org/athenas-idprotect-smart-card-and-middleware-approved-for-us-government-pivhspd-12-deployment/" target="_blank" style="color: rgb(0, 123, 255);">US government as PIV cards in 2008</a>. However, these chips are not really being sold any more and I would also expect only a low number to be still in use. The core problem exploited by the attack is a leakage of the length of numbers used in multiplications of ECDSA/EdDSA. When the multiplication is optimized, it will cause different timings and this can be measured from outside - it's easiest on the same processor but successful attacks were demonstrated when using network loopback or even across local networks. Each such measurement leeks information about the key and the Minerva attack requires between just hundreds of measurements, in some cases, to over hundred thousands. So - is the attack just an academic issue or is it a sign of a wider problem? After all, some developers like to optimize their code ...<h3>Expanding The Impact</h3>While the impact of the original attack was small, the team had a brilliant idea – what about testing the attack against software libraries. I would expect they were taken aback when they found quite a few popular libraries to be vulnerable.<ul><li><a href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-13627" target="_blank" style="color: rgb(0, 123, 255);">CVE-2019-13627</a>: Vulnerability in libgcrypt.</li><li><a href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-13628" target="_blank" style="color: rgb(0, 123, 255);">CVE-2019-13628</a>: Vulnerability in wolfSSL/wolfCrypt.</li><li><a href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-13629" target="_blank" style="color: rgb(0, 123, 255);">CVE-2019-13629</a>: Vulnerability in MatrixSSL.</li><li><a href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-2894" target="_blank" style="color: rgb(0, 123, 255);">CVE-2019-2894</a>: Vulnerability in SunEC/OpenJDK/Oracle JDK.</li><li><a href="https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-14318" target="_blank" style="color: rgb(0, 123, 255);">CVE-2019-14318</a>: Vulnerability in Crypto++.</li></ul>Most of them have been fixed in upgrades over the summer, but I’m pretty sure that there will be zillions of projects that will rarely upgrade versions of cryptographic libraries. So the impact may be long-term with smaller attacks to keep coming over many coming years. The list of libraries tested and found safe includes:<ul><li class="ql-align-justify">OpenSSL 1.1.1d</li><li class="ql-align-justify">BouncyCastle 1.58</li><li class="ql-align-justify">BoringSSL 974f4dddf</li><li class="ql-align-justify">libtomcrypt 1.18.2</li><li class="ql-align-justify">Botan 2.11.0</li><li class="ql-align-justify">Microsoft CNG</li><li class="ql-align-justify">mbedTLS 2.16.0</li></ul>While I was in no way connected to this attack, I like to think it all started in 2002.<h3>Humble Beginnings</h3>Back in 2002/3, mesmerised by successful side-channel attacks on internet encryption protocols, I decided to build a tool for systematic research and validation of some of these attacks. The result was an analytic toolkit for timing and power analysis of security chips.I remember sleepless nights trying to get the tool measure, instead of burning the chips, desperation, and an eventual success.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="SCSAT04 - a 2nd generation of the side-channel analysis tool" src="/storage/storage/wink/images/IYCpCgaWkDRwiuWrXo4quZ7rNjAUImtz0bAfpcs8.jpeg">SCSAT04 - a 2nd generation of the side-channel analysis tool</div>At the time I thought that just the timing attack has been a bit low-level, easy to defend against, and we really needed to combine it with other types of attacks power / glitch attacks. The SCSAT tools were designed to generate microsecond, high voltage peaks on power, clock, and data pins of chips. This was coupled with up to 200Mps, 12bit samples with microsecond trigger. The goal was to induce computational errors without triggering protection mechanisms of the high-security chips we were looking at.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="The 1st generation of the tool in use" src="/storage/storage/wink/images/olgC4GY42Lp0dZfMNmQAGBzatmMqwsjmq724Y8GZ.png">The 1st generation of the tool in use</div>The main memory of mine is of long hours of relentless work and it’s absolutely amazing that what we have started many years ago has been turned into absolutely fascinating results by <a href="https://crocs.fi.muni.cz/people/svenda" target="_blank" style="color: rgb(0, 123, 255);">Petr Svenda</a> and his team. By: Dan Cvrcek, founder of <a href="https://keychest.net" target="_blank">KeyChest</a>.

Publication Date: 2019-11-22 08:02:00

Automating HTTPS Issuance in KeyChest

KeyChest has automated issuance of first certificates this week. It makes the process as simple as possible with great certificate prices.

<a href="https://keychest.net" target="_blank">KeyChest's</a> business model is based on the management of HTTPS expiry. Automation of certificate issuance is for us an additional service that moves it closer to a complete service to manage your internal and external certificates. What it means in practical terms is that we simply pass on our cost of certificates to all paying users of KeyChest.In terms of user experience, our primary goal was to make issuance of new certificates as simple as possible while preserving enough flexibility for the more technical users.Let me start with the simple flow. The first step is to register a payment card. We need to pay issuers and so we need to charge you as well. This step will also add default values for the “company”, which are needed for certificate requests.The actual certificate request starts with you entering one or more domain names. It can be single domains, or wildcard domains.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="KEYCHEST - issuance of new certificates" src="/storage/storage/wink/images/YhNzRXCVY7hSUoQfk0KVqKbNM7n5lWjhwpPEY0cK.png">KEYCHEST - issuance of new certificates</div>KeyChest will automatically select products that satisfy your domain names and calculate expected pricing. You simply select the most suitable product from this list.We believe that email validation of requests is the preferred method wherever it’s possible. KeyChest will therefore obtain a list of allowed email addresses from a certificate issuer and you again simply select one from a list.We are now ready to create the actual request. This happens completely in your browser. Once you enter a password to encrypt a new private key, KeyChest will generate a new key, create a request (PKCS#10 / CSR), and encrypts the secret part of the key with your password.You can choose whether this secret key is stored on your disk or in the KeyChest database. Either way, a copy is stored in the local storage of your web browser. The request is then immediately sent out to initiate its validation by the issuer.At this point, you need to complete required validation steps – in most cases this would involve following instructions sent via email. KeyChest will keep checking the request status and download the new certificate as soon as it is available. It will also send a notification if configured.Voila – at this moment, you have all you need to install the new certificate on your server and KeyChest helps automating it.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="With one click, you can download files for Linux and Windows servers" src="/storage/storage/wink/images/W8ZE65gcdKumnsUiLepbWcv9vuV5ts0zN1Q3p2tm.png">With one click, you can download files for Linux and Windows servers</div>You can open your Orders and select the one you need. If your target server is an IIS, you click on “Download All as PFX”, enter a password to open the private key (if it’s still in your browser storage, it will be automatically found). Click the same button again and you get a password protected file you can import to IIS. It contains all the certificate and the key. The next step is simply to update the 443/https binding on your web site.<div class="inline_html" contenteditable="false"><iframe src="https://player.vimeo.com/video/374413887" width="640" height="414" frameborder="0" allow="autoplay; fullscreen" allowfullscreen=""></iframe> </div>You can see that we avoid sending any secrets over the network. However, you may not want to copy your secret/private key away from where it should be used. Possibly, you already have a CSR file and want to use that instead.If this is the case, you simply upload your CSR at the beginning of the certificate request. You don’t have to fill in domain names – they are in the CSR – and you can simply select the product and send the request to the issuer.Once the new certificate is ready, you can download the certificate files and install them on your server.Most domain-validated certificates will take a couple of minutes from entering a domain name to downloading a complete bundle for your server. The financial side is automated as well so you can download an invoice at the same time, if you need one.At the time of publishing:<ol><li>COMODO — $6.53</li><li>THAWTE — $25.07</li><li>GEOTRUST — $49.79</li><li>RAPIDSSL — $9.62</li><li>SECTIGO — $40.52</li></ol>

Publication Date: 2019-11-21 10:10:00

KeyChest – Unifying Public and Private Keys

KeyChest has started as an easy to use HTTPS monitoring service. What we are aiming for is a general purpose key management service, which can look after your public as well as internal web encryption keys.

KeyChest has started as an easy to use HTTPS monitoring service. What we are aiming for is a general purpose key management service, which can look after your public as well as internal web encryption keys.Ever since KeyChest got its first users, we were asked if it can be used for internal keys. Our initial approach was to provide on-premise instances, which gave potential users a complete autonomy in terms of separate database and dedicated processing power. This is what we still offer as the Enterprise option. But this option asks a lot from users as they have to, at least to some degree, look after their KeyChest.As we developed new features it became obvious that we need another option – more flexible, lightweight, low-maintenance. That’s where independent agents, extending the reach of KeyChest audit engines, come from. As we are close to launching KeyChest agents, here’s an introduction.Agents will be generally available from the beginning of July 2019.<h2>Proxy Agent</h2>We wanted agents to be as small and platform-independent as possible. The smaller they are, the easier they are to install, maintain, and update. The platform independence means there is almost no encryption inside (except for an ability to create a secure encrypted channel). Neither it can use low-level networking or other platform-dependent support. Agents will still have to be able to read / write to files and connect to the main KeyChest service, but even file-system access has to be limited to common functionality.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Communication and block diagram of KeyChest agents." src="/storage/wink/images/778XBs0QtPgdumuxZolrUXS8BwrLwPLPVa6bzduF.png">Communication and block diagram of KeyChest agents.</div> The core of KeyChest agents comprises the following 3 subsystems:<ol><li>Logging – a robust logging, which stores activity logs locally as well as posts them to the KeyChest service is simply a must for efficient management. Detailed information is what you need when in trouble – whether it’s for KeyChest users or for our support helping you out.</li><li>Proxy operation – the actual audit of secure services requires a strong control over the networking, something that is platform dependent and we are regularly updating it. It means agents must work as transparent proxies for traffic generated by the KeyChest Audit Engines.</li><li>Local control – agents are gateways into your internal networks and we want to give you as much control as possible over what they can be used for. We are putting restrictions on the ports they can connect to, the address ranges they can use, and so on. This information is in local configuration files, which can be locked-down so only you can change them. We also plan to give as wide access to the source codes of agents as possible.</li></ol><h2>Logging</h2>We see logging and ability to audit collected logs as one of the most important aspect of any software and network services. When we started implementing the agents, the first thing we did was a robust logging framework.Logging produces JSON formatted messages that provide much better quality of “raw” data. This has massive implications on log audits as all the parameter values from logs are natively stored as separate indexed items. (Internally, we use Splunk for audits. Since we started using JSON logs, visualization and inspection of tens or hundreds of thousands log records were actually fun. The simplicity of selecting any of the log parameters and focus analysis around them gives you a good chance to spot irregularities and issues to look into.)The success of any audit of logs depends on the quality of the data and being able to quickly identify where it came from. Each log entry therefore contains:<ul><li>time of logging</li><li>time of event</li><li>event code – severity of the message and a unique code (e.g., ERR0043)</li><li>host machine – name and its IP address</li><li>version of the software (agent); and</li><li>process – if the software uses multiprocessing.</li></ul><h2>Proxy Operation</h2>This is possibly a more interesting part of KeyChest agents and it still amazes me how simple can it look from outside.Each agent controls the traffic and any requests coming from the KeyChest.net service. Details of audit requests are sent to agents so they can block those not complying with agent’s local configuration file.Agents regularly connect to keychest.net to request audit “jobs”. When they receive a valid description of an audit job, they will launch a proxy, which connects to the audit target (downstream) and to KeyChest (upstream). Once the audit is completed, the proxy is terminated and the agent can re-use its port.Internal network discovery is treated separately. We plan to implement a range of discovery methods based either on:<ul><li>internal database of certificates (e.g., LDAP storage of your PKI system); or</li><li>internal DNS zone.</li></ul>Discovered services are sent to the KeyChest service so it can start scheduling regular audits.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Communication diagram of KeyChest agent in the reverse proxy mode " src="/storage/wink/images/sQ2qkrWhx9mUZ3WAhaLGFz0ICtFHWa15ApZsBskq.png">Communication diagram of KeyChest agent in the reverse proxy mode </div><h2>Agent Installation</h2>Agents are provided as one-file packages. You can run them by hand or invoke them with a scheduler like Linux cron. When it’s started for the first time, it will create an initial configuration and will try to register with KeyChest. Once completed, the agent will have its own unique API key and ID, which is in the form of an email address.It will occasionally push its configuration to the KeyChest service, but nothing else will happen until it is associated with a KeyChest account. Once that happens, the service discovery and regular audits will commence automatically.

Publication Date: 2019-06-24 10:02:00

KeyChest and Reliability

Those who have been with us for a while may know that we change the cloud provider to Digital Ocean in January. At the same time, we started experimenting with HA database cluster. And we learnt a lot.

Those who have been with us for a while may know that we change the cloud provider to Digital Ocean in January. At the same time, we started experimenting with HA database cluster. And we learnt a lot.A wise man would say that you learn most from mistakes and it's true that if things simply work, you are likely to be caught off-guard. We have been experimenting with HA database clusters for several months and believed we mastered the technology. Well, you can judge for yourself in this <a href="https://magicofsecurity.com/mysql8-cluster-and-networking-problems/" target="_blank">blog post of mine</a> :) One big lesson however was that you can't do much without information.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Netdata provides an excellent source of real-time system data." src="/storage/wink/images/ExX2VCY58lIiSEAzmTKNuxWXwyyjiR3CG86f0Mr8.png">Netdata provides an excellent source of real-time system data.</div>We have eventually decided to use Netdata, which so far satisfied all our requirements. We can access detailed reports when we need them, all alarms are forwarded to our Slack and we have also included alarms and main system stats to the KeyChest web site. You can access those at <a href="https://keychest.net/status" target="_blank">https://keychest.net/status</a> . This page also includes notes about problems and planned system upgrades when we expect downtimes or issues.Another aspect of the KeyChest monitoring is access to the application logs. We have completely re-factored logging in the KeyChest audit engine. The aim was to create data that is easy to analyse and query. We have decided to go for the JSON format, with simple text messages and separate parameters and implemented a new log dispatcher that collects logs from multiple audit processes.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="" src="/storage/wink/images/Xf9g3ndqb1yPAHX3e11ZdezDR4hAuvwYTR8COq3R.png"></div>Each log message uniquely identifies its source and the version of the KeyChest. It has two timestamps so we know when the logged event happened and when it was added to logs. All variable parameters are defined as separate JSON data items so we can automate analysis with very little loss of detail.The analysis is crucial for effective use of any logs. In our case, we store logs locally in files to ensure they don't get lost due to networking issues and we also push the data to a Splunk instance for visualization and querying.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Splunk visualisation of memory use by KeyChest audit engine" src="/storage/wink/images/Alqm9WbBU1fi6fRTEo3uKIvf1EMsnP3oLwtuJTTZ.png">Splunk visualisation of memory use by KeyChest audit engine</div>We really do put a lot of effort into making KeyChest a reliable service. Sometimes it feels like everything is stacked against us but we are making it better week by week. So if something doesn't work it only means we missed something and we'd love to hear from you!

Publication Date: 2019-06-17 13:58:00

Can blockchain remove the need for an SSL certificate authority?

Blockchain is basically a chain of signatures. You create a digital signing key and sign data/transaction with it. It has nothing to do with trust.

Let’s think about what blockchain does and what a certification authority (CA) does.Blockchain is basically a chain of signatures. You create a digital signing key and sign data/transaction with it. Each new signature is over 2 items:<ol><li>the data/transaction you want to sign - it could be a string with a “public key” and “name”; and</li><li>the previous signature you created.</li></ol>CA verifies that a certain “name” belongs to YOU, that you own a “public key”, and binds the “name” and “public key” together with a digital signature. YOU can be a domain name, email address, an organisation, a physical person with a postal address, and so on. CA will use any of a number of methods to verify that the “name” is YOU. The signature at the end is the easy part.So Blockchain can create a signature, but that is hardly enough to replace CA. But let’s say, we want use Blockchain. The first question would be - who verifies that the “name” is you, who will be responsible for the correctness, how you compare independent proves, if you expect more than 1.On a side-note, good CAs will sign all their operational log as a “blockchain”, i.e., linked together. :)

Publication Date: 2019-06-14 13:06:00

Is a password of 20 characters strong enough to use?

It could be but it may not, it depends on how random those 20 characters are. Let me demonstrate the thought with 2 common attacks on passwords.

It could be but it may not, it depends on how random those 20 characters are. Let me demonstrate the thought with 2 common attacks on passwords. These attacks are:<ol><li>dictionary - using dictionaries of passwords someone else used before or words from “normal” dictionaries, including concatenations of 2 or more words;</li><li>brute-force - trying all possible combination of characters (letters, numbers, special characters), possibly using templates, e.g., first letter is capital, last is number, lower-case in the middle.</li></ol>So if you use 2–3 common words to get 20 characters’ password, then it will be weak.If your password is random and not in any of the dictionaries (attack no 1 is impossible), it has to be long enough to withstand brute-force attacks.In this case, a 20 characters’ long password made up from 70 different symbols (lower case, upper case, digits, special characters) is as strong as today’s encryption keys. Which means, it is “cryptographically secure” and it doesn’t make sense to go any further.If you think Google represents industry standards, then its Google Authenticator uses secrets of 80 bits = 13 characters passwords as defined above. These secrets are used to compute one-time passwords.

Publication Date: 2019-06-14 13:01:00

What could be done if all current Encryption could be broken and cracked?

If all current encryption were suddenly broken, that would be the end of it for encryption as we know it. With one exception - one-time pad.

If all current encryption were suddenly broken, that would be the end of it for encryption as we know it. With one exception - one-time pad. One-time pad is a provably secure encryption that can’t be broken, but it has practical difficulties - it requires keys as long as the data.There are, however, alternatives to encryption and there is also a saying “If you think encryption will solve your problem, you don’t know what is your problem.”.If you want to “crack” encrypted data, you need to get the encrypted data first. The most likely defence would in this case be protection of the communication channels - whether by splitting data between 2 or more channels, making access to these communication channels hard, or hiding the fact that there is any data present (e.g., by mixing it up with random noise).Practically - you can’t do much. Current encryption algorithms are hard-coded to critical business applications and it would be impossible to change much in less than 6–12 months.We may be forced to think which data is sensitive and important and which data we actually don’t have to send or store in the first place. This is another great way of protecting us from getting hacked.

Publication Date: 2019-06-14 12:57:00

What Is Encryption Domain

Encryption domain is simply a set of computers or other computing devices (or even people :) ) who share encryption key(s) allowing them to trust each other.

Encryption domain is simply a set of computers or other computing devices (or even people :) ) who share encryption key(s) allowing them to trust each other.It can be a master authentication key which can easily verify identities of entities within the encryption domain. Or it can be an encryption key, which allows access to encrypted data or communication.The point of encryption domains is to create an environment where all entities within an encryption domain can automatically verify each other or share access to and create encrypted assets which can be accessed with the knowledge of “domain encryption key”.

Publication Date: 2019-06-14 12:54:00

What is the SHA-256 fingerprint?

SHA-256 fingerprint is a digital fingerprint we use to compare two documents or to check if a document has been changed.

It is a digital fingerprint we use to compare two documents or to check if a document has been changed.It has a fixed length of 64 bytes (or characters, if you want), which encode 256 bits, each bit is either 0 or 1. Once calculated, it is unique for the data or file for which it has been created. While it is relatively short, it is practically impossible to compute any other document, which would have the same fingerprint.The main properties of digital fingerprints, including SHA-256 are:<ol><li>It’s short - it is always 256 bits regardless of whether the original data is 100 bytes or 100 terabytes.</li><li>It’s impossible to compute a different data which would have the same fingerprint.</li><li>It’s impossible to compute the original data from the fingerprint.</li></ol>What it DOES NOT do - it doesn’t necessarily protect against discovering the original data, if the original data comes from a sufficiently small set. Attackers can simply try all possible data values and find a match. Let's take ZIP codes as an example. There are less than 100,000 5-digit zip codes in the USA and 1,000,000,000 zip+4 codes. It's trivial for your laptop to calculated SHA-256 fingerprints of all of them and compare them to the one fingerprint you want to "reverse". This approach is used to create "rainbow tables" - fingerprint tables of all possible values - used to crack our passwords.From practical point of view:- It DOES “hide” the original data (with the caveat above).- It simplifies data search - it can speed up database queries when they are done with fingerprints, rather than the data itself.- It is fast to calculate.- If it’s too long for you, you can use only part of the fingerprint and it still works (too much detail possibly, but beware of the birthday paradox if you don’t want to end up with collisions - <a href="https://en.wikipedia.org/wiki/Birthday_problem" target="_blank" style="color: rgb(43, 109, 173);">Birthday problem - Wikipedia</a>).The name also denotes the function used to calculated it (SHA-256), which is one of the family of functions called SHA-2 and which includes SHA-224, SHA-256, SHA-384 and SHA-512, where the number indicates the size of the fingerprints.

Publication Date: 2019-06-14 12:41:00

What is certificate pinning

Some people argue that certificate pinning is a must to protect against sophisticated attacks. Some will say that it's pain in the bottom. But what is it actually?

Some people argue that certificate pinning is a must to protect against sophisticated attacks. Some will say that it is a pain in the bottom. But what is it actually?Certificate pinning is a mechanism that introduces "direct trust" between your web browser and a web server. When the web server (e.g., quora.com) provides a "pinning flag", your web browser will remember it and will warn you when you try to connect to the quora.com server again and it shows a warning if it detects a configuration change (i.e., someone attacks you or someone made a configuration error).The usual verification of HTTPS is using standard X.509, which defines certificate verification via proxy, where the proxy is a “trusted authority”, or a CA. We, as users, don’t care which certificate is used so long as the trusted authority confirms it’s a good one. And as long as the trusted authority is trusted by our web browsers.Pinning means that you only trust the certificate you have seen before. You may use the proxy for the first validation, or you can just say whatever it is, I trust it.When you talk to the same server again, you only trust it if it uses the same certificate. If it’s a different one, it will raise an alarm even if the proxy says it’s a good one.In a way, the concept of certificate pinning is direct trust, just like in PGP (if you’re familiar with this system).

Publication Date: 2019-06-07 07:53:00

Reliability of Let's Encrypt

I have analyzed reliability of Let’s Encrypt using their own data in Spring 2018.

It’s a good question especially after a recent <a href="https://community.letsencrypt.org/t/may-19-2017-ocsp-and-issuance-outage-postmortem/34922" target="_blank" style="color: rgb(43, 109, 173);">downtime of LE</a> that lasted more than a day, with 16 hours of the service basically unavailable. This is quite a serious outage, which will probably put LE behind most of other certificate provides as it makes it uptime just 99.8% on its own for this year.As LE assumes that renewals are automated, you should never assume that LE services will be always available and design your own production automation accordingly.<div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Let's Encrypt planned updates and their actual duration." src="/storage/wink/images/bTp1ljxHT4oYcr6XKTB3GuGh5D7BQPEiFUGDQm3x.png">Let's Encrypt planned updates and their actual duration.</div>Having said that, I don’t believe there is a real possibility of an outage, which would last more than a week so in terms of issuance and certificate renewals LE is likely to remain reliable if you plan and renew according to recommended timelines.Other than that, you should consider operational limits of LE. I have written up <a href="https://keychest.net/content/letsencrypt_numbers_to_know" target="_blank" style="color: rgb(43, 109, 173);">many of them into one blog post here</a>.You may also be interested in assurance in terms of potential damage caused by incidents. I haven’t found any authoritative reference for any such assurance yet.I have analyzed reliability of Let’s Encrypt using their own data in Spring 2018. <div class="embedded_image" contenteditable="false" data-layout="default"><img alt="Duration of "full" disruptions per month with incident detail at the bottom." src="/storage/wink/images/071NXyAp95YZ5uEw8iKcTsZHLAPIbzA6QcTsyVxR.png">Duration of "full" disruptions per month with incident detail at the bottom.</div>I used a couple of charts above and the write-up is at <a href="https://dan.enigmabridge.com/lets-encrypt-uptime-is-99-9%E2%80%8A-%E2%80%8Aor-98-8-without-defects-in-2017/" target="_blank" style="color: rgb(43, 109, 173);">Let’s Encrypt uptime is 99.9% — or 98.8% without defects in 2017</a>. It shows that it was fully up and running only for 98.8% of time in 2017. Some disruptions were “partial” so only a subset of users would experience them.Another important aspect are changes to the protocol and its implementation. Many users experienced problem in Q3 as LE set IPv6 as the primary protocol. It was used whenever it detected existence of an IPv6 address, regardless of whether it actually worked.It certainly makes sense to deploy an external end-to-end monitoring to detect any potential issues early on - something like our <a href="https://keychest.net" target="_blank">KeyChest service</a> .

Publication Date: 2019-06-07 07:41:00

KeyChest blog KeyChest updates, information, and write-ups