KeyChest Status
Brief updates of KeyChest status and planned operational changes

KeyChest Service Status Updates (times are in UTC)

All Systems Operational Last change 7 March, 10:21am

Audit Engine

Operational

API

Operational

Web Interface

Operational

Note: just status updates now, we will be adding real-time status information in March 2019.

Planned Digital Ocean datacenter upgrade

Time: 26 Mar 21:00am - 27 Mar 3:00am

Expected impact: There may be a few minutes of increased latency as well as small amounts of packet loss while we shift traffic to redundant devices. We will endeavor to keep this to a minimum for the duration of the change.

KeyChest upgrade

Time: 17 Mar 23:40am - 18 Mar 3:35am

The service was taken down for upgrades and maintenance.

Service outage

Time: 15 Mar 7:40am - 15 Mar 13:05pm

A couple of things happened at the same time - but they were triggered by network issues between our virtual servers. Those caused a rapid growth of database synchronization data, which filled one of the servers' disks. The recovery was a matter of restoring from backups but it took a little longer, unfortunately.

Database node outage

Time: 12 Mar 9:40am - 13 Mar 10:30am

This was the second time within a fortnight and it became clear that we should not try to optimize for speed if it means losing resiliency. We have now disabled direct access to a local database nodes for reading. There is a small impact on speed of the web UI but KeyChest is more resilient as it fully utilizes its failure resistant database cluster.

Improving performance of the audit engine

Time: 9 Mar 11:39am

Ever since the KeyChest migration to Digital Ocean, we can see delays in audits while the CPU is only used at 30% of its capacity. We are working on removing bottlenecks so that we can show you first results for new domains more quickly.

One of database nodes down due to network errors

Time: 7 Mar 4:40pm

Update @18:42, 7 Mar: the database cluster is back to normal.

One of our database nodes dropped out from the database cluster due to network errors. We have restarted it and the system should be back to normal shortly. There may be a small impact on real-time notifications for servers added / discovered between 8:50am and 16:30pm UTC.

API changes - integration

Time: 7 Mar 10:14am

We are improving the integration of the API with online reports and the audit engine. The update will also allow users to manually add services with wildcard certificates under a correct main domain. It will improve visibility of wildcard certificates. It will also allow us improving online reports with regards to wildcard certificates.

Audit Engine incorrectly ignores valid domains

Time: 18 Feb 9:40am - 19 Feb 23:20

Update @23:20pm, 19 Feb: the audit engine has now updated all faulty records.

Update @10:32am: we have identified a probable cause of the problem. The audit engine will be updating results over the next 12 hours. We will probably re-send updated weekly summaries once the re-audit has been completed.

The effect of the error is out of date information in weekly reports. We are looking into the issue and expect to resolve it today. We will provide updates in due course.

Upgrade to version 1.7.7

Time: 18 Feb 2019, 08:40 - 08:45

Upgrade resolving some bugs and small improvements.

Upgrade to version 1.7.6

Time: 17 Feb 2019, 23:55 - 00:01

Upgrade resolving some bugs and small improvements.

Audit Engine running to its schedule

Time: 14 Feb 2019, 16:00

Periodic audits were behind the schedule before the migration and the migration caused an additional backlog. Faster server has now helped to catch-up with all the backlog and periodic audits of your services are now executed as configured - every 12 hours.

ROCA tester fails

Time: 12 Feb - 14 Feb 11:50 am

We have discovered a problem with the ROCA tester after the migration to Digital Ocean. The problem has been fixed and the tester should be now fully functional.

Upgrade to version 1.7

Time: 13 Feb 2019, 13:00 - 13:04

We have migrated KeyChest to a Digital Ocean datacenter.

Migration to Digital Ocean

Time: 9 Feb 2019, 23:50 - 10 Feb 2019, 5:00am

We have migrated KeyChest to a Digital Ocean datacenter.

Creating this status page

Time: 14 February, 2019

We decided to create a simple page for the service status updates. We will keep adding entries about discovered problems / bugs as well as about planned operational changes.