The Adventures of Accordion Guy in the 21st Century

I’m posting this here (as well as the forums on Tucows Discuss) since I know that some of our customers read this blog from time to time.

Here’s what I know — a more official notice should be coming your way shortly.

The actual applications, for the most part, continued to work.

Folks

with Blogware accounts could post to their blogs, Email Defense

continued scanning for spam and viruses, Certs kept working, Email was

queued for a little while and began processing again in the early

evening, and no mail was lost.

What stopped working during the outage: Provisioning and Management.

The

specifics are a little different for each service, but in general, it

was the ability to sell new instances of services or manage existing

ones. Blogware, being located in a different facility, was unaffected.

It’s very unusual for a redundant power system to fail like this.

The

“redundant” in “redundant power system” makes this sort of thing very

unlikely. For example, if you have two systems with a 1% probability of

failing, the probability that both will fail is one one-hundredth of 1%.

The

cause for the failure is as of yet unknown. We’re working with IBM to

identify it. Once we know, we will take steps to prevent this from

happening again and let you know what we will be doing in the future.

If you’re a Tucows reseller, this will be of interest to you:

Greetings,

Please read the following important information concerning OpenSRS.

1. Problem at Hosting Facility Impacting Service Availability

At approximately 20:14 UTC or 4:14PM EST today, Tucows redundant power

supply at the IBM data center failed. As a result, the majority of the

services have been impacted to some degree.

• Domain Names: provisioning, management, renewal, and transfers

  offline. This includes WHOIS.

• Email and Email Defense: No mail is lost. Mail delivery may be delayed. Provisioning and management are offline.
• Digital Certificates: Certificates operating normally.

  Provisioning and management are offline.

• Website Builder: Website Builder is operating normally for

  end-users. Provisioning and management are offline.

• Blogware: The service is UNAFFECTED.

Tucows is working closely with IBM to resolve the power problem. At this moment, there is no available restore time.

Tucows will provide regular updates as they are received. Tucows

will also update the Systems Status tool as information is received.

The Systems Status tool is available at:

http://status.tucows.com/.

We apologize for any inconvenience this may cause.

Thank you for your continued support of Tucows.

Sincerely,

Ed Gray

Director, Operations

If you’re a Tucows reseller and felt a great disturbance in the source

(“as if millions of packets suddenly cried out in terror and then were

suddenly silenced”), it’s because the redundant power system at our

co-location itself became redundant.

All services save Blogware have been affected, and crews are on the scene working on the problem. More details as I get them.