Here's what happened to short-media

primesuspect · August 2003

Okay, this is a post for the tech-minded curiousity seekers who may be wondering just what went on.

bart.short-media.com is the webserver that this site is running on. It is colocated with Sittech, Inc (Rob's company). This is the first server we've ever colocated off site. We decided to colocate due to projected bandwidth demands for this site (extrapolating from the 'old' icrontic's traffic) and figured that the 768K upstream that we have at our office wouldn't be enough.

We have a Sun server that is our company's primary web and mail server. It also serves as our primary DNS server. Needless to say, it was MUCH easier to set up short-media's mailer to go through this server rather than configure sendmail on short-media's server.. So since the beginning of this site's history, mail has gone through our main mail server in Detroit. It has worked so far

Now, enter the great blackout of '03 :rolleyes2

Now, since the Sun box did DNS, when the power died, the name resolution for short-media died as well. This was a design flaw on our part. We thought having DNS on a seperate box was a good idea, which I still believe it is, but our design flaw was this: We only had about 20 minutes of backup power at our Detroit location. Well, obviously those 20 weren't enough

On Friday morning, we didn't know much about the power situation. All we knew was that Detroit would probably be down until Sunday. We also knew that looting was occuring, and it was occuring on our premises. We made a flash decision to move the primary server to the datacenter in Southfield. We took all of our valuable equipment and packed it into the back of my truck and drove to Southfield (Southfield is a suburb of Detroit which is about 20 miles from our office) Rob was there, which was no surprise, since the datacenter was the only building for miles around that had power

After the previous night, seeing lights and feeling air conditioning was very very strange. So we hooked up the sun box on the rack, and switched all the relevant IP addresses. I then updated the server's nameserver host record with Verisign and crossed my fingers.

Well, this morning (Saturday morning) I checked the host record and sure enough, it updated, and that's why we're all here now

I don't know why the outbound mail died, but it seems to be working now.

I can't describe how apocalyptic the previous two days have felt. My five year old son said "Dad, why is the world different today?"

What a time. What a time.

Garg · August 2003

Thanks for all your work! I can't imagine what it must be like up there. Most of us take electricity for granted, and I'm just not sure I trust civilization to behave rationally in a crisis.

How's the looting/crime situation now? Most of the news has been really New York centric, I hardly hear about Detriot and other cities affected by the blackout.

Linc · August 2003

Awesome work as usual, Brian

primesuspect · August 2003

I can't believe how little the media is talking about Detroit. Power seems to be back for the most part, but there were still long lines at gas stations last night, and even some lines this morning. My phones are dead, and it's not because of lack of power. The CO is dead. If I call my office number right now, there is no ring. Nothing. Just silence.

Park_7677 · August 2003

me j00

(and everyone else)

primesuspect · August 2003

I have to give mad props where credit is due.

1: The colocation facility: www.switchanddata.com

These people are serious - as serious as they come. Not having a single second of downtime during the nation's worst power outage ever is testament to this. I cannot praise them enough. They kept us constantly updated on the power situation, and were scheduling fuel deliverys for days in advance. Top quality, professional, and dedicated.

2: Sittech, Inc. (www.sittech.com)
Rob was right there with us (big sacrifice! Air conditioning!

) but really, he always makes himself available, any time of the day, to help us with any technical issues. If you need something co-located, I cannot recommend a better company than his. You need game serving or colo services, do not talk to anybody without talking to Rob first.

True professionals

IC-Games · August 2003

Glad to see you guys back

Jimborae · August 2003

In Primes avatar

Fear ME, for I am w00t

Er no, we love you, for you are w00t

Thanx for all your effort.

Jim

profdlp · August 2003

I'll bet I was not the only one who suffered withdrawal symptoms during the outage. I never had any doubt that everything possible was being done to get things back in order - thanks for all of your hard work.