Here's what happened to short-media

primesuspectprimesuspect Beepin n' BoopinDetroit, MI Icrontian
edited August 2003 in Community
Okay, this is a post for the tech-minded curiousity seekers who may be wondering just what went on. is the webserver that this site is running on. It is colocated with Sittech, Inc (Rob's company). This is the first server we've ever colocated off site. We decided to colocate due to projected bandwidth demands for this site (extrapolating from the 'old' icrontic's traffic) and figured that the 768K upstream that we have at our office wouldn't be enough.

We have a Sun server that is our company's primary web and mail server. It also serves as our primary DNS server. Needless to say, it was MUCH easier to set up short-media's mailer to go through this server rather than configure sendmail on short-media's server.. So since the beginning of this site's history, mail has gone through our main mail server in Detroit. It has worked so far :)

Now, enter the great blackout of '03 :rolleyes2

Now, since the Sun box did DNS, when the power died, the name resolution for short-media died as well. This was a design flaw on our part. We thought having DNS on a seperate box was a good idea, which I still believe it is, but our design flaw was this: We only had about 20 minutes of backup power at our Detroit location. Well, obviously those 20 weren't enough :)

On Friday morning, we didn't know much about the power situation. All we knew was that Detroit would probably be down until Sunday. We also knew that looting was occuring, and it was occuring on our premises. We made a flash decision to move the primary server to the datacenter in Southfield. We took all of our valuable equipment and packed it into the back of my truck and drove to Southfield (Southfield is a suburb of Detroit which is about 20 miles from our office) Rob was there, which was no surprise, since the datacenter was the only building for miles around that had power :) After the previous night, seeing lights and feeling air conditioning was very very strange. So we hooked up the sun box on the rack, and switched all the relevant IP addresses. I then updated the server's nameserver host record with Verisign and crossed my fingers.

Well, this morning (Saturday morning) I checked the host record and sure enough, it updated, and that's why we're all here now :) I don't know why the outbound mail died, but it seems to be working now.

I can't describe how apocalyptic the previous two days have felt. My five year old son said "Dad, why is the world different today?"

What a time. What a time.:ninja:


  • GargGarg Purveyor of Lincoln Nightmares Icrontian
    edited August 2003
    Thanks for all your work! I can't imagine what it must be like up there. Most of us take electricity for granted, and I'm just not sure I trust civilization to behave rationally in a crisis.

    How's the looting/crime situation now? Most of the news has been really New York centric, I hardly hear about Detriot and other cities affected by the blackout.
  • LincLinc Owner Detroit Icrontian
    edited August 2003
    Awesome work as usual, Brian :cool:
  • primesuspectprimesuspect Beepin n' Boopin Detroit, MI Icrontian
    edited August 2003
    I can't believe how little the media is talking about Detroit. Power seems to be back for the most part, but there were still long lines at gas stations last night, and even some lines this morning. My phones are dead, and it's not because of lack of power. The CO is dead. If I call my office number right now, there is no ring. Nothing. Just silence.
  • Park_7677Park_7677 Missouri Member
    edited August 2003
    me <3 j00 :D

    (and everyone else)
  • primesuspectprimesuspect Beepin n' Boopin Detroit, MI Icrontian
    edited August 2003
    I have to give mad props where credit is due.

    1: The colocation facility:

    These people are serious - as serious as they come. Not having a single second of downtime during the nation's worst power outage ever is testament to this. I cannot praise them enough. They kept us constantly updated on the power situation, and were scheduling fuel deliverys for days in advance. Top quality, professional, and dedicated.

    2: Sittech, Inc. (
    Rob was right there with us (big sacrifice! Air conditioning! :D ) but really, he always makes himself available, any time of the day, to help us with any technical issues. If you need something co-located, I cannot recommend a better company than his. You need game serving or colo services, do not talk to anybody without talking to Rob first.

    True professionals :)
  • edited August 2003
    Glad to see you guys back :)
  • JimboraeJimborae Newbury, Berks, UK New
    edited August 2003
    In Primes avatar
    Fear ME, for I am w00t

    Er no, we love you, for you are w00t :)

    Thanx for all your effort.

  • profdlpprofdlp The Holy City Of Westlake, Ohio
    edited August 2003
    I'll bet I was not the only one who suffered withdrawal symptoms during the outage. I never had any doubt that everything possible was being done to get things back in order - thanks for all of your hard work. :thumbsup::respect:
  • SpinnerSpinner Birmingham, UK
    edited August 2003
    I second that.
  • SpinnerSpinner Birmingham, UK
    edited August 2003
    I second that.

    (okay, so I did this one on purpose ;) )
  • NecropolisNecropolis Hawarden, Wales Icrontian
    edited August 2003
    And I third it. Great job guys :thumbsup:
  • PreacherPreacher Potomac, MD Icrontian
    edited August 2003
    Thanks for continuing to feed my need for a daily Short-Media Fix!
  • Straight_ManStraight_Man Geeky, in my own way Naples, FL Icrontian
    edited August 2003
    Ya done good, Brian and team. Superb.
  • LeonardoLeonardo Wake up and smell the glaciers Eagle River, Alaska Icrontian
    edited August 2003
    I am impressed by your dedication and loyalty to the Short-Media members. You lead by example! Hat's off...:thumbsup::cool:
Sign In or Register to comment.