Downtime this evening

LincLinc OwnerDetroit Icrontian
edited December 2011 in Community
Me: " I need memcached"

Sysadmin 1: "You already have it."

Me: "No I don't."

Sysadmin 1: "Yes you do it's right here"

Me: "That's memcache (no 'd')."

Sysadmin 2: "OK I will install it but you're running PHP 5.1. Can I upgrade it to 5.2?"

Me: "I'm on PHP 5.3. WTF."

Sysadmin 3: "Then where is it?"

Me: "Try the 'locate' command. It's right where your sysadmin installed it 18 months ago."

Sysadmin 3: "This is confusing. Can I delete the copy of PHP 5.1?"

Me: "Sure. We only use PHP 5.3"

Sysadmin 3: "Turns out they were hooked together and I broke all the things."

[/scene]

Comments

  • primesuspectprimesuspect Beepin n' Boopin Detroit, MI Icrontian
    Amazing that we made it through the most intense migration ever conceived with no data loss and no fuck-ups... It's only AFTER the fact that they screw it all up.
  • TushonTushon I'm scared, Coach Alexandria, VA Icrontian
    Clearly, this wasn't read-only Friday.
  • JokkeJokke Bergen, Norway Icrontian
    I can imagine the rage. Matt must have popped a few blood vessels.
  • Straight_ManStraight_Man Geeky, in my own way Naples, FL Icrontian
    edited December 2011
    So THAT is what happened. Sheesh, no wonder things went nuts for while - and just when I needed an Icrontic forum fix in middle of night (A couple-three times, to tell the truth)....

    Pretty please, would someone tell the spell checker that Icrontic is a valid word - it keeps highlighting it, and I keep doing double-takes and checking the spelling. :D
  • RyderRyder Kalamazoo, Mi Icrontian
    Pretty please, would someone tell the spell checker that Icrontic is a valid word - it keeps highlighting it, and I keep doing double-takes and checking the spelling. :D
    The spell checker is in your browser, right click the word while highlighted and add it to the dictionary.

  • Straight_ManStraight_Man Geeky, in my own way Naples, FL Icrontian
    Thanks Ryder, fixed.
  • LincLinc Owner Detroit Icrontian
    We're back under control.

    There will be database maintenance tonight at midnight. This will involve up to 30 minutes of downtime, but hopefully only a few minutes.
  • LincLinc Owner Detroit Icrontian
    edited December 2011
    We experienced more crashing over the last 36 hours. From what I see, the cause is they neglected to bring APC (advanced PHP caching) back online after they restored PHP, then delayed fixing it after I asked why it wasn't on. I was too panicked and distracted to realize the thing they were dillydallying about was the most important factor.

    I also upgraded the server to have double memory last night, but that looks like it was probably a waste of money at this point.

    In any case, we're now protected (again, finally) by properly configured caching as of 11AM ET this morning and have 4 GB of spare RAM in case everything goes south. If the server crashes again, I'm retiring.
  • CantiCanti =/= smalltime http://www.youtube.com/watch?v=y9K18CGEeiI&feature=related Icrontian
    Sounds like they know about as much as I do about what they're doing (basically nothing). Get them to pay for the upgrade. If you retire I'm uninstalling the internet.
  • UPSLynxUPSLynx :KAPPA: Redwood City, CA Icrontian
    Sounds like they know about as much as I do about what they're doing (basically nothing). Get them to pay for the upgrade. If you retire I'm uninstalling the internet.
    Don't forget - Internet Explorer in the recycle bin = uninstalled internet.
  • LincLinc Owner Detroit Icrontian
    We had a close call yesterday afternoon from a residual problem (MySQL tables had become corrupted from the repeated crashes) but the additional RAM appears to have given the server enough cushion to recover this time (even though the site was unavailable for probably 20 minutes) so I'm keeping the upgrade. That, and it'd probably be another 90 minutes of downtime to downgrade so screw it.

    To their credit, the hosting team caught and solved the MySQL table issue with aplomb.
Sign In or Register to comment.