Relevant only to your beginning comment about never using a plain photo when a kitty photo is available:

I have a new addiction.  It's all your fault.  I am on the verge of randomly, throughout my day, starting to hum...

... What Kitty Doesn't Like                  

that I have helped spread Sylvester's message makes me happier than you know!

i am a talking kitty evangelist and you're my first convert.   

Anyway, emergency upgrades to a live production server on the fly in the middle of a work day resulting in angry end users aren't as relaxing as they may seem to those of you not in the trade.  

Sweet Baby J. I am so. fucking. sorry. 

Thanks.  As Buddy on Night Court used to say "I'm much better NOW."  Long story short table in live environment containing 340,000+ rows of job data shit the bed so came in Monday morning to all kinds of weird error messages from various users in most modules.  

Choice was a patch and reload 116 GB from backup.  Iffy stability and all data entered post backup re-entered manually (or mapped and imported - either way time consuming as hell.) or door number 2 which was move the upgrade planned for end of year to immediately and clean up the wonky table in the conversion process. 

Sad when the least painful option is a surprise upgrade on a Monday morning.  Of the 340k+ rows of data I got all back clean except for 809 and hoping the awesome top tier guy over there will help me in the morning.  Other than that it's just been the usual weird stuff, testing, verifying data and relinking tables to custom reports, setting defaults needing to be reset, permissions!  @Curious permissions are spiteful in all platforms!  

Anyway, if I can get back my 809 rows I'll unclench.  If not...hurling myself off a building or starting a new life in a country where I don't speak the language are viable options.    it'll be fine...kidding.

The fury from one of tptb isn't even bothering me.  This wasn't a screw up, it's one of those things that totally suck but happen.  If they didn't disaster recovery wouldn't be a thing and software companies wouldn't pay more in tech support than R&D if the default was trouble free point and click deployments.  Would be nice if people had the slightest idea of the complexity involved before complaining, but not going to happen.  

it was way more fun testing FJ with you.  No accounting and 

Fucked up the formatting and totally forgot I wasn't typing this in PM so apparently I should sleep now since typing is hard! 

Anyway I was going to say the best thing about testing on FJ is no accounting and no one who honestly believes applications run on wishes and magic spells.

OMFG. That sounds like the worst day ever. That's a big freeking table. out of that much data, 809 rows is an acceptable loss. I mean, getting it back would be awesome, but it's isn't the most terrible thing. Clearly you personally broke everything though, b/c you thought your job was boring, amirite?

I will neither confirm nor deny saying a small prayer to  the gods when I did the FJ upgrade though....so there was a wish or three and a magic spell done. That said, the server SO does not run on that.

Fucking permissions! 

Everything is back.  Because I was sleepy and a dumb-ass I miscalculated how much was missing and it was only about 25 rows which were regenerated by running the populating transactions again - I am hoping to unclench soon.

Actually considering the circumstances about 5 hours total downtime wasn't as bad as it could have been.  The clean up with little wonky stuff - the handful of transactions that went in during the dark window of corruption.  Have a few service calls still out - one for the data issue itself (5 items printing weird) and the the others because there is no documentation for certain permissions which now rely on other settings in the new version which didn't before.  PERMISSIONS!  

I did get the worst possible response from a tech support person in the history of tech support which I can kind of laugh at after the fact.  Vendor typically excellent on the tech support front, even for tier one, but man this chick needed to be vetted before being assigned to 24 hour emergency support.