Sandpanther (sandpanther) wrote,
Sandpanther
sandpanther

  • Mood:

Update...

So... I kind of fell off the face of the planet. I can't remember last week (this week sucked too much), but here is what happened this week:



I had an appointment with a mortgage broker that a friend of my mother's recommended. The short version of the story is that I've tossed him into a "don't call me, I'll call you" category. It was really weird...

The guy gave me bad vibes. It started out okay, but part way through I got the feeling that he suddenly decided that I was a waste of his time. I'm not sure why. The really weird thing is that when I first started getting these vibes was right after he was trying to push a 5/1 ARM on me using the reasoning that "anything could happen in five years. You might get married and start a family." I replied that it's not likely. After that, the weird vibes started. Really weird.

Anyway, after talking with some other people and doing some more research, I've decided that the guy must have thought me a source of easy cash. The rate he was looking at selling me for the second loan was a good 2% higher than what one of my coworkers was getting (and pretty awful, at 8.25%!) As an alternative to the second loan he was suggesting a primary loan that would avoid PMI, but would permanently add an extra .5% to the loan. All the loans I've heard of that do that lose the extra percentage once the value of the principle drops to below 80% of the value of the property. I specifically called him on that, and he said that no, it was a permanent part of the loan. Right. I'll just be taking my business somewhere else now.

After that he started subtly implying that he didn't trust that all of the information I was supplying him about outstanding loans and my income were correct. That was particuarly annoying since I had all the documentation he requested regarding income and debts with me.

Overall it was a really depressing experience. I've done all the math to make sure that what I'm asking is reasonable. I know that I'm really conservative fiscally, which means that if I think I would be a good risk, then most financial instutions will be trying to push more than what I'm comfortable with. I expected that securing a loan would be the easy part of the process. And I certainly did not expect to be left with a feeling like I'm asking something unreasonable, or that the amount I'm looking to borrow is too piffling to be worth someone's time. I know it's not on par with the million-plus figures that most homes around here go for, but sheesh!





As I was leaving for work I noticed a message on the answering machine that hadn't been there when I went to sleep. Some strange impulse prompted me to listen to it before leaving. It was from someone from work. Somewhere in all the rambling he mentioned that reports were not printing, and that he thought the server was down. Hm, potentially bad.

A minute after I got to my desk, several people came over to report a couple of critical problems which were interfering and/or halting production -- and had been since about 8:30 last night. (I thought it odd that they hadn't called until after I went to bed around 11. I also was amused that my roomie -- who I knew had to have still been in the living room when the call came in -- had not thought someone saying the server was down was reasonable grounds for waking me up. I love my roomies and their perspective!) It didn't take long to identify the problem -- a ton of critical files (and non-critical files) were missing from directories on two different servers.

Fixing the problem, though... Restoring files from backup isn't my responsibilty. The people whose responsibility it was were not in yet, and had no ETA. Folks in charge of Production were not happy. *shrug* Not much I could do.

Many hours later we got the necessary files back, but it took through Wednesday afternoon to finish getting the mess sorted out. Actually, the whole mess isn't entirely done, since we still have a security investigation underway to try and track down the individual who deleted the files (it has been determined that it was malicious damage.) And we haven't done anything to fix our security policy to prevent this from happening again. I sleep much better at night when I ignore that last part. We have a meeting scheduled next Tuesday to discuss how we want to approach locking down file access. Let it not be said that we are overly hasty in addressing the issue.

By the time I got home I was so exhausted and the weather was so terrible that I decided to err on the side of caution and not risk going out on the roads. Which meant that I didn't go to class. *sniff* I actually find my class relaxing, so that may not have been the best call. OTOH, getting into an accident because I was too tired to have acceptible reaction times probably would also have been a bad call.



Wednesday was mainly devoted to cleaning up the mess from Tuesday. Or the calm in the eye of the storm. I'm not sure which.



I've done longer than a 14 hour work day once before, maybe twice. In any case, Thrusday was probably the worst of them all, since it wasn't planned in advance.

The power went out around 10:30 AM. The UPS failed when the power went back on, which caused my server to crash. (Don't think too hard about that; I'm not even going to pretend that it makes sense.) That particular server has two known problems: the hard drive controller card is flakey and tends not to recognize the hard drives when the system is powered back on, and for months there has been a system message stating that system files are the wrong version, or are missing. These two together meant that the system wouldn't boot, and even once the controller card was working with the hard drives, the system was corrupted.

Actually, the OS being known to be unstable may have been unrelated to the problem. It turns out that they were in the middle of powering down the server when the power cut out. Either way, the end result was that the Registry corrupted.

It took about six or seven hours to realize that the system was screwed up enough that restoring a good copy of the Registry from backup wasn't going to fix things. Fortunately there was a new server that had been set up but not deployed yet which we ended up comandeering. Eleven hours after the power went off the server was once again available so that the production floor could work.

The chaos left in the wake of this outage was, well, profound. I've heard rough estimates tossed around that we lost about $100,000 by having the systems down for that long. I have a feeling that senior management is going to want to know what happened.

I had arrived at work at about 7:05 AM. I finally left at 9:15 PM with the sure knowledge that we had barely scratched the surface on the amount of work remaining to recover from the outage.

Oh yeah, and once again, I had to skip class. It's a good thing that it's not for a grade. OTOH, I'm not taking it for a grade, I'm taking it to learn things. I'm not exactly learning anything by not going. (No, I didn't even know how screwed up my workplace is. I already knew that. And I already knew how vulnerable this particular server was to having exactly this kind of thing happen. Which kind of made the staying late suck all the worse, since it really was a preventable outage.)





Friday was painful only because 1) everything that had happened previously in the week, and 2) I had to sit in a meeting and listen to someone who is at least as useless as Fluffy try to weasle out of responsibility for any of the problems that his lack of action allowed. The non-critical system restore issues that had been left over from the night before did not get solved because I spent the day stuck in a mire of politics. Dumb, really.

I did manage to get one of my coworkers to laugh. This pleased me. Especially since he was laughing at me swearing at one of his coworkers. It happened like this...

On my way back from getting a caramel apple cider (to fortify myself for the rest of the day), Special Boy approached me. I told him "no, not today." He is a complete idiot who does not pay attention to what he is doing, and does not follow procedure. As a side result of this, he doesn't do all the steps required for a given task, and screws the whole thing up. I told him that he was not a system-critical issue, and as such would have to wait until Monday, since I would be busy fixing critical issues for all of Friday. He kept persisting, and eventually stated that he though something was wrong with the new server. I refrained from stating that I thought something is wrong with his brain. However, I did tell him (while walking off) that if he wanted, I could put the old server back. Really, it would be absolutely no problem at all. When I walked into my area, I muttered "do no give me shit today!" One of Special Boy's coworkers looked up from getting a cup of coffee and laughed.

The only good thing I can say about Special Boy is that in the month that he's been here, he's managed to demonstrate gross incompetence to just about everyone he has encountered. With a little bit of luck, he'll get himself fired soon. Preferably before I do something permanent and fatal to him.

Actually finishing making the new server stable ended up being put off because Cotton Candy (who is in charge of the Network Infrastructure group and thus is the responsible party for all the outages from this week) called a post mortem meeting. Next time can I finish killing the issue before we hold the post mortem?

That meeting was kind of fun, in a masocistic kind of way. Cotton Candy hadn't bothered to get his fact right, so I took great delight in pointing out every time something was inaccurate in his repots. (CC: "The incident was reported at 11 PM." Me: "No, it was at 8:40 PM." CC: "They used such-and-such a login." Me: "Actually, we can neither confirm nor deny that login was used. Current evidence suggests that it is unlikely, though." *murmers of agreement from CC's staff*)

CC is such a fuckwit. He tried to blame the server crash on the file deletion issue, and claimed that everything on that server was suspect. He didn't really like it when I pointed out that there were two servers effected so by his arguement the other one was suspect as well. (The other one is our main fileserver. Rebuilding all of it... Heh.) It probably didn't help his case any when his own staff attributed the failure to the know hardware and OS issues. Like, duh.

The "best" part of the meeting was when CC started talking about the plan for moving all the critical functionality back onto the broken server. Yes, that's right, the server with the known unstable hardware and corrupted OS. Sounds like a great place to put a mission-critical system. CC actually seemed surprised when the entire room (including his own staff) basically looked at him and said "no."

CC must be the Amway supplier for stupidity. It's the only explaination I can think of for the excessive bulk quantities of the stuff that he seems to possess.







Boss-man called at 10:30 AM to report that one of the programs wasn't working. If we hadn't had all the outages earlier in the week, it wouldn't have mattered that it was down over the weekend. Of course because of the outages earlier in the week folks were in working and the program failing needed to be fixed immediately. It turned out that the only problem was that someone had shut down the program. Not sure who, not sure why. Since everyone who had reason to be on that server knows better than to close the program, I begin to wonder if someone had access they shouldn't have to the system and shut down the program. We will see. Somehow, I don't think we've heard the end of the gremlins.

Subscribe
  • Post a new comment

    Error

    default userpic
    When you submit the form an invisible reCAPTCHA check will be performed.
    You must follow the Privacy Policy and Google Terms of use.
  • 2 comments