Just when you thought it was safe

Another big part of my job these days is the matching and dispatching of reconditioned whole-unit exchanges.  We’ll use refurbs for exchanges when we can, ’cos they’re tons cheaper and faster to dispatch than new-building an equivalent system is (48-hour turnaround when we do it right, versus ten days to two weeks for a new build).  This is a function that had been outsourced to one of our teams in India, but there have been continual complaints about making incorrect matches or failing to make a match ’cos they didn’t understand which system is which one’s descendant, and excessive lead time to get a used dispatch out the door—at times as much as three weeks, which is unconscionable.

So one of my managers (I more or less report to two at once—it’s a weird org chart) has been working for months to get us positioned where we could bring used exchanges in the commercial/public segment back home to Circulith.  I volunteered to be one of the pilots because hey, it was something new I hadn’t ever done before, and I kinda enjoy wading into badly broken processes and making them work again.  The pilot has worked out well, and after the first of the year I expect he’ll start pushing India to stick to doing used exchanges only for the consumer segment, and let us do all the ones for commercial/public.

One drawback to bringing the function home is that the tools we have to use for dispatching refurb systems are antiquated.  One of them is a Windows 3.1-era relic, and another is a home-brewed application for which all the source code has been lost, so what it actually does has become something of a black box, and modifying it to meet changing needs is out of the question.  IT refuses to have anything to do with any of it ’cos none of it is IT-approved, so support is also very hit-and-miss.  Still, it’s all the tools we have, so that’s what we’ve used, and more or less it worked.

Or it worked until yesterday.  Yesterday, suddenly one of the macros in the home-brewed tool started to hang midway through each dispatch.  Another mainframe-based tool went on strike completely, returning “Error in server SEND” messages to anything you tried to do with it.  My counterparts and I, who learned how to operate these tools back when The World Was New and All, managed to work around the dispatching issue by hand (but in a slower, much higher-touch way), and flat ignored the issues with the other app, which isn’t critical to getting systems out the door—it does some internal housekeeping to transfer warranty contracts and such between old and new systems.

The team in India didn’t do nearly so well.  They’d never had to do dispatches by hand—all they knew how to do was to run the macros, which were now broken.  So they couldn’t dispatch anything at all, and the backlog of matched but un-dispatched exchanges began to pile up alarmingly.  Eventually one of them, who has figured out that I’m something of a go-to fix-it guy when stuff goes wrong, IM’d me and asked whether I knew any way to get these backed-up dispatches dispatched.  I said I did, and ended up having a fast con-call-cum-shared-desktop tutorial on how to dispatch a used exchange by hand, at which point I was proclaimed the hero of the day.

However, now I was identified as someone who knew Which Way Went Up with these tools, and someone who could explain clearly what the issues were and what was broken about them from the UI standpoint, and I began getting yanked into these five-way chats with programmers in India, corporate IT dweebs in Brazil, and various odd bods in Circulith.  The chats dragged on and on as people tried this and that patch, and meantime I’m trying desperately to scrub service calls hourly for a special project, cover a different set of dispatches for another team member who’s out all this week, and kick out as many of the backlogged dispatches as I could manage to push.  Eventually someone figured out that some code changes that IT pushed Saturday in our mainframe environment was what had caused all of this, mostly because nobody in IT even knew the tools we use still existed, much less that a major function was depending on them.  At 5:00 (by which point I’m two hours into overtime), we had identified at least two changes that had to be reversed if our tools were to work at all, but no one was sure they could persuade IT the situation was dire enough to undo what they did; they’re all wrapped up in future rollouts and I wouldn’t be surprised to have them try to tell us “sucks to be you” at some point in this mess.

By 6:45, when I clocked out so I wouldn’t go over twelve hours and land in double-time territory, I had pushed out a double fistful of used exchanges, created and dispatched two dozen others of one kind and another, cleared out one queue, and scrubbed the project service calls.  I never did get to the new exchanges I was supposed to set up, and now I’m expecting a huge lump of escalations to hit my desk tomorrow, and I’ll have to drop four other things to take care of them.

I like working where I am at the Empire, but at times like this, with very limited backup for what I do and nobody else willing to cross-train on handling the escalations, it gets overwhelming.