Confessions of a sysadmin: why honesty is the best policy

The year, 1996. I was working for a large multinational consultancy and had just been assigned to a project to maintain and develop financial reporting systems using Arbor Essbase (later taken over by Hyperion). This was an early database tool that took the base level dimensional data and summed it up into a “hypercube”.

The system we used was a “monster”. It boasted dual Pentium Pro processors and a massive 96MB of memory!

The problem with Essbase at that time was the re-calculation of the hypercube. To calculate a “fresh” cube, with just bottom-row data on each dimension, would take around four hours. If you recalculated a full cube, it would take between 12 and 18 hours.

So, the standard procedure was to take a dump of the bottom row data, erase the cube, reload the bottom row data and recalculate. The export and import took about ten minutes, but, compared to the recalculation time, we were still saving a minimum of eight hours per recalculation – which meant the cube was recalculated twice a day.

It was our job to dump the bottom row data, erase the cube, reload the data and calculate the cube during month-end reporting periods. The first few days went okay, but can you guess what I then did? Yep, I went straight to step two! Blam, cube deleted!

Luckily for me, we kept the last four exports on the system, so I could restore to how the cube was four hours earlier. Still, all the changes that users had made to the data in the past four hours were gone. Alternatively, Essbase had a transaction protocol.

I was a consultant by this time and there was a junior operator working on the project. I was supposed to be supporting him with the admin side of the contract, as well as the analysis and development of new cubes. So, as he had been there for two years, I asked him: “What do we do when we forget to do an export?” His answer: “Nothing, we just load the previous export and say it was user error.”

Confess or hide?

Instead, I went to the head of finance, told him that I had accidentally deleted the cube before doing the export. I explained that we could use the transaction log to rebuild most of the changes in the past four hours, that it would take an extra hour or so on top of the usual recalculation time, but the users wouldn’t have to re-enter all the data from the previous four hours. Or we could rebuild from the last export, the previous afternoon.

We went with the transaction log option and we lost a total of four changes, according to one user. Everybody else had all of their changes in the system.

I had been dreading confronting the Finance Director with my error, but, being new to the project, it actually got me in his “good books”. He trusted me. After all, everybody makes mistakes. It’s how we deal with mistakes that makes us who we are; somebody to be trusted or somebody to be avoided.

Moving forward, when I told him something wasn’t a good idea, or that if something would take longer than he expected, he believed me. I had been worried about losing my job when I admitted my error, but it ended up making the working environment there better and more rewarding in the long term.

Risky Business

Another time, I was preparing a presentation for a client. It was a detailed analysis of their current systems and areas where the system could be improved going forward.

My presentation was brutally honest about the problems I had found. My boss caught sight of some slides shortly before and nearly had a fit, saying I couldn’t tell them the truth, that we would lose the contract. I argued that I was pointing out weaknesses in their other systems, not ones we had written. Plus, it was too late: I was due to give the presentation in an hour.

I gave the presentation and it’s true that the facts presented by the slides were damning. But I was friendly and light-hearted, to the point of being funny (in my mind at least). Plus, I defused some of the problems shown on screen by detailing how they could be easily remedied. Not through replacing systems or putting in a lot of consultancy work, just changes in workflows to improve them and make them more secure.

At first, the room was cold, but as the presentation went on and they listened, they started to come over to my way of thinking. In the end, we won the extension to the contract and the implementation of a new budget planning system.

The customer actually praised me for my honesty and, after a few hours, my boss started breathing again!

Lies will come back to bite you

A friend helped me get a job at a company that manufactured and repaired industrial touchscreen terminals. He was head of IT and production. On my third day there, we were in a management meeting about the current state of play. It soon became clear there was one customer who was having problems.

The customer had sent back two terminals for repair and the CEO asked my friend what the status was, as the customer had called him. It was taking a while to get the terminals repaired and if another terminal broke, production would stop at the customer site.

My friend told the CEO that everything was under control and that the terminals would be repaired and sent out that day, that the work was already under way. The CEO punched the number of the production into the conference telephone on the table and asked the production, how far along the work was on the terminals…

My friend hadn’t actually informed them that it was a priority and they needed to go out that day. He was trying to cover himself and would have gone direct to the production after the meeting and got them to express repair the terminals. He was fired on the spot!