Dev error bricked 600 PCs, caused $1M CD-ROM write-off

One lucky engineer scored a free dinner during the frantic fix, the rest got to explain their mistake

Welcome to Monday, the day upon which The Register offers a new instalment of "Who, Me?" – our reader-contributed column that shares your stories of messing up, 'fessing up, and somehow finding redress.

This week, meet a reader we'll Regomize as "Oriel" who told us he once worked for a vendor of small business accounting software.

This story comes from a time when software shipped on CD-ROMs and users bought it off the shelf in actual shops. The vendor released a new version each year and always shipped it before a major holiday that's celebrated with a giant meal.

REG AD

Oriel was part of the team whose job it was to get the package ready on time.

REG AD

In the previous year, they flubbed the job because the program didn't always install itself on some PCs running Windows 95 or Windows ME machines system due to registry corruption.

"We wanted to increase our successful installation metrics, so in conjunction with the installation team, a key decision was made – by me – to force the updating of registry keys to enable the install," Oriel admitted. In his email to The Register, he asked the portentous question: "What could possibly go wrong?"

What went wrong was that not long after the software was shipped, the vendor's support lines started getting calls from customers who complained that after installing the accounting package none of their other software would work.

"Installing a new version of our application had borked the customer's machine completely," Oriel wrote.

The company decided the best thing to do was to tell affected customers they should go and buy a new PC – and that it would pick up the bill, no questions asked.

"In the week prior to the holiday, we replaced over 600 machines," Oriel told The Register.

Oriel and his developer colleagues were of course told to fix the problem but couldn't reproduce it nor figure out their mistake.

"To get a reproduction case we need a user's machine where the user had created a Windows Registry restore point prior to installing our application," Oriel explained. If the developers could find such a machine, the plan was to roll it back to the restore point, verify everything was in order before the customer installed the update, install the accounting software, and figure out the problem.

REG AD

The company tasked its support agents with finding a customer willing to surrender their PC, which proved difficult because few bothered to create a registry restore point.

The day before the holiday, a customer volunteered their machine.

This customer was located a few hours away by road, so an engineer got the job of visiting on the morning of the holiday to pick up the stricken machine, give the customer a shiny new system and configure it to their specifications.

Said engineer did the job, which took so long he was invited to stay for the holiday meal, then returned to HQ with the broken box.

The next day, Oriel and his colleagues got to work and found a fix, then created a version of the accounting application that wouldn't break any more PCs.

Oriel was, of course, summoned into a meeting with senior execs to explain the situation and watched as they decided the best course of action was to pull buggy CDs from stores – which meant trashing CDs and packaging valued at $1 million.

Have you made a mistake that broke customers' kit, or cost millions? If so, don't make the mistake of not sharing your story with The Register and click here to email the column. ®