How do you know?

It’s such a simple question. Yet, it’s one I wrestled to answer for way too long.

I want to share a story…

It’s the summer of 2009 and I’m a newly appointed general manager of a print manufacturing business. Our primary product line is chain of custody documents for the transportation and medical industries.

If you’re not familiar with chain of custody, think barcoded and numbered documents that are used for drug screening, and bill of lading forms.

Since these industries typically have many transactions, our typical order length hovered around 1 million + documents.

We made it part of our order process to split these large orders into more manageable batches.

The first batch was manufactured without incident.

The second batch was manufactured without incident.

The third batch was manufactured without incident… or so we thought.

I received a phone call from our customer stating there was a potential problem that was identified at the fulfillment center.

There was a duplicated set of numbers in one of the boxes that were opened and getting ready to be kitted.

When the call came in, the problem seemed unlikely.

I mean, we had processes and technology (systems) in place to prevent things like this from happening… yet, we never did any type of formal business systems testing.

Well, reality was about to hit this new general manager like a ton of bricks.

These were mission critical documents that cannot have duplicated numbers…

People could be denied employment.

People could be terminated.

People could receive the wrong shipment.

And a host of other potential problems I hadn’t even thought about yet!

So, I did what anybody in an operational leadership role would think to do.

I pulled all the documentation that we had on the batches in question. I gathered everyone in the organization that had touched the batches and we waded through every detail together.

We did a full 8D report and identified the root cause using the 5 Whys method.

At the end of all of that it was perfectly clear what happened…

Operator Error.

When the process of creating the number series was executed, the double check process was not executed. Worse yet, we identified that we didn’t actually have a double check process in place once the order and numbering information headed out onto the shop floor.

Armed with all of this information, I did what I typically did.

Pull everyone together and go through a bunch of retraining.

We walked through every aspect of the process to make the appropriate changes. We had follow-up meetings to talk through the process with anyone who touched or was adjacent to the process. We trained. We retrained. We had operators sign off on their training records.

When it was time to fill my boss in and find out what consequences I would face, before he could even ask, I took over the conversation and talked him through every single aspect of the investigation and laid out the preventative measures we had implemented.

I was confident that we nailed it and that we would NEVER see another problem like this in the future.

Without skipping a beat, my boss asked me the one question I was not prepared to answer…

“How do you know?”

I’ll never forget thinking, what do you mean how do I know, I just freaking told you how I know this will never happen again… we had meetings, we made process changes, we trained and retrained on processes, the team signed off on their training sheets.

WE WERE COVERED…

Then he asked again, “how do you know?”

This went back and forth for what seemed like an hour before it finally clicked and this question came out of my mouth: “Do you mean, I should intentionally try to introduce an error into the system and see if our process changes catch the defect?”

“Yes. That’s the only way you will know if your preventative measures work.”

Mind Blown. 🤯

The next day, we put the test into motion and my confidence level was high…

I just knew what we did had solved the problem.

Well…

IT DID NOT. WE FAILED.

Here are the 2 lessons I learned:

1. There is almost no such thing as an “Operator Error”. This was a System Failure in complete disguise. Taking a look across many examples, it’s easy to pass problems off as people problems or Human Failures but the reality is the underlying process and/or technology is more times than not the actual root cause of the problem.
2. Business systems testing is the one and only way to ensure the underlying processes and/or technology that you’ve deployed is doing what you “think” it’s doing. Don’t convince yourself you’ve done everything you can to prevent an issue from happening again through the typical training and retraining efforts, create or improve your system through real scenario testing.

That’s it for today.

See you all again next week!