How Do You Know Your Corrective Action Actually Worked?

You have had this moment. Something went wrong, you pulled the team together, you found the cause, you put a fix in place, and you walked away believing it was handled.That is where I was in 2009.

Prefer listening? Check out this week’s Solo Session where I go even deeper on the topic.

I was a brand new general manager at a print manufacturing business, and I had just finished cleaning up the worst quality problem of my career. I got my boss on the phone, ready to walk him through everything we had done and explain exactly why it would never happen again.

He asked me one question. Four words that I did not have an answer to.

The problem that started it

We ran long-run, multi-part, barcoded and numbered mission critical documents. In this case it was medical chain of custody paperwork for a laboratory client. The forms that follow a drug screen from collection all the way through to results. Serial numbers on the forms have to match the serial numbers on the specimen vials. When they do not match, the wrong person gets the wrong result, and somebody gets hired or fired on the back of it.

The order was over a million documents. We split it into three runs of about 400,000 each.

On the third run, the number series was duplicated. It got reused from an earlier batch.

We did not catch it. Our customer did.

What I did next is what almost everybody does

The investigation kicked off immediately. I brought everybody who touched the job into a room and we worked it the way you are trained to work it. Root cause analysis. We landed on operator error.

From there the playbook runs itself. Retrain everybody. Write new documentation. Build a double-check procedure. Get people to sign off on their training records so there is proof it happened.

We did all of it. Thoroughly. Then I got on the phone with my boss and walked him through every piece: what we found, who we talked to, what we changed, who signed what.

Then he asked me: “How do you know?”

I could still hear those four words today. At the time they did not even make sense to me. What do you mean, how do I know? I just spent twenty minutes explaining how I know. We did the investigation. We got everybody together. We found the root cause.

It felt like he was not listening. He asked it again. I got more frustrated. He asked it again.

What he was actually asking was: **did you test it?** Did you proactively confirm that the things you believe will prevent this from recurring will actually prevent it?

We introduced the failure on purpose

Once it finally landed, we did the only thing that answers the question. We deliberately recreated the error and sent it out to production to see whether any of our new processes would catch it.

Only two other people knew what we were doing. If I had told the floor what we were doing, everybody would have been watching for a phantom problem, and the whole thing becomes an open-book test, not a real one.

A week after all that retraining, all that documentation, all those signed training sheets, the job ran straight through. Nobody caught it. Which meant we would have had the exact same failure, with a real customer, on a real order.

The team was furious with me. They thought I had tricked them. We ended up holding a company meeting where I had to walk everybody through the whole story: the call with my boss, those four words, and why recreating the error on purpose was the only way to know anything.

That conversation did two things I did not expect.

It proved the root cause, definitively, instead of leaving it as the most plausible theory in the room. It also proved something bigger: the measures we had called “preventative” were not preventative at all, because every one of them depended on a person remembering to do something.

What we actually built

The real root cause was not the operator. It was the programming of the number sequence combined with the decision to split the order into batches. Nothing in the process tracked which series had already been used.

So we built software for the number programming that kept a record of every series that had been run. After that, producing a duplicate was not a matter of being careful. It was impossible.

Then we built a second piece that tracked the production run and checked it against the programmed series, because operators still had the physical ability to load the wrong series on their equipment. The first system removed the possibility at the source. The second one caught the human variable downstream.

Software took time to develop, so in the meantime we ran a new checklist and procedure at each production step. That was the bridge, and I want to be clear that it was a bridge and not the answer.

I also want to be careful about the word “software,” because that is the wrong takeaway. A system is Planning, People, Process, and Technology. Technology is the last pillar for a reason. We could not have built either of those tools without first understanding the process, the failure points, and the people running the equipment. The software was a big component of that particular fix because that is where the root cause landed. Your root cause may land somewhere else, and the fix will look like something else.

Then we tested the new system the same way we tested the old one. We tried to break it and it held.

That facility finished the following full year under a 0.1% error rate.

The same gap, in a completely different costume

Years later I heard about a company that got hit with an email phishing hack. Much larger operation, multiple entities across several states and several business sectors. I was not there for any of it, so this is second hand, but the shape of it is unmistakable.

Somebody clicked a link in an infected email. The attacker moved through the network undetected and started taking down the corporate ERP environment and Microsoft tenants.

Early on, the mood was not panic but inconvenience. Big problem, with lots of cleanup, but they had backups. They would be down a few days, restore, and get back to it.

Then they got to recovery. Tape backups in, hit restore.

Nobody had ever confirmed that the data inside those backup records was intact and functional. The backup job ran, got logged, and box got checked. What never happened was somebody actually creating a restoration point and verifying it came back.

The backups did not work. They lost roughly a month of ERP data, and they could not restore their Microsoft tenants correctly either.

Think about what that second part means for a mid-market company. All those inboxes, all those connection points between systems, built up gradually over years. Backing up is one button. Rebuilding all of it from scratch, from memory and conversation, is a different category of undertaking entirely.

Same missing step as my print floor in 2009. Different decade, different industry, different scale.

A closed corrective action that passed an external audit

Here is the part that should bother you, because this is not a small-company problem or a discipline problem. It’s a nearly everyone problem.

ISO-certified companies that have passed their audits with flying colors have had this problem. Recently we took a specialty manufacturers already-closed corrective actions and run them back through the root cause analysis engine in our Operations Workbench, so they could see how their conclusion compared to a structured investigation.

An engineer had set up a part in their ERP with the wrong base material selected from a dropdown. It affected a part on an open order and required rework to correct. The team’s investigation identified the cause as manual entry, and the corrective action closed.

Reasonable. Manual entry is genuinely where the error happened.

Run the same problem through a structured Five Whys and you get to a different place. Why was there nothing in the setup process to catch the wrong selection? Because users transcribe from customer emails and pick from a dropdown. Why is there no verification step to confirm the selection matches what the customer specified before it saves?

The answer, in writing: because the impact is not large enough to warrant a double check on every entry.

That is not a root cause. That is a justification. Nobody had measured the impact. The risk tolerance was never a documented decision anybody owned, it was an informal assumption that had never been examined and never been tested against actual rework data.

Notice where stopping at “manual entry” leads. It leads to a corrective action of retrain the users, which would not prevent a recurrence.

Seventeen years after my boss asked me those four words, on a completely unrelated process, in a different industry, inside a company with a certified quality system and a clean audit. Same answer = Retrain everybody.

Every framework already has this step

This is the part that gets me, because none of this is new information.

Look at the 8D report structure. Team, define, contain, root cause, corrective action, validate, preventative measures, congratulate. Validation sits right there in the middle, immediately after the corrective action.

Look at the Process Playbook approach I teach: prepare, document, train, audit, then process improvement. The audit step exists specifically to compare what is written against what is actually happening on the floor.

Look at P.E.R.R.R.: Plan, Execute, Review, Revise, Repeat. You cannot revise anything intelligently without the review.

Three different frameworks, built by different people for different purposes, and every one of them contains a verification step. Nobody skips it out of ignorance or even laziness. It gets cut because it comes last, it costs time you already feel like you overspent, and it has the emotional texture of doubting your own work in front of your team.

So we phone it in. We check the box. Yes, we did the retraining. Yes, backups are running. Yes, there was a problem with that particular operator and we handled it.

That is why problems repeat. Not because the frameworks are wrong but because the one step that closes the loop is the one we treat as optional.

The math nobody runs

The objection I hear most is time. We do not have time to dig into root cause, bring people together, and proactively test.

That 2009 error cost us north of $60,000. The test that proved the root cause and exposed our useless countermeasures cost less than a thousand dollars.

That ratio will not hold everywhere, and I am not going to pretend it will. But in my experience tests are usually a fraction of the cost of the problem they are checking for. We brush them off as expensive and inconvenient, and then we always find the time and the money to rework or remake the jobs that make it to the customer.

You have the time. You are not valuing the effort.

Go back to that specialty manufacturer for a second. The reason there was no verification step was a belief that the impact was too small to justify one, and nobody had ever put a number to the impact. That is the same trap from the other direction. We decline to measure the cost, then use our own ignorance of the cost as the reason not to check.

Where verification actually fits

Clarity, then consistency, then accountability. That sequence is the foundation under everything I teach, and verification is where most operations quietly lose the third one.

Most leaders think accountability means confirming the person did the task. Confirming the task happened is administration. Confirming it produced the outcome you needed is management. Without that second half, an audit trail is just an attendance record with better formatting.

This also crosses departmental lines, which is why it hides so well. Quality owns the corrective action. IT owns the backup. HR owns the training sign-off. Each one is locally efficient and individually defensible, and the gap between them belongs to nobody.

What to do Monday

Crawl, walk, run.

Crawl. Pick one high-visibility process, ideally one where a failure reaches a customer. Do not just trace it end to end. Ask specifically what would happen if the failure you are worried about were introduced today, and whether anything in the process would actually stop it. If you have an NCR or CAPA process, start there. Pull one closed corrective action, look at the preventative measures, and ask honestly whether they depend on a person remembering something. If they do, you have a checklist, not a control.

Walk. Attach a verification step to every corrective action going forward. A named owner, a date, and a success threshold defined as an outcome rather than an activity. “Zero repeat occurrences over the next sixty days” is a threshold. “Training completed” is not.

Run. Schedule the failure on purpose. This is the least comfortable step and the only one that actually answers the question. In IT this is already normal practice: there are firms whose entire business is sending controlled phishing emails to your own employees and scoring how many people click. Nobody in operations schedules the equivalent test on a process.

Remember this stuff doesn’t always require technology either. We worked with a pet food manufacturer on an intermittent product quality issue and identified the root cause at the end of their manufacturing process. If they had wanted to prove it, the test would have been asking one of their suppliers to run a sample batch deliberately out of spec to see whether the problem reproduced downstream. No software. Just a phone call that most people would rather not make, because it means asking somebody else to make bad product on purpose.

That discomfort is the actual barrier, not budget.

Final Thoughts

It is very easy to sell yourself on having done something. You held the meeting. You wrote the procedure. You created the backup. You closed the corrective action.

My contention is not that operations leaders are lazy about this. It is that most of us have never understood the real value of testing, and most of us have never had somebody ask us the question that forces it.

I got lucky. Somebody asked me, over and over, until I got frustrated enough to actually hear it.

Ask it about one thing this week. The process you would bet the company on. The backup you assume is fine. The corrective action you closed last quarter.

How do you know?

That’s it for today.

See you all again next week!

Dave