During on-site assessments for our clients, we often ask the question: “How do you know if your operational processes remain valid and effective without scheduled assessments of process qualification from time to time?” Allowing underperforming or redundant process operations to fall below the radar without periodic reassessment could result in significant losses—not only in productivity through the misallocation of resources, but also in wasteful and needless spending.
We typically recommend using Design of Experiments (DOE) for process validation to yield effective results in our clients’ experimentation and research. DOEs have been proven boost operational gains significantly in engineering quality and the ability to innovate efficiently. The following example scenario describes how we had uncovered insights about a client’s longstanding process operation by performing a quick and effective analysis using a particular technique of DOE, the paired comparison method.
The Client’s Scenario
During a past, on-site assessment for a manufacturing client in the automotive industry, I was able to observe a labor-intensive, quality assurance (QA) operation, the cost of which was 2% of the company’s revenues annually, which translates to about $1.5 million USD. This seemed to be a rather large amount to spend on merely one aspect of QA, so I was curious to find out what analyses and methods the client was using to its evaluation criteria and ensure effectiveness.
The on-site personnel gave me a brief overview of the procedures involved in the process operation. The purpose was to resolve defects on product parts that occurred after the injection moulding operation. The post-operation defects were bumps and blisters on certain areas of automobile fenders, quarter panels, and bumper covers. These defects were inconsistent with the original equipment manufacturer’s goal of delivering showroom quality to their parts.
To improve the final quality of the automotive parts, the defective areas were treated with a labor-intensive sanding and buffing operation after the injection moulding operation and before the parts were primed and cured. Not only did the operational spend seem vast on this quality assurance operation, but also the significant number of resources who were assigned to perform the operation, which was performed over 3 work shifts with 6 to 12 operators per shift.
I learned that the client was performing the longstanding operation without any recent experiments or studies to validate its usefulness. It is not unusual to find that companies perform operations without recurring studies to validate effectiveness.
The Opportunity for Quick Experimentation
I offered to do a quick, paired-comparison type of design of experiments (DOE) to show the client how effective a basic analysis can be, even with limited set of sample data. These types of studies are simple to do, easy to understand, and very powerful. Paired-comparison methods are powerful, largely because they eliminate virtually all noise factors such as variation in the materials, tool conditions, operator habits and other process conditions.
For the sample data set used in the paired comparison DOE, we chose several automotive parts that had defects, counted the number of defects, and indicated the defect positions on a diagram, similar to the one displayed below.
Closeup detail of defects in areas with and without treatment on an automotive part
Subsequently, I drew a line on the diagram that separated roughly the same number of defects on either side. A coin toss determined which side would receive the treatment (more specifically, sanding and buffing). The part would then be sent to undergo priming and curing operations and in turn undergo inspection once again.
There were two dimensions to the paired comparison. The first dimension was the before-and-after effect. Would the priming operation cause more blisters to appear (which is what the engineers had ) or could it actually reduce the number of blisters (which none of the engineers had expected)? The second dimension was to consider whether the treatment, i.e. the sanding and buffing operation, reduced or eliminated the blisters after priming.
The Revealing Results of the Analysis
Following the above-mentioned test operations, the analysis demonstrated that there was no difference in the result of the side of the part that had undergone treatment in comparison to the side that had received no treatment. Surprisingly, the priming operation had actually reduced the number of blisters in about half of the number of runs in the DOE. Had the QA procedures yielded improvement, it would have looked something like the left plot of the below comparison diagram.
Comparison results of the experiment using the paired comparison method
In fact, the actual results looked more like the chart on the right side of the comparison diagram (note the solid, red line near zero on the Y-axis). Whether the part was treated or not, the number of blisters after the priming operation was about the same on average. Examining the parts showed that some blisters had disappeared and others appeared, regardless of treatment or lack thereof.
After reviewing the results of the analysis, I shared them with the client’s engineers. I let them know that not only was the operation in question unnecessary, but also the analysis highlighted the evident need for the client to reassess its process characterization on a regular basis.
Some of the client’s engineers had suspected for some time that the quality improvement effort was a waste of time and money, but they didn’t know how to conduct tests to justify their thinking. Others correctly surmised that although the results might prove the point, they were only applicable to this particular automotive part examined in the given experiment. The engineers undertook similar paired comparison DOEs on other automotive parts, which ended up leading to the same conclusion.
Process Qualification Reassessment as a Best Practice
As a best practice of continuous improvement, we recommend that clients adopt the DOE methodology to perform periodic reassessments to validate their process characterization for higher productivity and quality. DOEs should also be conducted in the following scenarios:
- If you suspect that there may be changes in supplied materials or tool and equipment operation
- If you have recurring quality problems in products or processes
- If you are introducing a new process or changes to existing operations
If you’re interested in the topic of the paired comparison method and would like to learn how to apply it in evaluating your operational processes, please get in touch with me to start a conversation. For in-depth learning about DOE and paired comparison analysis, we offer a series of Design of Experiments training courses to visualize and explore practical use cases of DOE in greater detail.
