Many companies spend considerable amounts of money on customer and employee surveys every year. The survey results are used to amend strategies, design new products and services, focus improvement activities, target staff development activities and … to celebrate success.
The question is: Can we always rely on what we see?…
At MyInsurance, survey results have been collected in 2017, 2018 and 2019. The rating was done on a 10-point Likert ranging from 1 … very poor to 10 … outstanding. As always, the upper 3 points, i.e. 8, 9 and 10, are seen as customer is satisfied. All other ratings are undesirable. How did MyInsurance do over the years?…
Lean Six Sigma is linked to some myth that are unjustified and merely a result of half-knowledge or wrong assumptions. Therefore, let’s try to demystifying lean six sigma:
MYTH: LEAN SIX SIGMA IS THE LEAN, THE SLIM VERSION OF SIX SIGMA.
The truth is: Lean Six Sigma is the advanced version of Six Sigma. It combines two powerful toolboxes, Lean and Six Sigma.
Lean has been developed by Toyota starting after WWII under the name of Toyota Production System which has been contributing as a major factor to Toyota’s world-wide success in making the most reliable cars available for an affordable price.
Six Sigma was a major means during and the result of Motorola’s successful fight for survival in their bleeding TV business during the mid-eighties.
Whereas Lean focusses on waste reduction in all kind of processes by cutting out unnecessary activities, Six Sigma helps to reduce variation and increase predictability in the steps that are really necessary.
MYTH: LEAN SIX SIGMA HAS MANY TOOLS I DO NOT NEED.
Certainly, life is much easier without the need to choose between 50 advanced tools for problem solving. This is correct. However, if the only tool available is a hammer, everything looks like a nail. In reality, there is a multitude of process problems whose solution needs the appropriate application of a variety of tools. Not knowing these tools and concluding that they are not needed is called ignorance. Knowing these tools and their application and selecting those that add value and dropping those that are not needed in this situation is called making an educated decision.
MYTH: LEAN SIX SIGMA IS FOR MANUFACTURING ONLY.
The truth is there are applications in all kind of organisations in all industries. Yes, the methodology has been developed for the manufacturing environment. Yet, it has been used in basically every process landscape you can think of.
MYTH: LEAN SIX SIGMA IS FOR ENGINEERS, FOR SCIENTISTS.
To the contrary, tools for listening to and understanding customers, for collecting and analysing data about problems and their root causes, for making fact-based decisions about improvements, for managing and motivating project teams and for delivering results whilst developing staff at the same time should be part of every managers tool box.
LEAN SIX SIGMA IS AN OUT-DATED APPROACH.
The fact that this set of tools has been around for decades does not mean it is out-dated. It rather means it is much more than the flavour of the month. There have been a lot of developments around this toolbox. It has been customised for R&D activities, for sales processes, for administration functions and for many other environments. After having mastered the nuts and bolts of the approach, Organisations have tailored the selection of tools for their needs. With all these developments the methodology has been kept current to meet changing needs.
“We have great news for you. Our project is delivering results already.” The team is all smiles when they give this update during the project meeting. The carefully prepared graphs unveil a remarkably shorter time for the whole process, from customer request to delivery of results. “We have applied a hypothesis test and the result is significant with a p-value of flat zero!” They sound like they know what they are doing. When asked for the change in the process, they all give different answers. When asked what the root cause to have achieved this effect was, their smiles fade. “We actually only implemented some Quick Hits. They turned out to have a greater effect than we thought. Isn’t this a nice surprise?” They asked. Or, is it just a Hawthorne?
If the going is real easy, beware, you may be headed downhill. Anonymous
When addressing process issues, there are different constellations to describe the relationship between improvement solutions developed and the outcome:
Different Solutions and Corresponding Results
Effective Solutions that show Excellent Results. This is the dream of every project leader. This does not only refer to success. It also leads to the assumption that the solution has addressed the root causes for the problem and in turn, generated the expected result.
Seemingly Effective Solutions that don’t show Expected Results. There is a multitude of potential reasons for this. It always means going back to the drawing board and searching for the gap, be it initial wrong assumptions, biased data, insufficient analysis or solutions implemented the wrong way.
Toothless Solutions that don’t show Results. This certainly does not come as a surprise.
Toothless Solutions that deliver Excellent Results. This constellation does sound unlikely. However, it is not. Sometimes, even without implementing any solution, the process improves and results get better – often to a remarkable degree.
This can be attributed to the Hawthorne Effect. Hawthorne says that observing a situation will change the same. It will likely appear in all environments where humans are “watched”.
In our experience, it is even worse. Talking about a process is enough to trigger Hawthorne results because people running the process get more aware, focus more on their work and deliver better results. In some situations, only then they do what the SOP says. The Hawthorne Effect and results go away as soon as people don’t feel “watched” any longer and the attention is directed towards other things. In this sense, Hawthorne results are not sustainable.
Hence, the apparent disadvantage here is that we cannot distinguish between real project results due to changes in the process and those due to the Hawthorne effect. It becomes an extra variable in a situation where only one variable should exist.
The only way of making sure that improvement effects will last, is to establish a logically consistent chain from description of the problem via potential root causes, data collection and analysis to identification of the vital few tackled by related solutions. Lean Six Sigma is what these steps are. Follow them carefully and you will be successful – beyond Hawthorne.
Many companies spend considerable amounts of money on customer surveys every year. Customer survey results are being used to amend strategies, design new products and services, focus improvement activities and … to celebrate success. Because the impact of customer service results can be quite hefty the data driving important decisions shall be trust-worthy. And, the question is: Can we always rely on what we see?
A life insurance company – let us call them MyInsurance – with world-wide market reach was celebrating their success of improving their customers’ satisfaction in 2006. They proudly presented the results: “In Thailand we have achieved 58% satisfied customers as compared to year 2005 when it was only 54%.” This sounds good, right? In a market with millions of consumers, an increase in satisfaction of 4% would mean, the number of customers who would happily buy from MyInsurance again has increased by some ten thousands.
Such kind of conclusion could be too fast. Why? For obvious reasons, MyInsurance did not really ask millions of customers for their opinion. They only managed to gather the opinion from 280 customers. And, this is called sampling. Such approach is in use in every kind of company in every industry many times a day.
Sampling is based on a comparatively small number of customers, called “Sample”, and it is the foundation to draw conclusions about the “Population”. Population in this case refers to the entire pool of customers whose opinion we want to gather. Sampling has a huge advantage: it saves money and time and is especially useful when it is nearly impossible to collect data from the whole population or when the process of testing can destroy the object like drop testing of mobile phones. This advantage comes with a disadvantage: the “Margin of Error” or “Confidence Interval”.
Confidence Interval is the range in which we expect the population value to be. Since we do sampling, we can only guess what the “real” value is. In sampling, we never know. This Confidence Interval cannot be avoided, even with a perfectly representative sample under “ideal conditions”. However, this Interval can be reduced by increasing the sample size and by decreasing the variance in the population. The latter is usually not possible. Hence the only choice one has is to determine the minimum sample size for the Confidence Interval one is expecting.
Sampling M&Ms
A very simple experiment will help you understanding what sampling means:
First of all, buy one 200g package of chocolate M&Ms. Then, open your package and count the number of M&Ms in your package. This number – the population – in my case was 233.Confidence Interval Is Close to the Truth
Sampling means taking a small number of M&Ms out of the population in a representative way. Hence, I took a bowl and filled in my M&Ms. After some shaking and stirring, I turned around and counted a sample of 20 M&Ms out of my bowl – blindly. The first sample gave me no yellow at all. After that, I put my sample back into the population and counted a new sample with 20. Then, second sample revealed 4 yellow M&Ms. Eight more samples gave me 2, 3, 3, 6, 3, 5, 4 and 3 yellow M&Ms, respectively.
The Result
Finally, doing the math, my samples suggested that my population has 0%, 20%, 10%, 15%, 15%, 30%, 15%, 25% and 15% yellow, respectively. Which sample is correct?
Now, count the number of yellow M&Ms. In my experiment, this number was 43. It means I have got 18.5% yellow in my population. None of the samples is correct. All of the samples give only an indication for the real percentage of yellow in the population.
Consequently, we may conclude that sampling results vary even though the population is untouched. Drawing conclusions based on this variation may lead to expensive mistakes.
What does this mean in case of MyInsurance? With some simple statistics we can calculate the Confidence Interval for our samples based on the sample size we have got:
In 2005, the “real” customer satisfaction level was between 48% and 60%. In 2006, it was between 52% and 64%. So, can we still conclude that we have improved? We cannot!
If MyInsurance wishes to distinguish between a customer satisfaction level of 54% and 58%, they need to have confidence intervals for 54% and 58% that do not overlap. If they would overlap, we cannot distinguish between both. Hence, we need confidence intervals of +/- 2% or less for both.
The estimation of the sample size for this requirement tells us that we would need to involve nearly 2,500 customers in our satisfaction survey each year. Again, based on the sample of 280 customers we have taken it can easily be that there has been no change at all or even worse a decrease in customer satisfaction. We will never know until we have more data to give us a better result.
Unfortunately, in our example MyInsurance has no reason to celebrate success due to increase in customer satisfaction. This assumption could be totally wrong.
Very often important decisions are based on means coming from small samples of data. Sometimes these small samples of data are poorly collected or have a large variation. Unfortunately, we usually do not care a lot about variation in our daily professional or personal life. What matters most is the average, the mean. Because this mean is easy to calculate and everyone understands what it stands for. However, every mean coming out of a sample is only correct for a sample, it is “wrong” for the whole we are trying to make a decision about.
Hence, management would take a great leap in decision making by changing the way they look at data: Don’t trust the yield you have got for your production line, ask for the confidence interval for that. Don’t make an investment decision based on a small sample of data, ask for the minimum improvement this investment will give you.
Don’t trust means, they are lies.
Just some weeks ago, I filed my tax in Singapore. It took me about twelve minutes at my computer at home on a Sunday afternoon in April. It was not straight forward, I needed to make some amendments and additional inputs to what IRAS had already prepared for me. Yet, it was really easy to understand, very effortless to do and I have the strong feeling I did not make a mistake. Twelve minutes. Really.
Chatting with a Singaporean a while ago, we touched the taxation system in Germany. And, I have to admit that I may not know exactly how it works nowadays, because I have been out of Germany for many years. However, from what I remember about my motherland, I can easily conclude that nothing changes very fast. And the changes are not always for the better for the people. They are often designed for political window-dressing as a result of some half-hearted promises given before election.
Innovating Processes Every day
When I explained to my Singapore friend, that we have tax consultants in Germany who do not belong to the government, he did not understand immediately. “What do they do?” he wanted to know. I explained that the German tax regulations are so complex that no normal taxpayer understands them. Hence, when I was living in Germany, I paid a tax consultant every year to help me prepare filing my tax. “Isn’t it the job of the government to ensure people understand their regulations?” I told him that we have only 103,000 people working for the Ministry of Finance. No way.
“Can you not learn the tax regulations over time? After all, you file every year”, was his question. Yes, you file every year. But it is very likely they change the regulations just in the moment you thought you got it. They seriously do. The system in Germany is so complicated because every Müller, Meyer, Schulze – in Germany we don’t have Tom, Dick and Harry – gets his special tax exemptions or regulations. You have a strong lobby, you can save tax. This is a benefit of democracy. Or, democracy gone wrong, depending on who you are.
My friend further thought, that I might be an exception because of my job. “How many people need a tax consultant, really?” I told him that we have about 86,000 tax consultants who make a decent income from that job every year. I think it is a valid assumption, that many, many people need this kind of support. If the finance authorities would do their job, all these tax consultants would be out of that job – and could do something really value-added.
However, the chance that you catch German tax staff doing some process innovation, must be quite slim.
IRAS has been reinventing tax collection for many years. My friends tell me about the old days, when you could drive by IRAS and drop the tax returns from your car on your way to work. Now, tax collection in Singapore is a breeze. The best part is, IRAS is not leaning back and taking a well-deserved rest after all the accomplishments. No. They work very, very hard to get even better.
In my opinion, IRAS is one of the showcase institutions for continuous improvement and innovation to the benefit of tax payers in Singapore. And, highly effective processes are usually efficient, too. I.e., management and staff have been optimising their internal processes to be able to support the well-known excellent service delivery with minimal effort.
However, these results did not come over night. Over many years, they have been building a strong foundation of continuous change and innovation. They have a core of facilitators who have the capability to support this change and managers with the strong will to drive it – and to walk the talk. Their innovation initiative is supported by many small interventions, events and systems that all together form a homogeneous message: Get better every day.
Every blood donor of a large blood bank has to go through five process steps. These steps are Registration, Screening, HB Test, Donation and Refreshment. At the end of the process, that often takes around an hour, feedback forms are available for the donors. In one week, 210 donors have returned these forms with their satisfaction score for each process step. This satisfaction score is measured using a six-point scale. The desired rating is either 5 (satisfied) or 6 (very satisfied). The results are summarised in Figure 1.
Figure 1: Data for Chi Squared Test
From the results obtained, there is an obvious majority of “Not 5 or 6” ratings for Step 5. The question is, is there a significant difference to the other steps’ satisfaction score or are we observing just random variation. How can we test this?
One way is to use the so-called Chi-Squared test (also written Χ2 test, another method is Fisher’s Exact test). The Chi-Squared test is a statistical tool to check whether there is a significant difference between observed frequencies (discrete data) and expected frequencies for two or more groups.
To perform the Chi-Squared test, the following steps are necessary:
Figure 2: Column Chart for Chi-Squared Test Data
For any statistical application, it is essential to combine it with a graphical representation of the data. The selection of tools for this purpose is limited. They include pie chart, column chart and bar chart.
The column chart in Figure 2 reveals, that the customer satisfaction for Step 5 seems drastically lower (more “Not 5 or 6” ratings) compared to the other steps. Although, there does not seem to be any doubt about a worse rating for Step 5, it is a good practice to confirm the finding with a statistical test. Additionally, a statistical test can help to calculate the risk for this decision.
In this case, the parameter of interest is an average, i.e. the null-hypothesis is
H0: P1 = P2 = P3 = P4 = P5, with all P being the population proportions of the five process steps in this blood donation process.
This means, the alternative hypothesis is
HA: At least one P is different to at least one other P.
Since there is no reason for changing the commonly used acceptable risk of 5%, i.e. 0.05, we use this risk as our threshold for making our decision.
If there is a need for comparing two or more proportions, the popular test for this situation is the Chi-Squared test.
Finally, there is only one prerequisite for the Chi-Squared test to work properly: All expected values need to be at least 5. After the test, we will know whether this assumption holds true.
Figure 3: Chi-Squared Test Results
Using the Chi-Squared test, statistics software SigmaXL generates the output in Figure 3.
Since the p-value is 0.0000, i.e. less than 0.05 (or 5 percent), we reject H0 and accept HA. The risk for this decision is zero.
From the test result at Figure 3 we can conclude, that all expected counts are larger than five. This means, the test result is valid.
With this, the Chi-Squared test statistics means that there is at least one significant difference.
In order to interpret the Chi-Squared test results, three steps are necessary:
With this result, it is quite obvious, that interventions are needed to increase the customer satisfaction with Step 5.
Interested in the stats? Read here.
Three teams compete in our CHL business simulation, CHL Red, CHL Green and CHL Blue. After completing Day One, it looks like the teams show a very different performance (Figure 1). Although the means look very similar, the variation is strikingly different. This is surprising, since all teams start with exactly the same prerequisites. To test this assumption of different variability among the teams, the Test for Equal Variances is deployed.
Figure 1: Data of CHL Blue, CHL Green and CHL Red in a Time Series Plot
Finding a significant difference in variances (square of standard deviation), in the variation of different data sets is important. These data sets could stem from different teams performing the same job. If they show a different variation, it usually means, that different procedures are used to perform the same job. Looking into this may offer opportunities for improvement bu learning from the best. However, this only makes sense if this difference is proven, i.e. statistically significant.
To perform the Test for Equal Variances, we take the following steps:
Figure 2: Box Plot for CHL Blue, CHL Green and CHL Red
For any statistical application, it is essential to combine it with a graphical representation of the data. Several tools are available for this purpose. They include the popular stratified histogram, dotplot or boxplot. The Time Series Plot at Figure 1 does not clearly show the variability of the three teams.
The boxplot in Figure 2 shows much better an obvious difference in the variation between the three groups. A statistical test can help to calculate the risk for this decision.
In this case, the parameter of interest is an average, i.e. the null-hypothesis is
H0: σBlue = σGreen = σRed,
with all σ being the population standard deviation of the three different teams.
This means, the alternative hypothesis is
HA: At least one σ is different to at least one other σ.
Since there is no reason for changing the commonly used acceptable risk of 5%, i.e. 0.05, we use this risk as our threshold for making our decision.
Figure 3: Descriptive Statistics for CHL Blue, CHL Green and CHL Red
If there is a need for comparing variances, there are at least three popular tests available:
Since test that are based on a certain distribution are usually sharper, we need to check whether we have normal data. Figure three reveals, that CHL Blue does not show normality following the p-value of the Anderson-Darling Normality Test. Therefore, we need to run a Test for Equal Variances following Levene’s test
Finally, there are no other prerequisites for running a Levene’s test.
Figure 4: Levene’s Test for Equal Variances
Running the Test For Equal Variances, using the statistics software SigmaXL generates the output in Figure 4.
Since the p-value for Levene’s Test Statistic is 0.0000, i.e. less than 0.05 (or 5 percent), we reject H0 and accept HA.
With this, the Levene’s statistics means that there is at least one significant difference.
Additionally, this statistics shows which CHL is different from which other CHL. The p-value for Levene Pairwise Probabilities is 0.0000 between CHL Blue and CHL Green, as well as between CHL Green and CHL Red, i.e. there is a significant difference between CHL Blue and CHL Green as well as between CHL Green and CHL Red. The boxplot shows the direction of this difference.
Finally, the statistics informs that CHL Green seems to have a significantly better way to run the simulation with much less variation, i.e. StDev of 1.09min compared to 3.92 and 4.17, respectively. After further looking into the procedure, we recognise that CHL Green organises packages in First-in-First-out (FIFO) order, whereas CHL Blue and CHL Red do not ensure FIFO.
Interested in the stats? Read here.
Nearly all medium-size and large companies spend hundreds of thousands or even millions on customer surveys every year. Customer survey results serve to amend strategies, design new products and services and focus improvement activities. Gathering customer survey data is only the first step. The second step involves making best use of the expensive data, analysing them, drawing business relevant conclusions and making important decisions. How are we doing in this step?
A Home Appliances manufacturing company called upon their staff in the Customer Care Department to analyse new satisfaction data and to suggest actions to the management team. All satisfaction data have been gathered using a four-point Likert scale for satisfaction and “Buy Again” and a five-point Likert scale for “Recommend”, i.e. Net-Promoter-Score.
Some conclusions are obvious immediately:
Calculating confidence intervals for all results has proven significance of all changes, i.e. it makes sense to conduct more detailed analyses to find the culprit for the drop in Delivery Quality. Looking into the four major drivers for Delivery Quality revealed that Cleanliness and Punctuality leave room for action.
The question is: Does improving Cleanliness and Punctuality really drive the ultimate goal, i.e. repetitive business from existing customers and them recommending the company to their friends? The data may suggest it. However, are we really sure about this?
Kano Analysis is a tool – often mentioned in Six Sigma trainings, not so often applied in projects – that can greatly help to structure customer needs based on feedback given. It divides customer needs in four categories:
Information about these categories of customer perception for our product and service will be of enormous value for improving performance and gaining market share. How can we use our customer satisfaction data to establish a Kano analysis? The so called Jaccard Index of Similarity gives the answer.
Paul Jaccard developed an algorithm that enables regression-like comparison of non continuous data such as a Likert Scale. Additionally, this algorithm is able to filter “Musts”, “The More The Better” and “Delighters” out of the data. This enables the following conclusions:
Customer Satisfaction Data is not easy to come by. Therefore, it is self-explaining that we must full use of this data. By appointing Green Belts – or better Black Belts – to run analysis of the data helps avoiding data analysis mistakes. They will know how to deal with discrete Likert data.
Additional value can be added with tools like Kano diagramme and Jaccard index that are beyond the standard Six Sigma toolbox. Instead of relying on customer survey data providers for this analysis, we recommend to train your Black Belts on additional methods to gain flexibility and save costs.
There are multiple tools for analysing survey data in order to find the most appropriate method to show “patterns in data” that lead to conclusions.
Remember: Attaining the data is expensive, analysing them is cheap.
Consider a production process that produced 10,000 widgets in January and experienced a total of 112 rejected widgets after a quality control inspection (i.e., failure rate = 1.12%). A Six Sigma project was deployed to fix this problem and by March the improvement plan was in place. In April, the process produced 8,000 widgets and experienced a total of 63 rejects (failure rate = 0.79%). Did the process indeed improve?
Figure 1: Pie Charts for Two-Proportions Test
The appropriate hypothesis test for this question is the two-proportions test. As the name suggests it is used when comparing the percentages of two groups. It only works, however, when the raw data behind the percentages (100 rejects out of 10,000 parts produced and 63 out of 8,000 respectively) is available since the sample size is a determining factor for the test statistics.
To perform this test, we take the following steps:
For any statistical application, it is essential to combine it with a graphical representation of the data. The selection of tools for this purpose is limited. They include pie chart, column chart and bar chart.
The pie chart in Figure 1 shows that the percentage of defective widgets has gone down from January to April. However, there is not a large drop. Therefore, based on this plot, it is risky to draw a conclusion that there is a significant (i.e. statistically proven) difference between the defect rate in January and that in April. A statistical test can help to calculate the risk for this decision.
In this case, the parameter of interest is a proportion, i.e. the null-hypothesis is
H0: PJanuary = PApril,
with PJanuary and PApril being the real defect percentage for these two months.
This means, the alternative hypothesis is
HA: PJanuary ≠ PApril.
Since there is no reason for changing the commonly used acceptable risk of 5%, i.e. 0.05, we use this risk as our threshold for making our decision.
If there is a need for comparing two proportions, the popular test for this situation is the two-proportions test.
There are no prerequisites for the application of this test.
Using the two-proportions test, statistics software SigmaXL generates the output in Figure 2.
Figure 2: Results of Two-Proportions Test
Since the p-value is 0.0264, i.e. less than 0.05 (or 5 percent), we reject H0 and accept HA.
As a result, rejecting H0 means that there is evidence for a significant difference between the January and the April batch. The risk for being wrong with this assumption is only 2.64%.
In conclusion, we can trust the change in the widget production line and expect improved quality under the new conditions.
Interested in the stats? Read here.
Architect of High-Performing Organisations
Copyright © 2020 COE Pte Ltd. All Rights Reserved
Recent Comments