The Managers at Department E and Department F listen to the explanation given to their colleagues at Department C and D. They get smart and compare not only the percentages of favourable ratings – which are both at 60% – but they also compare the means, that show 3.6 and 3.4, respectively (Figure 3). The managers conclude due to the small difference between the means that their style in setting work targets might be very similar since the rating is similar.

They have learned that small differences will most likely not get accepted by statistics as significant. Therefore, they conclude that there is no real difference. So, there is no reason for concerns.

Even this seems to be a wrong conclusion.

It is always advisable to try to understand the patterns demonstrated in data of your survey results. Since the survey rating is only showing values between 1 and 5, plotting data of much more than 10 raters using bar charts or column charts may not show much pattern. Still, in our simulation, the patterns are quite obvious (Figure 3). Whereas Department E displays a homogeneous rating between 3 and 4 by all ten raters, Department F seems to have two groups of raters. One group rates very low, another group rates very high.

Calculating the standard deviation for each department could help shading some light. Consistent rating, i.e. rating by a homogeneous group of raters would typically result in a smaller standard deviation, i.e. StdDev of 0.5 for Department E. Inconsistent rating, in other words, rating of different groups of people combined in Department F results in a larger standard deviation, i.e. StdDev of 2.1.