12 Oct Is “Average” a Good Indicator?
As all agencies in the E&O plus program are no doubt aware, the issue of auditing is viewed as one of the top QMP components because it is one of the most effective means to assess whether the “quality agency staff” are adhering to the “quality agency procedures.” The audit questions are designed to judge the compliance level on each of the 20+ key components.
One approach that some agencies use to measure the compliance level is to determine “the average score” for all components when combined. The goal is to compare the averages for a division from quarter to quarter. This will show the overall trend for that division. While using the “average” approach has some merit and makes sense, it also potentially has some downside.
Using an “average” approach works well when all of the issues being evaluated are of the same value. For example, getting 100s in English, math, Spanish, and history in high school and a 64 in biology may have resulted in a 93 average. Still, it also meant you were probably taking biology in summer school.
When reviewing the E&O audit templates (using commercial lines as the example), it is fair to say that all of the questions are of tremendous value, which is why they are on the audit form. However, it is also fair to say that not all of the questions are of equal value. The exposure analysis checklist and documentation are definitely in the top 3. Is policy delivery important? Definitely, but in my opinion, not of the same value as ascertaining the client’s exposure on both new and renewal business.
I am not implying that agencies shouldn’t use “averaging” as an indicator or a key metric. It certainly has value. What I am suggesting is that “averaging” by itself is not enough. As in my “high school example,” relying on the average might lead someone to some misleading or incomplete conclusions. The additional key step is to evaluate the scores of each of the questions to provide a complete assessment of what is going well and what issues need to be addressed.
While E&O Plus looks for agencies to do auditing, the real value is clearly at the agency level. By identifying issues where the compliance is below the suggested 80% level (many E&O Plus agencies have elevated their expected level of compliance to 90%), the agency is taking a significant step to identifying the “hotspots” of the firm and the key areas where additional analysis and effort is needed to improve the results.
So, averaging certainly has value, but it potentially (and probably) does not tell the whole story. An E&O culture is “only as strong as its weakest link.” Identifying and addressing that “weak link” is the key.