Bima Hermastho

What Gets Watched Gets Done: How Metrics Can Motivate

In Uncategorized on Oktober 2, 2009 at 9:31 pm

The old BI adage "what gets watched gets done" suggests that when activities are observed and measured, people change their behavior to look good on the performance metrics. This is exactly what makes dashboards and scorecards so powerful: They focus organizational attention and motivate behavior.

This recognition is not new. Executives quickly grasp the power of metrics. For example, there is the old story about a steel baron who, at the end of the first shift, asked the foreman, "How much steel did you produce?" When told, the owner took a piece of chalk and wrote the number on the plant floor. When the next shift came in and asked about the number, they were told it represented the tons of steel produced during the last shift. When the second shift ended, the next foreman crossed out the previous number and wrote the new, larger number. This process continued for several days and led to exceptionally high levels of steel production.

Dashboards at 1-800 CONTACTS

1-800 CONTACTS provides a great example of the ability of dashboards to motivate behavior. The company uses dashboards in its call center and has seen significant improvements in operator performance. It also has an interesting twist in the way it uses dashboards to create a competitive environment for the operators.

The company was founded in 1995 by two entrepreneurs who offered contact lenses at low prices with convenient ordering and fast delivery. Today, 1-800 CONTACTS is the world s largest contact lens store, selling as many contact lenses as 2,500 retail optical shops combined.

Customers can order over the Internet, by mail or fax, at WaImart, or by phone. The company has almost 300 trained customer service agents.

Until four years ago, 1-800 CONTACTS had no formal BI initiative. A full-time DBA would create reports on demand when requested by management. When Jim Hill, director of data management, came on board with his team, their first task was to build a data warehouse to support executive reporting.

Soon after the data warehouse was built and a reporting system was in place, call center management approached Jim for help with developing a better incentive system for operators. The legacy system was largely manual and there were no good statistics or benchmarks available for assessing and rewarding the agents’ performance. Jim’s response was to create a dashboard that used warehouse data and displayed metrics such as operator closing rates and average order size.

When the prototype was shown to management, one executive said that it was fine but that he wanted to create more of a "horse race" where the agents could see how they were doing and compete against one another. The next version was close to what is in place today. The operators’ dashboards are updated every 15 minutes and they can see how they are doing on key metrics and in comparison with the other operators. The operators’ compensation is based in part on the metrics displayed and their performance relative to other operators.

The Operators’ Dashboard

Figure 1 shows a typical operator’s dashboard. The color- coded gauges on the left indicate how the operator is performing on the closing ratio, average sale, and calls /hour metrics for the day. The bar charts to the right are more detailed and show performance on an hourly basis. The straight lines are the operator’s average monthly performance for the metrics. This operator has been doing better than his month’s average closing ratio and average sale, but only in the 1:00 p.m. hour has he done better than his monthly average on calls /hour. At the right of the screen is a testimonial from a customer about favorable customer service. This serves as a motivator for the operators and is updated every month or so. The ribbon at the bottom of the screen identifies the top five performing agents for each of the three measures (and a few others).

There are two ways operators can see how they are doing in comparison to the other agents. First, for today’s performance, the gauges are color coded. In Figure 1, a green light for the closing ratio and average sale indicates that the operator is in the top quartile of operators. By way of contrast, a red light shows that when measured by number of calls answered per hour, the operator’s performance is in the bottom quartile.

The second way for an agent to see how he or she is doing is to select the metrics button from the top menu. This takes the agent to a dashboard (not shown) that provides month- to- date comparisons with the other operators on his team (operators are divided into teams of about 24) and the call center as a whole.

This page also shows the calculation of the operators Call Center Incentive (CCI) index, which is used in determining the agents’ monthly compensation. The CCI formula uses a closing ratio index, average order size index, hours worked, bonus points associated with selling solutions or setting up appointments, and a quality index that is based on audits of the quality of the operators interactions with customers. The indices are computed so operators are compared to other operators who are working at the same time.

At the end of the month, all of the agents are ranked based on their CCI scores. The top 80 percent of the agents receive bonuses based on their scores and the size of the bonus pool, which is tied to overall business success. A top-performing agent can earn a bonus of as much as $1,000 per month.

The Executives’ Dashboard

The executives’ dashboard (not shown) reveals how the business is doing at any time during the day, how the business is performing relative to four weeks and to a year ago, the performance of the various channels (e.g., Internet, call center), and the predicted sales for the day. The last item is particularly interesting. Using historical data, by 10:00 a.m. each morning it is possible to predict sales for the day within $2,000 (assuming no unusual event occurs). The company plans to add even more comparisons and trending information to the executives’ dashboards.

How does an executive use the dashboard? One example would be a marketing manager who knows the timing of a new marketing campaign. The manager can use the dashboard to monitor the campaign’s effectiveness and respond accordingly.

Impact on Performance

The operators’ dashboards had the anticipated impact. What gets watched gets done. After the dashboards were in place, revenues increased $50,000 per month and call quality remained high. Call center management attributed the increased sales revenue to the dashboards.

The Operators’ Reactions

The agents had an interesting reaction to the dashboards and to a compensation system that had operators competing against one another. Jason Cann, who was an operator when the dashboards were first implemented and is now a team supervisor, provided a first-hand perspective.

Nearly all of the agents welcomed the introduction of the dashboards. Before the dashboards were installed, competition for incentive dollars was based on the closing ratio. Many of the agents considered this to be unfair. Closing ratios tend to vary with the day and time of day, and those agents who worked at a time when closing ratios tend to be lower felt they were at a disadvantage. For example, in the afternoon there are relatively more calls about existing orders than new orders. Consequently, many of the agents liked the dashboards and the CCI index because they provided a better measure of performance. Interestingly, one high-producing agent who did not like the new system left the company. He had been a leader in closing ratio and sales, but did not score well on the CCI index because of poor-quality interactions with customers. As a result, he was no longer considered to be a top producer.

Most of the operators liked, and continue to like, the dashboards and compensation system. One reason is the agents’ ability to constantly monitor their performance and compare it to other agents’ performance. They just seem to like the scorekeeping and competition. The company also tries to hire people who seem likely to work well in a competitive environment.

Another reason agents like the system is that they can see whether their performance is consistent over time and at a high level. This allows the agents to be accountable to themselves. Jason says that the culture of the call center has changed because the agents now know where their performance stands and the dashboards help them to improve. Every 15 minutes the agents can see how far they have "moved the needle."

Most of the agents who find themselves in the bottom 20 percent (and thus do not qualify for bonuses) respond by working harder to improve their performance. The rest probably would not work hard under any system.

Finally, the dashboards allow the agents to see how their teams are performing against the other teams. Regular team competitions are fun for the agents and profitable for the company.

Choosing the Right Metrics

The metrics for dashboards must be chosen carefully. Metrics should:

* Be accurate

* Be easy to understand

* Be actionable

* Be controllable

* Be few in number

* Be supported by data

* Be presented in the context of targets, goals, and past performance

* Balance and reinforce one another

* Be quantitative whenever possible

* Measure the true event when possible, not an associated event

* Be periodically reviewed and revised

* Be owned by the person who is responsible for the outcome

* Include both leading and lagging measures

* Be aligned with the business strategy

Some of these are especially interesting as they apply to the CCI index. Metrics should balance and reinforce one another. For example, a concern with any set of metrics is that they may be "gamed"; that is, people may engage in behaviors that allow them to perform well on the metrics but, in a larger context, produce results that are undesirable for the organization. In the case of the CCI index, the closing ratio index and average sales index are balanced by the quality of the customer interaction. Because of the quality index in the CCI formula, operators are discouraged from pushing for high closing rates and sales at the expense of a pleasant customer experience.

Although the CCI index provides a good balance of performance metrics, it is relatively complicated. Operators cannot equate any particular score to their compensation. As a result, it violates the "easy to understand" characteristic of a good metric. Basing compensation directly on operator sales would be simpler, but management is hesitant to move in this direction lest there be an associated decrease in the quality of customer interactions. Management continues to consider a CCI index or other method for assessing overall operator performance that is both easy to understand and provides appropriate incentives.


Dashboards and scorecards are powerful ways to monitor performance and focus organizational attention. This has been shown to be the case at 1-800 CONTACTS, where the company has taken the use of dashboards one step further by using them to create a competitive environment. Call center operators compete for bonuses, and the dashboards foster this competition by showing how the operators are doing on the metrics used to determine those bonuses.

What is especially interesting is how well the operators have responded to the system. With few exceptions, the agents think highly of the system. This shows that employees like feedback on how they are performing and enjoy competing. It is important, however, that the metrics used in the competitive environment are well thought out and have a balance consistent with overall company goals.

[Author Affiliation]

Hugh J. Watson is Professor of MIS and C. Herman and Mary Virginia Terry Chair of Business Administration in the Terry College of Business at the University of Georgia.

Jim Hill is director of data management at 1-800 CONTACTS.


Tinggalkan Balasan

Isikan data di bawah atau klik salah satu ikon untuk log in:


You are commenting using your account. Logout /  Ubah )

Foto Google

You are commenting using your Google account. Logout /  Ubah )

Gambar Twitter

You are commenting using your Twitter account. Logout /  Ubah )

Foto Facebook

You are commenting using your Facebook account. Logout /  Ubah )

Connecting to %s

%d blogger menyukai ini: