An inch or a mile? Proven practices in measuring learning impact

Jeffrey Berk, KnowledgeAdvisors

Abstract

In today’s learning environment, learning and development (L&D) professionals continue to be challenged with the impact learning has on an individual or a business initiative. This paper will explore the most riveting and emerging topics that L&D professionals face and how measurement is playing a role in aiding them in demonstrating the impact of learning. Eight initiatives are discussed where measuring can be a catalyst for positive change.

Linking to Shareholder Value

Throughout ESI International’s (ESI’s) 30 years of experience with global, regional, and government clients, a high correlation has been seen between organizations that perform various initiatives to measure the impact of learning and their ability to secure funding. According to the vice president of the global project management office (PMO) for a global information technology organization, “part of my job is to get in front of the senior leadership team and show the impact of project management training every six months.” The need for more formal and systematic measurement strategies has exponentially increased in the past two to three years. ESI’s strategic partner in measurement strategies, KnowledgeAdvisors, in conjunction with McBassi Investments, found a positive correlation between the measurement of learning and market value. Over a multiyear period, the data shows that publicly traded companies that have high learning measurement acumen achieve 15% higher returns in stock market performance than the Standard & Poor’s 500 (S&P 500) Index (McBassi & KnowledgeAdvisors, 2008). The data also loosely links investments in learning with financial outcomes. When preparing a business case for measuring learning, it is important to demonstrate the impact of such measurement on the results, e.g., market value to support your case.

The Convergence of Talent Management

In years past, L&D may have been organized as a self-sufficient department yet still reported to human resources (HR). In current times, the reporting lines are fused and the emergence of talent management has taken hold. KnowledgeAdvisors has determined that this field of talent management comprises a set of complete processes that overlay the life cycle of the employee. These processes help the practitioner determine where measurement needs to occur. The processes are recruiting, learning, talent (competency management), leadership, engagement, and performance management (KnowledgeAdvisors, 2008a). What does this have to do with learning impact? It means L&D professionals need to access data from other talent processes to provide insight into their own strategy. Further, L&D professionals need to provide data to other talent processes so they may manage themselves better. Finally, the data from each process should be combined analytically to provide predictive metrics to create high-performing workforces. For example, the HR function of a firm may conduct a multi-rater competency exercise throughout its workforce. It may measure various competencies and determine that one, e.g., financial acumen, is performing poorly. This data should be shared with L&D so that new learning programs can improve performance of current workers in this area. It should also be shared with recruiting so that function can hire new employees with strong financial acumen.

One of the more difficult challenges for organizations that have identified project management as a key competency is how to plan for the future workforce in that competency. How do you obtain the tacit knowledge from the individuals who will not be with the organization in two to four years?

Showing Business Impact

At the heart of all challenges is this one: Leaders of PMOs, chief executive officers (CEOs), and L&D professionals have historically been willing to buy hope. They believed that if they invested in developing the skills of their people (project managers, project team members, and managers of project managers) then the individuals would benefit as well as the organization. Nowadays, these same leaders are no longer willing to buy hope. They want proof — proof that the $500,000 they spent internally, or with a partner, to develop the skills of their people will yield at least that much in bottom-line benefit to the organization. Even if they are still willing to invest, their CEOs are demanding proof to approve funding.

This issue is amplified due to the expected shortage of talent worldwide. Much of the workforce is aging and planning to retire soon. The current economic crisis has probably delayed this exodus but it is imminent. In addition, there is a new generation entering the workforce that needs a nontraditional development plan to leverage their technical savvy and relate to different mindsets. Motivating this group to remain at an organization is a challenge. Given these two worldwide phenomena, many are predicting a large turnover in the workforce. Although there is no consensus on the number, one example that may highlight the severity of this is from a large global organization of 140,000 employees. It conservatively estimates that 280,000 employees will leave the organization during the next eight years. This puts a different set of pressures on leaders to identify their talent management, or human capital, strategy in order to be a viable business in the next five years. Without an effective measurement plan to assess the impact of various interventions, such as training, on the bottom line and strategic alignment, an organization might be heading toward bankruptcy.

Return on investment (ROI) is a term often heard. Most people associate a dollar amount or a percentage with ROI. Think of a different way to show the impact — value of investment (VOI). VOI is a collection of many data points, some financial and some not, that when put together can make the business case and show the “proof” in a credible conservative fashion that will make sense to even the most critical CEOs. In addition to making the business case, the VOI will be actionable to help decision makers adjust solutions to maximize impact.

Whatever the metrics selected, one universal necessity is to baseline the current state. Without a baseline, it will be hard to show improvement or help build your VOI. This may seem elementary, but it is surprising how many global 500, or Fortune 1000, organizations fail to meet this basic requirement.

There are many schools of thought on this but six approaches will be briefly described. The first three are more practical and repeatable while the last three are more precise but have a larger investment.

Turn Smile Sheets into Smart Sheets

Turn smile sheets (customer satisfaction ratings) into “smart sheets” (powerful predictive data). Use leading methodologies but in practical ways at the end of training and in follow up. Take advantage of metrics with low change management to build a consistent and comparable database of evaluation performance measures not just for satisfaction but also for effectiveness and predicting impact and results. For example, asking a person how much of the training was used on the job and comparing it over time is a healthy continuous improvement approach.

Human Capital Approach

Leverage the ROI process (Fitz-enz, 2000) to estimate change in performance, isolate it to training, and adjust for bias. This inserts questions onto the smart sheets that can then determine the change in performance due to training. This benefit figure is then monetized using salary (i.e., the economic value of human capital) and can then be subtracted from cost to yield a reasonable ROI calculation.

Business Results Evaluation Approach

This approach leverages the prior estimation, isolation, and adjustment process but is specific to a business result vs. performance in general. Results include sales, quality, cycle time, productivity, customer satisfaction, and business risk. A reasonable calculation of change can be predicted on the smart sheets and compared against programs and over time to make decisions.

Business Impact Templates

These templates allow learning professionals to be consultants with stakeholders to prepare a business case prior to training investment. If you ask for a metric before training and a predicted metric after training, then the business result changes. A root-cause exercise isolates it to learning and an adjustment factor makes it conservative. Unlike the prior models, this approach uses the actual business results.

Correlations

One can measure the outcomes of a business result before training and then again after. If the aftereffect is positive, there is a positive correlation. If a naturally occurring control group exists, it is even more credible. This too uses actual results data but may take time to wait on the aftereffect results. So, while more precise, there is a delay in the final analysis.

Causal Modeling

Causal modeling (Bontis, 2009) uses significant volumes of historic data to statistically link training to business impact. The higher the coefficient of determination (R2) the greater link of training to impact. This too requires actual data and a statistical competency plus voluminous historic data. However, it can be very powerful in showing the business impact from learning.

Scrap Learning

This concept is very troubling in learning today. About 65% of training is not optimally applied on the job (KnowledgeAdvisors, 2008b). The core problem is that managers are not properly preparing learners for training nor are they supporting them afterward on the job. The result is wasted (i.e., scrap) learning. Scrap can be reduced through better manager engagement. Managers who set expectations with employees prior to learning and follow through afterward with projects and measurable goals overcome the scrap learning crisis.

Performance Optimization/Six Sigma

To those outside of L&D, measurement is a process improvement tool. L&D should adapt process improvement models to help in its measurement endeavors. A popular approach is Six Sigma. Using define, measure, analyze, improve and control (DMAIC) is what measurement is all about, for any process (KnowledgeAdvisors, 2010). Defining what learning will measure, measuring it, reviewing results, making changes, and measuring again is the spirit of Six Sigma and helps to reduce defects that can be harmful to a process.

Predict and Forecast, Then Follow Up

Consistent with smart sheets is the paradigm of predicting impact vs. waiting for it to happen, or not. Smart sheets collect evaluation data that forecasts the future. For example, will you use this on the job? In addition, one can take a business result for a short period and extrapolate it into the future for predictions. The theme is to provide reasonable information to make forward-looking decisions.

Actionable Metrics

Metrics must enable decision making, i.e., they must be actionable. There are five ways to optimize this. First, leverage both internal and external benchmarks. These motivate by example. Second, leverage goals. If they are challenging yet attainable, they too motivate. Third, trend results over time. Fourth, use dashboards, which are concise yet visual summaries that tell a story. Fifth, talk to people about the data instead of e-mailing it to them. A dialogue goes a long way in engaging people in using metrics for decision making.

Informal Learning Measurement

Informal learning was around before it became formalized, and in today’s world of social networks, it is important to embrace new ways of learning and measuring impact. If a person goes to a blog and learns something new, how is it measured? The goal is to measure quickly via a poll at the point of access. Then, measure more formally with a sample of the population at a key milestone or periodically and preferably in a way that is consistent with other learning deliveries for comparison. The fact is that organizations that measure have more budgets allocated toward informal learning approaches than those that do not (KnowledgeAdvisors, 2009).

References

Bontis, N., & KnowledgeAdvisors. (2009). The predictive learning impact model. Chicago, IL: Knowledge Advisors.

Fitz-enz, J. (2000). ROI of human capital: measuring the economic value of employee performance. New York, NY: AMACOM Books.

KnowledgeAdvisors. (2008a). Human capital analytics: measuring the people process from recruitment to retirement. Chicago, IL: Knowledge Advisors.

KnowledgeAdvisors. (2008b). Manager engagement best practices. Chicago, IL: Knowledge Advisors.

KnowledgeAdvisors. (2009). Executive research:informal learning. Chicago, IL: Knowledge Advisors.

KnowledgeAdvisors. (2010). Performance optimization framework. Chicago, IL: Knowledge Advisors.

McBassi Investments, & KnowledgeAdvisors. (2008). Human capital measurement and its impact on stock performance. Chicago, IL: Knowledge Advisors.

This material has been reproduced with the permission of the copyright owner. Unauthorized reproduction of this material is strictly prohibited. For permission to reproduce this material, please contact PMI or any listed author.

© 2010, Raed S. Haddad and Jeffrey Berk
Originally published as a part of 2010 PMI Global Congress Proceedings — North America

Advertisement

Advertisement

Related Content

  • PM Network

    Digging Deep member content locked

    By Parsi, Novid Iron ore mining projects are on the rise again in Australia. After prices plummeted in 2015, the sector is expected to see an increase in export volumes of iron ore from 847 million metric tons in…

  • Project Management Journal

    Who Supports Project Careers? member content locked

    By Huemann, Martina, | Ringhofer, Claudia | Keegan, Anne E. This exploratory research examines who supports what aspects of career development on projects. Our main finding is that, although project professionals receive support from formal and informal…

  • Project Management Journal

    Selecting Organizational Partners for Interorganizational Projects member content locked

    By Braun, Timo | Sydow, Jörg The selection of partners is critical for the success of interorganizational projects. Based on conceptual reasoning in light of prior research, as well as illustrative empirical insights into an…

  • PM Network

    Chain Demand member content locked

    Manufacturing, retail and logistics executives are clamoring for more real-time, end-to-end transparency of their supply chains. That desire is driving demand for digital supply chain investments,…

  • PM Network

    Raise the Roof member content locked

    A serious labor shortage is threatening to slow housing projects across the United States. In May, builders reported shortages in 12 of 16 labor categories, with carpenters, framing crews,…

Advertisement

Publishing or acceptance of an advertisement is neither a guarantee nor endorsement of the advertiser's product or service. View advertising policy.