An inch or a mile? Proven practices in measuring learning impact

Abstract

In today’s learning environment, learning and development (L&D) professionals continue to be challenged with the impact learning has on an individual or a business initiative. This paper will explore the most riveting and emerging topics that L&D professionals face and how measurement is playing a role in aiding them in demonstrating the impact of learning. Eight initiatives are discussed in which measuring can be a catalyst for positive change.

Linking to Shareholder Value

Throughout ESI International’s 30 years of experience with global, regional, and government clients, a high correlation has been seen between organizations that perform various initiatives to measure the impact of learning and their ability to secure funding. According to the vice president of the global project management office (PMO) for a global information technology organization, “part of my job is to get in front of the senior leadership team and show the impact of project management training every six months.” The need for more formal and systematic measurement strategies has exponentially increased in the past two to three years. ESI’s strategic partner in measurement strategies, KnowledgeAdvisors, in conjunction with McBassi Investments, found a positive correlation between the measurement of learning and market value. Over a multiyear period, the data show that publicly traded companies with high learning measurement acumens achieve 15% higher returns in stock market performance than the s&p 500 (McBassi, 2008). The data also loosely link investments in learning with financial outcomes. When preparing a business case for measuring learning, it is important to demonstrate the impact of such measurement on the end results (e.g., market value to support your case).

The Convergence of Talent Management

In years past, L&D may have been organized as a self-sufficient department yet still reported to human resources (HR). These days, the reporting lines are fused and the emergence of talent management has taken hold. KnowledgeAdvisors has determined that this field of talent management comprises a set of complete processes that overlay the life cycle of the employee. These processes help the practitioner determine where measurement needs to occur. The processes are recruiting, learning, talent (competency management), leadership, engagement, and performance management (KnowledgeAdvisors, 2008a). What does this have to do with learning impact? It means that L&D professionals need to access data from other talent processes to provide insight into their own strategy. Furthermore, L&D professionals need to provide data to other talent processes so they can manage themselves better. Finally, the data from each process should be combined analytically to provide predictive metrics to create high-performing workforces. For example, the HR function of a firm may conduct a multi-rater competency exercise throughout its workforce. It may measure various competencies and determine that one (e.g., financial acumen) is performing poorly. These data should be shared with L&D so that new learning programs can improve the performance of current workers in this area. It should also be shared with recruiting so that function can hire new employees with strong financial acumen.

One of the more difficult challenges for organizations that have identified project management as a key competency is how to plan for the future workforce in that competency. How do you obtain the tacit knowledge from individuals who may not be with the organization in two to four years?

Showing Business Impact

At the heart of all challenges is this one: Leaders of PMOs, chief executive officers (CEOs), and L&D professionals have historically been willing to buy hope. They believed that if they invested in developing the skills of their people (project managers, project team members, and managers of project managers) then the individuals would benefit as well as the organization. Today, these same leaders are no longer willing to buy hope. They want proof— proof that the US$500,000 they spent internally, or with a partner, to develop the skills of their people will yield at least that much in the bottom-line benefit to the organization. Even if they are still willing to invest, their CEOs are demanding proof to approve funding.

This issue is amplified due to the expected shortage of talent worldwide. Much of the workforce is aging and planning to retire soon. The current economic crisis has probably delayed this exodus but it is imminent. In addition, there is a new generation entering the workforce that needs a nontraditional development plan to leverage their technical savvy and relate to different mindsets. Motivating this group to remain at an organization is a challenge. Given these two worldwide phenomena, many are predicting a large turnover in the workforce. Although there is no consensus on the number, one example that may highlight the severity of this is from a large global organization of 140,000 employees. The client’s Vice President of Human Resources conducted their own analysis and estimated that either the organization would, over 8 years, replace the whole workforce twice or half of their workforce would change four times. This puts a different set of pressures on leaders to identify their management talent, or human capital, strategy in order to be a viable business in the next five years. Without an effective measurement plan to assess the impact of various interventions, such as training, on the bottom line and strategic alignment, an organization could be headed toward bankruptcy.

Return on investment (ROI) is a term often heard. Most people associate a dollar amount or a percentage with ROI. Think of a different way to show the impact—value of investment (VOI). VOI is a collection of many data points, some financial and some not, that when combined can make the business case and show the “proof” in a credible conservative fashion that will make sense to even the most critical CEOs. In addition to making the business case, the VOI will be actionable to help decision makers adjust solutions to maximize impact.

Whatever the metrics selected, one universal necessity is to baseline the current state. Without a baseline, it will be hard to show improvement or help build your VOI. This may seem elementary, but it is surprising just how many Global 500, or Fortune 1000, organizations fail to meet this basic requirement.

There are many schools of thought on this, but six approaches are briefly described in this paper. The first three are more practical and repeatable, whereas the last three are more precise but have a larger investment.

Turn Smile Sheets into Smart Sheets

Turn “smile sheets” (customer satisfaction ratings) into “smart sheets” (powerful predictive data)! Use leading methodologies, but in practical ways and at the end of training and in follow up. Take advantage of metrics with low change management to build a consistent and comparable database of evaluation performance measures, not just for satisfaction but for effectiveness and predicting impact and results. For example, asking a person how much of his or her training was used on the job and comparing it over time is a healthy continuous improvement approach.

Human Capital Approach

Leverage the ROI process (Fitz-enz, 2000) to estimate change in performance, isolate it to training, and adjust for bias. This inserts questions into the smart sheets that can then determine the change in performance due to training. This benefit figure is then monetized using salary (i.e., the economic value of human capital) and can then be subtracted from cost to yield a reasonable ROI calculation.

Business Results Evaluation Approach

This approach leverages the prior estimation, isolation, and adjustment process but is specific to a business result versus performance in general. Results include sales, quality, cycle time, productivity, customer satisfaction, and business risk. A reasonable calculation of change can be predicted on the smart sheets and compared with programs and over time to make decisions.

Business Impact Templates

These templates allow learning professionals to be consultants with stakeholders and to prepare a business case prior to training investment. Asking for a metric before training and a predicted metric after training yield the business result change. A root-cause exercise isolates it to learning and an adjustment factor makes it conservative. Unlike the prior models, this approach uses the actual business results.

Correlations

One can measure the outcomes of a business result before training and then again after. If the aftereffect is positive, there is a positive correlation. If a naturally occurring control group exists it is even more credible. This too uses actual results data but may take time to wait on the aftereffect results. So, although more precise, there is a delay in the final analysis.

Causal Modeling

Causal modeling (Bontis, 2009) uses significant volumes of historic data to statistically link training to business impact. The higher the coefficient of determination (R2) the greater link of training to impact. This too requires actual data and a statistical competency plus voluminous historic data, but it can be very powerful in showing the business impact from learning.

Scrap Learning

This concept is very troubling in learning today. About 65% of training is not optimally applied on the job (KnowledgeAdvisors, 2008b). The core problem is that managers are not properly preparing learners for training nor are they supporting them on the job once training is over. The result is wasted (i.e., scrap) learning. Scrap can be reduced through better manager engagement. Managers who set expectations with employees prior to learning and follow through afterward with projects and measurable goals overcome the scrap learning crisis.

Performance Optimization/Six Sigma

To those outside of L&D, measurement is a process improvement tool. L&D should adapt process improvement models to help in its measurement endeavors. A popular approach is Six Sigma. Using DMAIC (Define, Measure, Analyze, Improve, and Control) is what measurement is all about for any process (KnowledgeAdvisors, 2010). Defining what learning will measure, measuring it, reviewing results, making changes, and measuring again is the spirit of Six Sigma and helps to reduce defects that can be harmful to a process.

Predict and Forecast, Then Follow Up

Consistent with “smart sheets” is the paradigm of predicting impact versus waiting for it to happen, or not. Smart sheets collect evaluation data that forecast the future. For example, will you use this on the job? In addition, one can take a business result for a short period and extrapolate it into the future for predictions. The theme is to provide reasonable information to make forward-looking decisions.

Actionable Metrics

Metrics must enable decision making (i.e., they must be actionable). There are five ways to optimize this. First, leverage both internal and external benchmarks; these motivate by example. Second, leverage goals; if they are challenging yet attainable, they too motivate. Third, trend results over time. Fourth, use dashboards, which are concise yet visual summaries that tell a story. Fifth, talk to people about the data instead of using e-mailing. A dialogue goes a long way in engaging people in using the metrics for decision making.

Informal Learning Measurement

Informal learning was around before it became formalized, and in today’s world of social networking it is important to embrace new ways of learning and measuring impact. If a person goes to a blog and learns something new, how is it measured? The goal is to measure quickly via a poll at the point of access. Then, measure more formally with a sample of the population at a key milestone or periodically and preferably in a way that is consistent with other learning deliveries for comparison. The fact is that organizations that measure have a larger budget allocated toward informal learning approaches than those that do not (KnowledgeAdvisors, 2009).

Bontis, N., & KnowledgeAdvisors. (2009). The predictive learning impact model. Retreived from http://www.knowledgeadvisors.com/archives/knowledgeadvisors-announces-the-predictive-learning-impact-model/

Fitz-enz, J. (2000). ROI of human capital: Measuring the economic value of employee performance, New York, NY: AMACOM Books.

KnowledgeAdvisors. (2009). Executive research: Informal learning.

KnowledgeAdvisors. (2008a). Human capital analytics: Measuring the people process from recruitment to retirement.

KnowledgeAdvisors. (2008b). Manager engagement best practices.

KnowledgeAdvisors. (2010). Performance optimization framework.

McBassi Investments & KnowledgeAdvisors. (2008). Human capital measurement and its impact on stock performance.

This material has been reproduced with the permission of the copyright owner. Unauthorized reproduction of this material is strictly prohibited. For permission to reproduce this material, please contact PMI or any listed author.

© 2010, Raed S. Haddad and Jeffrey Berk
Originally published as a part of 2010 PMI Global Congress Proceedings — North America

Advertisement

Advertisement

Related Content

  • PM Network

    A Foot in the Door member content locked

    By Scott, Lindsay Project management recruitment professional Lindsay Scott, answers questions about making the most of informal meetings and looking for new career opportunities beyond the usual sectors.

  • PM Network

    On Your Own member content locked

    By Thomas, Jen For project professionals, a boss can be the biggest advocate—or biggest obstacle—on the career path. A helpful manager will mentor and champion project wins, promote face time with high-level…

  • PM Network

    Bridging Ambition and Ability member content locked

    By Parsi, Novid Agile is everywhere, but not every organization is ready for it. According to the 12th annual State of Agile survey published by CollabNet VersionOne, 41 percent of organizations cite a lack of…

  • PM Network

    How to Get Hired in the Age of AI member content locked

    By Thomas, Jen From self-driving cars to search engines, artificial intelligence (AI) is slowly transforming the world around us. Now, it's starting to play a role in how project professionals land new gigs.…

  • PM Network

    Conference Connections registered user content locked

    By Scott, Lindsay Project management recruitment professional Lindsay Scott gives advice about searching for jobs at professional events, acing interview presentations and leadership skills.

Advertisement

Publishing or acceptance of an advertisement is neither a guarantee nor endorsement of the advertiser's product or service. View advertising policy.