A capability maturity model for training & education
This paper applies the concept of a Capability Maturity Model to Training and Education – the realization that how you design and deliver training (your process) is as important to your success as what you deliver (your product). It is based largely on the classical CMM and CMMI developed by the Software Engineering Institute (SEI). Other associations and vendors have sponsored related models – among them The Gartner Group, Bersin & Associates, Zeroed In Technologies, and the Center for Effective Performance.
The time has come to advance the concept of a unified model that can someday lead to an industry-accepted standard for training maturity. The methods and metrics of the training CMM (which we nicknamed “TEMMPO”, for “Training & Education Maturity Model in Projectized Organizations”) are encapsulated in the concept of training analytics, and made visible and tangible through performance management dashboards. With the tools and techniques currently available, the time is right to start building a community of people who utilize and value CMM in the training framework.
Introduction – What's a Maturity Model, and Why Should You Care?
You just took a training class and it was great. Why? Because you really, really liked it. And you felt you really learned a lot.
BUT… Did you really learn as much as you think you did? How do you know? Did you like it for reasons that had anything to do with the business goals that drive your department, and your job? And were you able to apply what you learned to what you do on the job?
You just taught a training class and it went really, really well. Why? Because you looked at the evaluations and the participants really, really liked it.
BUT… Did the participants take away value that they will apply on-the-job? Are their bosses likely to agree? Will these managers happily send their staff to the next training program you propose? And will you receive high-level support from senior management to develop more programs?
As division chief, you just approved a large-scale training program that will commit a lot of staff and resources for some time to come. The training department has assured you that it will be worth every penny. And you feel really, really good about it.
BUT… Is the training program truly in alignment with your business goals and requirements? Did the various costs exceed the net benefits? Will you recoup your investment and get the results you hope for? And how will you prove the observed business gains came from training, and not from coincidental factors such as improving markets or a successful advertising campaign? There's hope on the horizon….
This paper posits a Capability Maturity Model as a framework for creating, managing and measuring an organizational training program. It asserts that the way you create and manage training is as important to your success as the training itself. In other words, it values process as well as product in assessing the output of your efforts. How you do it becomes as important as what you do, and what you produce.
The model also focuses on demonstrating the essential relationship of education and training to business results. Learning has been described as an “ecosystem” that touches all other aspects of the business. The sustainability of a training program… and, to a great extent, a trainer's career, hinges on the alignment of the learning organization to the broad business goals and strategic objectives of the overall organization.
The maturity concept adds value on several levels: it improves the organization's ROI through results-oriented learning initiatives. It validates the contribution of the training department and its programs. It makes it easier to secure funding and management support. And it enhances the career path and skill set of trainers and trainees alike. It's a win-win-win situation all around
Distilling Wisdom from Data
Most people find it easy to “connect the dots” between the achievement of business goals and the fulfillment of personal goals – continued employment, recognition, remuneration, advancement, and prestige. The hard part is figuring out how to do it, and how to verify your accomplishments. Acquiring the skills and discipline of project management is a solid first step in achieving this internally, but it may not integrate into the environment outside of your departmental unit. You need to capture and present your achievements in the language and vocabulary of senior management - the language of metrics and measures. A solid maturity model adds precise methods, measures and metrics –learning analytics – to your training management repertoire, and displays them in an organizational dashboard that can be used throughout the organizational chain of command.
Zeroed-In Technologies CEO Chris Moore exhorts us to “view learning as an investment, not an expense” and to directly align learning with the needs of the business to show a measurable return on that investment. Further, he decries learning systems that are “data rich but information poor”, that ‘rarely approach reporting from a measurement perspective” and, consequently, “can't link or align your activity back to your strategic initiatives, or can't do so without a lot of external wizardry… Put simply, today's learning systems don't store information about business goals, objectives, measures, targets, and projections.” (2005, p. 4)
Using a maturity model – or, at least, adopting a maturity approach – will help prove the value of your training initiatives to your sponsor (and to your employer, in case they are not one and the same). Adding metrics to the mix will help you quantify the business impact of training in terms these executive will salute. It will bring the predictability, repeatability, clarity, elegance, stability and (dare we say) tranquility of project management methods and best practices to your instructional design and delivery process. It will generate a cycle of continuous improvement that extends and amplifies these rewards into the future. And it will provide useful analogies to help you manage your own career – meeting your personal objectives while satisfying those of your organization.
Many Models, a Common Core
Given the popularity of CMM and maturity in general, it's not surprising that several credible sources have jumped on the maturity bandwagon and issued their own versions, often with particular reference to eLearning . And, since PMI has a long tradition of undertaking responsibility for proposing and maintaining standards in different areas of project management, we may someday see a formal, widely-accepted standard carved out from the proprietary standards that are springing up among vendors and other stakeholders. For now, we can synthesize a credible working model from what's out there, and adapt freely to our own individual circumstances.
We call our model TEMPPO – Training & Educational Maturity Model in Project-based Organizations – because, well, everything deserves an acronym! It extends the concept of the original CMM created by SEI and its offshoot, CMMI, as well as subsequent maturity models (People Maturity Model, Organizational Project Management Maturity Model (OPM3®)). It compares and attempts to align them with classical training paradigms, such as Kirkpatrick's Evaluation Model. It also proposes an assessment you can conduct now, and actions you can take to bring the benefits of the CMM concept to your training program, whether or not you are able to immediately launch a full-scale implementation.
The Foundation – The Capability Maturity Model
The Capability Maturity Model, which has proven so valuable in IT settings, is now being imported into other settings not traditionally associated with “hard” metrics. CMM is defined in SearchCIO.com as “a methodology used to develop and refine an organization's software development process. The model describes a five-level evolutionary path of increasingly organized and systematically more mature processes. CMM was developed and is promoted by the Software Engineering Institute (SEI), a research and development center sponsored by the U.S. Department of Defense (DOD).” Similar to the ISO 9001 standards that specify an effective quality system for software development and maintenance, CMM “establishes a framework for continuous process improvement.”
SEI took CMM to a higher level with the “Capability Maturity Model® Integration (CMMI), defined as “a process improvement approach that provides organizations with the essential elements of effective processes. It can be used to guide process improvement across a project, a division, or an entire organization. CMMI helps integrate traditionally separate organizational functions, set process improvement goals and priorities, provide guidance for quality processes, and provide a point of reference for appraising current processes.”
SEI has also released the People Capability Maturity Model (People CMM). Now in Version2, “People CMM is a framework that helps organizations successfully address their critical people issues. Based on the best current practices in fields such as human resources, knowledge management, and organizational development, the People CMM guides organizations in improving their processes for managing and developing their workforces. The People CMM helps organizations characterize the maturity of their workforce practices, establish a program of continuous workforce development, set priorities for improvement actions, integrate workforce development with process improvement, and establish a culture of excellence. ” (SEI, 2005, ¶3)
There are five levels in the basic Capability Maturity Model:
- INITIAL: Processes are unpredictable, often reactive, not yet defined and documented, even disorganized. They are hard to replicate on the next project. . Success may depend on individual efforts, and occasional “heroics”. Individual contributors may be so talented and productive that the organization does not yet realize that lacks process and control.
- REPEATABLE/MANAGED: Basic processes have been established, defined, and documented. They can be repeated on subsequent projects. These processes are applied mostly to free-standing projects and are often reactive.
- DEFINED: The organization now uses internal control processes on individual projects and archives them. It pursues standardization and integration across projects. The approach is becoming proactive, but still resides largely within “silos”.
- PREDICTABLE: The organization analyzes, measures and controls processes across departmental units.
- OPTIMIZING: The organization focuses on continuous improvement of its processes. It seeks out responsive innovations to better serve organizational needs.
The Maturity levels are linked to a web of Key Process Areas (KPAs) – related activities that contribute to achieving goals - as well as Key Practices, goals and activities. Taken as a whole, they enable the organization to continuously improve the processes of the activities and the products, or outcomes, of those activities.
CMMI describes the Five Levels a little differently: Performed, Managed, Defined, Quantitatively Managed, and Optimized. For the purposes of our initial discussion of a training model, these differences are not material, although they may provide fuel for discussion and consideration later on.
Organizational Project Management Maturity Model (OPM3®)
Project Management Institute (PMI®) leveraged its original standards, including A Guide to the Project Management Body of Knowledge (PMBOK® Guide) and the Project Management Competency Development Framework, to create a new standard for applying project management principals at the organizational level. This Organizational Project Management Maturity Model (OPM3®), combines three elements: Knowledge (the contents of the standard, including maturity and best practices), Assessment (a method for comparing a real organization to the standard), and Improvement (how to change the organization, build capabilities and implement best practices).
Training plays an important role in every OPM3 implementation, according to Tom Keuten, PMI's 2008 OPM3 Update Project Manager. “In order for an organization to become more mature in its project, program and portfolio processes, training is required for the people responsible for executing those processes and there are specific capabilities in OPM3 that discuss this.” These best practices, and their officials IDs, are:
5200 - Provide Project Management Training. The organization provides project management training appropriate for all levels within the project hierarchy.
5210 - Provide Continuous Training. The organization provides continuous training in the use of tools, methodology and deployment of knowledge.
However, Keuten explains, the decision to create a “sub-set of the model dedicated to training maturity” would only take place in the context of organizational project management. The OPM3 governance team is not planning a formal sub-model for training. So, the field is still wide open for dedicated models for training maturity.
A Brief History of Maturity
The headline might have read: “Hard Metrics Invade Soft Skills!” The concept of maturity, like so many other project management ideas, developed first in the technical realms of IT, engineering and construction. These concepts then trickled down to other, less-traditional areas, in effect taking metrics “where no metrics have ever gone before.” The evolution of HR Management Information Systems and the People Capability Maturity Model are two examples. But there are other, earlier glimmers of a maturity outlook.
Such a glimmer took shape in paradigms such as Maslow's Hierarchy of Needs. The Maslow model originated as a personality theory that described the priority in which people seek to satisfy the known range of human needs. The hierarchy climbs from physiological needs, to safety and security, to belonging, to esteem and finally to self-actualization.
We could debate whether most professionals are sufficiently protected from physical fear and abject material want to freely focus on the higher levels of Maslow's hierarchy. Although most of us are not being chased by real bears, we might also argue that mergers, reorganizations, downsizings and generic job stress have replaced the primordial tigers, droughts and plagues that tethered our ancestors to Maslow's lower tiers.
Those lucky individuals who achieve a sense of harmony and security in the workplace are likely to pursue esteem satisfiers. On a basic level, equates to “status, fame, glory, recognition, attention, reputation, appreciation, dignity, even dominance.” In a more advanced form, esteem might be expressed as “self-respect… confidence, competence, achievement, mastery, independence, and freedom.” (Boree, 1998, Theory ¶9)
The highest level of the Maslow hierarchy, Self-Actualization, engages our desire to fulfill potentials, to exert a positive influence in the world, to “be all that you can be.” Although we cannot, in this paper, explore the full richness of the Maslow Hierarchy, it is fairly obvious that it echoes a personal maturity model that parallels professional development and business growth. And, interestingly, it pegs individual maturity with that of the surrounding environment.
While Maslow's Hierarchy is generally and generically applicable to business is all its aspects, Bloom's Taxonomy provides a hierarchical structure to specifically categorize learning and education. As described by Don Clark in his online article “Learning Domains or Bloom's Taxonomy”, Benjamin Bloom and his colleagues identified three domains of educational activities: cognitive mental skills (knowledge), affective feelings or emotions (attitudes), and psychomotor manual or physical skills (skills). (2001)
Blooms’ Taxonomy pairs categories and key words to facilitate the development of test questions and learning objectives. While not a normative system, Bloom's Taxonomy is highly regarded by many trainers are a mature way of thinking about educational programs, and should be included the in a broad “cosmology” of resources behind training maturity.
The Legacy of Kirkpatrick
The training profession has long relied on an almost iconic model developed by Donald Kirkpatrick, the Four Levels of Training Evaluation. These four levels – reaction, learning, behavior, and results – map roughly to four basic questions about the learners’ experience – Did they like it? Did they learn it? Can they do it? Was it worth it?
The fourth level -- Results -- asked whether the learning activity produced the desired change in business operations, but didn't go far enough. One critical question remained unanswered: Did the learning deliver the desired results for less cost than the monetary value of those results? Over time, a putative Fifth Level has been added, Return on Investment, to address this critical concern. However, the separation of time (and, often, physical distance) from the original training event makes it harder to measure, and harder to map, to training results.
Dr. Kirkpatrick first published his model in 1956, based on what became known as the “Kirkpatrick Assumption” – the belief that “the immediate objective of any training class can be stated in terms of the desired knowledge and understanding that the program is trying to impart in the students.” Kaliym Islam, Director of Customer Training & Information Products for The Depository Trust & Clearing Corporation, suggests that this assumption is no longer completely valid, or sufficient, to describe training's impact. (2005)
The Six Sigma Alternative
Writing in LTi Newsline, Islam explains “… The Kirkpatrick Assumption essentially mandates what is measured and what is reported to project stakeholders. This mandate takes place regardless of the perspective or expectations of the process partners.” For example, high evaluation scores might not be relevant to a CEO/sponsor who spent $1 million dollars to implement a $750,000 program. An operations manager who missed a deadline because key staffers were away at a training class would have a different perception, as well. (2005, ¶13)
Islam suggests an alternative approach, Six Sigma, as a way to synchronize training results with corporate objectives. Six Sigma has gained wide popularity and success because of its focus on customer service and stakeholder satisfaction. “One immediate benefit of applying Six Sigma is the business credibility to learning programs that the methodology brings. Unlike… Kirkpatricks's evaluation methodology, that [is] proprietary to the training world, Six Sigma brings a methodology that is universally accepted by businesses across various industries… Because of its customer focus, Six Sigma provides ideal solutions for showing and proving the business impact of training.” (2005, ¶14)
Other Reference Models – DMAIC, Gartner, Bersin, CEP
Another well-known model used in the training industry goes by the acronym of DMAIC, and stands for Define, Measure, Analyse, Improve and Control. It originated as a problem solving cycle within the Six Sigma movement. According to Trevor Durnford of Kaizen Training, “this tool helps teams approach problems in a logical, stepwise method, ensuring that the problem is clearly identified at the outset; that root causes are explored and measured before solutions are implemented.” DMAIC is sometimes used as an instructional design framework. Among its benefits are the opportunities to capture and share lessons learned among colleagues who are working on similar projects throughout the organization.
Gartner's eLearning Maturity Model
The Gartner Group has sponsored a Maturity Model for eLearning that consists of SIX levels: Non-existent, Initial, Repeatable, Defined, Managed, and Optimized. No surprises here! Gartner further identifies six characteristics of successful eLearning programs -- characteristics that might well apply to training across-the-board, and are presumptive indicators of overall maturity. These characteristics, highly evocative of “project scope”, are Vision and Strategy, Sponsorship, Roles and Responsibilities, Funding, Business Processes and Policies, and Technology.
The Gartner levels echo the overall trajectory of classical CMM. At Level 0 (Non-existent), use of eLearning is intermittent, somewhat casual, focused on IT applications, and not widespread. At Level 1 (Initial), awareness kicks in and the organization starts to perceive eLearning as a worthwhile endeavour, but it may still be pursued in “silos”. At Level 2 (Repeatable), eLearning is now regarded as critical and gets funded and sponsored in specific business “domains”; many of the eLearning deliverables may be reused and shared among these domain.
At Level 3 (Defined), an enterprise-wide eLearning strategy has been defined and communicated, and is used not just for training skills but for developing career competencies. At Level 4 (Managed), eLearning plans are integrated into the organization's strategic plans and the function is regarded as a *program*, not just as a string of individual projects. Development activities are well-defined and automated, and a continuous feedback loop (replete with metrics) assures continuous improvement. At the top of the chain, Level 5 (Optimized), training is completely integrated into the organization's strategic fabric and the approach is pro-active, student-centric, and highly leveraged, with reusable content and LMS-supported tracking and measurement.
An important concept emerges from this model and its counterparts: that the maturity concept doesn't just help us track what has already happened; it helps guide our path going forward. According to Gartner researcher Kathy Harris, “Gartner's maturity model provides a framework to evaluate the current state of your eLearning program. However, the greater value of the model is in using it to chart a path toward a more effective eLearning program.”
Bersin's Four Stages of eLearning
Josh Bersin of California-based Bersin & Associates presents a four-stage maturity model focused on eLearning, but also broadly applicable to a wide range of training scenarios. The first three stages are Getting Started, Expansion, and Integration and Alignment. (2005)
The first stage applies to organizations starting a new eLearning program. Largely motivated by the opportunity to save time and money, they mostly rely on off-the-shelf catalogue content housed in a fairly basic LMS. During the second, expansion phase, organizations grapple with issues related to wider access, outreach, support, branding, customization, and blended learning
After three to five years of eLearning experience, the organization will enter the third phase, characterized by integration and alignment. In this scenario, eLearning is no longer a standalone solution, but is woven into the fabric or the organization and its work processes, including performance management. A fourth, somewhat hypothetical phase-of-the-future posits the widespread adoption of eLearning that makes full use of rapid eLearning, simulation, an active LCMS, a continuous improvement process and learning analytics to measure and control learning activities.
Center for Effective Performance
The Center for Effective Performance (CEP) offers a five-phase Maturity Model to “provide a strategic framework for becoming a performance-based… organization (that) is structured around defined and measurable process that are aligned with organizational goals.” The six phases are: Discover, Plan, Manage, Optimize, and Sustain. In a recent webinar, CEP vice president Paula Alsher presented an assessment plan and Ten-Step program to achieving maturity.
CEP asserts that training is only part of the solution to most performance problems. A cornerstone of the CEP model is the Criterion-Referenced Instruction Methodology. Developed from behavioural science research, it is also closely aligned with job performance, broken down into three areas: Skill, Motivation (Will), and Operational (Hills). When all three areas are engaged the organization can achieve high performance and efficiency. Under the pragmatic CRI scenario, training is derived directly from the job, moulded to the needs of the learners, based on clear learning objectives, and covers only what is required to meet those objectives. Such training mirrors actual job conditions and provides immediate practice and feedback.
Making it Measurable – Learning Analytics
Maturity, management and metrics are inseparable and indispensable. Learning analytics represents is the place where all three come together, and the “rubber hits the road”. Learning Analytics comprise a subset of business intelligence that measures the impact of training and professional development programs on actual business results. Like a Work Breakdown Structure, learning analytics break down, or decompose, learning data through logical analysis to generate meaningful management-focused information.
Zeroed-In Technologies employs a set of Key Learning Indicators (KLIs) to measure and predict goal attainment. According to Zeroed-In CEO Chris Moore, “each KLI describes some critical, must-achieve business outcome, such as operation excellence or compliance.” And, like most goals that reflect project management sensitivity, they must be S.M.A.R.T. goals – specific, measurable, achievable, realistic, and time-based. The seven KLI's in the Zeroed-In model are:
- Operational Excellence – maximum Return on Effort (ROE) (as measured in cost per student day, ration of training costs to full time employees, enrollment and fill rates, facility costs, training revenue, development timelines, etc.)
- Learning Effectiveness – optimal results from instructional design and delivery (a measured in satisfaction surveys, progress indicators, completions, comparative pre- and post-assessments, return on investment (ROI), etc.)
- Compliance – compliance with all internal and external mandated regulations, with a robust audit process, etc.)
- Change Readiness – prepared and receptive workforce, able to respond to strategic change (as measured in cross-training and communication characteristics and climate surveys, etc.)
- Time to Competency – minimum amount of time needed to master the requirements of a job (as measured in proficiency levels, usage of “just-in-time” learning, ratio of job time to learning time, duration of orientation program, etc.)
- Workforce Proficiency – level of knowledge and competency in the workforce (as measured through test scores, skills gap analysis, productivity, quality ratings, 360-degree reviews, etc.)
- Point of Engagement – level of employee engagement, motivation, and commitment (as measured by retention, attrition, punctuality, customer surveys, results, etc.)
These KLI's can be matched to successive levels of the maturity model. At the higher levels, training becomes part and parcel of overall corporate planning, baked into the company's balanced scorecard and reflected in the “performance management dashboard”, sometimes rolled up into a broader “organizational dashboard.” (2005)
Training Impact on the Extended Enterprise
In addition to measuring the internal maturity of the training unit, training may be assessed by what it contributes to other departmental units and to the extended organization – development, sales, support, customers, suppliers, subcontractors, distributors, channel partners and others in the “value chain”. Cushing Anderson and Lyn C. Maize, writing in Chief Learning Officer magazine (2005), identify eight cycles in the value chain that may interact with training: product development, supply, logistics, sales, implementation, maintenance, customer relationship and support. These eight cycles, in turn, suggest a new range of metrics that may be embedded in a maturity assessment. For example, training of maintenance personnel may be linked to reduced costs, faster repair times, up-sell awareness and reduced breakage. Customer service training may be measured against customer loyalty, brand value, and sales cycle. To the extent that the elements of the value chain cycles are prized in organizational maturity, the impact of training efforts in meeting value chain objectives will become equally important to training maturity.
Making it Visible – Organizational Dashboards
An organizational dashboard provides a single, integrated view for monitoring key initiatives. Or, as described on the PMOStep web site, it is a “concise, visual representation of consolidated status information and metrics. It may contain graphs, tables and charts that show the overall status and health of the organization and projects. Many dashboards allow you to click on each graph and chart to see the detailed metrics that make up the total.”
Some of the metrics that can be captured on an organizational dashboard, specifically as they relate to training, include efficiency (numbers served per money expended), effectiveness (completions and scores), compliance (attainment of mandatory goals) and quality (process improvement and measurement). The dashboard can e used to assess, monitor, predict and drive your training organization.
Zeroed-In's Chris Moore reminds us that “You're not steering a go-kart; you are driving a race car, a competitive machine with paid sponsors and onlookers. Go-karts have few controls and no dashboard. The pace is slow and the mechanics are simple. A race car is a complicated system with electronics and devices to aid the drive in maneuvering and decision making. The dashboard is the central monitoring center for the driver. You are the driver of your learning organization.” (2005, p. 10)
Now that you've had a chance to get acquainted with the concepts and tools of the Capability Maturity Model applied to a learning organization…. Are you going to drive a go-kart or a race car? Are you going to dazzle your boss with ad hoc heroics, or support your organization with a sober and serious alignment to its long-term goals? Are you going to coast in your area of specialization, or are you going to stretch and climb strenuously up the ladder to higher maturity? Some people will enjoy success with a totally Level One approach, just like some people will roll out of Las Vegas with pockets stuffed with their casino wins. But do you want to bet your career on it?
This paper, with its limited scope and depth, is intended to set the stage for further discussion, debate and development of a future, unified standard for training management maturity. (A preliminary self-assessment questionnaire and comparison chart of the various approaches are being prepared for delivery at the PMI Global Congress, as first steps in creating a unified and practical training maturity model.)
We have tried to present a rounded and broadly comprehensive survey of what's out there, and what can be done. Organizational maturity is not achieved in a straight, seamless trajectory; there are bumps, bruises and glitches all along the way. Progressive elaboration at its best! And each organization will progress unevenly, more rapidly in some areas than in others; it may achieve some Level Four qualities before it completely masters Level Two. And so this maturity concept will stretch and grow unevenly, as it strives towards its own maturity.
It has been said that a mind, once stretched, never returns to its original shape. I hope that *your* mind has been stretched by the concept of training maturity, and that it will pay increasing dividends as you continue your journey in project management and in learning.
Alsher, P. (2005, August) Center for Effective Performance, Inc. “Building a Best Practice Performance –Based Organization”, PowerPoint slide deck from webinar [http://www.cepworldwide.com/]
Anderson, C. & Maize, C. L. (2005, December), Educating the Extended Enterprise, Chief Learning Officer 4(12) 24-56.
Bersin, J. (2003, Summer) Actional Information for Business Improvement, Training Analytics Series Oakland, California:Bersin & Associates
Bersin, J. (2005, October) The Four Stages of eLearning Oakland, California:Bersin & Associates.
Bersin, Josh (2003, May) Training Analytics: The Time is Now Oakland, California:Bersin & Associates
Boeree, Dr. C. G. (2005), Biography of Dr. Abraham Maslow. Retrieved from http://www.ship.edu/~cgboeree/maslow.html on December 10, 2005
Brandon Hall, various reports and resources, Retrieved from http://www.brandon-hall.com
Carnegie Mellon Software Engineering Institute, What is CMM? Retrieved from http://www.sei.cmu.edu/cmmi/general/general.html on December 25, 2005
Clark, D. (2001), Learning Domains or Bloom's Taxonomy. Retrieved from http://www.nwlink.com/~donclark/hrd/bloom.html on December 10, 2005
Durnford, Trevor. “KaizenTip #64: pssst…‥ DMAIC is flawed - pass it on!”, published in Kaizen Training's online newsletter, December 5, 2005
Harris, Kathy (2004, March) Using a Maturity Model to Make the Most of eLearning Gartner, Inc.
Islam, Kaliym A., (January 26,2005), True Return of Investment, Learning & Training Innovations (LTI Newsline). Retrieved from http://www.ltimagazine.com/ltimagazine/article/articleDetail.jsp?id=144536
Kirkpatrick's training evaluation model. Retrieved 12/07/05 from Business balls.com Web Site: http://www.businessballs.com/kirkpatricklearningevaluationmodel.htm
Moore, Chris (2005) Zeroed-In Technologies, Inc. “Seven Key Learning Indicators Your CEO Really Needs To Know”. Retrieved from http://www.getzeroedin.com/resources/Seven_KLI_v1_4.pdf on January 31, 3006, pages 4 and 11
People Capability Maturity Model – Version 2,. Retrieved from http://www.sei.cmu.edu/cmm-p/ on December 21, 2005
PMOStep Project Management Office Framework™, Retrieved from http://www.pmostep.com/0.0.0PMOStepHomepage.htm
Project Management Institute (2003), Organizational Project Management Maturity Model (OPM3®), Best Practices Directory. Newtown Square, PA: Project Management Institute.
SearchCIO.com. Retrieved from http://searchcio.techtarget.com/sDefinition/0,,sid19_gci930057,00.html on December 21, 2005
© 2005, Hollis Nan Wagenstein, PMP
Originally published as a part of the 2006 PMI Global Congress Proceedings – Madrid, Spain