APPS assessment method


As project managers we deal with people on a day-to-day basis. People are the major factor in the success or failure of a project, yet, we have very few tools and models that can quickly and easily assess teams and the factors that influence their ability to complete the tasks at hand. Our goal is to get the job done on time and within budget; however, there will be those instances where individual team member productivity is negatively impacted and will require analysis, planning, and communications. What are the available tools that allow the project manager to determine quantitatively the impact of project failures and significant negative events on an individual's productivity?

Both the APPS Methodology and Tools were developed by the author out of necessity. There was a major need to understand how the emotional state of team members after a project failure or significant negative event impacts their productivity and influences the viability of the project. This allows the realistic planning of the future tasks and new implementation dates.

The APPS Methodology was used to resurrect a failed project with a statewide bank in Tennessee for the rollout and deployment of a statewide 170+ branch network, including hardware, software, phone switch upgrades, wiring, etc. The project was approaching the six-month mark, and a major pilot of the system was about to occur.

The evening finally arrived! Everything was a “go” for the system pilot. The time drew near, and all were present, from the company and client CEOs down to every available technician. After all, this was a high visibility project for both of the organizations. The time arrived, and the system was started. The lights began to blink and the wheels began to spin. The monitor pressed through the normal sequence of events, and all appeared normal, except there was no data on the screen. The screen was black. There were nervous smiles all around the room as the frantic search for the problem began. The minutes turned into hours, and finally everyone called it a night. The system pilot deliverable had failed.

The next morning the project was turned over to the author to manage. The new project manager took an inventory of the team, an inventory of the client relationship and a review of the overall project. The APPS Methodology was developed and used to obtain reasonable feedback on people, facilities, and project timing. APPS projected a three man-month negative productivity impact. Based on information gathered in other meetings, an additional delay of six to nine months was projected due to equipment and other constraints. The APPS findings, along with other supporting information and projections, were presented to the client, and a sixmonth contract extension was obtained. This extended the entire project duration to two years.

In a lessons learned review performed after project completion, the results of the use of the APPS methodology originally projected 12 weeks of reduced productivity. Both the client and operational management agreed that the process had missed the projection by two weeks, for only 10 weeks of decreased productivity due to factors outlined instead of the projected 12 weeks.

The APPS methodology has also been used on several other projects, most recently for a Manufacturing Plant Relocation project. The project was to relocate a corporate division starting in June 2001, and completing by December 2001. The manufacturing facility to be moved was located in New York City in Manhattan, 10 blocks from the site of the World Trade Center. Following all of the chaos and confusion on September 11, 2001, the methodology was utilized to provide a base estimate of delays and to evaluate options. The projections of two weeks of little to no production in the New York facility and a one-month project delay were realized. The model projected that productivity would only be 45% versus their normal planned 100% during the four-week period being monitored for impacts. The actual productivity reached 50%. These projections and the realization of a looming deadline prepared the team to make decisions and modifications to the project. On November 30, 2001, the New York facility was closed and production relocated to Nashville, Tennessee. The project did not reach some of its identified goals; however, it did reach the schedule constraint of relocating the manufacturing plant activities by year-end.

The Methodology

APPS stands for “Assess, Ponder, Plan and Share,” which are the phases of the methodology (see Exhibit 1). These phases provide a structure to be utilized in the collection, analysis, planning, and communications related to project risk, analysis, and recovery. APPS provides a tool to assist the project manager and a methodology to organize how the tool is used. Unfortunately, it is still not a crystal ball; it is only a tool to assess possible outcomes.


The “Assess” phase of the methodology involves the use of surveys for data collection, which is used later to support the “Ponder” phase. A survey (See Exhibit 2) should consist of five or more questions in each of three major areas: Emotional Factors, Physical Factors, and Functional Factors. Each factor is further subdivided into direct and indirect aspects. For example:

Exhibit 1

Exhibit 1

Emotional Factors

Direct Aspects

Loss of an immediate relative

Loss of home or job

Indirect Aspects

How does this person react to stress? Talkative, breakdown, etc.

How does this person affect others? Good or Bad

Physical Factors

Direct Aspects

Physical Building Loss or Damage

Physical Office Loss or Damage

Indirect Aspects

Building Access

Work Area Access


Functional Factors

Direct Aspects

Tools Available

Other Job Restrictions

Indirect Aspects

Utilities Available

Support Systems Available (i.e., Mail Service, Shipping and Receiving)

The survey can be conducted at any of several different levels of the organization. On large projects with extremely large teams, the survey can be provided to the team leader, where on smaller projects it may be given to individual team members. The survey can also be used to assess locations and groups. The strategy of using the survey is to extract quantitative information that could impact the productivity of the team members on the project. Additional project information can be gathered from all available sources, even if it spans across multiple vendors, and be utilized in the “Ponder” phase. If the project team has been performing structured project management techniques, most information may be stored in the Project Workbook. Depending on the size and complexity of the project, this may or may not be easily available or may require contacting multiple vendors.

If a project has failed or certain components have potential to fail, many individuals on the project team may already be overworked, unrecognized and/or just plain exhausted. These elements have a significant impact on the timing and continued effort required to deliver the project's requirements and should be considered.

Other items to be gathered and utilized would be the project schedule, budget, lessons learned, issues and risks. Additional information could be obtained with the use of a formal analysis tool, such as Root Cause Analysis. This process allows one to analyze whether there are reoccurring issues, indicating that only the symptoms of a problem have been resolved and not the root cause. If the project team has not been using such a formal tool, a more thorough investigation will be required through interviewing and research to determine the historically significant issues and problems. The goal of this step is to understand the potential failure points so they can be properly categorized and documented for the tool usage. Not having this information readily available will make the project management Superhero's job more difficult, but not impossible.

Exhibit 2

Exhibit 2

Survey questions should be direct. Additionally, survey questions should be developed with an integrated impact scale. This includes positive and negative values reflecting their affect on the project (i.e., response = –3, etc.). A “10” represents no impact, and a “0” represents maximum impact. The survey questions need to cover a specific time frame as it relates to the project (i.e., 3 days, 3 weeks). The transfer to the next phase is simplified by starting at 10 and adding and subtracting identified impacts. Transferring data between the “Assess” and “Ponder” phases is usually the most administratively intensive part of the process. Online and PC-based survey tools can help to facilitate the data transfer.

Using these and other possible analysis tools will provide the project manager with the necessary input into the APPS tool. The overall goal of the Assess phase is to determine “What” needs to be measured. Now it's time to review the Ponder phase.


The survey analysis spreadsheet (Exhibits 3 and 4) is a simple tool that can be modified to meet the needs of the most demanding projects. The transfer of data from the survey to the spreadsheet is usually the most tedious and time-consuming activity of the process. Survey design is therefore a critical step in the process. An effectively designed survey for the APPS methodology allows the information to flow easily into the three major factors of emotional, physical and functional. By dealing with these three factors and their direct and indirect aspects, a system of weighting can be utilized and a productivity index calculated for individuals, groups and/or the project as a whole. In the example spreadsheet for Manufacturing Plant Relocation project, the individuals represented three functional organizations in two locations. Productivity indexes for all six functional groups, as well as for both locations, were developed (Exhibit 4). This demonstrates the tool's flexibility and scalability.

Exhibit 3

Exhibit 3

To generate the index, the spreadsheet has a two-step approach. The first step is an individual summary calculation generating a value for each functional area. For both the functional and emotional areas, the calculation is as follows:

“2 of 3 Index Rating”? ((2*direct)+indirect)/3)/10)

Another example of this calculation could be:

“3 of 5 Index Rating“? ((3*direct)+(2*indirect)/5)/10)

Establishing a weighting system that reflects the project risks and outside factors surrounding the project is crucial. The weighting helps to place emphasis on the direct aspects, which tend to have a greater productivity impact, while still accounting for the indirect aspects. Both systems rank direct impacts as greater than indirect and both the “2 of 3” and “3 of 5” weightings have been used. In the past, consistent factors have been used for the calculations on the areas. An area for future research and planning could be the use of a mixedweight model along with further validation of the previously used weightings.

To generate the value for the physical area, the same calculation is used if neither the direct nor indirect value is equal to “0.” If either value is “0,” the value of that calculation is “0.” The thought process here is that if there is an impact to the physical location making access to that location impossible or improbable, then little or no work will be accomplished for that time period. In the case of those working from remote locations, a number greater than “0” should be used unless there is a physical reason that keeps work from continuing (i.e., direct aspect, physical factor building loss, permanent loss of network connectivity, toxic waste).

Now that the Emotional, Physical, and Function detail has been stored, the second part of the calculation is averaging and creating an index by individual, group or as a whole. The Productivity Index calculation that has been used is as follows:

((emotional + functional)/2)*physical

This generates an index that is then applied to productivity or hours, project plans and/or productivity analysis. The nature of how this index is applied and to what areas will be the directive of the project manager and the project team. Additional analysis will need to be completed to reflect the team and any outside factors. By applying the index, the productivity, or productive hours, of each group is reflected in the highlighted column on Exhibit 4.

Exhibit 4

Exhibit 4


Some experts may argue that Planning is traditionally a project management activity that should come first and Analysis (Assess Phase) is something that should follow. The author feels these activities should be reversed when project failures occur. In the event that a Project Manager has been brought in to recover the failed project, the sequence of Analysis first, then Plan second is a must in order for the Project Manager to transition into his or her new role. By following the APPS Methodology, the Assess Phase is completed where Factors and Aspects were identified and measured via a Survey. The Ponder Phase is completed when the quantitative productivity indexes were calculated. Now it's time to plan the recovery.

At a minimum, the Recovery Plan should include all of the new documentation created to date, including the inputs used for the “Assess” Phase, including historical documentation, copies of completed surveys and summaries. This information constitutes the facts upon which the future recovery plan will be based. Additional documentation may include an Executive Summary, a newly formed Work Breakdown Structure, Schedule, Budget, Risk Management Plan, Communication Plan, etc. A sample list of deliverables can be found in the PMBOK® Guide, 2000, pages 44–45.

The value of using the APPS tools is that quantitative data on team members' productivity is provided that contributes significantly to predicting reasonable outcomes. The methodology is a model that allows many factors to be quantified and then calculated to create a Productivity Index. Like “Earned Value,” the Productivity Index is used to assist in planning and creating projections related to the overall project. These tools require more iterations and tests before becoming as consistent and widely accepted as the Earned Value Models. The APPS tools are available to the project manager to transform unquantified items into a quantifiable, workable form. With “Earned Value Analysis,” quantifiable predictions can be made on total project cost. With APPS, quantifiable predictions can be made on team members' productivity.

Over the past several years in Corporate America, the author has had positive and negative experiences associated with “Emotional Decision-Making,” decision-making based on how one intuitively “feels” about an issue or subject rather than based on quantitative data or facts. In some cases, making decisions based on emotion is necessary and acceptable where the decision-maker is very intuitive, visionary, and capable of making decisions with little information and where information is not readily available (also, in some cases, pure luck); however, the author and most managers agree that the best decisions are made from quantitative information. Data-driven organizations and data-driven decisions are vital for predictable and repeatable results. The underlying basis for 3-sigma and 6-sigma Quality Control tools is quantitative data.

Now that the Plan Phase, where outcomes have been predicted and forecasted, is complete, it is time to begin the Share Phase of communicating the next steps.


Face it, there has been a significant project failure, some negative event or act of God that has put the project, project deliverables, budget or timing at risk; otherwise, the APPS Methodology and tools would not be needed.

Now that the Recovery Plan is complete, knowing what to do with the information is fundamental to fulfilling the project manager's obligation to the project. Knowing Who needs to know What and When they need to know it is vital to recovery efforts.

A Communications Plan is critical in the APPS methodology. If one has not already been created, then it's appropriate to create it now. This can be difficult in projects that occur involving multiple vendors or in a globally dispersed project teams. It's been the author's experience in managing service sector projects with multiple vendors that some organizations will not adequately communicate weaknesses in their organizations as effectively as they sell their strengths. Many articles and books have been written on the subject on Communications and how it important they are to the project's success. (See PMBOK® Guide, 2002, Chapter 10, pages 44–45.)

Bottom line, at a minimum, the Communication Plan should state:

1. What needs to be communicated (historical information, decisions, schedule impact, survey results, tools used for assessment, assumptions, etc.)?

2. Who needs it (Project Sponsor, other Executives, other Project or Program Managers, Vendors, Customers, Project Team Members, potentially affected supporting organizations, and any additional Stakeholders, etc.)?

3. When and at what frequency (daily, weekly, monthly, etc.)?

4. Who owns the creation of the communication (Project Manager, Project Team Members, etc.)?

5. Where it will be stored for future reference (Project Work Book, Shared area on a Local Area Network, etc.)?

6. How it is to be distributed (Email, Newsletters, Meetings or Debriefings, electronic or paper, etc.)?

The approach to communications will have to be agreed upon at all levels of the stakeholders. When the draft communication plan has been developed, it should then be discussed with the project stakeholders in order to obtain their consensus. Keep in mind, the stakeholders should identify any additional communications items they need to know, and when they need to know it.

With the advent of email, over-communication can bring as much detriment to the project just as too little communication. On one project, the author participated in a Rapid Assessment and Recovery of a failed project with approximately 50 project team members with seven different vendors providing multiple services. It was common to receive over 150 emails daily. The common routine was to click on the “Reply to All” command in email and send your response! In many instances, Executives and Project Sponsors from the various companies who should have been operating at strategic levels, allowed themselves to get drawn into day-to-day operational project activities and issue resolutions. The project team became more and more dependent on leadership for the simplest of decisions, and they were not attempting to solve their own problems. There are times when it is appropriate for project team members to escalate and elevate issues to Leadership for resolution. This is typically when the project team cannot resolve the issues. However, be very careful when asking for Executive level help. A cliché has been identified and used by the author: “When you decide to dance with the Bear, you need to be prepared to dance as long as the Bear wants to dance!” When Executives and Project Sponsors get drawn into the day-to-day problems and issues, their natural tendency is to want to resolve the problem for the team. The author has witnessed the best intentions of the Executives to resolve problems with less than adequate solutions while no one had the courage to challenge the solution. This can be extremely detrimental to the project.

Numerous communication tools have been developed, whether they're embedded in Enterprise Program Management Software or just the simple use of a spreadsheet. Even simpler, a large chalk board or white board that may be placed in the Project Room, Project War Room, or in a hallway in close proximity to where the project team works. Some projects are large enough to have websites developed just for the purpose of information dissemination. This concept is illustrated with the “Digital Dashboard” concept that is emerging in the Information Technology and Manufacturing Sectors today.


In the process of resurrecting a failed project or determining the appropriate response to a significant negative event, critical decisions are required. In many instances, obtaining reasonable and accurate data is difficult, if not nearly impossible. APPS is a methodology that is represented by four phases: Assess, Ponder, Plan, and Share. It captures information in a consistent and organized process covering the emotional, physical, and functional aspects of the project team members and other stakeholders involved in the project. The methodology provides a tool to input this information into planning and communication phases. It is flexible enough to be used on projects large and small projects with minimal resource consumption.

Since the project has failed, a deliverable was missed or some act of God has threatened the project, the major responsibility of the Project Manager is to instigate change. In the case of project failure, if changes are not implemented, it is important to note that many of the same results will probably occur again. It is a cliché in today's society to define insanity as “expecting different outcomes by continuing to do the same thing.” Change must occur on failed projects or the project is most likely doomed again.

One of the most noted human elements is the resistance to change. Project Managers are typically expected and empowered to institutionalize change in the project's activities. Through the use of the APPS Methodology and Tools, knowing the impact of change in productivity and change to the project will be one of the more important elements in structuring the project's future success.

There are many facets to this type of methodology and the decisions required from the information obtained. The survey and the analysis spreadsheet serve as inputs into the larger scheme of planning and communications. The method has been tried and tested yielding results within the project's tolerance, and it provided a means of planning the recovery and then working that plan. One question the author has for the readers: “Would a more detailed analysis and information associated with the process prove beneficial to individuals and/or project management organizations?” Please forward any email comments to [email protected] or [email protected].

This material has been reproduced with the permission of the copyright owner. Unauthorized reproduction of this material is strictly prohibited. For permission to reproduce this material, please contact PMI or any listed author.

Proceedings of the Project Management Institute Annual Seminars & Symposium
October 3–10, 2002 • San Antonio, Texas, USA



Related Content