Abstract
Information technology (IT) projects are still failing at an alarming rate and IT project management maturity levels are not increasing. The implication is that IT project managers are managing projects the same way over and over without taking cognizance of previous and current projects. Lessons learned should be taken into consideration during project execution as well as before the initiation of new projects. An audit was done on 717 projects to determine which best practices are followed by IT project managers and which ones are not. The results provide an indication of how IT project managers are managing their respective projects in relation to best practices. The results highlighted some serious concerns especially with regard to planning and risk management. It is evident from the results that IT project managers are not necessarily following best practices and that IT project success rates and maturity levels can be attributed to this lack of commitment. It also implies that audit results are not necessary fed back into lessons learned and that the same mistakes are made over and over. The results raise some fundamental questions about the capability and professionalism of IT project managers.
Keywords: information technology; lessons learned; project audits; South Africa
Introduction
“An audit in any project management environment should measure results and identify the contributing causes to those results” (Hill, 2007). This statement implies that learning take place during the execution of a project as well as after the completion of a project. Learning thus take place at two levels. The first level is to identify the contributing causes of project results and the second level is to identify the causes when projects are not successfully delivered. When learning takes place, then mistakes in general should not be repeated.
This seems not to be the case when it comes to the implementation of information technology (IT) related projects. Research over the past decade indicates that neither the success rates nor the maturity levels of IT projects have improved (Eveleens & Verhoef, 2010; Glass, 2005; Ibbs & Kwak, 2000; Jugdev & Thomas, 2002; Marnewick, 2013a, 2013b; Sonnekus & Labuschagne, 2003). This begs the question whether learning took place at all.
The research results reported on in this article highlight and elaborate on the fact that project managers in the IT domain do not necessarily follow the best practices and standards available to them. These results underline the facts that IT projects are only successful in 30% of the instances and that the maturity levels have not matured beyond level 3, that is, processes are standardized and integrated. This research fills the current gap between the theories around project audits and empirical data highlighting the gaps in IT project audits themselves.
The issue at hand is that there is currently little to no research available with regard to IT project audits and their related results. A literature review highlighted that project auditing is not really a topic of research. Most of the literature focusing on project auditing in general forms part of books focusing on project management and project management offices. Even project management standards do not elaborate on the issue of project audits. The available literature places project auditing under the function and role of project reviews (Hill, 2007; Schindler & Eppler, 2003; Stanleigh, 2009). The role of project auditing and project auditors is discussed, but none of the literature establishes an empirical link between the results of project audits and project success, as well as project management maturity levels. It is evident from the restrictive literature that there is a lack of empirical data confirming whether projects are adhering to the project management processes and thus contributing to the lack of project success.
The benefits of project audits are described and discussed at length in the limited theory available, but are these benefits realized in practice? Without the knowledge and wisdom of empirical data to substantiate the obvious benefits, the benefits of project audits will not be realized. This implies that organizational learning does not take place, resulting in the same mistakes and evidently leading to project success stagnation.
The aim of this research was to gather empirical data about IT projects and to determine whether project managers are following and applying best practices. Objectives flowing from the aim were to determine (1) whether there are certain areas that project managers are adhering to or not and (2) whether causality can be established between the auditing results, project success, and maturity levels. An auditing checklist was used to analyze 717 IT projects. The results were analyzed for inferential and normative statistical results. These results were used to determine correlations between various aspects of the project audit as well as to provide descriptive analysis highlighting areas of concern. This article is divided into five sections with the first section focusing on project auditing within the IT domain. The second and third sections focus on the research methodology and subsequent results. The results are analyzed and discussed in the fourth and fifth sections within the domain of IT project management, and implications of this study are identified.
Project Auditing
“An independent audit committee fulfils a vital role in corporate governance. The audit committee is vital to ensure the integrity of integrated reporting and internal financial controls and identify and manage financial risks” (Institute of Directors Southern Africa, 2009). This statement implies that project audits fulfill a vital role in project governance. Project governance provides the structure through which the objectives of a project are set, and the means of attaining those objectives and monitoring performance are determined (Turner, 2006). Project audits, on the other hand, determine whether project governance is in place, that is, determining and attaining the project objectives as well as monitoring the results of the objectives.
Project auditing originates from financial auditing, which is defined as “an independent, objective assurance and consulting activity designed to add value and improve an organisation’s operations. It helps an organisation accomplish its objectives by bringing a systematic, disciplined approach to evaluate and improve the effectiveness of risk management, control, and governance processes” (The Institute of Internal Auditors, 2001). By nature, auditing is a reactive process and the main objective of auditing is early detection of errors and fraud (Kumära & Sharma, 2011). It is important to highlight that there are different types of audit activities and also different types of auditors.
In fulfilling their duties, auditors must adhere to auditing standards, principles and the code of ethics defined by different standard-setting organizations and bodies to which they belong.
If the focus of attention moves to current project management standards, then it is obvious that there is not really emphasis on the auditing of projects. Project audits are defined as the monitoring of compliance with project management standards, policies, and procedures (Project Management Institute, 2013). Another definition is that the purpose of a project audit is to provide an objective evaluation of the project (Association for Project Management, 2006). Neither PRINCE2 nor the P2M make any reference to project auditing per se.
Gray and Larson (2008) and Hill (2007) state that project auditing enables the following:
- Monitoring of project management contributions to the achievement of the business objectives,
- Identification and response to weak and troubled project performance,
- Overseeing of quality management activities,
- Maintaining of professional and best practices within the project management environment, and
- Compliance with organizational policies, government regulations, and contractual obligations.
A distinction can be made between the various types of project audits. A project management audit provides a comprehensive examination of project management performance. The primary purpose of a project management audit is to ensure that the team and the project manager have put into place both business and technical processes that are likely to result in a successful project (McDonald, 2002). In contrast to the project management audit, the project audit represents a detailed examination of the financial and business aspects of the project. Another important audit is that of project management methodology. This audit validates the content and effectiveness of the established project management methodology. The highest ranked benefit of this type of auditing has been found to be client confidence, followed by enhanced accountability, reduced project costs, and disputes in that order of significance (Sichombo, Muya, Shakantu, & Kaliba, 2009).
Project audits, conducted routinely during project execution and after project closeout, can help ensure project success through the identification of major risks that are likely to be faced by the project manager, stimulating the project manager and the project team to address the risks before it is too late to have a positive impact (McDonald, 2002). A project audit is of no value to the organization when the audit findings are not addressed. The purpose of the project audit findings is to identify lessons learned that can help improve the performance of a project or of future projects (Stanleigh, 2009). The lessons learned can be applied to all new and current projects within the organization as well as the development of strategies that will increase the likelihood of future projects being managed successfully.
A theoretical project audit process can be summarized as per Figure 1.
Figure 1: Theoretical project audit process
Project audits should take place during the project life cycle. The results of the project life cycle are incorporated into the project, ensuring the successful completion of the project itself as well as the delivery of the project product. A post-project audit is performed as part of the lessons learned exercise and these audit results are then incorporated into future projects that will increase project success and ultimately the success of the organization.
There is, however, a flaw in the presentation of the literature as well as that of the theoretical project audit process. The flaw is that the results of project audits are not incorporated into learning and the same mistakes are made over and over again. This is evident from two perspectives, that is, project success and project management maturity levels.
Marnewick (2012) reported on a longitudinal study focusing on the success rates of ICT projects. The results of this study are illustrated in Figure 2.
Figure 2: IT project success rates (Marnewick, 2012)
It is evident from these results that the delivery of successful IT projects is still eluding organizations. These results are in line with those of the Chaos Chronicles (Eveleens & Verhoef, 2010; Glass, 2005). The Chaos Chronicles report that on average 30% of all IT projects are perceived as failures.
The second perspective focuses on the maturity levels of IT projects. Ibbs and Kwak (2000) have determined that the maturity level for the information system industry is 3.06. A longitudinal study by Marnewick (2013a) confirms these maturity levels as per Figure 3.
Figure 3: IT project management maturity levels (Marnewick, 2013a)
Once again, it is evident that IT projects have not matured beyond level 3. Given the results shown in Figures 2 and 3, it can be concluded that learning does not take place, and the results of project audits are not incorporated into institutional learning. The successful delivery of projects is the ultimate goal of any organization as this ensures its ultimate success and survival.
The following four research questions were developed based on the literature review:
- Are project managers following best practices?
- Will the results from the project audits shed some light on the underlying reasons why projects are failing?
- Is there a relationship between the audit results and the project success factors/criteria?
- Is there a relationship between the audit results and the maturity levels?
The next section contains a discussion of the research methodology that was used to answer the four research questions.
Methods and Materials
To conduct a project audit effectively, it is suggested that a comprehensive analysis be conducted in five areas: technical objectives, cost and budget, human resources, project termination, and technical and managerial implications of the project (Roman, 1983). The elements in each of the five areas are subject to variation and degree of intensity of importance, depending on the nature of the project.
A questionnaire was circulated among a purposive sample of potential respondents. The population consisted of IT or related projects conducted in various organizations and industries.
Each respondent was requested to provide information on various projects they had been involved in. Data regarding 717 projects of various durations, budgets, and levels of complexity was received and analyzed.
The questionnaire was divided into three sections:
- General project information, which covered the cost and budget information.
- Section A requested project information to be provided on the following:
oProject goals,
oProject activities,
oProject management,
oHuman resource management, and
oRisk management.
- Section B requested project information to be provided regarding:
oLeadership style,
oDirecting operations,
oProject reporting, and
oProject execution.
Sections A and B comprised 102 questions related to project management practices. The respondents needed to indicate whether a particular practice was employed in a particular project or not. The results were tabulated to provide a project audit score out of a possible 102. The results for all projects and individual steps were tabulated to provide a basis for descriptive statistics as well as correlations.
Results
The average project in the sample scored 71.3% or 72.8 out of 102. This reveals that IT project managers only applied 71.3% of recommended practices.
When increasing the level of detail, Figure 4 shows how each of the nine areas of the audit contributed to this total average:
Figure 4: Averages of all areas
Clearly the areas of project goal, project management, and project execution contribute to a higher project audit score as the majority of project activities are performed by project managers. Activities relating to the definition of the project goal, the presence of formal methodologies, the project manager’s competency, and project management support seem to be under control in the majority of IT projects audited.
However, grave concerns arise from project activities, human resource management, and risk management. Project activities questions related to the creation of a work breakdown structure (WBS), methods for scheduling, and use of milestones in planning. Human resource management questions pertained to information regarding the planning for human resources and the practice of resource leveling. Risk management questions were included to determine which practices are employed in the management of risk. All of these areas performed below the average of 71.3%.
The results for each individual area are analyzed and discussed in the following sections.
Project Goal
It seems as if the project managers were very diligent in applying best practices. This area scored an average audit score of 85%. Goal setting scored consistently over the total audit average. This area was divided into six sections.
The first section focused on how well the project goal was defined, as illustrated in Figure 5.
Figure 5: Clarity of goal statement
The results in Figure 5 indicate that 80% - 90% of all the audited projects have a clear project goal. This means that the project managers know exactly what should be delivered as a final project deliverable.
The second section focused on the project goal itself and determined how it conformed to the SMART principle (Clements & Gido, 2008). The results are depicted in Figure 6, and it seems that in 86% - 95% of the audited projects, the project goal conformed to the SMART principle.
Figure 6: SMARTness of the goal statement
It is very interesting to note that respondents considered the stated goals as being realistic and achievable, and they perceived the project goal as well described in terms of cost, quality, and time.
It is also evident from Figure 7 that the respondents were very confident (91%) that all of the deliverables resulting from the project had been identified and documented. However, ancillary consequences resulting from the project had not been identified. This may indicate that the stakeholders were very much more concerned about results internal to the project, rather than what effect the project might have on the external environment.
Figure 7: Project consequences and deliverables
Compared to the overall project score of 71.3%, projects performed very well in ensuring that project completion criteria were defined. This was the focus of section 4. It is, however, noteworthy that both these measurements scored below the area average of 85%.
The results from Figure 8 correspond with those from sections 1 and 2 where it was found that the project goal was well defined and adhered to the SMART principle.
Figure 8: Project completion criteria
The next section deals with establishing assumptions, dependencies, and critical success factors (CSF) as per Figure 9. It does seem that documenting CSFs has been neglected in a moderate sense. It is also clear that projects are not viewed in relation to other projects and how projects affect one another.
Figure 9: Assumptions, critical success factors and dependencies
The last section focuses on the project stakeholders. Best practices regarding project stakeholders in planning were also investigated and the results are depicted in Figure 10.
Figure 10: Project stakeholders
Best practices regarding project stakeholders in planning are diligently performed when compared to the overall project average of 71.3%. However, only the identification of all stakeholders outperforms the project goal area’s average of 85%. The activities relating to stakeholder needs and requirements seem to be lacking. A concern does arise that requirements resulting from stakeholder needs were only included 80% of the time, whereas project managers were sure that they had included all interested stakeholders 96% of the time. The same concern is raised for identifying stakeholder needs and garnering their approval for project scope and goals. This could mean that although stakeholders are readily identified, they are not necessarily consulted with the same rigor.
Area 1 scores compared very well to the overall project average, but there are certain areas of concern, especially where the project managers need to identify what the external effect of the project is likely to be.
Project Activities
The area of project activities achieved on average a score of 67%. This area focuses on the planning of project activities such as establishing a WBS, developing the project schedule, and defining milestones.
Figure 11 shows the results from the section focusing on the WBS and it is evident that this section shows some concerns.
Figure 11: Work breakdown structure
Respondents indicated that few of their WBS contained all relevant assumptions or project management effort. Neglecting to include this information may give rise to actual variances during the execution phase. Even though an organization expends little effort during a project’s planning phase, the project manager spends a great deal of time and energy during this phase (Schwalbe, 2010). Conventional project management wisdom holds that the increase of project planning efforts provides superior results (Steven, Joseph, & Fred, 2011).
It is a best practice to include a WBS dictionary detailing the approach, deliverables, and estimates of each work package (Schwalbe, 2010). Slightly less than two-thirds of projects utilized a WBS dictionary. The Project Management Institute (2006) states that the WBS dictionary is a key document that accompanies the WBS and carries critical project information. Given the fact that only 36% of the audited projects had a WBS dictionary, the question should be raised as to how much critical project information is lost.
Scheduling is a logical output of the WBS. The results for the scheduling of projects are shown in Figure 12.
Figure 12: Scheduling
The scheduling of IT projects indeed raises concerns. Firstly, one-third of the projects did not use a visual representation of the schedule. This may impact the clarity of communication and reporting once execution has started. Secondly, many schedules did not indicate the effort required to complete certain phases. This may impact the preparedness of the responsible parties when they plan for resources to be available. Thirdly, the WBS for many projects does not contain a clear link to the schedule. Since a schedule is derived from the WBS, it is then unclear how these schedules were developed and planned for (Burke, 2010; Clements & Gido, 2008; Marchewka, 2012).
A WBS should have milestones at certain key points within the project. Respondents gave the following indications in the establishment of milestones:
Figure 13: Milestones
Most projects included a series of milestones to assist in monitoring and controlling the project. It is, however, clear that many projects did not include milestones for project reviews where the sponsor provided ongoing input regarding project performance. This may result in sponsor concerns being raised very late which could impact completion times or overall project success. It also implies that lessons learned were not applied at these key points in the project.
When a cross-tabulation is done between the involvement of the sponsors at certain milestones and whether all the stakeholders have been identified, then Table 1 paints an interesting picture.
Table 1: Stakeholder and milestone planning correlations
| Do the milestones indicate project reviews, requiring project sponsor input? | ||
| Total | Percentage | |
| Have all stakeholders with an interest in the project been identified & documented? | 467 | 65% |
| Have all their needs been captured in the project goal statement? | 410 | 57% |
| Have all the requirements consequent from their needs been included in the goal statement? | 403 | 56% |
| Have they reviewed and approved the project scope and goal(s)? | 422 | 59% |
Only 65% of the audited projects involved the identified stakeholders at key decision points such as project reviews. The remainder of the results in Table 1 highlight an area of concern with less than 60% of the audited projects indicating a correlation between the needs of the stakeholder and the verification of these needs. It seems from the results that, in 40% of the cases, the stakeholders’ needs were captured, but the stakeholders were not involved during project reviews to determine whether the projects were still meeting these needs and requirements. The implication is that, although the respondents claimed that they identified and involved project sponsors as per Figure 10, these project sponsors are not involved in milestone decisions. This raises the question of the continuous involvement of the project stakeholders.
This area of the audit did not measure as well as the others. As this area is concerned with the establishment of a schedule, this should raise serious concerns. If a project is managed using a deficient schedule, project success would be compromised.
Project Management
This area audited the project manager’s competencies and abilities as well as the status of project management as a discipline within the organization. This area achieved a rating of 79%. The aspects investigated revealed the results shown in Figure 14.
Figure 14: Project management
It is interesting to note that more than a quarter of projects operated in an environment where no formal project standard or methodology has been adopted by the organization. These standards and methodologies provide best practices on what needs to be done in order to complete a successful project. Only two-thirds of projects clearly defined each team member’s roles and responsibilities. This obviously leaves room for error where an individual is not clear about his or her role in a project. The fact that less than two-thirds of projects had a formal governance structure should also give pause. Governance guides behavior to achieve a desirable objective (Bevir, 2013; Muller, 2009). Without the assistance of formal governance structures, it may become problematic to ensure that desirable behaviors are exhibited by team members that would contribute to project success.
When seeking the relationship between project management best practices in this area and developing a schedule, it was surprising to find weak relationships.
Table 2: Cross-tabulation between planning best practices and a formal methodology or standard
| Does the organization have a formal project management methodology or standard? | ||
| Total | Percentage | |
| Does the WBS contain an estimate of the effort (work) that each task requires? | 390 | 54% |
| Does the WBS contain project management effort? | 365 | 51% |
| Is there a WBS dictionary that supports the WBS? | 217 | 30% |
No strong relationship exists between these aspects, indicating that there is little correlation between how a project schedule is developed and what project management best practices prescribe. Given the fact that project time management as a Knowledge Area explicitly focuses on the WBS, scheduling, and the WBS dictionary, then it is quite disheartening to realize that the audit reveals disturbing results.
Only 54% of all the project managers that did have a formal methodology in place had a WBS with task estimates. This implies that almost half of all the audited projects were executed without being estimated. It also means that these project managers ignored the best practices as prescribed by the formal project management methodology or standard of the organization. This directly addresses the ethical behavior of the project managers and project governance within the realm of corporate governance. The behavior of the project managers becomes more disturbing when it is realized that only 30% of the audited projects did have a WBS dictionary.
It can be concluded that project management as an audited area did fairly well but there are some concerns that need to be addressed. This is especially the case where organizations do not follow a formal project management methodology or standard. Another concern is the lack of a governance structure. A governance structure will ensure that the project managers adhere to the best practices as prescribed.
Human Resource Management
This area of the audit deals with issues regarding the planning for human resources in the project. Respondents were asked to indicate what best practices they employed in estimating resources and resource leveling. Human resource management achieved an average rating of 65%, indicating that human resource management has serious concerns that need to be addressed (see Figure 15).
Figure 15: Resources
When evaluating available resources, it becomes evident that not much consideration has been given to training requirements. Only half of all projects deemed it necessary to conduct training needs assessments. It does, however, appear that if a training assessment is done, it should follow that stakeholders will then receive appropriate training.
There is a strong significant correlation (p = 0.796, r = 0) between a completed training needs assessment and training received. This indicates that there is diligence in providing training where a need is identified, and project planners can be commended for this.
When exploring the relationship between the remaining human resource factors and the WBS as a planning tool, the results are indicated in Table 3:
Table 3: Cross-tabulation between WBS and human resource factors
| Does every activity in the WBS have a team member’s name against it? | Have project assignments been based upon individual strengths & weaknesses? | |
| Does a list of ALL the tasks to be done to complete the project exist? | 67% | 60% |
The cross-tabulation in Table 3 indicates two concerns. The first concern is that in only 67% of the audited projects, all the activities have resources allocated to them. This begs the question: who is executing the activities in the remaining one-third of the audited projects? If no resource is allocated, then the activity cannot be executed and the project cannot be delivered. A second concern highlighted by the audit is that only in 60% of the projects, resources are allocated to activities based on skills. How are resources allocated in the other 40% of the projects? Is there a process in place or is it a direct consequence of the IT skills shortage in South Africa?
The second section of human resource management focuses on resource leveling. Resource leveling forms part of the Human Resource Management Knowledge Area as per the A Guide to the Project Management Body of Knowledge (PMBOK® Guide) (Project Management Institute, 2013). It does seem that resource availability over the duration of the project has not been strongly taken into account (see Figure 16). This raises the question whether the project managers have the adequate skills and knowledge to apply project management standards.
Figure 16: Resource leveling
Many times it seems that project team members’ availability was not considered in terms of their personal and professional commitments. South Africa’s many public holidays, as well as religious periods, were not taken into account when developing a project schedule. This will lead to days lost and schedule slippage. Many of the audited projects did not account for these fluctuations in resource availability through resource leveling. This may elevate the resource cost or extend the duration of the project. Resource leveling serves to make the most efficient use of resources over the project duration (Clements & Gido, 2008).
There is an indication that human resource management is not done efficiently in projects. The time lost and expenditure wasted may impact the possibility of future project investments and may cause the project to fail on the technical grounds of budget and schedule. In the worst case scenario, it may even be possible that poor resource planning could lead to total unavailability of resources, causing the project to be abandoned as a total failure.
Risk Management
This area achieved the lowest rating of all the sections, with an average of 51%. Figure 17 highlights these low averages ranging from 68% for the identification of risks to 28% for the appointment of a project risk manager.
Figure 17: Risk management
The importance of risk management in projects cannot be overstated. Risk management forms part of the quintuple constraint that determines technical project success (Clements & Gido, 2008) and entire sections of PMBOK® Guide are dedicated to it (Project Management Institute, 2013). Project managers that neglect to plan for risk are severely handicapping the probability of success in their projects. The proper management of risk contributes to the success of the project as it allows project managers to think of mitigation strategies (Camilleri, 2011).
The aspects of particular concern are that risk management actions are not entered into either the WBS or project schedule. However, many respondents indicated that there was no clear link between the WBS and the schedule in any event, as per Figure 12. The implication is that the majority of project managers fail to take risks into account in developing the WBS and the schedule.
The audit results again highlight the aspect that project managers are not doing what they are supposed to do and are not following the best practices. Project Risk Management as a Knowledge Area clearly indicates that risks should be continuously identified and classified. The average of 50% for this area highlights the total disrespect for Project Risk Management as a Knowledge Area.
Leadership Style
This area seeks to audit the leadership style of the project manager. This is also the first area of the implementation section as per the questionnaire. This area has an average of 72%.
Figure 18: Leadership style
Three concerns emerge from Figure 18. Firstly, less than two-thirds of project managers performed a stakeholder analysis. This analysis would assist the project manager in managing the expectations of the various stakeholders and would also improve project communication (Schwalbe, 2010). It is also contradictory to the PMBOK® Guide, which added a tenth Knowledge Area, namely Project Stakeholder Management (Project Management Institute, 2013). This Knowledge Area focuses on the identification of people that could impact or be impacted by the project.
Secondly, most project managers did not use reward or recognition in attempting to motivate project team members. Reward or recognition systems are used to promote and reinforce desired behaviors and this can only be effective when rewards and recognition are based on a team member’s activities.
Thirdly, in only 38% of the audited projects was a personality analysis performed on team members. If no such analysis is performed, it would be difficult for the project manager to understand what motivates each team member and how each team member’s personality contributes to project success (Cohen, Ornoy, & Keren, 2013).
It is, however, clear that the respondents felt that the project managers understood the organization they operated in. This is indicated by the high ratings achieved in whether or not the project manager understood the organization’s structural, political, human resources, and cultural frames. This should lead to project managers taking the entire organization into account when planning a project. A cross-tabulation between these four aspects and the question regarding interdependencies with other projects is reflected in Table 4.
Table 4: Project manager understanding of the organization and interdependencies with other projects
| Have all the interdependencies with other projects been identified and documented? | |
| Does the project manager have a good understanding of the organization’s structural frame (organizational chart)? | 63% |
| Does the project manager have a good understanding of the organization’s human resources frame (balance between organization and individual needs)? | 56% |
| Does the project manager have a good understanding of the organization’s political frame (power and leadership)? | 56% |
| Does the project manager have a good understanding of the organization’s cultural frame (business and operational)? | 59% |
Project managers need to understand the project they are managing as well as where this project fits into the larger organization. The audit results indicate that, in 40% of the projects, this was not the case. The results in Table 4 clearly indicate that although project managers understood the structural, political, and cultural frames of the organization, they did not see the interdependencies with other projects within the same organizational frames.
This may indicate that an understanding of the organization will have very little or no bearing on whether or not other projects are taken into account for planning purposes. This very internal and individualistic characteristic undermines coordination within an organization. This may lead to resource availability issues later on in the project where many initiatives may be vying for the same resources. This in turn may lead to project managers playing a political game within the organization and will force leadership to assign resources to prioritized projects, causing lower-priority projects to fail.
When exploring the question of stakeholder analysis, the correlations in Table 5 were found regarding project stakeholder factors from the first area.
Table 5: Stakeholder analysis and stakeholder factors
| Has the project manager performed a formal stakeholder analysis? | ||
| Have all stakeholders with an interest in the project been identified and documented? | Pearson correlation | 0.068 |
| Significance (2-tailed) | 0.070 | |
| Have all their needs been captured in the project goal statement? | Pearson correlation | 0.225* |
| Significance (2-tailed) | 0.000 | |
| Have all the requirements consequent from their needs been included in the goal statement? | Pearson correlation | 0.074** |
| Significance (2-tailed) | 0.046 | |
| Have they reviewed and approved the project scope and goal(s)? | Pearson correlation | 0.160* |
| Significance (2-tailed) | 0.000 | |
* Correlation is significant at the 0.01 level (2-tailed).
**Correlation is significant at the 0.05 level (2-tailed).
Clearly, there is almost no correlation between a performed stakeholder analysis and the identification of stakeholders, their needs, or their approval of the scope. This may imply that when stakeholder analysis is performed, it may be ineffective or simply not referred to ever again in the management of a project. A stakeholder analysis serves the purpose of ensuring that all relevant stakeholders’ needs, requirements, and communication criteria are included for project planning purposes (Schwalbe, 2010). Failure to do so may have a detrimental effect on planning and delivering a project that will meet the needs and requirements of the users of a project deliverable and may lead to project failure.
Directing Operations
This area rates the aspects of how projects are controlled and monitored. This area also averaged at 72% but significant variances were detected in subsections. The first subsection focuses on the project plan as a means to an end, which is the successful delivery of the project. The results are shown in Figure 19.
Figure 19: Using the project plan as instrumentation
The first two audited activities yield good averages, implying that project managers confirmed the status of the project on a daily basis. This is good practice as it will alert the project manager if there are any deviances from the schedule.
A concern is that in 26% of the audited projects, the project managers were not using a project management tool. This raises the question as to what tool was used to track the daily progress of the project as confirmed in the first two questions. Another concern is that 42% of the project managers did not look ahead regarding the detailed schedule. The impression is created that this is not an activity that project managers are engaging in. The benefit of this activity is that the project manager can plan for future issues and risks when a long-term view is implemented.
When a cross-tabulation was performed between the project manager’s focus in the next 6 to 8 weeks and when all the tasks are in the schedule, a shocking picture is painted. Only 52% of all the projects had a list of all the activities to be focused on the next 6 to 8 weeks. This may indicate that either the task list is transposed into a weak WBS that is not being used for future planning of near-term tasks, or that not all tasks are in fact included. Project managers also did not seem to report on the health of their project based on the task list. This may lead to the mistake of mainly reporting on work-in-progress, rather than on actual tasks completed (Clements & Gido, 2008).
When a cross-tabulation is done between the questions Does a visual chart or representation showing work tasks phased over time exist? and Does the project manager use a project management tool to record regular progress on current project activities?, then a visual chart is used by the project manager to track the status of the project in only 60% of the projects that did have this chart. This again raises questions: What is used to track the progress of a project? Secondly, if a visual chart does exist, why is it not used? The use of dashboards, charts, and other such visual aids assists stakeholders who have little project management experience in understanding current events in a project (Turban & Volonino, 2012). This aids communication and will assist the project manager in effectively and clearly relaying messages to the project team and stakeholders.
However, given the fact that very little attention is paid to developing a stakeholder analysis, the project manager may not be aware of what the relevant message to a particular stakeholder should contain. This lack of communication is further confirmed by the fact that many projects did not have formal change control processes, as depicted in Figure 20.
Figure 20: Project monitoring, tracking, and control
Change control processes ensure that only formally accepted changes affect the project (Clements & Gido, 2008). Informal changes are not documented and do not ensure that the project schedule, budget, or scope is updated to reflect these changes. This gives rise to what is known as scope creep (Schwalbe, 2009) and may eventually lead to a bloated scope or unauthorized time extensions that may very well lead to project failure.
These change control aspects are correlated against the creation of a WBS and schedule in Table 6.
Table 6: Correlation between WBS and schedule factors with change control factors
| Does the schedule show the amount of work/effort in each phase? | Is there a clear link between the WBS and the schedule? | ||
| Is what the project plan says should be happening reflected in the actual project? | Pearson correlation | 0.211* | 0.264* |
| Significance (2-tailed) | 0.000 | 0.000 | |
| Is there a formal, documented change control management process within the project? | Pearson correlation | 0.330* | 0.287* |
| Significance (2-tailed) | 0.000 | 0.000 | |
| Is the formal, documented change control management process followed? | Pearson correlation | 0.273* | 0.240* |
| Significance (2-tailed) | 0.000 | 0.000 | |
* Correlation is significant at the 0.01% confidence level (2-tailed).
Weak significant correlations exist between scheduling and WBS aspects and change control aspects. At least there is some relationship between the established change control processes and the schedule. The weak correlations may indicate that this is not managed effectively and could lead to unauthorized and undocumented changes being implemented that may influence the project negatively.
Although the directing operations factor seems to be performing on average, it is clear that the underlying issues revolving around change management are very serious. This again raises the issue of governance on the projects.
Project Reporting
This area investigates aspects of project communication and, more particularly, reporting of progress in project implementation. This area was rated an average of 71%, as shown in Figure 21.
Figure 21: Project reporting
Clearly, the persistent problem of poor communication is reflected in the results shown in Figure 21. Many projects did not have a formal communications plan or the reports are not distributed regularly. The reports may be ineffective by not providing a historical comparison. Most project managers had no mechanism for ensuring that reports with important information were actually being read. The project communication process is not being managed. Previous research has indicated that communication is a very important factor for project success (Erasmus & Marnewick, 2012). Project leadership must be commended for ensuring that project reports are disseminated as much as they are; however, the quality and effectiveness of this communication remain dubious.
Project success is dependent on the lessons learned during project implementation. The results shown in Figure 21 highlight some areas of concern, all relating to reporting. Reporting highlights issues and concerns, and assists the project manager and team in learning from mistakes so that these unfortunate events can be prevented in the future. It must be noted that 83% of the projects stated that daily tracking of activities was done. A cross-tabulation based on the results in Figure 19 and the areas of concern in Figure 21 resulted in the following:
Table 7: Cross-tabulation between daily activities and reporting activities
| Does the project manager’s daily work practice include checking on tasks starting, tasks finishing and tasks in progress? | |
| Are project status reports issued weekly? | 60% |
| Are project status reports properly structured with a clear account of the activities for the week? | 59% |
| Do the project status reports contain historical information? | 54% |
| Is the project performance reported on in terms of business benefits? | 53% |
The results show that between 53% and 60% of the project managers included the daily tracking of activities into status reports. The concern is once again that 40% - 47% of the project managers did not follow best practices and did not relate the daily tracking of activities into informative reporting.
The last area of the audit focuses on project execution.
Project Execution
For this area, respondents were questioned on how the project was executed. This area was rated 80% by the respondents.
Figure 22: Project execution
This area performed relatively well except for one specific concern: that of managing change. Wholly apart from the fact that changes were not well documented in the area of directing operations, it is clear that changes were not well controlled. Figure 20 already raised the issue that only 60% of the project managers followed change management processes. Figure 22 emphasizes this phenomenon where only 54% of the audited projects employed the best practice of controlling changes during the execution of a project. Table 8 shows the cross-tabulation between Figures 20 and 22, focusing on the change management processes.
Table 8: Change control factors and project changes
| Were changes to the goal properly controlled and plans adjusted accordingly? | ||
| Total | Percentage | |
| Is the formal, documented change control management process followed? | 313 | 44% |
| Is there a formal, documented change control management process within the project? | 310 | 43% |
The results in Table 8 are quite disturbing. It seems as if IT project managers do not care or worry about change. This is especially concerning in an industry where changes happen frequently in terms of user requirements and the IT environment itself. Change control and its related processes form part of project integration management and should be managed throughout the project life cycle. It is quite disheartening that project managers in the IT industry do not adhere to these best practices.
It is to be noted that all the steps related to project implementation scored on or around the average for total project audits. The main variances are caused by the planning phase steps.
Overall, an interesting trend emerges that there are strong correlations between factors within a specific area. However, as soon as a relationship is sought between factors from different areas, as demonstrated, the relationships break down. This may indicate very little integration of project management best practices across the whole project.
Discussion
The audit results do not bode well and highlight concerns within the IT project management discipline. The first concern highlighted by the audit results is the lack of compliance by IT project managers. There is more than enough evidence that IT project managers are not following best practices. These best practices form part of standards and methodologies and provide project managers with the necessary processes to manage a project. This lack of compliance raises concerns about project governance and ultimately corporate governance. Given the emphasis that project governance has been receiving over the last couple of years, it is disconcerting that IT project managers are not complying with project governance.
A second concern is around the notion of the WBS, scheduling, and milestones. The WBS and subsequent schedule form the basis of all other project management activities, such as costing and resource allocation. If the audited projects do not have a complete list of all the activities, this jeopardizes the success of the project. Omitting activities leads to changes that have an impact on costing and duration. It might be that the complexity of IT solutions contributes to this incompleteness and could be an area for further research, or it might be that IT project managers are not diligent in creating a WBS and schedule.
Marnewick (2013a) states that Project Risk Management is the least mature Knowledge Area and has never gone past the maturity level of 3. This is evident from the auditing results as well, where risk management activities are not performed. It can be concluded that risk management in totality is ignored by IT project managers. Again, the question is: why is this the case? It could be that IT systems are not critical or life-threatening when they crash or it might be that IT project managers are not doing what they are supposed to do.
A fourth concern is resource management. Resources are not allocated to activities and resource availability is not taken into account. This is a serious oversight by the IT project managers as it is a well-known fact that there is an international skill scarcity of IT skills. Not allocating resources places project success under jeopardy. Why is this not done if processes are provided by the PMBOK® Guide (Project Management Institute, 2013)?
Four research questions were raised and from the results and evidence provided, it seems as if the overwhelming conclusion is that the low IT project success rates and maturity levels can be attributed to IT project managers not performing their fiduciary duties. Again, why is this the case?
It is evident that more questions are being raised than answers are provided. Are these results forming part of lessons learned and ultimately part of organizational learning in project management? It makes one wonder.
Conclusion
The literature indicates that a project audit can provide insight into which practices are followed by project managers and which are not. However, the result of the project audit evidently does not contribute to organizational learning. If project managers did indeed learn from their mistakes, then the average ratio of successful ICT projects would not have remained in the vicinity of 30% as per previous studies (Erasmus & Marnewick, 2012).
The same can be said of project management maturity stagnating at level 3, decrying the integration of formalized practices (Marnewick, 2013a). Clearly, if learning did take place, the status quo would have changed before now.
The average audit rating of 71.3% is not benchmarked against any standard in literature, as none exists. This rating merely indicates that 71.3% of project management best practices were indeed employed by project managers in this study. At first glance, a rating of 71.3% seems acceptable for an industry. However, an audit rating of 71.3% cannot be accepted, given that the average industry project success rate is around 30%. It may very well be that the 28.7% of practices that are not regularly employed can make all the difference in vastly improving project success. It must also be noted that respondents were not queried on how well they performed the 71.3% of project management best practices. It may be the case that even though 71.3% of best practices were performed, these were not being performed well at all.
A third possibility could be that the best practices identified are not best practices at all. This disconcerting notion opens up the possibility for professional project managers to critically evaluate every action taken to ensure that it adds value and increases the chances of project success.
Change may have been viewed as something to avoid in the management of projects. And while unnecessary and unauthorized changes are definitely not desirable, it seems that change can induce an improvement in how many best practices are performed. The reason may be that the particularly traumatic changes of project leadership and project goals force a greater focus on such a project. This could have the effect of all stakeholders experiencing increased pressure to perform and perhaps being placed under closer scrutiny. This may indicate that a greater level of ethics and integrity is required in order to achieve greater project success.
The establishment of a project goal seems to be done well across respondents and projects. Having a clear goal in mind ensures that the project gets off on the right foot. However, this momentum seems to be lost very soon once the initiation and planning phase gains complexity or is concluded.
The area of Project Risk Management performed particularly badly. Effective Project Risk Management is addressed from every project management arena as one of the greatest factors to influence project success (Clements & Gido, 2008; Erasmus & Marnewick, 2012; Schwalbe, 2009, 2010). Unfortunately, for some reason, project managers do not practice what they preach. This is evidently the least implemented area of practice in the industry and urgent attention it is required if increased return on investment is to be seen in projects in general.
The solution may lie in ensuring that project best practices are adhered to and implemented by way of a governance model. More than that, the quality of performed practices needs to be scrutinized through a similar mechanism. Additionally, real organizational learning must take place for improvement to realize. Mechanisms need to be put in place to ensure that feedback is indeed assimilated and applied in future projects. A governance mechanism may be established to ensure that learning takes place as part of its mandate.
Further research is required to understand how these best practices relate to project success factors. Additionally, research into a new mechanism that stimulates project success is required. The question of how organizations can gain greater momentum to break into higher levels of project management maturity remains open. More importantly, organizations need to learn how to leverage project management maturity in order to achieve greater success in projects.
