Abstract
Because of resource limitation as well as the nature of the projects that are being implemented, project managers in many organizations are tasked to lead more than one project at a time. In this research, they are referred to as multiple-project managers. While this multiple project management (MPM) practice has been used widely in industries, MPM research at the project manager level is still limited. The objective of this empirical study is to identify the factors influencing the effectiveness of MPM. Previous research has found that to be successful in simultaneously leading different projects, project managers need supports from their organization, such as reasonable assignments, sufficient and sustainable resource allocation, and specific organizational culture. They also have to possess a distinct set of competencies and have a good utilization of project management processes. Based on a survey with IT project managers of one of the world's largest financial institutions, the author found that teamwork-oriented culture and competency of project managers are important factors for effective MPM practice.
Introduction
Multiple project management (MPM) is referred to in this study as a management practice in which a project manager is assigned to simultaneously lead multiple projects. With resource limitation, this practice has been popular in many organizations, since it helps improve efficiency in managing projects. Typically, a multiple-project manager leads projects that are not mutually dependent in terms of objectives and goals (Archibald, 1975, Ireland, 1997). An example is the assignment of a product improvement project, an internal process improvement project, and a small IT upgrading project to one project manager. These managers are tasked with making decisions lower in an organization hierarchy and have interrelationships with multiple functional units from which they draw resources (Galbraith, 1994). While MPM was referred to in this study as the management of multiple projects at a project manager level, other researchers took an organizational view and recognized MPM as portfolio management (Fricke & Shenhar, 2000; Pennypacker & Dye; 2002; Platje, Seidel, & Wadman, 1994) and the management of multiple projects at a functional level, a.k.a. managing resources across multiple projects (Engwall & Jerbrant, 2003; Levy & Globerson, 1997; Liz, 2000)1.
At a project manager level, simultaneously leading multiple projects can be a significant challenge. In dealing with multiple projects at the same time, a multiple-project manager leads several teams for projects of different objectives. A multiple-project manager also faces a challenge of switch-over from project to project (Fricke & Shenhar, 2000). While multiple-project managers share many characteristics with project managers who lead one project at a time, with the aforementioned challenges, the assumption that managing several concurrent projects is no more than the sum of managing individual projects may appear to be incorrect (Levy & Globerson, 1997).
In the research domain, empirical research on management of multiple projects, especially at the project manager level, is limited and well behind its rate of utilization in the industry. With the challenges of managing multiple projects, the focus of this study is to answer the question “What is needed to enhance the effectiveness in managing multiple projects?”
Literature Review and Research Hypotheses
Literature related to the factors influencing MPM effectiveness is reviewed in this section, which is arranged into three parts. The first part discusses factors in the organizational level that influence the effectiveness of MPM. The second part focuses on the influencing factors at the operational level. The third part reviews the factors used to measure MPM effectiveness.
Influencing Factors at the Organizational Level
In the literature, several authors have discussed issues related to the organizational context of MPM, especially, the issues regarding interactions between projects and their organizational environment. To promote effectiveness at the organizational level, researchers focus on the need for realistic portfolio management, typically in conjunction with formal project selection, prioritization, and resource allocation (Fricke & Shenhar, 2000; Payne, 1995). Projects should be selected based on their relevance to the organization's business objectives, the appropriateness of the project size, the schedule and technical feasibility, the financial viability, etc. Along the same line as Payne (1995), Adler, Mandelbaum, Nguyen, and Elizabeth (1996) emphasized that it is very ineffective when an organization implements too many projects to be handled by its available resources.
After the selection process, projects must be properly assigned to project managers. Kuprenas, Jung, Fakhouri, and Jreij (2000) proposed that the effectiveness in MPM depends on the number of projects a multiple-project manager leads at a time. In manufacturing support environments, a study found that assigning two to three “major” projects to an engineering project manager is an effective maximization of his or her productivity (Fricke & Shenhar, 2000). With regard to the assignment process, researchers assert that an effective assignment process should consist of the steps toward the understanding of project priority, the match between competencies of multiple-project managers and project requirements, and the recognizing the organizational/personal limitations (Patanakul, Milosevic, & Anderson, 2007; Patanakul & Milosevic, 2006). Such a process enhances MPM effectiveness (Patanakul & Milosevic, 2009). Having projects that match with their expertise and with an appropriate workload, multiple-project managers would be more effective in leading projects to their success. Thus:
Hypothesis 1: The greater the effectiveness of project manager assignments, the greater the MPM effectiveness.
Besides the assignment of the project manager, the allocation of resources is also another managerial challenge (Engwall & Jerbrant, 2003; Fricke & Shenhar, 2000). Several studies proposed tools and techniques for scarce resource allocation, which include integer programming, heuristic methods, queuing theory, and so on (Dean, Denzler, & Watkins, 1992; Hendriks, Voeten, & Kroep, 1999, Levy and Globerson, 1997, Morse, McIntosh, & Whitehouse, 1996). However, these techniques were proposed for use at the functional level— the allocation of functional resources across multiple projects. Regardless of resource allocation tools and techniques, research has also shown that having sufficient and sustainable resource leads to the effectiveness in MPM (Patanakul & Milosevic, 2009). Thus:
Hypothesis 2: The greater the resource sufficiency and sustainability, the greater the MPM effectiveness
At the organizational level, studies have shown that organizational culture plays a critical role in supporting the organization performance and effectiveness (Barney, 1986; Cooke & Rousseau, 1988; Schein, 2004). Denison and Mishra (1995) proposed that the culture that includes involvement, consistency, adaptability, and mission is a significant predictor of quality, employee satisfaction, and overall performance of the organization. With a focus on project management, studies have shown that organizational culture has an impact on project management activities (Chapman, Ward, & Harwood, 2005). The teamwork-oriented culture that supports commitment, communication, teamwork, and reward for performance seems to have an impact on team effectiveness (Abdel-Hamid et al., 1994; Campany, Dubinsky, Druskat, Mangino, & Flynn, 2007; Denison, Hart, & Kahn, 1996; Jassawalla & Sashittal, 2000; Jha & Iyer, 2007; Maltz, 2000; Moenaert, Caeldries, Lievens, & Wauters, 2000) and project management effectiveness (Patanakul & Milosevic, 2009). The culture was also found to be a foundation for the development of project management competence (Suikki, Tromstedt, & Haapasalo, 2006). In addition, the organizational culture can be a basis for the project culture, which is claimed to be an important element of project management (Marrewijk, 2007; Schacht, 1997). Thus:
Hypothesis 3: The more that the organization establishes teamwork-oriented culture, the greater the MPM effectiveness
Influencing Factors at the Operational Level
In addition to the support from the organization, at the operational level, the MPM effectiveness may stem from the proficiency of multiple-project managers in project management processes. While project management processes are proposed in the literature, those processes are relevant to the management of an individual project (e.g., Project Management Institute [PMI], 2008a) or program (e.g., PMI, 2008c). When multiple projects are presented, the interproject interaction becomes an issue (Eskerod, 1996; Payne, 1995). As a result, multiple-project managers need to focus on interdependencies among projects in order to maximize the objective of group success as opposed to individual project success. When there is a lack of management attention to project interdependency, local suboptimum reigns. Several pieces of research suggested that the concept of portfolio management should be applied to an operational level so that functional managers, multiple-project managers, and team members can couple the planning and control cycles for single projects and the portfolio of projects. This will help identify the balanced tradeoff among the interests of multiple-project managers and functional managers in a team effort (Platje & Seidel, 1993; Platje, Seidel, & Wadman, 1994). The idea is to manage all projects as a collection, by adjusting and linking their schedules to match available resources, and removing unnecessary variation in the workloads of multiple-project managers (Adler et al., 1996). Some see multiproject scheduling as an efficient way to link projects (Ireland, 1997).
In sum, while the use of an individual project process is important for leading each project, multiple-project managers also utilize an interproject process and a process for managing interdependencies among projects. The typical individual project process may follow a set of standard such as A Guide to the Project Management Body of Knowledge (PMBOK® Guide)—Fourth edition (PMI, 2008a). The interproject process, on the other hand, is highly tactical in nature. Several multiple-project managers create their own approach (Patanakul & Milosevic, 2008). Interproject process refers to a process that helps multiple-project managers integrate the work of several projects—for example, milestones and deliverables to reduce the magnitude of multitasking (Patanakul & Milosevic, 2009). Having such a process also helps multiple-project managers manage interdependencies and interactions among projects related to shared milestones, resources, and technology. Therefore:
Hypothesis 4: The greater the systematic management of the individual project process and the interproject process, the greater the MPM effectiveness
To lead a group of multiple projects effectively, studies show that multiple-project managers should possess certain competencies. Tullett's study (1996) suggests that a typical manager of multiple projects has an innovative thinking style. He or she is likely to be less concerned with the attention to detail. A systematic approach is required to plan and manage such projects successfully. Multiple-project managers should be capable of multitasking, minimizing the switch-over time cost (Rubinstein, Meyer, & Evans, 2001) when shifting attention from project to project. Also, handling multiple projects is likely to increase pressures on multiple-project managers as they lead and build multiple teams at the same time. They should be effective in simultaneously leading multiple-project teams (Fricke & Shenhar, 2000; Patanakul & Milosevic, 2005). In addition, because of the unstable relationships (e.g. change in priority, unsustainable resources) that occur in a multiple project environment, multiple-project managers must know how to deal with conflicts when they arise (Payne, 1995).
Patanakul and Milosevic (2008) propose that in an MPM setting, multiple-project managers should possess a combination of hard and soft competencies to be effective in managing multiple projects— i.e., in managing each individual project and coordinating among projects. They suggested that typical competencies such as administrative, intrapersonal, interpersonal, business, and technical competencies help project managers to be effective in leading individual projects. This list is similar to what is suggested in the literature (Brown & Eisenhardt, 1995; Thamhain, 2004), although most of the project managers' skills proposed in the literature are the skills of a single-project manager who leads one project at a time (Frame, 1999; Pettersen, 1991; Thamhain, 1991, 2004). Patanakul and Milosevic (2008) also indicated that competencies such as the ability to manage interproject process and project interdependencies and interactions, the ability to multitask, and the ability to simultaneously lead multiple project teams are needed when a project manager leads multiple project teams. Thus:
Hypothesis 5: The greater the multiple-project managers' ability to lead each individual project and to coordinate among projects, the greater the effectiveness in MPM
Measuring MPM Effectiveness
Typically, a key reason that an organization implements MPM is to achieve better efficiency and management of projects. The expected output from multiple-project managers, besides meeting time, cost, performance, and satisfying customers, is the effective use of organizational resources (Fricke & Shenhar, 2000; Ireland, 1997). At a project manager level, the effectiveness can also be measured in terms of employee satisfaction (Denison & Mishra, 1995). Job satisfaction, career growth, and financial benefits are typical measurement criteria (Fricke & Shenhar, 2000; Kuprenas et al., 2000).
Patanakul and Milosevic (2009) indicate that the criteria for measuring MPM effectiveness can be grouped into criteria from organizational, project, and personal perspectives. Resource productivity and organizational learning are the measurement at the organizational level. The project-level metric includes the project success measures in terms of time and customer satisfaction. Personal growth and satisfaction are the effectiveness measures at the personal level.
Research Method
The objective of this research is to empirically identify the factors that influence MPM effectiveness. Data collection has been conducted using survey questionnaires. Each question was rated using 1-7 Likert scales (1 = strongly disagree, 7 = strongly agree). Multiple regression analysis was used to empirically test the hypotheses.
Sample
Samples were collected from the project managers in North America Program Management Offices (NAPMO) of one of the world largest financial institutes. These project managers are tasked to lead projects related to information technology both in technology and infrastructure categories. Questionnaires were sent to all 98 project managers of NAPMO. Of 78 returning questionnaires, 75 were usable. With five independent variables, the ratio of observations to independent variables is 15:1. This ratio meets the desired level, which is between 15 and 20 observations for each independent variable for multiple regression analysis (Hair, Anderseon, Tatham, & Black,1998).
For these project managers, the median of the experience as project managers is 11 years (mean is 14 years). On average, they simultaneously lead five to six projects. Of the total combined projects led by these project managers, about 60% had a budget less than US$1 million, 60% had durations of 4 to 12 months, and 50% had a team size of 10 members or less
Independent and Dependent Variables and Their Measurement
According to the research hypotheses, the impacts of five independent variables (IV) are tested in this study. The IVs are 1) project manager assignment, 2) resource allocation, 3) teamwork-oriented organizational culture, 4) project management processes, and 5) competency of multiple-project managers. The dependent variable (DV) is the overall effectiveness.
Project manager assignment was measured based on whether projects were assigned to multiple-project managers with the consideration of 1) project priority, 2) a match between the skill levels of multiple-project managers and project requirements, and 3) some constraints. Examples of items used in the questionnaire included: “When project were assigned to me, my manager understood the strategic importance and the project priority”; “My manager always took into account my personal competencies when he or she assigned projects to me”; and “There were always an appropriate number of projects assigned to me so that I was able to handle them effectively.” Eight questions were rated in total. The reliability test of the scales indicated the Cronbach' s alpha value of 0.84.
Resource allocation was measured in terms of the degree of resource sufficiency and sustainability, using five questions. Questions such as: “I had enough resources to complete my projects” and “Resources stayed with me as long as there was a need for them” were used. Cronbach's alpha value was 0.72.
Teamwork-oriented organizational culture was measured using nine questions. Examples of the questions included: “Members in my organization had a strong sense of contribution to project development”; “The working environment encouraged team members to communicate with each other for a mutual goal within our organization”; “There was a strong sense of working as a team”; and “There was a clear reward policy for outstanding performance.” Cronbach's alpha value was 0.87.
Project management processes were measured based on the use of individual project process and interproject process, using 10 questions. Examples of the questions included: “When leading each project, I followed our established project management process”; “I had a way to integrate my projects and that helped me manage them more effectively”; and “I understood that all projects under my responsibility had interdependency among them.” Cronbach's alpha value was 0.79.
Competency of multiple-project managers was measured with regard to administrative, business, interpersonal, intrapersonal, technical, and multiple project management competencies, using 41 questions. Examples of the questions included: “I was able to effectively develop a project plan and project schedule”; “I knew the business perspective of my projects”; ‘My teams respected me as their leader”; and “I was organized. I knew where my project documents were. I was in the top of them”; “I understood the technical aspect of my project”; and “I solved projects in such a way that the solutions benefited all of my projects as much as possible.” Cronbach's alpha value was 0.93.
Overall effectiveness was measured in terms of resource productivity, organizational learning, project success, and personal success, using 25 questions. Examples of the questions included: “My productivity increased since I simultaneously led multiple projects”; “While managing multiple projects, I used the knowledge I have gained from managing one project to manage other projects”; “All of my customers were highly satisfied'; and “Leading multiple projects improved my managerial capacity.” Cronbach's alpha value was 0.89.
Results
Table 1 illustrates the correlations between all variables in this study. The results show that the competency of multiple-project managers (IV) has the highest correlation with the overall effectiveness (DV) at the value of 0.670, followed by organizational culture (0.646). Project management processes and project manager assignment also have significant correlation with overall effectiveness (0.494 and 0.411, respectively). The correlation of resource allocation and overall effectiveness is smallest (0.219) among the five independent variables, and it is significant at the 0.06 level. These correlation results alone do not provide a full test of the hypothesized relationships. However, they are generally consistent with the expected pattern of results.
Table 1: Correlations among independent and dependent variables.
| Project Manager Assignment | Resource Allocation | Organizational Culture | Project Management Processes | Competency | Overall Effectiveness | |
| Project manager assignment | 1.000 | |||||
| Resource allocation | 0.494** | 1.000 | ||||
| Organizational culture | 0.631** | 0.424** | 1.000 | |||
| Project management Processes | 0.555** | 0.209 | 0.470** | 1.000 | ||
| Competency of multiple project managers | 0.316** | 1.163 | 0.466** | 0.441** | 1.000 | |
| Overall effectiveness | 0.411** | 0.219 | 0.646** | 0.494** | 0.670** | 1.000 |
| ** Significant at 0.01 level. | ||||||
Stepwise multiple regression analysis was conducted to test Hypotheses 1 through 5. The results are illustrated in Table 2. The regression model suggests that the teamwork-oriented culture and competency of multiple-project managers are the significant predictors of the overall MPM effectiveness (p < 0.000). Hypotheses 3 and 5 are supported. Together the model explains 57.7% (Adjusted R2 = 0.577) of the variance in the overall effectiveness (F = 21.197, p < 0.000).
The model also indicates that project manager assignment, resource allocation, and project management processes are not the significant predictors. Hypotheses 1, 2, and 4 are not supported.
Table 2: Regression model, dependent variable: Overall effectiveness.
| Model Summary | |||||
| R | R2 | Adjusted R2 | Standard Error of the Estimate | ||
| 0.778* | 0.606 | 0.577 | 10.02 | ||
| ANOVA | |||||
| Sum of Square | Df | Mean square | F | Significance | |
| Regression | 10,643.62 | 5 | 2,128.72 | 21.197 | 0.000 |
| Residual | 6929.27 | 69 | 100.42 | ||
| Total | 17,572.89 | 74 | |||
| Coefficient | |||||
| Predictor | B | Standard Error | Std. Coeff. Beta | T | Significance |
| Constant | 7.819 | 15.124 | 0.517 | 0.607 | |
| Project manager assignment | -0.132 | 0.251 | -0.059 | -0.524 | 0.602 |
| Resource allocation | -0.110 | 0.280 | -0.035 | -0.391 | 0.697 |
| Organizational culture | 0.842 | 0.209 | 0.431 | 4.037 | 0.000 |
| Project management processes | 0.258 | 0.177 | 0.142 | 1.460 | 0.149 |
| Competency of multiple project managers | 0.340 | 0.071 | 0.431 | 4.817 | 0.000 |
| * Predictors: (constant), project manager assignment, resource allocation, organizational culture, project management processes, competencies | |||||
Discussion and Managerial Implications
In this section, the research results are discussed in collaboration with the literature. Managerial implications are also presented.
Competency Matters
The research result indicates that the competency of multiple-project managers is a significant predictor of the overall effectiveness. While the extent literature stresses the significance of the project managers' competencies on project success (e.g., Archibald, 1975; Einsiedel, 1987; Frame, 1999; Gaddis, 1959; Pettersen, 1991; Thamhain, 1991), those studies focused on the competencies of project managers who lead one project at a time. The result of this study is unique in that it focuses on the competency of multiple-project managers and empirically tested its significance to MPM. Previous studies (e.g., Fricke & Shenhar, 2000; Patanakul & Milosevic, 2008; Tullett, 1996), put forth its significance based on the qualitative study.
To be effective in managing multiple projects, project managers should possess a set of competencies that help them be proficient in both leading individual projects and coordinating several projects under their responsibility (Patanakul & Milosevic, 2008). For leading individual projects, multiple-project managers should possess administrative, intrapersonal, interpersonal, business, and technical competencies. Administrative competencies refer to the knowledge, skills, and experience of a project manager in planning, organizing, and controlling projects. Intrapersonal competencies are the qualities internal to a project manager's character (e.g., being organized, responsible, proactive, mature, and flexible). The knowledge, skills, and experience of a project manager in interacting with other project stakeholders are referred to as interpersonal competencies, which are essential to project managers because they often have to exert their influence on project team members without having direct authority over them. Business competencies include the knowledge, skills, and experience of a project manager in addressing the business/strategic aspects of projects. Having business sense, customer focus, strategic thinking, and profit/cost consciousness are included. The importance of the business competencies in project management has increased as a result of the growth of the acceptance of projects as basic business vehicles in the business community (Forsberg, Mooz, & Cotterman, 2000; Frame, 1999; Pennypacker & Dye, 2002). Technical competencies are the knowledge, skills, and experience of a project manager related to the technical facets of the project product.
In coordinating projects, having multiple-project competencies (e.g., multitasking, simultaneous team management, and interproject process) helps. Once multiple-project managers are proficient in multitasking, they are able to estimate their own resource capacity in order to set priorities and effectively switch contexts to multitask among different projects. Multiple-project managers should also be effective in managing interdependencies and interactions among projects related to shared milestones, resources, and technology. Simultaneous team management is also an important competency. It includes the ability to put the team together and gain team accountability in a quick manner, to lead the teams in a discontinuous manner, including selecting and using different leadership styles specifically for each team, and to communicate to the teams in a concise and formal manner.
For managerial implications, the organization should ensure that all of their multiple-project managers possess the competencies listed above. It may be a good practice that an organization develops a standard list of competencies and uses it to evaluate their project managers. Those who lack certain competencies on the list may require future training and development. This list of competencies can also be used in the project manager assignment process. While administrative, interpersonal, intrapersonal, business, and technical competencies are important for any project manager, multiple-project managers in particular should possess MPM competencies.
Organizational Culture Counts
In addition to the competencies we have just discussed, organizational culture is another a strong predictor of the overall effectiveness of MPM. The teamwork-oriented culture that cultivates commitment, communication, teamwork, and reward for performance enhances MPM effectiveness. In collaboration with the literature on team effectiveness, this finding may not be unique in revealing that commitment, communication, teamwork, and reward for performance have some impact on team effectiveness (Abdel-Hamid et al., 1994; Campany et al., 2007; Denison et al.,1996; Hoegl & Gemuenden, 2001; Jassawalla & Sashittal, 2000; Jha & Iyer, 2007; Maltz, 2000; Moenaert et al., 2000). The uniqueness of this finding, however, lies in the fact that MPM is the unit of analysis.
Project commitment should come from top management and is supported by every level of the organization. This commitment-driving culture encourages project team members to be committed to their projects and to have a strong sense of contribution to project development. Organizational culture that promotes communication is also important. Management should create a working environment that encourages organization members to communicate with each other for a mutual goal within the organization. This includes creating a useful means by which knowledge and experience can be shared across different projects and effective communication channels to support information- and knowledgesharing. Organizational culture should also support teamwork. The culture should promote the strong sense of working as a team and should encourage the team members to help and support each other by sharing their knowledge and expertise. Last but not least, the organization should create the culture that supports reward for performance. A clear reward policy should be set. This includes having clear performance measures and a system to recognize and show appreciation to the ones who perform beyond expectation. Management should also ensure that the members of the organization have a strong sense of reward for devoted commitment and outstanding performance.
Should Project Manager Assignment, Processes, and Resource Allocation Be Ignored?
The answer is “probably not.” Researches on project manager assignments are still limited. Recent studies (e.g., Patanakul et al., 2007; Patanakul & Milosevic, 2006), proposed a systematic process and methodology for project manager assignments. The process includes the understanding of project priority, identifying a match between project requirements and competencies of project managers, and understanding the organizational and personal limitations regarding project manager assignments. While such a process was assessed in this study, practitioners may not follow the process and the assignments were done in a subjective manner. This may be the reason why project manager assignment does not have strong predicting effect on the overall effectiveness of MPM, even though project manager assignment is one of the important decisions in project management.
As already mentioned earlier, while the process for leading individual projects is important (e.g., the processes of the PMBOK® Guide), this study also assessed the interproject process and a process for managing project interdependencies and interactions that are also important for MPM. Since there is no established standard for interproject process, multiple-project managers are likely to develop their own approach. While the measurement of interproject process was developed based on the results in the first phase, other multiple-project managers may or may not use a similar approach. It seems also counterintuitive that resource allocation, which was measured in terms of resource sufficiency and sustainability, is not a predictor of MPM effectiveness. However, because of the nature of MPM, project managers may not always have the luxury of sufficient and sustainable resources, and this may be the reason why the predicting effect of resource allocation is insignificant. More research is needed to investigate these issues further.
For managerial implications, while the research results do not suggest project manager assignment, project management processes, and resource allocation as significant predictors of overall MPM effectiveness, management should not ignore these factors. Research has shown that the project manager is among the project success factors. The assignment process that leads to the selection of the right project manager to the project is therefore important. The organization should also put forward the interproject process that would help multiple-project managers to be more efficient in the use of their time and in managing interdependencies and interactions among projects. In addition, a reasonable level of resources should be provided to multiple-project managers.
Conclusion
This study investigates the factors driving the overall MPM effectiveness. The results suggest that team-oriented organizational culture and competency of multiple-project managers are significant predictors of the overall effectiveness of MPM. While the study is in its preliminary stage, its results contribute to the state of knowledge and suggest meaningful implications to practitioners. However, the limitations of this research that should be noted are centered on its research methodology. First, the study has a small sample size, although the ratio of the observations and the independent variables is in the acceptable range. Second, the data was collected from one organization. This may posit a weakness of the study in terms of the generalizability of the results. The researcher recognizes that this study is in its preliminary stage, and is collecting more data across industries. Even though the data at this stage is limited to one organization, the organization considered in this study it is one of the world's largest financial service organizations. While the results may be specific to this organization, this study should also provide sufficient insights into the effectiveness of MPM that is worth sharing with others. In addition, the data were collected from experienced project managers of this organization across North America. With these potential limitations, the researcher suggests that the contingency approach should be applied when using the research results.
