A case study on taming the wicked problem of portfolio management
Dr. Andrew Guitarte, CBAP, PMI-ACP, PMP
Enterprise Business Architect, Wells Fargo Bank
Why is portfolio management a wicked problem? Portfolio management and wicked problems exhibit similar characteristics that elude traditional linear problem solving. Wicked problems resemble complex phenomena among intelligent agents who adapt and learn (Rittel & Webber, 1973; Mitchell, 2009). Problem solvers use agent-based models to manage complexity in social contexts such as corruption (Jain, 2001), policy reform (Nunberg & Abdollahian, 2005), and transport planning (Le Pira et al., 2013). Similarly, portfolio managers demand a novel and cost-effective approach to portfolio analysis and prioritization. The author recommends business architecture models (Guitarte, 2012; Guitarte, 2014) and a capability-based portfolio analysis technique to develop “good enough” solutions for portfolio management. The added rigor and discipline in decision making can decrease costs by increasing portfolio management team productivity.
I received encouraging feedback from readers when I first published my paper on this same topic (Guitarte, 2015), but they wanted more. They wanted to find out how to apply these concepts by using a real-life example. Better yet, they wanted to find out if there were any success stories that I could share. Hence, I came up with this case study presentation of a U.S.-based national bank's experience on managing the wicked problem of portfolio management.
Why is portfolio management a wicked problem and how do we manage it? This presentation seeks to answer those questions. By the end of the presentation, participants will be able to identify the characteristics of portfolio management as a wicked problem, describe current approaches to project portfolio prioritization that fall short, and apply business architecture models and techniques to manage complex project portfolio prioritization issues through a case study walk-through of a U.S.-based national bank.
Portfolio Management's Problems
A business architect eats ambiguity for breakfast, chaos for lunch, and complexity for dinner.
– Andrew Guitarte, business architect
Every system interacts with other systems. One output may be an input to another system. Such open characteristics of systems reflect the reality of doing business today. Each department, division, subsidiary, conglomerate, industry, and nation intertwines to form the fabric of modern society. No longer is there an island of single interest but an archipelago of business partners in a vast, though at times turbulent, ocean of opportunity.
The term wicked problem was introduced in 1967 by C. West Churchman but formally described by Horst Rittel and Melvin Webber in 1973. Based on Rittel's idea (Churchman, 1967) wicked problems, as opposed to “tame” problems, which are solvable, “are a class of social system problems which are ill-formulated, where the information is confusing, where there are many clients and decision makers with conflicting values, and where the ramifications in the whole system are thoroughly confusing.” Rittel sought an alternative to the linear, step-by-step design process composed of problem definition and problem solution. Both steps follow the analysis-then-synthesis progression of problem solving applicable only to “determinate problems which have definite conditions” (Buchanan, 1992, p. 15). In contrast, wicked problems are indeterminate.
The Standard for Portfolio Management – Third Edition defines portfolio management as “the coordinated management…of interrelated processes by which an organization evaluates, selects, prioritizes, and allocates its limited resources to best accomplish organizational strategies (PMI, 2013, p. 5).” One of the key tasks is portfolio prioritization, which is prone to missteps because of the complexities involved in decision making.
There are several reasons why portfolio prioritization may fail. I found the following scenarios a common occurrence across all sites of a U.S.-based national bank. In a highly homogenous group, there is a real threat of “groupthink.” Irving Janis (1982, p. 9) coined this term to refer “to a deterioration of mental efficiency, reality testing, and moral judgment that results from in-group pressure.” Factors that contribute to this malady include too much power vested in a leader or a faction, lack of procedures to accommodate contrary views, and pressure to make rushed decisions when stakes are high. Portfolio managers who are insulated from outside opinions fall into this trap by thinking that they're invulnerable and that the group's judgment on important issues must always be right.
But the bigger problem surfaces from highly heterogeneous groups. Diverse interests and clashing organization cultures etched in spoken and unspoken rules can railroad any effort to arrive at group consensus within a reasonable period of time. Decision makers either postpone making decisions or rush to make one, to the detriment of group morale.
“Plurality of objectives held by pluralities of politics makes it impossible to pursue unitary aims.”
– Rittel and Webber (1973, p. 160)
“Ay, there's the rub,” laments Shakespeare's Hamlet. We're all too familiar with the homogenous organization where we are supposed to share common values, interests, and goals. Any company vision and mission embody this principle. Such aspirations are noble because these assume that every person in the organization promises to hold dearly these tenets. But the practice is farther from the theory. Ask any employee of our case study to clearly articulate the company mission and how the company plans to implement it. Ask another and that other person will probably offer a slightly different version of the answer. Multiply these with the number of employees and you can see the organization is more fragmented than united. It's far from being homogenous.
To find a final solution to a problem may be easier in a homogenous group. But where diversity is the norm, finding solutions may be a fleeting goal. It's like fixing a flat tire while the automobile runs on the freeway. It's not impossible, but it's also highly improbable. But decision makers accept this as the truth and the way of doing business. It's rightly so. Problems of this complex nature defy definition. If we can't define them, how do we know how to solve them?
Decision making involves a single judgment, whereas problem solving involves many decisions. When a group, such as a jury, team, or committee makes a decision, the group usually answers the “who, what, where, or when” question. A single decision does not necessarily solve a problem. In fact, problem solvers make plans that involve multiple decisions to eventually solve the problem. It's a process rather than a single event. A group makes better decisions and solves problems more efficiently than an individual as long as there is room for validation of ideas, clarification of options, and testing of solutions.
Group Problem Solving
Nicholas Rescher's (1998) Delphi methodology, usually effective in business forecasting, cultivates the best decisions based on advice from a panel of experts. This is similar to the nominal group technique developed by Andre Delbecq and Andrew Van de Ven (1975), where members generate ideas individually and vote anonymously to diminish any bias. Product managers and system architects in our U.S.-based national bank case study, for example, successfully use this technique to build their business models. On the other hand, voting can be divisive, especially in controversial topics. Another approach is through consensus building, but it may take too much time in order to accommodate all ideas in the discussion and come up with the next best alternative that everyone can support.
Asking questions allows a problem solver to make better decisions. There are questions of fact, conjecture, value, and policy. A combination of these four types of questions provides the best approach to understand the problem.
John Dewey's reflective thinking process, outlined in his seminal 1910 book, How We Think, is a problem-solving technique that involves fact-finding as a critical step in the process. LaFasto and Larson (2001) boil down problem solving to a collaborative process of crafting a single question using the single question format. Both approaches highlight the need to gather facts and analyze the issues surrounding the main problem (or single question). This avoids the pitfalls of groupthink where the group arrives at a solution too early without understanding all the critical components of the problem. Stephen Toulmin (1958) developed a method of building arguments called the Toulmin model of argument, which specifies that evidence is a critical part of the argument. When well-positioned, an argument almost always has facts to back it up. Evidence can take the form of facts, opinions, definition, description, examples, illustrations, and statistics.
There is another approach to problem solving that may help address the issue of complexity. Creative problem solving involves information gathering as well as thinking of solutions “out of the box.” Insights form once the group incubates the more creative ideas. Then the process repeats itself to improve on the previous solution.
A Wicked Problem
Every attempt to solve a wicked problem such as portfolio management using traditional problem solving methods can fall short and result in unintended consequences. Ranking information technology (IT) projects, if poorly executed, for example, can yield more problems downstream. Wicked problems display important characteristics as outlined by Rittel and Webber (1973) that confront and befuddle portfolio managers. Let's tackle some of these characteristics as applied to our U.S.-based national bank case study.
- You can't define it since you need to have an idea of the solution first to frame the problem. In other words, it depends who you ask. Opinions on how to prioritize a portfolio based on importance and value vary among portfolio managers because of their values and organizational culture. What is important and valuable depends on what worked in the past, and that varies among groups of stakeholders.
- You can't tell when you're done. There's only a “good enough” solution, which will never be final or optimal. Change constantly stirs this pot of analysis without knowing with certainty when it's cooked. Good portfolio managers know that there is no right or wrong way of prioritizing a portfolio. There are only bad, good, or “good enough” ones. Some even end their analysis not by arriving at some logical conclusion but only because they like what they see or they run out of time.
- You can't fully test the solution. In fact, you may be surprised by unintended consequences. How do portfolio managers know if they did a good job? They can't immediately see the results of their trade-off decisions and what else was affected. A seemingly balanced portfolio that performs well over time may even come as a surprise.
- There is no trial and error option; every solution is a “one-shot operation” with significant consequences. Some failures are even irreversible. Investment decisions as a by-product of portfolio analysis involve large sums of money. Once invested, it is difficult to retract due to the “sunk cost effects” common in large organizations. Once retracted, additional wicked problems may arise.
- Unlike mathematical puzzles or chess, there are no rules for coming up with solutions. “Anything goes” seems to be the maxim since there are no criteria to exhaust all possibilities using all possible permutations. Portfolio managers or a team of experts rely on good judgment to solve for the “good enough” mix of portfolio components that adequately manages risks without being absolutely sure if they covered all possible options.
- Each problem is essentially unique and you can't classify problems in nice little buckets. There's always something different about a new problem compared to the last one. That's why old solutions won't work all the time. Reusing the logic portfolio managers used in a previous prioritization exercise can't guarantee success in the succeeding ones. A fresh batch of projects to be prioritized carries its own set of burdens and concerns. It seems that you're always beginning.
- Everything is a symptom of something else. This is where a dilemma exists. Solve the higher-level problem and you end up with generalized solutions that are impractical. But try to cure a symptom instead and you may perpetuate the root cause or worsen it. Portfolio managers will demand better software, for example, to cure a symptom of failed prioritization instead of fixing a broken communication process.
Business Architecture Solution Approach
All models are wrong, but some are useful.
– George E. P. Box (Box & Draper, 2007, p. 414)
Complexity science studies problems or phenomena that exhibit the same characteristics of wicked problems. An interdisciplinary field of research called complex systems “seeks to explain how large numbers of relatively simple entities organize themselves, without the benefit of any central controller, into a collective whole that creates patterns, uses information, and in some cases, evolves and learns” (Mitchell, 2009, p. 4).
Even scientists who study nature to explain complex phenomena such as Newton's law of gravity or Einstein's theory of relativity construct and study models of nature, which are mere representations of the “real” phenomena. When we study and learn the complexities of portfolio management, we model it using well-known frameworks, mechanisms, procedures, and objects. These models allow us to grasp the magnitude of the problem, simulate effects, and in some cases predict future behavior.
One way to solve a wicked problem, according to some authors (see Conklin, 2005), is to seek a sub-problem and define that as the problem to solve. In the case of portfolio management, we tackle the problem of portfolio analysis and, more specifically, the problem of portfolio prioritization. The process of project selection based on a set of criteria can be ad-hoc and lacks formality in some organizations. Quality of output suffers and hampers the portfolio. Reasons for poor prioritization processes as outlined by Moore (2010) include insufficient data, commitment bias or effects of “sunk costs,” organizational inertia, and cognitive dissonance.
The obvious opportunity where we need to prioritize is when portfolio components compete for the same resources. Just like managing a financial portfolio, the portfolio manager needs to make trade-offs by balancing and diversifying the risks across the portfolio. The portfolio manager also needs to balance the goals of the portfolio so that cost-cutting goals, for example, do not reduce business performance and revenue growth. A robust and transparent portfolio selection process ensures that all projects align with the organization's strategic direction.
Business Architecture Models
Business architecture, as an emerging practice within or outside the realm of enterprise architecture (EA), creates “a blueprint of the enterprise that provides a common understanding of the organization and is used to align strategic objectives and tactical demands” (Object Management Group, n.d.). Business architecture's value proposition, unlike that of other disciplines, is to increase functional effectiveness by mapping and modeling the business to the organization's business vision and strategic goals. An important domain where business architecture can provide novel and rigorous methods of problem solving is in portfolio management.
As previously stated, a model is a depiction of what's happening in the real world. It reduces the macro to the micro with the inherent caveat that this representation will fail at some point when analyzed in greater detail. But business models, for example, serve a higher purpose than just to display a set of boxes and lines in a fancy diagram. They help the decision maker and problem solvers structure their plan of attack using this “blueprint of the enterprise.” The planner relies on the blueprint of the organization to identify pitfalls and manage risks before the actual implementation.
Let me highlight two models that business architecture teams in our U.S.-based national bank case study use for this purpose.
Business Capability Architecture (BCA)
The BCA is a blueprint of the organization that ties all the important components of running an organization into a cohesive whole in order to achieve synergy. Exhibit 1 shows the BCA as a hexagonal frustum with business capabilities as the apex of this three-dimensional design (Guitarte, 2012). A business capability describes what the business does that creates value for customers. It encapsulates the people, process, technology, and information, the essential building blocks needed to produce business outcomes. A frustum is the most appropriate geometric shape to describe the BCA since it shows that business capabilities are the ties that bind all other views and concerns of the organization.
The elevated prominence of the BCA suggests a convergence regardless of a stakeholder's starting viewpoint. A product manager, for example, can use the BCA as the “rallying point” to trace products to customer needs and business strategy. This line of sight gives stakeholders a common taxonomy through which conversations about impact analysis or strategic alignment bear more fruit.
The BCA can network with other BCAs to portray the concept of an extended enterprise. Exhibit 2 illustrates the relationships among the BCA components. It is dynamic and attuned to the shifting landscape of business. It goes beyond the confines of the corporation and interlocks with business partners, regulatory agencies, and third parties. It also traces the routes one can take to achieve complete traceability from strategy to operations. This is important for impact and dependency analysis of complex scenarios. In other words, the BCA is the tie that binds all.
Portfolio prioritization can use this paradigm to map out the traceability between projects to strategy by first mapping projects to impacted business capabilities. This prevents product owners from “gaming” the system by vaguely relating all projects to a strategy. It's difficult to argue against such mappings because there's almost always an element of truth involved. However, the BCA adds rigor and discipline to this alignment exercise. Since business capabilities are mutually exclusive by design, we can systematically identify a primary business capability impacted by every project based on the documented project requirements. Business capabilities in turn will map to the organization's strategy model of mission, vision, goals, and tactics. This provides the portfolio manager a more transparent and comprehensive validation of the true nature of alignment between strategy and the project portfolio.
Enterprise Business Architecture Framework (EBAF)
As shown in Exhibit 3, EBAF provides a more business-oriented and customer-centric framework than any of the existing legacy EA frameworks today (Guitarte, 2014). It's business-oriented because it highlights business outcomes instead of service levels, interfaces, and anything that closely identifies with IT operations. It's customer-centric because everything should revolve around meeting customer or stakeholder needs and wants. Believe it or not, customer is still king in this day and age. The author explains further:
It encapsulates what's essential in the business model, business context, strategic intent and goals. The focus of EBAF can be the stakeholder segment, value proposition, channel of distribution, and the set of core and supporting business capabilities that operationalize the strategy where strategy can emanate from the top and from down in the trenches. EBAF should be able to capture these and show clear traceability to business outcomes. (Guitarte, 2014)
Exhibit 3 illustrates the EBAF concept using the following design principles. First, form follows function. A framework is built because of an unmet need. It's there because you need it and you use it because it's there. Second, begin with the end in mind. Ask the difficult but important question, “Why bother?” Executives want quick, actionable insights. How do we deliver these to them using our diagram? Third, think out of the box, literally. Design something different from what we're used to, especially for those of us who grew up thinking in technical terms. But relate it to the familiar so that we don't need a manual to explain how to use it.
The author developed this framework to highlight change as a constant and help organizations understand the dynamics of operating amidst chaos and complexity. It attempts to capture the shifting priorities of the organization and the resulting feedback. A framework that's designed from the ground up, illustrates dynamism, provides actionable insights, and speaks the language of business can help structure the conversations related to managing complex problems. It is a springboard for stakeholders to hold meaningful discussions about achieving business outcomes, which we see happen almost on a daily basis in our U.S.-based national bank case study.
Freeman's stakeholder theory (1984) prescribes methods to identify and consider the interests of those affected in the operations of the organization, addressing the principle of “who and what really counts.” Effective stakeholder participation in decision making enhances the organization's ability to tackle complex issues. However, stakeholder analysis sometimes relies on less rigorous intuitive methods such as the one developed by the Overseas Development Administration (1995) entitled, “Guidance Note on How to Do Stakeholder Analysis of Aid Projects and Programmes.” These methods produce sensible results but may not be robust enough to make policy or investment decisions. What they lack is a sharper focus on fact-finding, analysis, and reporting to systematically support decision-making and problem-solving processes such as portfolio management.
Agent-based modelling (ABM) attempts to manage the chaotic nature of complexity and wicked problems. An agent or stakeholder is a person, group, or organization that has an interest in a given issue and may be affected by decisions related to that particular issue. By modelling the preferences and behavior of stakeholders, ABM provides the analyst a robust tool to design solutions. Social scientists apply this tenet to social issues such as corruption, policy reform, and transport planning.
The main agents in the area of corruption, for example, are the political elite, the administrators, and the legislators (Jain, 2001). An agent-based model can forecast the future actions of these agents or stakeholders based on a set of assumptions and criteria.
World Bank researchers developed an agent-based stakeholder model (Nunberg & Abdollahian, 2005) to predict the outcome of introducing procurement reforms in a fictional country called Anyland. They identified the need for a robust tool to help policymakers understand the preferences and behaviors of key stakeholders before introducing civil service and anti-corruption reforms that may be doomed to fail from the start. Using the same principles of ABM, the researchers identified key agents that include the prime minister, domestic leaders, interest groups, and international organizations. These agents are influential in defining the future course of any policy reform implementation.
In the complex process of transport planning, Le Pira et al. (2013) simulate and predict behavior of the following agents to arrive at either a state consensus or non-consensus. These agents are decision makers, experts, stakeholders, and citizens. The researchers use a computer application called NetLogo to operationalize the variables for consensus building and create “what-if” scenarios of possible decisions that can emerge over time. It's a more advanced form of ABM using social network analysis where stakeholder networks mesh with opinion dynamics to simulate information flow and how (or if) consensus will be reached.
Portfolio Management Agents
Similarly, we can identify the types of stakeholders who have a vested interest in project portfolio management. In our U.S.-based national bank case study, these include the product managers, the administrators, and the implementers. Product managers serve as proxies of the company owners, or sponsors, in the case of non-profit organizations. They have the most to gain oftentimes in monetary terms when the organization operates according to their well-defined strategies. Administrators seek a healthy balance between risk and reward and strive to achieve an optimal solution at all times. This group includes finance, legal, risk, operations, and vendor-relations personnel. Implementers include portfolio managers, program managers, project managers, technology teams, customer-support personnel, and third-party vendors.
Any portfolio analysis technique should embody the interests of these agents or stakeholders. Product managers earnestly pursue strategic alignment, while administrators seek a healthy balance between opportunities for growth and business constraints, while implementers operate on delivering high-quality products on time and within budget. One group of stakeholders asks, “Is it strategic?” while another asks, “Is it valuable?” and yet another group asks, “Is it doable?” A robust agent-based model for managing a complex problem such as portfolio management should systematically answer all three questions at the same time.
Portfolio Prioritization Technique
Business architecture can address complexity in portfolio prioritization using the capability-based portfolio analysis (CBPA) technique. For portfolio prioritization to be successful, the decision maker should have a credible list of criteria that all stakeholders can buy into. The criteria should include three critical dimensions of the business, namely strategic alignment, contribution to business value, and implementation complexity. Current methods of portfolio prioritization mostly rely on only two of three key criteria.
For example, in our U.S.-based national bank case study, product managers and executives often ask the questions related to an idea's contribution to organizational value and alignment to strategy. But they postpone the important question of feasibility until a later point in time. This is understandable since those who will eventually implement the idea may not necessarily be engaged from day one. However, executives need useful information that's not only accurate and relevant but also timely.
On the other hand, project ideas emanating from the technology teams may be doable from their point of view and possibly aligned with some business strategy. Only later will they find out their ideas were rejected or shelved as “nice to have” due to a poor business case. In both cases, the organization lost valuable time waiting in limbo for all the pieces of the puzzle to form. There has to be a better way to simultaneously and systematically capture all information related to the important dimensions of portfolio prioritization. This redounds to more cost-effective decision making.
Another benefit of the CBPA technique is the ease of data visualization. Using the portfolio optimization quadrant (POQ), the prioritized portfolio can be plotted in a graph to show “wow” projects that are the least complex to implement but will generate the most value for the organization. Conversely, the graph can also show complex but low-value projects as candidates for rejection or the backburner, aptly called the “no-brainers.” The “must have” projects have low complexity but still should be done even though they're of lower value. Examples include regulatory compliance, minor content enhancements, and so on.
Note that the POQ does not display the dimension for strategic fit. We simply assume that all projects in consideration pass the test for strategic alignment. Those that do not trace to a specific strategy should be rejected. A more advanced refinement can include the continuum for strategic fit by assigning scores for the least up to the most amount of fit.
Any wicked problem solution is almost always a “one-off” solution. The context and environment of a wicked problem change constantly so that by the time the problem solver arrives at a final solution, the original problem no longer exists. Instead, the problem can morph into another form that presents itself as a totally strange new problem.
The same is true for portfolio management. Portfolio managers can lock a prioritized list of portfolio components as optimal to achieve organizational goals. Portfolio components often undergo change, which is inevitable in the business context due to shifting customer demands and resource constraints. When this happens, the optimized portfolio of the past can no longer be considered optimal unless portfolio managers conduct a thorough review and re-evaluation. The process to derive an optimal solution at all times is costly. That's why most portfolio managers would rather settle for a “satisficing” solution (Simon, 1969) that is “good enough” for all stakeholders.
Wicked problems are indeterminate and essentially unique, where endless solutions cannot be fully tested. Portfolio management behaves in a similar fashion. Portfolio managers can utilize business architecture's decision support models such as the business capability architecture (BCA) and enterprise business architecture framework (EBAF) to help them wrestle with this problem. Business architecture's unique value proposition is to increase an organization's functional effectiveness by mapping and modeling the organization with its vision and strategic goals. Business architects add rigor and discipline to the complex nature of aligning the project portfolio to strategy using a capability-based portfolio prioritization technique. In the process, business architecture decreases costs by increasing the portfolio management team's productivity.
Solving the wicked problem of portfolio management is a dilemma. It's like making a choice among bad choices. But effective portfolio managers use good judgment to pull off good enough solutions. In portfolio prioritization, they can employ a transparent, comprehensive, and robust mechanism such as the capability-based portfolio analysis (CBPA) to align the portfolio to the organization's strategic directions. The case study of a U.S.-based national bank proves this scenario and shares the lessons learned.
Box, G. E. P., & Draper, N. R. (2007). Response surfaces, mixtures, and ridge analyses (2nd ed.). Hoboken, NJ: John Wiley & Sons, Inc.
Buchanan, R. (1992). Wicked problems in design thinking. Design Issues, 8(2), 5–21.
Churchman, C. W. (1967). Wicked problems. Management Science, 4(14), 141–142.
Conklin, J. (2005). Dialogue mapping: Building shared understanding of wicked problems. New York, NY: John Wiley & Sons, Inc.
Delbecq, A. L., Van de Ven, A. H., & Gustafson, D. H. (1975). Group techniques for program planning. Glenview, IL: Scott Foresman & Company.
Freeman, R. E. (1984). Strategic Management: A stakeholder approach. Boston, MA: Pitman.
Guitarte, A. (2012). Business capability architecture is the tie that binds all. Cutter IT Journal, 25(12), 24–31.
Guitarte, A. (2014, April 25). Wanted: A new business architecture framework. Retrieved from http://www.bainstitute.org/resources/articles/wanted-new-business-architecture-framework
Guitarte, A. (2015). Business architecture tames the wicked problem of portfolio management. Cutter IT Journal, 28(2), 11–18.
Jain, A. (2001). Corruption: A review. Journal of Economic Surveys, 15(1), 71–121.
Janis, I. (1982). Groupthink: Psychological studies of policy decisions and fiascoes (2nd ed.). Boston, MA: Houghton Mifflin.
LaFasto, F., & Larson, C. (2001). When teams work best. Thousand Oaks, CA: Sage.
Le Pira, M., Ignaccolo, M., Inturri, G., Garofalo, C., Pluchino, A., & Rapisarda, A. (2013, July). Agent-based modelling of stakeholder interaction in transport decisions. Paper presented at the 13th World Conference on Transport Research, Rio de Janeiro, Brazil.
Mitchell, M. (2009). Complexity: A guided tour. New York, NY: Oxford University Press.
Moore, S. (2010). Strategic project portfolio management. Hoboken, NJ: John Wiley & Sons, Inc.
Nunberg, B., & Abdollahian, M. (2005, March). Operationalizing political analysis for development: An agent based stakeholder model for governance reform. Paper presented at the International Studies Association Annual Meeting, Honolulu, HI.
Object Management Group. (n.d.). Mission and overview: What is business architecture? Retrieved from http://bawg.omg.org
Overseas Development Administration. (1995). Guidance note on how to do stakeholder analysis of aid projects and programmes. Retrieved from http://www.sswm.info/sites/default/files/reference_attachments/ODA%201995%20Guidance%20Note%20on%20how%20to%20do%20a%20Stakeholder%20Analysis.pdf [INSERT URL]
Project Management Institute. (2013). The standard for portfolio management – Third edition. Newtown Square, PA: Author.
Rescher, N. (1998). Predicting the future. Albany, NY: State University of New York Press.
Rittel, H. W. J., & Webber, M. (1973). Dilemmas in a general theory of planning. Policy Sciences, 4, 155–169.
Simon, H. A. (1969). The sciences of the artificial (2nd ed.). Cambridge, MA: MIT Press.
Toulmin, S. (1958). The uses of argument. London, UK: Cambridge University.
© 2015, Andrew Guitarte
Originally published as a part of the 2015 PMI Global Congress Proceedings – Orlando, Florida, USA