Navigating the minefield
estimating before requirements (for project managers)
Creating auditable and realistic project estimates in any industry or specialization involves planning, assumptions, knowledge, and experience. However, it is rare to find all four components present and complete at project initiation, yet project estimates are becoming the gauge against which project managers are raised, praised, and rewarded. When project estimates are made using the worldwide “date driven estimating” method, projects start off balance and only get worse as time goes on. Soon, even the most experienced project managers can find themselves, and their teams, in trouble – behind schedule, over budget, and with chaotic and uncontrolled scope creep. What's a sane project manager to do? This paper outlines how to take a requirements-based approach to project estimating in order to create both auditable and realistic project estimates from the beginning – even before requirements have gelled, and is applicable especially in information technology projects, but also applies well to any project governed by customer requirements and expectations.
Project managers have long been challenged to create auditable and realistic project estimates. To do so, they make preliminary project plans that include an articulation of requirements, the extent and scope of work to be done to address those requirements, and schedules are devised. Knowledge and experience normally provide the first task-based estimate and may be augmented by formal estimating methods specific to the industry and specialization. For example, a project manager in building construction uses estimating models that pertain specifically to the construction industry, while project managers in information technology use estimating models specific to software and systems development projects. The challenge to estimate overall cost, effort, and project duration becomes particularly difficult when the project introduces some sort of “novelty” in terms of new types of customer requirements, new quality constraints, or new requirements for design, construction or implementation. In software project estimating alone, some of the formal estimating models ask for input values for up to 200 project variables. In the situation where a project has many unknown values (as in the case of novelty projects), it can be almost impossible to come up with a reasonable estimate.
Projects involve customer requirements of varying degrees of clarity and completeness according to Meta Group Vice President Howard Rubin: a) The Known Knowns; b) The Known Unknowns, and c) The Unknown Unknowns. (Rubin, 1999) Each of these is explained in the following paragraphs.
The Known Known Requirements
The “known known” requirements are those which are obvious and are part of every project within the particular industry. For example, on a building project – one of the known known requirements is that the project must adhere to the building code of the county in which the project will be built. In information technology, one of the known knowns on a banking system would that bank accounts must be maintained, recorded and stored by the software. Known known requirements involve the smallest risks on a project because they are the foundation requirement(s) of the industry and type of project.
The Known Unknown Requirements
The “known unknown” requirements are those which are typically not fully articulated by customers, but which are absolutely predictable based on prior experience with similar projects. For example, in homebuilding, even if the homeowner/customer does not explicitly specify the number of bathrooms, from experience the builder knows that the number of bathrooms is typically the same or more than the number of bedrooms in any house exceeding 2000 square feet. In software projects, it is a known unknown requirement that any software system where human life is at stake must pass through level(s) of government inspection and approval, such as the software involved in pacemakers and other human implantable devices. The known unknown requirements incur a higher level of risk than the known knowns, but the risk can be mitigated by relying on past experiences to augment theoretical estimating models.
The Unknown Unknown Requirements
The “unknown unknown” requirements are typically the source of project overruns, schedule slippage, and overoptimistic estimates. Because these requirements generally arise during the project, they are nearly impossible to predict and estimate. Given a database of projects where novelty and innovation dominated the requirements, at least a scope creep percentage can be allocated over the entire suite of requirements to account for this level of requirements. However, a well controlled and tracked change management process will also alleviate the unknown unknowns especially when they arise (as they generally do) as a result of unforeseen business changes such as due to legislative or governmental regulatory changes.
To maximize project outcomes and achieve a level of product quality, project managers must strive to manage their projects to maximize the Known Known requirements and minimize the Unknown Unknown requirements. How can they do this in the face of pressures to come up with early-in-the-project and auditable and realistic project estimates? One of the keys lies in preparing requirements-based estimates instead of the typical task-based project estimates.
Project Estimates Vary by Discipline
As one of the first authors to recognize that software engineering differs from traditional engineering, David Card stated, “Engineering projects usually can wait until after design to provide an estimate, while software engineering requires an estimate before design.” (Card, 1999) In the author's experience, software projects can be even worse -- some projects base their budgets on (guess)timates that are prepared before requirements are even documented!
The folly of management demands for premature project estimates -- sometimes even fixed-price ones -- can derail projects before they are even named. Even tried and true construction projects with established processes can be affected. To illustrate, consider these fictional dialogues:
Dialogue 1: Home Construction Example
Potential homeowner: How much would it cost to build me a house with three bedrooms and two bathrooms?
Builder: Well that depends…
Potential homeowner: Depends on what?
Builder: On the size of the house, style (single or two stories), amenities and desired rooms, and what you want to have included.
Potential homeowner: Ok, consider a standard type of house with three bedrooms and two bathrooms, I just need a ballpark figure.
Builder: Ok, if you use one of our 1200 square foot floor plans it will cost you between $100. to $150. per square foot depending on options.
Potential homeowner: So, if I get a construction mortgage for $120,000, that should be enough?
Builder: It probably should be if you choose the basic options, (but I cannot really guarantee anything until there's a floor plan and specs).
Result: The homeowner remembers the dialog and assumes that $120,000. is the upper limit on the price – despite the range of costs that would place it in the lowest end of pricing range at $120. / square foot. Problems will ensue both when the homeowner goes to sign the contract (likely at a higher price due to selections) and also throughout the building process as the homeowner makes changes and change orders are created.
Dialogue #2: Software Project Example 1
User Manager: How much would a new software system cost me – it has to produce a pile of financial reports and do some up-to-the-minute financial processing? (I won't hold you to the estimate…)
Software Project Manager: Well that really depends on a bunch of factors like the language we use, the skills, and how much functionality you need in the first release.
User Manager: Just give me a ballpark figure – you should be able to do that quickly. After all, you've built lots of software.
Software Project Manager: Without requirements or any understanding about what we will build, I can't really give you an estimate – I can only tell you that it's likely to cost anywhere from $100K to $500K given the sketchy information you've given me.
User Manager: Ok. Thanks for the estimate (I'll remember the $100K price)
Result: The user/customer remembers this dialog and assumes that $100,000. is a reasonable (and likely) price – despite the range of costs that would place it in the most optimistic pricing. Problems will ensue as the requirements grow and increase in complexity (as usually happens with software). Increases in pricing will typically not be understood by users – even if the overall functionality increases from the preliminary, first estimate.
Dialogue #3: Software Project Example 2
User Manager: We just announced to the state government leaders that the new motor vehicle registration system will be operational by July 1, 2005. Do you have any idea how much it will cost?
Software Project Manager: I don't know why you always announce dates before we have any real idea of what the project will entail. We don't even have any requirements for the project and I really think that July 1, 2005 is an impossible date to deliver anything – after all, it's barely 5 months away.
User Manager: Well we made the announcement and the state legislators got so excited about the possible 2005 delivery that they promised to fund the project no matter what it costs. So, can you give me an idea of costs now?
Software Project Manager: Well if we have to be finished by July 1, that means we need to be finished testing at least a week prior to that, with coding done at least a month before that, design before that….. with about 15 project team members (of course, we'll pull all our best people off of their other projects), working for about 4 months solid, that's 60 person months of effort --- hmmm…. About $600,000. give or take a few hundred thousand for the unknown factors and changes.
User Manager: I knew you could do it. I'll tell the legislature to budget $800K. Gosh, it's great to have a project manager that can make things happen like you can.
Software Project Manager (to team): Yikes, we're doomed again!
Result: Both sides will feel frustration with the unrealistic schedule and pressures that only increase once the project begins. Likely testing and flexibility for change will diminish as the project proceeds, and the schedule and pricing go awol.
Realistic estimates depend on a realistic description of the problem as stated by the requirements. Without such information about the product or solution functionality and performance, an estimate can only be a “(guess)timate” – or a guess of what the project entails, and usually the result is overly optimistic.
Cost Estimating Challenges (based on requirements)
While software estimating is easier once the requirements are known, the following challenges still prevail: (Dekkers, 2004)
- Accuracy: How accurate are the productivity and cost driver inputs? Estimates are only as accurate as +/- 50% of the least accurate input variable
- Availability: Can all input variables even be provided to any reasonable level?
- Applicability of historical data If historical data is available, how applicable are the data points? How can one be sure to ensure an apples-to-apples comparison?
- Completeness: How complete are the requirements? Do project costs include or exclude hardware acquisition, software tool purchases, and other costs – or simply the software development labor costs?
- Risks: Have risk factors been considered and evaluated? Are they even included in the project? Risks must be calculated and predictive.
- What: What work effort tasks are included / excluded, and whose time is included?
In spite of these challenges, project managers are expected to produce estimates of duration, cost and work effort to a level of accuracy and realism – as early as project initiation or sooner. Estimates made early in the software development life cycle are subject to large variations due to factors mentioned above. While it should be obvious that estimates based on guesses of input variables are unreliable, many managers and users ignore this fact and treat the estimates as predictive project forecasts. In the US, part of our American culture is that if something is too good to be true, it probably is (Hammond, 1996) – yet we have an insatiable optimism that maybe, “just this once” it might come true. Time and time again overly-optimistic estimates are revised as dates slip, functionality is reduced and project budgets surpassed because the appropriate amount of time to complete the work was not allocated .
All project managers and customers would all do well to remember several key points related to project estimates:
- An estimate is only as good as its least reliable input variable;
- Garbage entered into an estimating tool produces a garbage estimate;
- Even if the corporate desire is for faster, better, and cheaper software development – an overoptimistic estimate won't make it happen;
- Just because a project manager provides an estimate does not mean that it is realistic or achievable, When estimates are based on flawed data, they generate a false sense of security that the estimate is reality. This in turn gives rise to pressure on the cost estimator and project team to “prove” the goodness of the estimate by working desperately to meet it – even if it is grossly in error, and to make it come true.
Project Requirements – Demystified
Given that (software) projects in the United States spend almost half of their time doing rework - in excess of 40% (Sanders, 2002), it makes sense to examine what can be improved in the requirements process – and in turn improve the estimating process(es). Rework figures are often factored into engineering project estimates in the form of contingencies (often allocated as 25% or higher).
Any type of project can be examined in terms of its requirements. Requirements can be easily broken down into three distinct types – and, in so doing, increase their understandability with both users and the project team. The three types of requirements are: (Dekkers, 2004)
Functional requirements. These represent WHAT will be built in terms of business requirements for the product. In a construction project, this is the floor plan. In software it is the functions or processes performed by the software, (e.g., record ambient temperature) and include what functions the software must do. These requirements are part of the users’/customers’ responsibility to define. Functional requirements can be thought of as similar to the product's Floor Plan. Functional requirements can be documented with words and diagrams that outlined what will be required functionally.
Non-functional requirements. These represent how the product must perform once it is built. These include the “ILITIES” : (Suitability, Accuracy, Interoperability, Compliance, Security, Reliability, Efficiency, Maintainability, Portability, and Quality in Use) as described by ISO standards ISO/IEC9126 series. While these requirements are also the responsibility of users/customers to ensure proper definition, they are often not articulated explicitly (or at all) but rather are “sprinkled” throughout requirements documents in dribs and drabs. In construction projects, the non-functional requirements are the contracted specifications, while in software projects they represent how the software will perform.
Technical (build) requirements. These types of requirements address how the product or solution to the functional and non-functional requirements will be built. Technical requirements include tools, methods, work breakdown structure (tasks), type of project (new construction vs. renovation), etc. This is where homebuilders and software developers have the most affinity with the requirements because this is where the other project requirements (the two above) are combined and a “blueprint” is drawn.
In software, modern development approaches, such as use cases and agile development, attempt to keep these three types of requirements distinct and separate—when used correctly. Unfortunately in a manner similar to the contractor who only has a hammer and everything looks like a nail, some software developers cannot overcome the need to insert technical requirements into modern method deliverables such as use cases and agile user stories.
Estimating Before Requirements
When a cost estimator or project manager is asked for estimates before solid requirements are articulated, guesses take the place of solid information. What types of challenges arise when doing this type of “estimating”:
Pre-functional requirements: Often cost estimators or project managers may be asked for an estimate based on functional requirements scrawled on paper napkins or other informal, non-standard presentation media. One can only guess at what the software really will do based on assumptions such as: Kind of sort of like…(another system); or rough “ideas” that are conceptual at best.
Pre-nonfunctional requirements: Assessment of the “ilities” of how the product will be required to perform can be done based on comparing them to other projects already completed for the same project department or business area. Unless there are major influences across the gamut of these non-functional requirements, they are not often seen as required input parameters needed to increase estimate accuracy. To overcome this flawed assumption, cost estimators and project managers often underestimate the complexities that the project will bring.
Pre-technical requirements: On many construction and software development projects today, the project teams use a standard suite of building toolsets and technology aids. The technical requirements area is generally the least dangerous of the three areas in project estimating. Assumptions are generally made that the tools and techniques to build the solution are pre-selected along with other technical aspects of the project.
When faced with the daunting task of producing an estimate for a project for which there is little or no known input variables, what are the options for a project manager (PM)? He/she could attempt to do one of the following, however, the response from management would likely be negative and could cost the PM his/her job:
- Refuse to do an estimate (too early);
- Delay estimate repeatedly until requirements are at least partially done;
- Wild guess(timate);
- Use “Kind of sort of” actuals as the estimate;
- Cite “professional” ethics and hide out…OR…(and this is the preferred method);
- Document assumptions and use them together with the estimate (guesstimate) to substantiate the estimation results.
When looking at software projects, even at the preliminary, pre-requirements stage, the guesstimate can still address and document the assumed (but unknown) aspects of functional requirements. As such, these requirements can still be sized using a measure called “Function Points” – which are equivalent to square feet for software. To do so at this early stage, an approximate size can be determined by assuming that each maintained data group in the software will typically follow the Add, Update, Delete, Inquire and Output (the AUDIO) profile given that the functionality is updating/maintaining the particular data.
For non-functional requirements, documented assumptions are critical here too. Overlooking or underestimating the complexity of the non-functional project requirements can cause major problems in keeping to the original estimates produced – because they would have emerged without documented and defensible assumptions.
For technical requirements, it is important to document novelty items (methods, tools, lack of skills, etc) for the technical aspects of the project. The impact of good or bad personnel skills on project teams can increase or decrease morale and project productivity (thereby altering the delivery speed and project completion either positively or negatively).
Recommendations for Estimating before or during Requirements
The following list emphasizes the need to document and substantiate guesstimates and estimates when a project manager is faced with having to produce and estimate before the requirements have been fully fleshed out:
- Review and identify as many of the known knowns and known unknown requirements as possible;
- Separate and document project requirements as the three distinct requirements types;
- Document all of your assumptions (and then don't forget to validate them again later on the project for future releases and associated pre-requirements estimates.)
- Create a range of “guesstimates” when there isn't enough information to generate an informed and reliable estimate. (AND, ensure that the implied accuracy of an estimate is not misconstrued – just because your project estimating software includes numbers with decimal places in the results, use common sense and realize that the estimate is only as good as its least input;
- Ensure you use standard estimating models that are proven for your environment (and ensure that the model is appropriate – e.g., for homebuilding projects, one model may be good for new construction, while another may be more appropriate for renovation projects);
- Communicate that the guesstimate or estimate is only a ball-park number because the requirements are not yet complete;
- Label results as “Preliminary”;
- Level with your Customers – there is no magic estimating wizard who will direct the project (no “Magic”) -- ensure that they understand that an “estimate too early in the project” cannot remain fixed throughout the project, nor can it be accurate.
- Despite your best wishes, project managers cannot create estimates out of “ether” or unknown requirements
- However, (guess)timates based on documented assumptions are a step forward from making guesses based on undocumented assumptions.
What Can You Do to Improve Project Estimation at YOUR company?
In summary, project estimating can get better if you follow some simple advice:
- Document as many of your assumptions about the project as you can, and revise them and the estimate according to these same assumptions to establish estimate traceability.
- Document the requirements clearly and objectively:
- Divide into 3 types;
- Consider recommendations presented herein;
- Consider objective measures of project size by requirements type;
- Practice, document, follow-up, learn from prior “guesstimates” and estimates.
- Share estimating and measurement data with others both in and out of your organization (benchmark using non-proprietary benchmarking databases such as ISBSG for software development).
We will never have the perfect environment where all requirements will be clear and complete before estimates are required, but we can improve the life of projects and project managers by documenting assumptions on which early estimates are made. It is through that knowledge and taking a retrospective look at that data that can bring about positive process improvement and increase estimating accuracy.
Army Software Metrics Newsletter, (2003, Summer).
Card, D. N.,(1998) The Role of Measurement in Software Engineering
Dekkers, C. (2002), Tutorial: Use Cases, Function Points, and Software Quality – Where's the Fit? Dekkers and Quality Plus Technologies, Inc.
Hammond, J. & Morrison, J., (1996) The Stuff Americans are Made of: The Seven Cultural Forces That Define Americans – A New Framework for Quality Productivity & Profitability, New York: MacMillan..
Hill, P. R., Ed. in conjunction with the International Software Benchmarking Standards Group (ISBSG), Software Project Estimation – A Workbook for Macro Estimation of Software Development Effort and Duration, 1999.
Nelson, M., Clark, J., & Spurlock M. A. (2002, November-December) Curing the Software Requirements and Cost Estimating Blues, PM, November-December, 2002
Rubin, H. (1998, August) Y2K. IT Metrics Strategies, Cutter Consortium.
© Carol Dekkers
Originally published as a part of PMI Global Congress Proceedings - Singapore