Abstract
Without a meaningful definition of quality, how does the business know the project team has achieved their desired quality milestones? The Project Management Institute (PMI) distinguishes between quality and grade (PMI, 2008, p. 190), and that distinction can help answer this question. People expect different experiences from “prime” cuts of beef than from “select” cuts of beef. In software development, business stakeholders should define quality differently for different grades of software, such as prototypes, production releases, and proofs of concept. This paper explores how one agile team dramatically improved its quality practices with a set of simple tools and a focus on the important concept of “grade.”
Measuring Quality: Quality vs. Grade
Beeson and Sitarski (2009, pp. 161–166) described why the distinction between grade and quality is important:
Determining a single definition of quality for a project is challenging, yet it is vitally important to understand and define the goal before a project can be judged against it. Try the following experiment: Ask five people to describe what makes a quality roller coaster ride. Their judgment of quality is going to be affected by their expectations and prior experiences, thus their definition of quality may vary greatly. The same is true for users of software. Ask any software user to describe a quality email application. Their definitions will vary depending on how often and for what purpose they use email. Quality is subjective.
On a typical software project, a QA team might judge quality by simply verifying that the application meets the specifications. A more mature QA team might consider two important perspectives for judging quality: (1) Does the application meet the business needs? (2) Does it meet the needs of the user? While at times these both may be in alignment, it is more often the case that they are at odds with one another. The challenge is to describe in tangible and measurable terms a set of goals that meet a defined set of user and business needs. To do so, we must define the grade of the application.
PMI defines grade as a way to distinguish functionally similar items that do not share the same quality requirements (PMI, 2008, p. 190). For example, consider the following three ways to access the internet: dial-up, cable, or DSL. We would not expect the inexpensive dial-up connection to be fast and reliable. However we would have those expectations of the more expensive cable or DSL options. Already we have established three basic criteria for determining the grade of an Internet connection: cost, speed, and reliability. These, in turn, can be used to define a measurable set of criteria for judging quality. In other words, one must define grade in order to determine measurable quality requirements.
Why Is Measuring Quality Important?
Ask a client what level of quality they would like to achieve for a product. More often than not they will answer with some version of “It has to work.” It is a rare client who will directly tell you that they are willing to put up with 1,000 bugs if it means that they have enough functionality to get the next round of funding. However, this is often the reality of project planning.
A project plan is constrained by three factors: scope, schedule, and budget. Project teams must be careful to avoid fixing all three at a level that makes it impossible to achieve success. This usually means that one of the three is declared, implicitly or explicitly, to be the flex point for the project plan. Because of its position as a vendor, Menlo Innovations' projects almost always have a firm budget constraint. Schedule is often firm, but not usually as firm as budget. It is scope that must remain flexible. This poses an ever-challenging question to our clients: What scope must be cut in order to meet the schedule and budget constraints? To answer such a question, one must have an idea of what “done” means. Is a piece of functionality “done” enough to fulfill its purpose? Does it meet a sufficient grade?
It is too easy to sink a project simply by getting caught up in the need for “high quality” when “adequate quality” would suffice. This is where using grade to define adequate quality becomes an essential tool for minimizing cost and maximizing scope.
Examples of Grade vs. Quality
Vacation House
Our expectation of a vacation house is not the same as a permanent residence. Many vacation houses are not winterized. Why bother when no one will live there during the winter months? It may be smaller than your average permanent residence. Who needs a big house when there is an ocean or lake to swim in? It will most likely have the basic appliances (washer, dryer, refrigerator, stove …); however, they may not be high-end appliances considering they'll only be used a few weeks out of the year. We may expect a vacation house to offer a wonderful view of the ocean or lake we spent so much money to be near, while we'd rather our permanent residence be functional.
All are factors in determining the grade of a house. The appropriate grade of a house is determined by its purpose; vacation vs. permanent residence. A house that meets the appropriate grade for its purpose is considered a quality house.
Beef
You may be familiar with the USDA's quality grades of prime, choice, and select beef from your grocer. Perhaps you have even thought about what those grades mean. More likely, you would react instinctively that it would be foolish to use prime grade porterhouse steaks in a beef stew. In fact, your favorite stew recipe may not taste as good using prime porterhouse as it would with the select grade stew beef the recipe called for. After all, the seasonings were tailored to the stew beef, not to the porterhouse. Likewise, serving select grade steaks off the grill to your boss may not be the most satisfying experience. In other words, we pick the appropriate grade beef for the purpose of using it in a particular recipe, resulting in a quality meal.
Brooklyn and Manhattan Bridges
The Manhattan Bridge in New York City will be 100 years old on December 31, 2009. The Brooklyn Bridge, directly to the south, is 26 years older. During those 26 years, the automobile and airplane were invented and the first lines of the NYC subway were built. Both bridges carried train traffic at one time; only the Manhattan Bridge still does so. Train traffic was halted over the Brooklyn Bridge in 1950 because of the wear it was causing on a bridge that had been built only for wagon and trolley traffic. In contrast, the Manhattan Bridge's designers had anticipated early 20th-century trucks and subway trains, but did not anticipate late 20th-century loads. In fact, beginning in 1985, train tracks on the bridge were closed intermittently over the next 18 years while the bridge was rebuilt to accommodate the heavier loads of the late 20th-century. Is the Manhattan Bridge a low-quality bridge because it had to be rebuilt? Is the Brooklyn Bridge a low-quality bridge because it can no longer support train traffic? Again, the issue is fitness for purpose.
Tools for Defining Quality and Grade
At Menlo Innovations, we have several tools we use to help our clients define and describe their quality expectations for a product. We begin every project, large or small, with some degree of utilization of each of the following tools.
Personas
Personas are used to describe the potential users of a product. As is often the case, products will have several different types of users. However, budget constraints make it difficult to build a product that meets the needs of every conceivable user. Thus, Menlo Innovations asks that its clients decide who will be the primary, secondary, and tertiary personas for the product. Once this is determined, the persona can be used as a tool for keeping functionality in the context of its purpose. After all, how can we be building a quality product if we don't know or understand its users?
Problem Statements
One of the most effective but least frequently asked questions in project management is, “What problems are you trying to solve?” Asking this question should result in a constructive debate among an organization's management team. This debate is an important tool in building a collective agreement on the problems to be solved and their priority for the organization. Exposing discrepancies among the management team and resolving them as early in the project as possible will almost always lead to a higher quality product.
Goal Diagrams
Goal diagrams offer a view of the intersection of the goals of a project's personas and the functionality that can be provided by the product. They give insight into the question of “Why,” giving the functionality a context and purpose. To build a quality product we must know why the user wants to achieve a particular goal and why the business wants or needs the user to achieve that goal.
Business Object Model
A business object model describes the top-level business objects and their relationships to each other. These relationships are the foundation of both business processes and the tools to support those processes. It is through this model that we are able to understand the question of “What” from the business perspective. Thus, it is important that the business stakeholders are involved in the creation of this model in order to establish a common understanding of the objects and the language to describe them. We have found that one of the biggest quality problems in software is a mismatch between the relationships of the objects in the object model and the relationships of the entities, or objects, that make up the real-world business processes. At Menlo Innovations, we believe that a common understanding of those relationships is one of the most effective tools for building quality software.
Workflows
High-level business process workflows are important for establishing a common picture of both the sequence of activities in the business process and the terminology used to describe those activities. Within the context of the overall business processes, they provide insight into what the persona is trying to accomplish. As with the business model, we believe that a common understanding of the sequence of activities is crucial to building quality software.
Grade Map
Menlo Innovations uses a grade map as a tool to help our clients prioritize the risk to the business or the user in the event a particular goal cannot be accomplished. It poses the question of “How bad is it to my business or users if the primary persona cannot do X?” The grade map opens a constructive dialog about quality by avoiding the question of quality.
Planning Game
Menlo Innovations believes in and practices a rigorous scope management process. Every project is decomposed into deliverable-based work packages called story cards. Story cards are prioritized and approved by the client on a weekly basis in a transparent and regularly-scheduled change management process called the planning game. Of particular interest are the story cards that are never approved. This is an explicit decision by the client about the quality and grade of the product.
Using These Tools to Define Quality Goals
At Menlo Innovations, we have found that our most successful projects are those for which we have devoted a significant portion of the design phase of a project to building and refining the previously mentioned tools. This process can last anywhere from four weeks to six months depending on the size of the project. It is, however, challenging at times to describe to our clients the importance of each tool when the most common approach to (and expectation of) software development is to start writing code as quickly as possible. Even our best clients have had the inclination to prioritize developers working on code over defining expectations first. The key is to find the right balance of discovery vs. build. It is our belief that diving into code without a firm understanding of who the product is for, what they will be doing, and why results in certain failure. This is not to say a project is certain to be successful using these tools, rather we believe they greatly increase the chances of success.
One objective of our analysis and design phases is to create a common picture of quality without asking the question, “What level of quality do you want in this product?” Because the answer to that question will always be, “high quality, of course.” These tools are not really for Menlo Innovations. They are tools used by Menlo Innovations' clients to aid in describing the product they are not just looking to build but the product they need to build. Menlo Innovations facilitates the conversations, gathers the information from all sides, and re-articulates it back to the client stakeholders, usually in the form of a picture. We are rather accustomed to being wrong. In fact, one of our greatest strengths is giving the client the opportunity to constructively tell us in what way we are wrong. After all, the conversations that result in our being wrong are usually where we learn the most useful information. Taking the iterative and incremental approach, we then change or refine the picture, all the while gaining a better understanding of client expectations. This approach is what allows us to strike the right balance between discovery and build. Once we have refined the picture to the point that the client stakeholder deems it adequate, we move on to the next priority.
Over the course of a project, the Menlo Innovations team builds a habit of using and re-using these tools, constantly revisiting their output. This keeps us looking for changes to the project's foundational assumptions, enabling us to identify when they have changed. Adjusting accordingly and deliberately is the true hallmark of agile.
The tools represent a small number of artifacts that describe the project from several important perspectives. The wider perspectives they provide keep us from focusing too narrowly on a single aspect of the project and help define the criteria that make up the quality grade of the application. Not one of these tools will solve your problems. When used effectively, they merely highlight problems more quickly or more completely. The team then describes possible solutions in the form of story cards. Armed with these tools and concepts, the client can use grade to make an informed decision about which solution best fits the purpose at hand.
