In the U.S., over 250 billion dollars are spent on information technology (IT) projects, and over half of those projects fail (Reiber 1999, p. 17). A critical shortcoming in the IT industry is a grasp of IT specific project management principles, which some attribute to information technology's reliance on the engineering body of knowledge (Kapur 1999, p. 1). One of the major complications in IT development projects is the highly cyclical, recursive nature of project stages (hereafter referred to as iterative processes), which invalidates many of the traditional management principles IT managers apply.
In a flexible methodology, system design is continually revisited as development takes place. This process should provide increasingly clearer requirements as the project moves forward, and the customers should be highly involved as iterations of design take place. However, this approach is not as successful if inefficient management strategies are applied to the complexities of iteration. An iterative process begs a project approach that embraces the changeability and flexibility of IT, so that we can fully exploit the benefits of IT. Given the tremendous time demands of the Internet economy, it becomes more vital than ever to create a parsimonious (unified, direct, lean) project methodology. With that in mind, this paper presents a conceptual model, called The Blur Box, for understanding iterative processes such as Object-Oriented development, Rapid Application Development, or complex integrations. This model is not a methodology, nor is it a project management tool. It is a way of looking at iteration so that informed business decisions can be made when attempting such a project. The model is essentially a business communication tool, which could be used in many ways: convincing upper management that certain documents are unnecessary, showing new project managers what they are in for, or explaining to the customer that the random nature of calls they'll receive is not haphazard but good.
The Blur Box model is presented here from a development-centric viewpoint. However, the concepts apply equally well to management, implementation, scoping, etc. The concepts are meant to be generalized.
The Blur Box Backstory
Imagine you are overseeing two projects with equal staff (25 developers, 4 business analysts, one project manager, support staff for training, testing, and database) and equal budgets (15 million dollars). The goals of the projects are identical. The customer is in-house. You trust your development staff because they have proven reliable in the past, but the problem this time is one that your company hasn't faced before. The business requirements are somewhat clear and you know on which platform you will implement.
The manager of the first project has done much preparation. She has provided you with detailed work breakdown structures for each phase of the project and allocated her resources accordingly. There is a solid timeline with phases broken down into weeklong time boxes. The developers follow a clearly defined methodology: they perform analysis and document their findings, then design the system, then code the system, then test it, then implement it on the delivery date. There is a meeting and a status report generated before each phase begins so that everyone is on the same page. When customers need to be consulted, meetings are scheduled in advance and agendas drawn up so that the customer knows what will be covered.
The manager of the second project has a pretty good idea of what his team will be doing in the next couple weeks, but is vague about anything thereafter. You have asked for some indication of scheduling or resource allocation, but he says he has no idea what his team members are doing at any given time. Furthermore, requests you stop asking him for schedule updates. The team is one-third of the way into the project and hasn't produced a single working piece of software. When customers need to be consulted, developers call up and ask if they can meet that afternoon because they still need to analyze part of the system. There is no clear documentation of how the team plans to develop the software, just a list of features they plan to implement. You get concerned and call a meeting to discuss the status of the project, only to find that there is no work breakdown structure, no resource allocation chart, and no timeline.
Which one is the better project? These are admittedly sketchy examples, but according to the Blur Box conceptual model, the second project is on the right track.
One possible subtitle for the Blur Box model is “Why We Need a New Business Culture.” This model is an attempt to describe why it is impossible to apply traditionally successful management strategies to certain types of projects, particularly iterative projects. Iteration occurs any time a team revisits a design or plan in light of new information. The essence of the model is this: the development process is inherently iterative, and thus changes rapidly. It looks like chaos or anarchy, and the very act of measuring it slows it down. But it is not chaos, and if you know how things work you can better manage them. The problem is that your instincts and training are working against you.
Exhibit 1. The Blur Box
Model Structure
Exhibit 1 depicts the basic structure of the Blur Box. The Pre-Launch phase contains all of the pre-technical elements, such as requirements gathering, scope delimiting, budgeting, hiring personnel, defining roles and responsibilities, etc. The system testing, user acceptance testing, and implementation are fairly traditional, straightforward, and linear, and are not the focus of this model. The four “phases” in the middle, however, are not clearly delimited. Developers move between, among, and around them as necessary during the entire project lifecycle. The basic idea of each phase is similar to the traditional waterfall definition: Analysis is communication with the customer about the system and their needs, General Design is nontechnical design about how the system will function, Detailed Design is technical specification of the system, and Development is actual work on the system. Do not confuse the Development phase with actual System Development, which is programming. For example, in Business Process Redesign, Development could refer to the creation of new business practices.
A project is like an empty trough that needs to be filled with water. There are many ways to get the trough filled. The waterfall methodology attempts to fill the trough with five or six huge buckets: The analysis bucket, the design bucket, the development bucket, etc. Having poured all of those buckets in, you may find that you filled the trough to overflowing and much of the water has poured off into the ground. The Blur Box says filling the trough is more like a steady drizzle of fresh raindrops. Each particular raindrop could be an analysis, design, or development raindrop, but it will keep raining only until the trough is full enough. Think of the water in terms of resources and the raindrops in terms of incremental development.
Exhibit 2. The Flow of Time
Time Flow
The approximate flow of time through the phases is represented in Exhibit 2. Pre-Launch is pretty much complete before Analysis begins, and most of Analysis, General Design, Detailed Design, and Development is complete before System Testing begins. Phases are not distinct, however. Because the process is iterative, some pre-launch activities may carry over into development. For example, if researching the customer's business needs reveals the need for a business change, role definition may change in the middle of the analysis phase. System testing may begin before all coding is complete, in order to get as early a start as possible.
Threads of Work
A very important concept is “threads of work” (Yates 2000). A thread is a concerted effort that is somewhat autonomous. This is not to be confused with a task: threads of work have tasks. A thread also differs from a subproject in that it does not imply dedicated resources. It is called a “thread” because it passes through all the phases of the project, weaving in and among the other threads. Exhibit three depicts several threads moving through the project: Business Process Redesign (BPR), Systems Development, and Operational Data Storage (ODS). According to this model, all three of these processes are threads because they move back and forth between analysis, general design, detailed design, and development as needed. Each of these threads has different tasks to be accomplished by different people, who may or may not be involved with another thread.
Thread Independence—One reason why the Blur Box is so named is because threads are independent. In other words, the development of the system is not dependent upon completion of BPR, and the ODS implementation does not have to wait until development is finished.
Exhibit 3. Threads of Work
Thread Divergence—These threads occur simultaneously, but none of them can start before the Pre-Launch tasks are “finished.” There is a point in Exhibit 3 where the Pre-Launch arrow meets the begin points for all of the threads. This point is the Divergence Point. The ride is about to begin. After this, the controlled chaos of multiple, simultaneous threads ensues. The Divergence Point is an important time because it is the last traditionally manageable point; a good time to schedule one last team meeting.
Thread Convergence—At some point these various threads must come together so that logical analysis of the entire system can occur and testing can begin; don't trade analysis paralysis for unending developing. Specifying a Convergence Point lets all threads know when they must be finished. What does the Convergence Point mean for our hypothetical three threads? In the case of BPR, all redesign should be complete, taking into account the coding that has been done and the best possible business solutions. System development should have produced all code for the expected functionality and done unit testing on the code. The operational data store should reflect the changes made by the coders, and take into account the new business practices detailed by the BPR thread. Because of the complexity of thread convergence, it is not really a single point in time, but rather a date to begin unification. It is only at thread convergence that the team truly can sit back and see what it has.
Thread Dependencies
Of course, threads do not exist in a vacuum either. Exhibit 3 shows the threads neat and parallel with each other, but the reality looks different. The Divergence Point is not truly the starting gun for all of the threads. Work on one thread may have to start and stop because of dependencies on another thread. For example, System Development cannot begin the Analysis phase until BPR has progressed through iterations of Analysis, General Design, and Detailed Design, because the system needs to support the new processes.
Exhibit 4. Thread Dependencies
Eventually BPR work will have to stop until System Development reaches a certain point—the details of BPR must wait until the capabilities of the system are known. ODS development cannot begin until System Development and BPR are reasonably far along. Otherwise, the ODS thread is trying to hit a moving target. Thread dependencies cause the model to look more like Exhibit 4. The dots represent a thread needing to start or stop based on another thread. All threads need to come together at system test time, hence the Convergence Point.
The Anatomy of a Phase
Wedge Proportions
As a thread moves through the project timeline, the time spent in analysis decreases while the time spent in development increases. This is depicted in the model by the wedge shapes of the phases. Exhibit 5 breaks the Blur Box down into thirds to depict the rough proportions of time spent in each phase as the project moves along. There is not a direct time correlation, but the top third reveals that there is no time spent in Development. The middle third has all four phases in roughly equivalent portions. The bottom third is dominated by Development, although some Analysis can still occur even at this late stage; at any time a person or group of people can revisit design or analysis if it makes sense to do so for their part of the thread. Do not get too attached to the notion that threads follow a noticeable pattern, however. Threads do unpredictable things.
Divergence Anchor
At the beginning of a thread, it is almost certain that analysis will occur first, then design will follow. But as soon as some design has been completed, the path is unclear. The team may decide that more analysis is warranted. They may stay in Design awhile. They may jump right into development. Or one person might start developing while three analyze and the rest design. The Divergence Point is the only real anchor.
Exhibit 5. Wedge Properties
To visualize this concept, pretend that the threads are all pieces of string and the Divergence Point is a thumbtack stuck through the ends. Now turn on a fan and try to predict where the threads will go next. It is impossible to do so. You might be able to concentrate on one thread and guess where it will go, but then you lose comprehension of the other threads.
The Anatomy of a Thread
Uncertainty Principle
In physics there is a principle called Heisenberg's Uncertainty Principle. In simple terms, you can know the speed or location of a particle, but not both at the same time. This is how the Blur Box model treats human resources. You can know in what general direction a person or team is going, but not exactly where they are in the process. You could know if you stopped the process and had everyone analyze where they are, but that would be counterproductive. How is this reflected in the model?
Consider BPR, the first thread. The figures so far have depicted a line moving back and forth through the stages. But what that line actually represents is the general direction of the team working on BPR. The “line” is actually a team of people all doing different things that take the thread in a general direction. It is not clear exactly when the team is in General Design as opposed to Analysis, because they could hop back and forth between General Design and Analysis several times within the same day. Therefore, the model looks something like Exhibit 6. Instead of seeing a distinct line, you see a rather hazy impression of the team moving.
Exhibit 6. The Uncertainty Principle
Through Thick and Thin
At this point you may still have the idea that you can track where the team is going and roughly when they will get there. But there is still a little more reality to contend with.
A thread is a concerted effort that is somewhat autonomous. Remember that it is not called a subproject because that would imply dedicated resources. What is actually happening with resources, and how does it make the thread look?
A thread has a collection of resources, such as customers, database administrators, managers, developers, and analysts, all working toward a goal. However, all of the resources available to a given thread will not be working on that thread simultaneously. Perhaps everyone will be following the same path at the beginning, then will break off to pursue other tasks. There are also other threads, which means at times no one will be working on a particular thread. A thread actually looks more like Exhibit 7. The thread is thicker when more people are working on it, then it frays as people move to thread-specific tasks or to other threads. There may be large gaps as people leave a thread, or large clumps as many people join a thread. You simply cannot tell where a thread is going with any real clarity.
Multiple threads actually cloud the picture even more. If someone leaves a thread and works on another for a brief amount of time, it impacts two threads. If you were to stop and ask for a status report, you might find that the report is an almost complete reversal from the last report. Last week John was analyzing BPR, this week he is developing the ODS.
Teleportation
Just to drive the point home, even thinking of a thread as having an observable direction is misleading. Exhibit 8 shows a possible path for System Development. Once finished with a round of General Design (point 2), the team may decide to skip to Development (point 3) for rapid prototyping. When the prototyping has progressed a while (point 4), they may need a more specific design (point 5), which causes them to realize they need more customer input (point 6). With that input comes clarification, (point 7), so the team jumps right back into development (point 8). The team does not actually pass through the phases depicted by the dotted lines, they jump directly to the required phase. As illustrated by point 5, they may not actually remain in a phase for very long. It is important to note that this skipping around is not the result of poor design or analysis in the previous phases. It is iteration brought on by learning new things or facing unknowns. This model embraces the concept that the team doesn't know where it will go: sometimes, the team doesn't even know what it needs to know.
Exhibit 7. Resource Movement
The Fuzzy Blur
All of the above concepts are meant to illustrate how iteration looks and how it can be conceptually confusing. Needless to say, many traditional management practices are ineffective when applied to such a project. One way to describe the process of iterative development is as a fuzzy blur.
Time Stop
There is a game called freeze tag where everyone is running around, but when someone calls “freeze” everyone stops in place. If you wanted to, you could stop the process of iteration and figure out who is doing what. But is that productive? It depends on the frequency and duration of the iterations. At the beginning of a thread, most people are working on the same thing in the same phase, and stopping time would allow you to get a good idea of where everyone is. But the most productive iterations also tend to be the fastest. If a team is in the heat of iteration, one person could teleport between phases almost hourly. If you were to stop time and ask where everyone is in the process, it might take longer for them to deconstruct where they are and how they got there than the actual iteration would take. This status check would also disrupt the process, probably slowing it.
Exhibit 8. Phase Telportation
Tapestry on a Loom
It might be best to think of the process like weaving a tapestry. You have a handful of threads and you want to weave a tapestry. You could opt for a manual loom. You would have control over each thread, dictating when each thread starts or stops. You would know where everything is and can explain to others where each thread will go next. You can even control exactly how the tapestry looks. But it would take a long time and your fingers would get tired. If you used an automated loom, you would select your threads and position them. You would tell the loom how you want the tapestry to look. Then you would start the loom and watch the threads dance. You could try to follow the threads’ movements, but you might get dizzy. You could explain to someone where a particular thread is going, but by the time you had explained it, the thread would be going somewhere else. As the process went along, you wouldn't know precisely how the tapestry was shaping up until the loom had stopped and you'd clipped the threads. You would lose control over the details of how the threads went together, but the finished tapestry would look similar to what you had envisioned.
The Third Dimension
Exhibit 9 reveals that this blur of activity is multi-dimensional. Individual developers can be working singly or together, but will rapidly work through analysis, design, and development— in time as well as space. They may jump threads or work on either side of the main thread trend, or up and down within the box. One person may be laying the groundwork for another, and this may repeat itself many times throughout a day or week.
Exhibit 2 showed the approximate flow of time. It is the same in Exhibit 9, in that the Z-axis is time, and the X- and Y-axis are space. A developer can be at her desk coding, or she might be at someone else's desk designing, or she might be at the customer's office analyzing. People move through the phases, time, and space in an unpredictable way.
Exhibit 9. The Blur Box
Conclusion
Pause a second and consider how you felt about this model. If you found yourself nodding your head in agreement, perhaps yours is the sort of development project that could benefit from this approach. In this case, the utility of the model is to explain to others what they can expect from your development project. If you found yourself indignant at the absurdity of trying to apply this model to the realities of project management, then perhaps yours is not the type of project that this approach would benefit. But consider the possibility that if this model irked you, then perhaps your organization needs a different perspective on development; thus, the utility of the model is to help you think about iteration in a new way.
The main benefit of the Blur Box model is avoidance of the “late discovery” syndrome of the waterfall methodology. Design, user requirements, and development are revisited so frequently that there is a very low probability of implementation failure. Of course a large part of the success of such a framework is having experienced developers and an experienced chief architect. Without such leadership and guidance at the helm of development, the guidelines presented from this model will not work.
The reality of iterative development demands a culture change within the project development team. The primary concession is that IT needs to not only endure change, but also embrace it. What does this mean practically?
Use a sliding timeline (Exhibit 10), periodically revisiting the sliding three- to six-month plan and updating it. The meaning of “periodically” changes as the timeline moves. At first revisit it weekly, but near the end of the project update it daily or even more frequently. Do not plan tasks or deliverables at all beyond the three- to six-month sliding plan, because there is no way to know what those will be.
With all of the threads of work and iterative development, a task list or resource allocation chart is futile. Staying on top of it would be nearly as much work as the development effort itself. The way progress is demonstrated is through release cycles. A release is a discrete chunk of functionality delivered at a specific time within the project timeline. Exhibit 12 shows a typical release schedule. Developers determine a feature set to be implemented at Alpha 1, Alpha 2, etc., and provide a date for that release. Progress is measured by whether implementation occurs on that date and whether all functionality is delivered. Working in this way gives managers, customers, and the project team clear targets and clear progress indicators. What this means practically is that six months into the project, there may be very little actual development to show. Building frameworks and base classes (such as in Object-Oriented technology) allows production of a lot of functionality near the end of the timeline. This makes people nervous if they do not understand the nature of the team's development philosophy. Ironically, the point at which iterative development begins to yield the richest rewards is the same point that a project is in danger of cancellation due to perceived insufficient progress.
The most critical success factor is communication between developers, testers, business analysts, managers, customers, and other stakeholders. Communication is one of the difficulties of most business processes, but in the case of iteration it is both more critical and more complicated.“…Software development companies do not function themselves as systems. Instead, they function as a set of totally asynchronous and rarely communicating divisions which succeed only by accident, not by design” (Davis 1999, p. 1). The role of management should shift from managing development to communication and facilitation. At some point the development team has to be trusted to deliver the functionality it promises. Managers who focus on managing communication are more likely to succeed in the middle of the Blur. The development team has new responsibilities as well: the development team must take responsibility to inform all stakeholders of what is going on. Specifying formal communication responsibilities is a possible way to enforce communication. For example, the development team has to communicate with the technical managers and business analysts, who are responsible for notifying other managers and customers. Customers are not off the hook either. Customers must become inquisitive about implementation status rather than delivered functionality. It is clear that the current strategies of application development rarely work. That is not to say there aren't successes, but IT hasn't achieved repeatable standards for project management success. In an age of e-commerce where delaying an implementation by a month can mean complete failure, IT needs to reassess which management tools are necessary and which are merely artifacts of the need to feel control over a project. Projects must be managed, but the key question is “how”? It is up to each individual to consider how an iterative process affects him or her. With that understanding, managers can build a management strategy that will help avoid disaster and increase the likelihood of project success.
Exhibit 10. Sliding Plan
Exhibit 11. Release Cycles
