Abstract
We all work in environments in which government funding is stretched, program resources are syphoned, disparate applications are waiting on future funds for cycles to implement backlogged features, and legacy applications are teetering with release patch development. In these environments, the project management office (PMO) becomes a resource for a better tactical project management strategy for mentoring project managers toward custom-off-the-shelf (COTS) solutions. Cookie-cutter templates give way to strategy techniques, Agile and collaborative monitoring, and tracking.
Determining the best value added COTS solution means obtaining better product requirements based on valued outcomes and requiring integration milestones and checkpoint activities to ensure both vendor and in-house deliverables are on the same integration track. COTS projects are more than just contract administration. This paper examines the COTS project management strategy and techniques for mentoring of project managers supported by the PMO.
Introduction
The planning of purchases and acquisitions considers whether, how, what, how much, and when to acquire project scope that is best met outside the project owner’s organization. (A Guide to the Project Management Body of Knowledge [PMBOK® Guide], 2004, p 269)
COTS solutions are third-party solutions that are bought, licensed, or acquired; often, they are integrated into a larger system. COTS-based systems (CBS) are COTS solutions in which at least 50% of a system is based on a COTS product. COTS products often have customized configurations, but may also be solutions in which the base product deviates into a modified-off-the-shelf (MOTS) solution.
“The term commercial off-the-shelf (COTS) is very generic; it can refer to many different types and levels of software, e.g., software that provides a specific functionality or a tool used to generate code. COTS may be one of the most diversely defined terms in current software development.” (Morisio, Seaman, Basili, Parra, Kraft, Condon, 2009, p 1)
The procurement category reviews whether the procurement is a major complexity, minor complexity, or routine custom-off-the-shelf (COTS). These three categories may also be an interdivisional or teaming arrangement buy. Teaming arrangements are agreements that are legal corporate contracts with other entities for strategic reasons. Interdivisional buys are agreements for procurement within the same company due to unique abilities in other divisions. (Fleming, 2003, pp 14–22)
Procurement strategy for COTS solutions for the Alaska Department of Health and Social Services (AKDHSS) identifies factors that need to be considered in purchasing a COTS solution (Exhibit 1). Prior to even considering COTS, the “make or buy scope” decision must look at the work breakdown structure (WBS), risk breakdown structure (RBS), and cost breakdown structure (CBS).
Exhibit 1: State of Alaska Procurement Components for Contract Planning (Douglas, 2008, p 2)
COTS Project Management Strategy Quandary
“By choosing to use COTS, an architect takes on an additional risk that he or she cannot control. Vendors may go out of business, choose not to support the existing components, etc.... COTS components are just another factor that an architect must consider when choosing or designing the architecture of a system. Use of COTS has its associated risks, as COTS components can be thought of as black boxes that just work. The tradeoffs between COTS components and homegrown components are development time versus flexibility and control. Because COTS components are restricting by nature it is critical for appropriate stakeholders to be more involved in those aspects of the system that are not controlled by a software architect.” (Software Architecture, 2010, p 1)
COTS projects are surrounded by known and unknown risks, both for implementation and for ongoing sustained engineering and maintenance. One of the biggest questions after the procurement decision is how to address the underlying COTS paradigm regarding which project management strategy or strategies to best manage the project.
In part, this revolves around the development or implementation requirements for COTS: Are we dealing with Glue Code, tailoring effort, or assessment effort? Glue Code is the requirement for custom development by an application that is external to the COTS package to allow integration. Tailoring effort requires custom code by the vendor within the COTS package to allow integration. Assessment effort evaluates the suitability of using the COTS right out of the box with only altering the human workflow processes to use the COTS (Yang, Bhuta, & Port, 2011, p 5). Often COTS projects are a combination of all three efforts.
After deciding on a COTS solution, the underlying COTS paradigm then must answer: What standard development practices fit the solution for integration of a COTS implementation? What is the best project management strategy (or strategies) to wrap around the entire process to manage the integration process proactively?
Complexity compounds managing COTS projects. COTS systems may combine highly diverse technologies; may incorporate multiple COTS packages, require modifications on the contractor side of the COTS configuration, and require modifications on the in-house application. Additionally, the COTS integration may span numerous skill levels and is compounded by testing complexity. Vendor contractors may have customers with higher priorities. Schedules for the vendor and in-house may also be impacted by security and operating system upgrades, service packs, and bug releases.
Background
The IT project initiation strategy at AKDHSS begins by filling in a SharePoint Project Initiation Form. Of the 42 projects that have been submitted in the past two years, 23 have entailed purchasing a COTS package, configuring or tailoring of a COTS system or an integration between COTS systems and export for data warehouse reporting (Exhibit 2).
Exhibit 2: AKDHSS COTS Projects Initiated 2009–2011
This trend seems to indicate that many of the department’s divisions have found COTS solutions for their core applications, which did not exist 10 or even 5 years ago. The American Recovery and Reinvestment Act (ARRA), Medicaid Information Technology Architecture (MITA), and other federal grants are requiring faster and greater return on investment for functionality or modularization deployment. Another large factor has been limited in-house developer and technical resources for new product development, as well as the needed skill sets.
Another trend for AKDHSS in the past two years has been that requests for proposals (RFP) for COTS have been in conjunction with services that collect the initial data for the system, mentor staff on how to collect the data, and configure the COTS package. The Capital Assets Management System (CAMS) project is an example: a functional requirements matrix was produced that included the physical facilities building assessments as well as those for the package that would manage these data. The Office of Children Services (OCS) Infant and Learning program (ILP) was looking for a vendor to come up with a training program and a COTS solution for deploying it.
Pitfalls and Hurdles
As the department PMO is maturing, one procedure that the Department Procurement Offices has enforced is to make sure that RFPs revolving around IT go through the IT PMO at the start of the project initiation process, which is done to ensure that the scope of work (SOW) requirements are being detailed and that project planning and oversight are parts of the request (Exhibit 3).
Exhibit 3: AKDHSS Project Initiation Process
The RFP SOW must be detailed sufficiently to allow prospective sellers to determine if they are capable of providing the item, service, or solution. (Fleming, 2003, p 25)
“[T]he Scope Definition, WBS, Cost estimating, and Risk Identification processes should occur prior to, or in conjunction with, the Plan Purchases and Acquisitions process. This scope, WBS, cost estimating, and risk identification planning allow the Project Manager to present the Program Manager and\or Sponsor with information to make funding decisions on what scope items to procure versus which scope items to be performed by staff within the Department. In providing information for purchasing scope the Department Procurement staff is then able to verify, review, and assist in decisions that are best for the Department, mandated by procedures and guidelines, as well as providing the understanding of marketplace conditions.” (Douglas, 2007, p 4)
Vagueness in requirements should be addressed by setting performance standards in regard to the goals and comparing results with the standards addressed by the SOW to apply corrective actions and controls for the quality of scope, schedule, and cost to allow opportunities to be exploited. (Hallegren & Maaninen-Olsson, 2005, pp 17–18)
COTS solutions begin with the WBS of deliverables. From the WBS, the deliverable “requirement abilities” or desired features or outcomes can be put into functional and technical requirements matrices or performance matrices that are provided by a COTS solicitation. These matrices make it easier for the vendor to identify if they have an exact match, similar functionality, need to make a modification, identify if there is a future release that will have the functionality, or that their solution cannot provide the specified requirement due to base system constraints.
Three recent AKDHSS COTS requests contained matrices that helped the vendors better understand the requirements and risks for better cost proposals and solutions. The matrices also made it easier to identify best fit and added-value solutions by the proposal evaluation confirmation (PEC) teams. An example of comparing WBS with the functional requirements matrix for the CAMS project can be seen in Exhibit 4.
Exhibit 4: Relationship of WBS to COTS Functional Requirements Matrix
COTS Reality
COTS solutions compound the complexity in which combinations of environments and multiple COTS and in-house applications are integrated in ways that have not been done before. Gaps in COTS features and functions may also require in-house developed modules, in addition to tailoring and glue code. Integration of COTS solutions also compounds testing and often requires a mix of skill sets that may or may not already exist in-house or for the contractors that need to work together.
Federal matched funds for solutions may also demand additional requirements and Gap analysis and assessments, such as MITA, for further delineation of what effort is COTS, what effort is configuration and development, what functionality is not covered by federally matched funds, or federally enhanced matched funds, and what effort does not qualify for federal funding. Added effort then requires further assessment of AS IS architecture and TO BE architecture and additional planning such as a Pre-planning Advanced Planning Document (PAPD). This was the case for the AKDHSS Senior and Disabilities Services Replacement project, which needed to reflect the Centers for Medicare and Medicaid Services (CMS) requirements for funding.
COTS solutions may also experience turf wars in which the in-house development staffs desire to build systems in-house and where COTS solutions may not seem as glamorous as development. This was the case for a few of the larger projects that came through the IT PMO project initiation process for the replacement of legacy systems. Turf wars may also occur once projects have been launched that include integration (glue code, tailoring, or configuration) when the COTS cycle differs from their normal software development cycles.
The AKDHSS eGrants’ Grants and Contract Replacement Project and the Senior and Disabilities Services Replacement Project are examples of projects that required the development staff to have “change awareness” and “change reinforcement” to assist understanding the timeline and level of requirements comparison for the number of functions and features they could develop versus a COTS application that could offer an implemented solution sooner. The developers had a substantial investment in supporting and enhancing the systems; however, the architecture and platforms did not easily facilitate enhancements and modifications, and the return on investment for the time to make modifications was substantial and testing intensive. The skill shift and training required them to focus on data warehouse implementation and COTS integration.
The COTS Paradigm
The COTS paradigm shifts in-house development resources to activities that proactively study for the best COTS solution match to the desired product requirements and to processes that integrate the chosen solution. Subcategories of COTS can be databases, hardware components, application systems, networking and middleware.
“The requirements activity changes considerably as compared to traditional development. Projects reported that, with COTS, not all requirements are equal. Some are ‘free,’ or provided by the COTS. Some are not immediately available, but can be worked out with reasonable extensions and add-ons. Others are incompatible with the COTS.” (Morisio, Seaman, Basili, Parra, Kraft, & Condon, 2009, p 5)
High-level COTS solutions considerations require asking questions, such as:
- What standard development practices fit the procured solution integration?
- What project management strategy is the most effective?
- What cost savings are expected not just for procurement and implementation, but for ongoing maintenance and upgrades?
- What metrics should we be collecting to know we are progressing appropriately?
- What internal effort needs to be happening in parallel and how early does it need to start?
- Do we have the internal organizational capabilities to handle COTS volatility for interdependency?
COTS feasibility “should consist of complete requirements definition, a high level architecture, an effort estimation, and a risk assessment model. The high level architecture allows the team to sketch dependencies amount COTS, incompatibilities, and integration effort.” (Morisio, Seaman, Basili, Parra, Kraft, & Condon, 2009, p 9)
The COTS Processes
The project management strategy must bring added value, be based on data, prioritize objectives, understand resource availabilities and capabilities, and go for the maximum benefit. “Post development surveys tend to show that most (around 50% of) custom solutions don’t meet original expectations and quite regularly completely fail for technical, political, and other reasons.” (Golarath, 2009, p 2)
COTS project management processes are challenging because of the integration of the vendor’s software and hardware as well as the extension of the in-house system or environment. Understanding the threat and value brought to the organization by COTS also requires a highly flexible plan to incorporate the dynamics. (Bechtold, 2007, p 252)
In-house non-COTS deliverables, COTS configuration support, and COTS tailoring activities must correlate to the COTS vendor milestones. COTS solutions still require some type of software development methodology to allow parallel activities of vendor and customer. The five project management process group activities are shown in Exhibit 5 for the overall project as well as the project phases or iterations.
Exhibit 5: COTS Process (modified based on Morisio, Seaman, Basili, Parra, Kraft, & Condon, 2009, p 8)
Alignment for integration of the COTS solution between the customer in-house deliverables and vendor COTS deliverables cycle for integration testing and checkpoint integration milestones must be included in COTS contracts (Exhibit 6). Checkpoints should be integrated into the overall project schedule to allow for appropriate learning, discovery, and prioritization.
Exhibit 6: Comparison of Software Development and Technology Project Life Cycles (Wysocki, 2006, p 59) and (Bonnal, Pierre; Gourc, Didier; Lacoste, Germain, 2002, pp 14–17)
COTS Complexity
Combinations of diverse technologies and incorporation of a wide variety of COTS vendor packages, middleware, and hardware may be requested for the project that may never have been exactly meshed before in the requested environment. The resultant integration may span numerous skill levels and testing requirements. COTS integration complexity may also deal with vendors that are only versed in their own products who are now being asked to implement their product in a mix of other COTS vendor solutions with a different configuration onto a new platform.
An example of COTS complexity is the AKDHSS Desktop Virtualization Proof of Concept Project, which involved Citrix Receiver Connection, NetScaler, XenDesktop, XenApp, IE browser, the State’s Custom Access Gateway, SOA WAN Wireless Connection, PAM Cards, Servers, Storage, (various clients –iPad, Wyes Terminal, PCs with PXE boot), LDAP authentication, and the Health Insurance Portability and Accountability Act (HIPAA) security requirements. Connection configuration discovery further required developing the Gold images for thick clients and for thin clients which required decoupling the desktop, operating system, streamlining the application and user profile due to latency for different locations.
Compounding application and network or hardware support complexity is the fact that there is often an inability for the hardware and software disciplines within the team to work together effectively. Often this is due to the lack of understanding of how they impact each other (Garton & McCulloch, 2006, p 58). Hardware COTS solutions also entail compliance and sustained engineering considerations.
Software applications such as the CAMS application also need to deal with the imperfect match of features to requirements by adapting their business practices.
Strategy, Not Just Schedules
Strategy defines direction, requires knowledge of desired outcomes, and influences decisions on the allocation of time, people, and money. For a COTS solution strategy, there needs to be a diligent assessment process: between the requirements and software or technology being implemented; the IT or technology/development life cycle of in-house and COTS vendor; and the project management approach to meeting customer needs, delivering business value, and being as effective and efficient as possible. The strengths, weaknesses, opportunities, threats (SWOT) approach should be also be considered in the contract negotiations for the COTS solution from these perspectives.
Project management for COTS solutions depends on the integration and glue code for existing components and applications that are often executed based on past implementations, with anticipated adaptations based on the unique circumstances of the current requirements and COTS limitations. As seen in Exhibit 6, it is possible that the customer and vendor may be operating on different development life cycles; however, the testing and integration points need to align and may also require adequate critical resource scheduling buffers. This alignment may be more complex than the individual development life cycles experience on their parallel paths and could possibly require a ’spike’ of paired (customer and vendor) efforts to resolve the discrepancies.
“As important as determining architectural mismatching is calculating integration effort” (Jilani, 2008, p 205). The complexity of the integration of the COTS into a requested environment constraint depends on the percentage of exact match pushes the integration effort toward a ‘controlled Agile environment’ from the two teams (COTS vendor and in-house) for the cycles/phases that require the groups to work together.
If there is a 95% to 100% match, the likelihood of integration may be more straightforward and linear or incremental between the two teams and can follow a traditional project management life cycle—Wysocki quadrant 1 (Exhibits 7 and 8). However, the further the match is from out-of-the box COTS, the greater the likelihood the joint solution will require more adaptive or extreme project management strategies (Wysocki, 2006, p 37). Collaboration and communication are vital at the integration points.
Exhibit 7: Project Life Cycles 4 Quadrant Complexity-Uncertainty (Wysocki, 2006, p 37)
Exhibit 8: Project Management Strategies 4 Quadrant Goal and Solution (Wysocki, 2006, p 19)
COTS IPEMCC Considerations
Continuous improvement follows Shewart and Demming’s quality improvement practices for plan-do-check-act (PMBOK® Guide, 2008, p 191) using the project management Process Groups: Initiating, Planning, Executing, Monitoring and Controlling; and Closing. In working together with the contractor and in-house staff (regardless of which software or technology life cycles are being used), it is important to understand where the dependencies of schedules of different work packages and joint efforts integrate and how those check points are handled and accounted for on the overall larger joint project schedule.
The more complex the integration, the more critical are the client checkpoints. Client checkpoints are “not the time to be politically nice or correct. It is time for the client team and the development team to put all their cards on the table and make some hard business decisions. The decisions they make now will set the tone and direction of the project for the next and all remaining cycles” (Wysocki, 2010, p 219). Checkpoints can be seen in Exhibit 6 for the five software development life cycles and the four technology life cycles that may need consideration for technology components. The checkpoints must align to ensure system integration between COTS vendor and in-house development\integration efforts.
COTS solutions do not always conform to the in-house software development cycles and require project management strategies that lean heavily on Agile adaptive and extreme strategies. Often, the integration points require probative iterations that require next or future iteration/cycle planning and possibly update of the requirements breakdown structure/product backlog/sprint backlog. (Wysocki, 2010, 225)
Assumptions + Constraints = Risks
COTS project management strategies include documenting and addressing assumptions and constraints. “Assumptions are factors for planning purposes that are considered to be true, real, or certain without proof or demonstration.” (PMBOK® Guide, 2008, p 427) They help gage and understand product and project risks for inaccuracy, inconsistency, incompleteness of scope, success measures, and quality. Constraints may include the design of the product or come from the project itself. (PMBOK® Guide, 2008, p 429) Assumptions and constraints can be organizational, environmental, and external. COTS projects require the identification of risks. The RBS interaction with WBS and CBS (see Exhibit 1) necessitates the understanding of assumptions and constraints to enable vendors and customers to make better judgment calls for the best fit solutions.
In considering and analyzing COTS and COTS integration risks, the largest factors are: selecting inappropriate COTS components or solutions, looking at the downstream integration problems, developing adequate built-in checkpoints, and defining success measures prior to implementation (Jilani, 2008, p 206).
COPQ: Quality Planning
Quality planning defines quality metrics for what will be measured and how it will be measured. Error capture and diagnostic management routines are essential for IT projects. Built-in quality prevents errors from happening. Sufficient but not excessive error management allows for recovery and corrective options.
The Cost of Poor Quality (COPQ) is twofold. There may be internal failure costs, which include scrap, rework, and repair. There may be external failure costs that surround the customer’s dissatisfaction for a product that doesn’t meet their requirements. These sunk costs can often be attributed to processes ineffectively managed due to planning, integration, quality metrics, or handoff.
The hidden costs of poor quality often outweigh the visible costs.
Testing
Testing everything is virtually impossible for COTS projects and it can be extremely challenging dealing with various levels of testing: in-house environment, diverse technologies, interfaces, integrated glue code or tailored code, as well as customization and configuration. Additional difficulties include: the end users’ lack of availability to test, their inability to provide actual data for testing, production data that do not have the newer fields, working with multiple vendors, and possibly even employing new business processes required by the implementation (Bechtold, 2007, pp 250–251).
Agile testing brings testing in as early on as the detail design phase and continues through each iteration. Project situations also dictate the “just enough” testing and entail developer unit tests, scenario tests, acceptance tests, and paired testing (like paired programming). The client checkpoint at the end of the iteration reflects what has been done, learned, and needed for the next iteration. Frequent and effective testing increases the cost in the short term but reduces the cost of poor quality at handoff. Waiting for end game testing does not allow the team enough time to act on defects found at the end. (Wysocki, 2010, pp 229–231)
COTS testing requires collecting intelligence, prioritizing objectives, understanding resources, and going for maximum impact. Collecting intelligence entails identifying and documenting the transaction dialogues (test scripts) mapped to one or more system requirements and deciding which ones are the most important, and then, as time allows, go for the less important dialogues (Bechtold, 2007, p 254). These COTS dialogues should be parts of the product backlog and their dependencies should be considered during iteration planning and should be prioritized along with the work items selected for the iteration. Some dialogues may need to be more comprehensive than others depending on the business case scenario.
As the transaction dialogues are identified, it is important to analyze and determine the user groups that invoke the dialogue and the average number of times per minute, day, month, and year that the dialogue would occur. If a dialogue creates 75,000 transactions per hour, it is more vital for testing prioritization than one that is not as prevalent; however, those dialogues that are critical to the organization’s core mission, although not as frequent, can be just as vital for prioritization.
Test dialogue matrices help run the appropriate tests for designated dialogues in priority order and for sets of tests. “Lifeboat ranking” aids in the judgment for prioritization and includes not just features, but performance, stress, overload, security, safety, and historical defects associated with a dialogue. Test plans need to be reviewed for revision, and unscripted exploratory testing may also give better coverage under certain scenarios.
Testing Resources
Quality assurance requires that objectives and standards are in place and also that a test plan be tailored to the unique needs of the project and establishes traceability to track progress. The test plan needs to consider the available test resources (both human and automated). The objective should be the highest priority dialogues, risk reduction, and user benefit based on the available resources. Quality control selects what to control and compares results with standards for corrective action and delivering the maximum benefit.
The testing environment for the resources must reflect as close to the true conditions as possible with realistic test scripts/scenarios. There is nothing more frustrating for testing resources than to find out that their testing effort was a time sink, because the developers did not have the environment set up appropriately.
Client Checkpoint
Delivering the maximum benefit is the purpose of the client checkpoint. As seen in Exhibit 8, the checkpoints for integration testing between the vendor team and in-house team must align. The client checkpoint is the joint review with the client of what has been done for the iteration and where serious questions need to be answered for integration and how to close the gap for a solution. Any functionality or features of integration that need to be addressed and prioritized can then be integrated into the next or future iteration.
COTS Deployment
Deployment plans of COTS solutions must be included as a deliverable for the handoff/transition rollout. There may be a silent rollout in which the integrated solution is rolled out for production use by a few customers and then a formal announcement is made later. A silent rollout is often the case in which preliminary data need to be built up prior to other groups of customers entering their data. A pilot rollout is a more incremental rollout by designated groups, possibly by units or regions. An Agile rollout occurs when integration and useable solutions are rolled out to production after client checkpoint approval, as features are available when they can start providing value back to the organization.
Regardless of the type of rollout, each project team member, operations team member, and customer must understand their roles and responsibilities for production support. Rollout also requires a training plan. Training needs to occur prior to usage in production.
Project handoff of the COTS solution should include handing over the knowledge base and any other information needed for support and management of the system. This includes the warranty requirements and the release management and the operational level agreement. The operational level agreement is vital for post implementation support, upgrades (in-house and vendor) clarifying who is responsible for what, and for handling the integration test dialogues.
Conclusion
Value added COTS solutions are complex and require integration of milestone activities. COTS project strategy involves pre-planning and the assessment of best value added solution, including glue code and tailoring. COTS solution solicitations must contain requirements to determine the best feature match and what integration effort and risks must be accounted for in the project plan. SWOT assessments of vendor and in-house capabilities must be considered in the COTS selection.
The reality of COTS solutions is that the project management strategy must be flexible in order to handle the challenging integration environment as well as vendor and in-house development life cycle alignment. Cost of poor quality can be attributed to ineffective processes, and quality metrics must be applied throughout the project with the client checkpoint being taken seriously for integration alignment by both the in-house and vendor team staffs.
COTS solution handoffs at rollout must consider the operation team, customer, support and operational level agreements for upgrades (in-house and vendor), and integration dialogues.
