Forecasting the performance outcome of clinical trials using earned value analysis as a method
Clinical trials in later stages of pharmaceutical product development are money and time consuming. Quite often, the estimates of total cost of a trial and total time needed are planned over-optimisticly, which may lead to multiple corrections of schedule and / or budgeting. Usually, planning for resources is a repetitive undertaking, while the trial is ongoing. Earned Value Analysis supports project managers by providing forecasts on total final cost and timeframe of the clinical trial and gives a performance based and regularly updated view on resources needed. Cost and time estimations are also input to risk management and public communication. This paper describes the methodological approach chosen and demonstrates the results.
Productivity in the pharmaceutical industry is going down. While the expenditures for one new compound on the market reaches a billion dollars, the number of new drugs per year is in a steady decrease (DiMasi, 1991).
Performance evaluation has become an ever increasing issue in pharmaceutical industry. Only few concepts and approaches are really suitable to describe and to dimension what is going on. Among these, Earned Value Analysis (EVA) has been introduced into performance monitoring of development projects and individual work-packages like clinical trials (Seifert, 2005)
Earned Value Analysis is an important tool for project managers and as such is described in A Guide to the Project Management Body of Knowledge (PMBOK® Guide) (PMI, 2000, page 41) published by the Project Management Institute (PMI®), Newton Square, PA. PMI says that Earned Value Management (EVM) is “one of the techniques used to both integrate the various processes and to measure the performance of the project as it moves from initiation through to completion”. Literature on Earned Value Management and Analysis (EVA) is abundant (Christensen, 2006; Fleming, 2000).
While EVM and EVA have been widely applied especially for huge projects/programs throughout the industry for performance evaluation and forecasting of project duration and final costs, it is a wonder, that it has not been used that widely –if at all- in the pharmaceutical industry. This opens some room for speculation, and the assumptions that pharma has not had the need to care about performance, or that there has been a lack in professional education of the pharma project managers would not always go too far. Adding to these, there has not been a clearly defined context and no „ready-to-use” application for that purpose.
The term “Earned Value” (BCWP, Budgeted Cost of Work Performed) paraphrases the “money earned” for the “money spent”. Ideally, and referring to a dedicated piece of work, the cost for that work should directly reflect what that work is worth. In terms of cost analysis, if the worth is less than the money spent, the result shows a negative variance: the activity experiences a cost overrun.
A similar statement can be made for time. If a piece of work was to be achieved today, but what was achieved today reflects only part of the work to be performed, then the amount of work scheduled is greater than the amount of work performed, and a negative schedule variance is the result: for every bit of money scheduled to be spent only a fraction was “earned”.
As an example: When being asked what the current status of a running clinical trial is, the replies often consist in:
…“We have set up m centers and already included n patients, and the number of monitors will probably not suffice…”
Earned Value Analysis instead would have given the amount of work completed (in percent of the total), the Key Performance Indicators (KPIs) for cost and schedule (time), and forecasts on the final total costs and the date of finishing the trial. This also allows for adaptive budgeting: until the calculated termination date, a calculated budget per time is available, starting from the date of analysis.
For a better understanding, in addition to EVM and EVA the following terms and formulas are being introduced:
Terms and definitions
Retrospective EVA: Analysis of a finished Work Object (WO). Purpose: evaluation of quality and past performance
Prospective EVA: Analyses while a WO is running. Purpose: forecasting of cost and time and risk evaluation.
Start of Work Object (SWO): Actual start (Date)
End of Work Object (EWO): Baselined end (Date)
Budget at Completion (BAC): Authorized work; the total money granted for the project / activity; When there is only an annual budget, then the annual budgets are to be cumulated in order to reflect the total budget for the WO. This can be done during the course of the WO.
Actual Cost (AC): Actual cumulative cost of work performed (cost incurred)
Degree of completion (DC(%)): Estimation of how much of the overall work has been performed.
Earned Value (EV): Budgeted cost for work performed (BCWP); the inner worth of a piece of work, irrespective of the actual cost. Budgeted cost is derived from a thorough planning process, based on best knowledge / expertise.
Formula: EV = BAC*DC. If BAC = 1000 and DC = 50%, then EV = 1000 * 50 / 100 = 500
Planned Value (PV): Budgeted cost of work scheduled (BCWS); the cost distribution along the timeline for scheduled pieces of work
Cost Variance (CV): Monetary deviation from nominal value in terms of cost Formula: CV = EV-AC. If EV = 500 and AC = 750, then CV = 500 – 750 =-250
Exhibit 1: Cost variance in EVA
Schedule Variance (SV): Monetary deviation from nominal value in terms of schedule
Formula: SV = EV – PV. If EV = 500 and PV (at this point in time) = 600, then SV = 500 – 600 = -100
Comment: From the variances above, it can be concluded, that the Work Object is 250 above cost and 100 behind schedule.
Exhibit 2: Schedule variance in EVA
Cost Performance Indicator (CPI): Signals the relation of value received for value spent Formula: CPI = EV / AC. If EV = 500 and AC = 750, then CPI = 500 / 750 = 0.67 For 750 spent, only 0.67 of that amount were earned.
Schedule Performance Indicator (SPI): The value received today is equivalent to what point in time Formula: SPI = EV / PV. If EV = 500 and PV (today) = 600, then SPI = 500 / 600 = 0.83 = there is a 17% delay (referring to the schedule between SWO and EWO, which is baseline)
Estimate at Completion (EAC): Forecast of final total costs at date of completion.
Formula: EAC = AC + (BAC - EV) / CPI. EAC = 750 + (1000 - 500) / 0.67 = 1496.3
Note, that this equation calculates EAC on the basis of cost performance only. Usually this reflects the lower borderline of total expenditures. Other methods divide by the product of CPI * SPI, which gives the more negative scenario, or they use more subtle adaptations.
Schedule Projection, Projected Completion Date (PCD):
Formula: PCD = SWO + (EWO - SWO) / SPI
If EWO – SWO = 100 days, and SWO = 1.1.2000, then
PCD = Jan 1, 2000 + (100 / 0.83) = Jan 1, 2000 + 120 days = April 10, 2000
Exhibit 3: KPIs in EVA
As stated above, there are principally two ways to apply EVA:
- the retrospective analysis (a piece of work had been finished anywhen and the analysis was carried out) for historical performance evaluation and estimation of the planning quality
- the prospective analysis for evaluation of current performance, forecasting of final total cost and schedule, and risk management activities, if imminent.
This type can be performed repetitively during the course of e.g. a clinical trial.
At first, the retrospective analysis is considered.
This type of analysis compares actual versus plan, after a work object was finished. Thus, the Degree of Completion is set to 100%.
From the inputs SWO, EWO, actual date of completion, the planned and the actual duration are calculated, their ratio reveals the SPI, while the ratio of BAC and AC results in the CPI.
Any result close to “1” represents excellent performance and appropriateness of planning, while results > 1 signal a better cost effectiveness (CPI) or schedule performance (SPI) than foreseen. Results < 1 represent the opposite: a worse performance than planned.
It remains open which factors influence a deviation to the better or the worse. This may have been an issue for project risk management. A performing and well managed organization, however, would appreciate the fact, that there was no relevant deviation between planning and fulfillment.
Retrospective EVA is most useful for evaluation across work objects, e.g. various types of clinical trials, or clusters of clinical trials according to phases, study types, indications or others. It may also be used to get a quick idea on the performance and continuity of contract research organizations (CROs) and how they carried out work for the sponsors.
In principle, retrospective analysis is one instance of prospective EVA, namely an analysis exactly at the end point of the work.
Prospective analyses may be carried out whenever appropriate while a work object is being performed. Whenever appropriate means: On a periodic basis (every first Monday per month; whenever a relevant milestone was achieved; when there is a specific management request and others more). Prospective analyses continuously allow the project manager to identify whether everything is in accordance with plan, and once deviations were detected to initiate the risk identification and consider counteraction.
Input to the prospective EVA are: start date (SWO), baselined end date (EWO), date of analysis, degree of completion (DC), budget at completion (BAC) and actual cumulated cost (AC).
The results do not only consist of the Key Performance Indicators CPI and SPI, signaling the current performance, but most relevantly also the Estimate at Completion (EAC; the forecast of the final total cost) and the Projected Completion Date (PCD). Either may differ from plan. In some cases, the differences become relevant, namely if the budget is at risk, or if the timely performance endangers dependent activities, like the product's submission to the authorities within an already announced time frame. There, risk management processes usually have to be involved.
One peculiarity, however, of prospective EVA is the dimensioning of “Degree of Completion”. In the following example this is demonstrated with a clinical trial. To what extent is a clinical trial completed, once the recruitment has started, or the last patient had the last visit, and many stages more.
There are various approaches to solve this issue:
The most conservative way: “Although progressed, a work object has no value unless totally finished”. In terms of clinical trials, this translates into: “We consider a study only of value, if the final report was approved, as only the report can be used for the submission file”. Degree of Completion would remain at “zero” throughout the course of the trial, and with the final report available jump to 100%. With this approach, however, no forecasting would be possible.
The alternative way was to introduce more or less detailed milestones into the course of the trial and use the achievement of a milestone for Degree of Completion to perform EVA.
While there should not be many impediments to identify milestones, the dimension of achievement deserves some deliberation.
We prefer the following generic milestones, that can be used for most if not all clinical trials, independent of phase or indication:
Exhibit 4: Milestones for modeling Degree of Completion
Exhibit 5: Degree of Completion
The dates of (actual) SWO and (planned) EWO are relevant key parameters in EVA; the achievement of a milestone may trigger an immediate analysis.
In Exhibit 5 the three mentioned models are outlined. It is evident that only the full modeling along milestones reveals a sufficient amount of descriptors for progress. In this case, the percentages listed are the result of expert advice and represent a compromise between various more or less differing recommendations. Other approaches to model progress comprise stratification into study design types, or indications, or even drug types. This could be achieved best by evaluation of benchmarking database information. Companies participating in such programs should be encouraged to request such information.
The following diagrams from a Phase III trial in a cardio-vascular indication show the results along the course of analyses (displayed on the horizontal axis). Note that each new result reflects the cumulated performances of all previous analyses. All of the following diagrams bear the identical x-axis, the dates of analysis.
Exhibit 6: Variances of cost (blue line) and schedule (violet line) from Earned Value CV = EV – AC; SV = EV – PV Legend: Ref.: Target variance; Start, End: Start and planned end of trial
Values above 0 signal better performance than planned. While the schedule variance was close to plan, cost variance showed an ever decreasing deficit. The study earned far less than it costed.
Exhibit 7: Key Performance Indicators for cost (blue) and schedule (violet), Degree of Completion (DC; light blue); Ref: Target value for KPIs
CPI = EV / AC; SPI = EV / PV
Values above 1 signal better performance than planned. The study completed ahead of time, also demonstrated by a positive Schedule Performance Indicator. Cost performance was less than half of what it should have been.
Exhibit 8: Cost parameters, esp. Estimate at Completion (EAC), forecast of final total cost (blue line). Legend: CostInitPlan: Commissioned initial budget; EV: Earned Value; AC: Actual cost; BAC: Budget at Completion; BCWS: Budget Cost of Work Scheduled; Start, End: Start and planned end of trial.
At the end of the trial, Actual Cost (AC, green line) converged with the Estimate at Completion (EAC). The provided budget was less than half of what the study finally costed. The first analysis was performed once a number of patients had been included. Already at this point in time (around May 2002), EAC results signaled a serious budget risk.
Exhibit 9: Schedule parameters, SWO (green line) and EWO (red line), forecast on projected completion date (blue line), projected on y-axis.
From the first to the last analysis, the forecast of the projected completion date remained stable and prior to the baselined completion date (horizontal red line).
Where to apply EVA and what to consider?
The approach presented here is a top-down, model based approach. There is a clear distinction to the bottom-up approaches used in standard EVA literature (Christensen, 2000), which we feel to be too tedious for the delicate clinical development environment. Top-down EVA can be applied to any task in development, either retrospectively on a finished task, or prospectively on e.g. running clinical trials. The retrospective analysis makes comparable finished tasks within a function and across functions, and also allows for the evaluation of external service providers.
Prospective analysis reveals forecasts on final cost and completion date, which we consider important as thereof serious consequences may be drawn in terms of risk management. If we assume a long lasting pivotal trial in a business-critical project and announcements made about an early submission, and the EVA forecast would not support the announcements, then the company may encounter some risk on the financial markets. If cost estimates exceed the granted budget, then countermeasures have to be taken in order to avoid endangerment of the trial.
It should not be underestimated, that especially forecasting is quite evidently depending on the quality of modeling the degree of completion with every milestone achieved. To improve appropriateness of the models with reality, it is not fully satisfying to jump from milestone to milestone, when an interpolation could be applied. This is especially true for the period between first patient first treatment and last patient last visit (= patient completed). We have made good experience to interpolate this phase and especially give the number of patients being treated a weight of one third, while the number of completed patients receives a two third weight.
In this model of a clinical trial, we also fully consider the duration of the treatment phase for calculation. In some benchmarking programs and also in-house evaluations, the treatment phase remains invisible, as this is said to be beyond influence of the sponsor and having a fixed duration anyway. However, this is the phase where to most money is being spent and where interaction and/or support of the trial centers can be most beneficial, esp. under poor performance conditions.
Any single company may possibly not be able to collect sufficient data to model their specific clinical trial spectrum. Here, information from benchmarking may be supportive, and at least summary cycle times from clinical trials in specific therapeutic areas or phases are available. Of course, clients to the benchmarking companies have access to this information.
We have shown, that the tool of Earned Value Analysis, which has been applied in other non-pharma industry, is also applicable to the pharmaceutical industry. Standardized tasks with generic milestones can easily be tracked and evaluated. EVA reveals a solution for performance evaluation by provision of Schedule and Cost Key Performance Indicators, and it provides the so important forecasts on final total cost (Estimate at Completion) and the finishing date (Projected Completion Date). Thus, it is a perfect tool not only for the project and study manager and team, but also for risk management.
Christensen, D.S. (2006) Comprehensive bibliography of Earned Value Literature, retrieved on July 15, 2006 from http://www.suu.edu/faculty/christensend/ev-bib.html
DiMasi, J. (1991) The Cost of Innovation in the Pharmaceutical Industry. Journal of Health Economics, 10,107-142
Fleming, Q.W., Kopelman, J.M. (2000) Earned Value Project Management. Newton Square, PA: Project Management Institute
PMBOK. (2000) A Guide to the Project Management Body of Knowledge (PMBOK® Guide) (2000 ed.). Newton Square, PA: Project Management Institute
Seifert, W. (2005, June). Earned Value Management and Analysis: A generic approach to functional performance and risk evaluation in clinical trials. DIA 41st Annual Meeting, Washington D.C., USA
Seifert, W. (2005, February). Using the Earned Value Analysis (EVA) Tool to Evaluate Clinical Trial Performance. IIR 3rd Annual Clinical Performance Management, London, UK
© 2006 Wolfgang Seifert
Originally published as a part of 2006 PMI Global Congress Proceedings – Seattle, Washington