Malcolm Bertoni, FDA: Conversations on Using Analytics to Improve Mission Outcomes

In the late 1980’s and early 1990’s, the Food and Drug Administration (FDA) faced a mountain of criticism. It was thought that the public health safety precautions built into its drug evaluation procedures in reaction to the Thalidomide tragedy two decades earlier were responsible for delaying consumers’ access to vital new drug therapies. Particularly in light of the growing activism around fighting AIDS, critics argued that the FDA procedures were born out of disaster and therefore extremely overcautious. In response, Congress passed The Prescription Drug User Fee Act of 1992, which enabled FDA to charge user fees to drug companies in order to fund the work required to improve the agency’s evaluation process performance. The legislation also required that the FDA report to Congress regularly the agency’s progress in meeting the goals. The FDA began using analytics to review new drug and medical device applications and to better report on its performance targets for the Prescription Drug User Fee Act. Mr. Bertoni describes his road map for implementing a successful analytics program: Executive Sponsorship and Specific, Clearly Defined Goals. Initially, there was some apprehension when the FDA engaged various program areas to collaborate on how to improve performance. Fortunately, there was very clear executive sponsorship as the law requires regular reports to Congress and the President on performance outcomes. The need for transparency, as well as the fiduciary aspect of the user fee structure, also imposed a mutual responsibility between the FDA and pharmaceutical companies. This all fostered collaboration from a business perspective, which resulted in the early establishment of specific, clearly defined goals. Partnerships. FDA then built a cross-agency partnership with those where performance management processes were already in place. Whether drug or medical device trials, both sets of processes involved collecting huge volumes of data and evaluating that data against specified criteria. The definition of metrics, whether that metric is the number of successes within a drug trial cohort or the numbers of drugs approved per year, is a scientific process with which FDA staff are very familiar. Mr. Bertoni reported that the FDA made sure to emphasize the real goal of instituting performance management – ensuring that the American public receives the fastest possible access to advances in medical science and treatment of disease – and to emphasize that it is NOT in any way punitive. The scientific methodologies already in place at FDA helped internal stakeholders see the value of standardization and to create sets of well-defined processes that both enabled better delivery of the mission and improved performance management. Key Performance Indicators. Once the stakeholders were all engaged and some initial standards and processes defined, the process of defining key performance indicators began. Mr. Bertoni relayed that a combination of Top Down and Bottom Up contributors were needed to develop a strategy for performance improvement. The measures needed to be both efficient and relevant. Efficient measures were clearly supported by data that could be captured and analyzed relatively easily. Relevant measures were easily understood in the context of the process they were applied to. According to Mr. Bertoni, without the involvement of the process actors at the agency level, the importance of some measures could have been overlooked. By discovering the relationships between the processes themselves and performance measures, a set of KPIs were developed. Culture Shift. It became clear that the performance management initiative was more about defining goals than assigning blame. Mr. Bertoni told us, the FDA staff “discovered the power of actually measuring and tracking something . . . all of a sudden, they have an ‘A HA!’ moment and can see . . . the value of applying the same scientific principles used in application review or risk analysis to actually manage the program and the resources.” That culture shift from silos of ownership to collaboration across organizational lines with real acceptance of performance goals as part of the public health mission was the main enabler for FDA’s success in using analytics to measure and improve performance. As proof of this success, Mr. Bertoni cites the program FDA-TRACK, which is accessible from the FDA website. FDA-TRACK provides a "regular conversation about performance" as well as a dashboard where one can drill down through performance data. To listen to Mr. Bertoni’s complete podcast and to read excerpts from his interview, visit the “Conversations on Using Analytics to Improve Mission Outcomes” page. In my next blog, I will highlight the insights gleaned from an interview with Carter Hewgley, Director of Enterprise Analytics at the Federal Emergency Management Agency.