Return To Blog

The Key to RBM and my New Year's/Decade's Resolution: Ask the Right Question

A conversation with Physician Scientist-Mary Flack M.D.

By Penny Manasco - January 6, 2020

Eight years ago, I had my first real experience working with a Sponsor that exemplified the principles of Risk Based Quality Oversight in her study implementation expectations. That person was Dr. Mary Flack, an NIH-trained physician-scientist and VP of Clinical Research, at NanoBio.  

Dr. Flack had had enough of high priced CROs that performed SDV and missed critical systematic errors (i.e., incomplete dosing at sites and one site that performed its primary efficacy endpoint analysis incorrectly, resulting in that site (a high enrolling site) being excluded from the primary endpoint analysis).  She needed a different approach to determine whether her investigational product worked or not.

Ask the Right Questions! Dr.  Flack was very clear about her High-Risk areas—activities that, if not performed correctly, would sink her trial or give equivocal results—requiring additional multi-million-dollar trials.

The critical questions she asked and the answers MANA RBM provided were as follows:
  1. Q: How can I know if an error occurs as soon as it happens, not months later?  
                   A: Use eSource to review the data in near real time. 
                       Use analytic tools to rapidly identify when an error occurs.  
  1. Q: How can I know when a site rater is not following the natural history of the disease in their rating scales in time to effect a change?
A: Change the way data  is reviewed (no need for SDV due to  eSource) so evaluation focuses on the critical errors that must be identified.  The methods to do so are as follows:
  • Ensure all raters were trained.  
  • Review rating scales for each subject AT LEAST weekly to identify errors as they occurred.  This included checking the pattern of data errors, not just whether data were entered. 
  • Identify where there were systematic differences in performance by investigators (e.g., differences in rating of spontaneous resolution of symptoms across investigators at each site).
  1. Q: How can I know when a subject is not getting the recommended dose of the study medication as soon as it occurs?
A: Add more training, data collection, and oversight around study drug administration and dosing.  Rapidly identify errors and correct those errors precluding errors from becoming systematic .  

Dr. Flack clearly enunciated the critical issues needed for successful implementation.  MANA RBM developed an oversight approach that met her needs. The results, including a 50% drop in deviations from previous trials run by traditional CROs, exceeded her expectations.

Now fast forward nearly a decade.  I recently asked her about that trial and what she saw as the future of Risk-Based Monitoring.

“I think we have to start asking the right questions to drive effective clinical trial oversight. I trained as a scientist and while at the NIH, and afterwards, I ask the questions a scientist would want to know. You could call it Science Driven Oversight or SDO, since we have to have an acronym.” 

She further expounded on her thoughts.  “I think much of our challenges with clinical trial oversight stem from the questions we ask. Just as we did nearly a decade ago, I want to know what activities in a trial will result in my ability to successfully complete the protocol as designed.  I want to know what errors occur that can result in censoring subjects, incomplete understanding of whether the subject received the correct dose, and all activities related to the primary endpoints—not just were the values collected, but did they follow the required processes as defined in the protocol.  And I want to know if the errors we see are systematic—a one off error is one thing, but a systematic error that is not identified indicates that we didn’t ask the right question.”

Science Driven Oversight, as Dr. Flack described it, is asking the questions that will affect the outcome/analysis of the trial and subject safety first and foremost.  “You can see that if your question is: “Were all pages SDV’ed or reviewed and  were all queries closed” will result in a very different outcome for oversight than asking “Were the processes for collecting the primary endpoint followed and how do you know? or Were there systematic errors identified at any site that can affect our ability to determine the safety of our investigational products and how do you know?”

Finally, Dr. Flack opined on SDO’s orthogonal view of clinical trial oversight, “When we focus our questions and oversight on the most important areas:  Did X, affecting primary efficacy endpoint, occur?, or Did Y, affecting subject safety, occur?, or did Z occur multiple times?, we can be sure to find the high impact, low frequency errors and systematic errors that affect trial success.”

Using a scientific construct to determine the critical questions we ask for clinical trial oversight is really what the FDA envisions as Risk Based Monitoring.  We ask questions to determine whether errors occurred in the  areas of highest risk for successful trial integrity.  

So, for the new decade, I adopt and encourage everyone working in clinical research to adopt Dr. Flack’s mantra of Science Driven Oversight.  I will think like a scientist, ask the questions a scientist would ask, and analyze study operations with the same rigor used in discovering innovative new medicines.