The absence of standardised monitoring and evaluation tools has been keenly felt by the patient engagement community. In response to this gap, the PARADIGM Patient Engagement Metrics Framework was developed, with the multi-stakeholder consortium endeavouring to provide a map for monitoring and evaluating patient engagement across multiple decision-points in the medicines development lifecycle. Having been published after several years of development, the growing user experience will see it become a highly actionable and accessible tool that anyone can use in day-to-day patient engagement work.
The co-creation process continues, as PFMD works to transform the framework into a tool that can be customized and used by anyone who will plan, develop and assess patient engagement initiatives.
A session held at the Patient Engagement Open Forum in April aimed to push this process along. Feedback and insights gathered at the well-attended session will help to inform the next steps of the development as attendees were asked to actively contribute to this work and shape the future of this necessary tool.
Hayley Chapman, program director of PFMD, chaired the interactive and lively online discussion. She noted that 20 years ago, relationships between pharmaceutical companies were not only unpopular, but they also weren’t even considered. “We have come a long way since then.”
Now not only are patients increasingly engaged with industry and other stakeholders, there are multiple ways of measuring, monitoring and evaluating this type of engagement, she explained.
The Patient Engagement Metrics Framework allows assessment of engagement between patients and industry, regulators, HTA bodies and other stakeholders, she explained.
Some quick poll questions served to highlight the uneven adoption of monitoring and evaluation in a formal way across organisations. When asked “what is the current status of measurement and evaluation of patient engagement in your organisation/or at an organization that you collaborate with?”, some 42% said there are varying degrees of support for measurement and evaluation and 30% said they feel it is important yet no action has been taken as of yet. Just 13% said there is a solid M&E strategy in place and 3% said they are collecting solid data in relation to this.
Almost half of the attendees said they were familiar with the PARADIGM Monitoring & Evaluation Framework, with a “general or high-level awareness” of it while a further 15% are considering ways to implement it.
The first presenter was Paula Wray, Senior Manager NIHR, Applied Research Collaboration Oxford and Thames Valley, UK. In her view, monitoring and evaluation is crucial because it will
improve PE practice, making it more inclusive and capturing a broad range of activities.
Yet she acknowledged that defining impact can be difficult, and offered the NIHR definition: “The changes, benefits and learning gained from the insights and experiences of patients, carers and the public when working in partnership with researchers and others involved in NIHR initiatives.”
Paula said that impact can be considered in terms of positive or negative, the time is taken, the units of analysis, or the attribution contribution to the overall impact. Many equate impact with change, which it often is, but the impact is depicted as a linear process when it usually isn’t – there are interdependencies and feedback loops, she noted. Paula also stressed that a wide spectrum of impacts need to be considered – individual, quality and trust – from health to education to economic to the environment to cultural, among others.
“What are we ultimately trying to achieve? We need to learn from this and we need to take on board these learnings and if we don’t follow up with people we won’t ever know what the real situation is.”
According to Paula, an impact assessment toolkit contains a number of elements, several of which are PFMD or PARADIGM tools, for example, the Book of Good practices and of course the Monitoring and Evaluation Framework.
Wray explained the reasons why monitoring and evaluation of impact is essential:
- To be accountable and transparent
- Make the case (advocate) for continued investment in research/patient engagement
- activities
- Enhance understanding (analysis)
- Inform allocation of resources
- Identify areas for continuous improvement (adjustment
Lidewij Eva Vat, researcher and lecturer at the Athena Institute, Vrije Universiteit Amsterdam, said for her, monitoring and evaluation is “a tool for learning – by learning together we can enhance the meaningfulness and sustainability of PE”. She spoke of the genesis of the PARADIGM project and how having conducted a gap analysis across all stakeholders, they strongly agreed on the need for a broader set of metrics to capture the value for all involved and guidance on how to assess progress.
Now that the PARADIGM project has concluded, Lidewij explained, there is a tangible output in the form of the PE Monitoring and Evaluation Framework, as well as a host of additional resources. The Framework is a tool to help partnerships self-evaluate the progress and impacts of patient engagement and provides an easy-to-follow map for monitoring and evaluating patient engagement across multiple decision-points in the medicines development lifecycle. While the framework includes 87 different metrics, it allows the user to select only the metrics they wish to measure and “tailor” the framework to suit them, she explained.
A common question from industry and patient organisations is how to develop an evaluation strategy. Her advice was as follows:
- Step 1 Determine the objectives of patient engagement and the purpose of evaluation from each stakeholders’ perspective
- Step 2 Develop roadmap from input to impact and select
- appropriate metrics for each stakeholder
- Step 3 Identify suitable methods and create an evaluation plan
- Step 4 Establish a feedback loop and consider context factors
Lidewij also recommended that organisations begin with a pilot evaluation. Her key takeaway messages were that no one size fits all set of metrics and what set of metrics is appropriate depends on your roadmap for change and how you want to use evaluation data. “Metrics should be relevant to all involved. Customization with your partners is key!”
Looking ahead, Lidwij acknowledged that the tool will evolve over time as it becomes increasingly digitised and incorporates more patient feedback.
She was followed by Ana Diaz, Project Officer with Alzheimer Europe, one of the patient organisations involved in the PARADIGM project. Ana explained the importance of patient engagement for patients; “we want things to change, we want things to be better for patients.. We want better medicines and interventions, we want research to be more inclusive for patients. We want to make a difference .”
When it comes to monitoring and evaluation from a patient perspective the important thing is that they don’t want to simply “tick a box” – monitoring and evaluation allows them to see how they are making a difference both in the short and long term. In this context, feedback is critical, Ana said, and the patients must be part of the feedback loop. “There must be a relationship of mutual respect.”
While monitoring and evaluation is something that requires a lot of planning and effort, it is a critical activity and one that must be shared between all stakeholders, Ana noted.
“It might seem overwhelming to think about a framework but even just opening these but the more important message is that this is something we have to do together,” she concluded.
During the ensuing discussion, Lidewij confirmed that PARADIGM worked with over 30 companies and organisations to apply and test the framework. Since PARADIGM ended and the tools were published, they have received numerous requests from people and organisations keen to implement the framework and interest continues to grow. It can be used in different settings and across different collaborations
‘
During the co-creation session, a number of questions were asked of the audience to determine their thoughts on some key elements of the framework as it evolves:
- Reasons for wanting to monitor and evaluate patient engagement?
- Make the case for continuous investment in PE activities
- Identify areas for continuous improvement
- Under what 3 scenarios are you most likely to use patient engagement monitoring and evaluation tools?
- Larger group initiatives like advisory boards and focus groups
- At the initial research stage
- Ongoing initiatives at regular intervals
- Which are the 3 components of the PARADIGM Monitoring & Evaluation Framework that are most helpful for you?
- Impacts
- Input
- Learnings/outcomes
- What would improve the user experience of the PARADIGM Monitoring & Evaluation Framework?
- Interactive guidance
- Integration with a PE planning tool
- Set of metrics for most relevant scenarios
