Blog - capturing the Covid-19 lessons in real time

In this blog, IPC Associate Dr Melanie Henwood outlines her observations on the importance of systematically capturing the current impact of Covid-19 and experiences of health and social care through Real Time Evaluation (RTE). This is vital if we are to fully understand the story, to have a comprehensive view of this rapidly evolving situation, and to shape our learning and future response effectively.

In the weeks and months ahead, we will begin to emerge into a new post-lockdown world, or into something with a less restrictive social-distancing regime, and to witness reducing rates of new infections of Covid-19 and falling mortality figures. That, at least, is the hope and expectation, and as things improve it will be vital to reflect on lessons learned and ‘what worked’. This is not about blame or recrimination; hindsight will always permit greater clarity about what could or should have been done. Rather, it is about ensuring we understand what has happened and taking that knowledge forward to achieve permanent change in systems, processes and – crucially – behaviour. None of that will happen without attention being paid to the questions that have to be asked while the crisis is ongoing.

Some of this will (or should) be undertaken by government, reviewing the impact of its policies to flatten the curve of infection rates by stopping transmission, and the ‘Stay home; protect the NHS; save lives’ mantra. But understanding what is happening is multi-faceted and requires a similar range of analysis and review at multiple levels. For this is not a single intervention that is being assessed, but multi-layered responses that come within the remit of different central, local and community agencies.

We have no expertise or recollection of anything remotely similar to the current Covid-19 challenge, as Nicholas Timmins recently observed, it is more than a century since the Spanish flu epidemic, and over 50 years since Hong Kong flu in 1968: "And it is clear that this lack of memory is one of the reasons behind the dark side of the Covid-19 outbreak. Its absence has made it remarkably hard to get some people (far too many people) to take seriously the life-saving warnings about ‘social-distancing’."

Capturing what happens, as it happens, matters both for organisations to reflect and learn, but also to provide a resource for future reference. This is the world of Real Time Evaluation (RTE), and – not surprisingly – it is a methodology and approach that has been developed particularly through humanitarian relief agencies responding to natural disasters and introduced by UNHCR in the early 1990s.

The model of emergency relief RTEs is not one that can simply be imported into different contexts and parameters, but there are some conclusions that are transferable and relevant. Such evaluation should be happening now rather than as an ex-post retrospective exercise; an RTE supports learning in action and enables a dynamic response to be adapted and refined as circumstances change. Typically, an RTE is a formative evaluation that provides immediate feedback in the course of fieldwork.

Much of the detail of what is happening, and how it is changing on a day to day basis in the Covid-19 response, will be lost, or overtaken by events, if it is not captured in some systematic form. The essence of the RTE is to assess the relevance, quality and timeliness of response, and if those core domains are applied to the current emergency, some key questions can be identified. The following are particularly relevant:

• What is the response?
• How has it been carried out?
• What has been the experience of implementation (what are the challenges)?
• What are the outcomes?
• What are the unintended consequences?
• What adaptations and revisions have been made?

These questions should be asked of all levels of intervention if central and local government and other agencies are to have an understanding and grip of the impact of their actions. What is apparent with the current public health emergency is that the central response is emerging iteratively. While the overall strategy remains with the core focus on ‘flattening the curve’, the means to the end are evolving, and the necessity of responding to the impact on the economy, on businesses, employees, the self-employed, charities and communities, is emerging on a daily basis. All of these elements need the RTE approach, and then there is a further question:

• How do these components fit together and how are they coordinated?

The 750,00 army of ‘NHS volunteers’ recruited to help with a range of practical support in the community, as well as providing telephone contact for the socially isolated, is a classic case in point on the challenges of logistics and coordination. Despite the overwhelming enthusiasm from prospective volunteers, delivery has been slow with people failing to be allocated tasks.

Moreover, at a time when the public health message is still ‘stay home’, a large number of people are now being encouraged to get out into the community and have close contact with others– bringing them home from hospital; taking them to appointments etc. It seems unlikely the volunteers will have adequate PPE given the lack of supplies available. It won’t be possible to maintain social distancing inside a car, and the failure to expand antibody testing at scale means no one will know whether or not the person they are helping has Covid-19 (or indeed if the volunteer is a carrier). Watch this space. In addition to the risks of volunteers spreading the virus are the inherent risks of working with volunteers who have been recruited at speed, been subjected to minimal vetting, and received no training beyond brief written guidance. Things could go badly wrong; charities that work with volunteers in health and care (including the RVS) know how they need to invest in training and supporting their volunteers, and how much time this can take.

While it may be argued (and frequently is by government ministers and officials) that the overall strategy is evidence and science-based, there is also a great deal of disjointed incrementalism, and the lack of integration across the health and social care divide is once more a deepening fissure.

Far from evaluation being built in to current strategies, there is a sense of monitoring and feedback coming largely anecdotally and predominantly through social media reporting. Despite the daily Downing Street Briefings broadcast on radio and TV, there is a growing unease that we do not have adequate understanding or knowledge about what is happening or what the strategy looks like to get us out of the current lockdown. In part, this is a reflection of the powerlessness and absence of agency that most people are experiencing as the familiar routines of life and work are removed. But it is also an indication that we have incomplete information, and the public consensus and tolerance which has so far provided a high degree of compliance with social distancing demands may start to break down if this is not addressed.

We are told to trust the experts, and each day two of these flank a Minister at the briefing to present the latest data and analysis. But the data are not telling the whole story; the daily death toll is increasingly out of step with the numbers actually dying; the recorded deaths are only those that occur in hospital. Under reporting the fate of people in residential and nursing homes who are not being admitted to hospital but left to die where they are is an emerging tragedy, and a situation where the numbers could increase exponentially, particularly with continuing problems around the supply of PPE to social care workers. The limited but emerging evidence suggests this is an international issue with under reporting of Covid-19 infections in care homes, but high mortality among those infected.

There are reasons to believe that the strategic inteventions in Britain were seriously behind the curve and slow to respond to the situation. The failures of global science to respond to the threat of a pandemic are striking, and as Dr Richard Horton (Editor of The Lancet) has pointed out: "Failure to escalate the risk assessment led to mortal delays in preparing the NHS for the coming wave of infections (…) the government knew the NHS was unprepared. It knew it had failed to build the necessary intensive care surge capacity to meet the likely patient demand."

At present we continue to trust and hope that the lockdown will see the flattening of the curve of infection spread, that the numbers of deaths will peak and then begin to drop; but it is hard when the numbers continue to climb and on 10 April the daily deaths reached 980, higher even than the worst daily figures previously seen in Italy and Spain, and when some other countries – notably Germany - seem to be achieving far better outcomes and fewer deaths. Whether we can recover the lost ground in delaying lockdown remains to be seen, but meanwhile the recurrent litany is of the same failings: the poor scaling up of testing for people with symptoms of Covid-19, quite apart from the test for anti-bodies which seems to remain elusive; and the supply failings of PPE for the NHS and – much more so – for social care.

How do we begin to emerge from this? The lockdown is set to continue, but there is no clarity on how it will be removed. It seems highly unlikely that we can go from total control to unrestricted freedom of movement, but a strategy for waves of relaxed control needs to be addressed. The academic Neil Ferguson, from Imperial College, whose team has been advising the government with its modelling, has suggested there could be a flexible approach with segments of the population released first (by age or geography perhaps); combined with more testing and contact tracing. Apple and Google are also pointing to the potential use of Apps to enable tracking of sections of the population, and automatic alerts for inadvertent contact with Covid-positive people. Health and Social Care Secretary Matt Hancock has indicated that the NHS will work closely with tech companies to develop a contact tracing app, but these are early days and the reliability of a self-reporting app fulfilling this role is dubious. So far, the government has not set out any comprehensive exit strategy for emerging from lockdown or indicated when this might happen; but we need to know what to expect and when.

One thing is clear: if we are now paying the price for a delay in responding to the pandemic, and if the next stage of emerging from lockdown is proved to be misjudged allowing the infection rates to soar once more in what the WHO warns would be a “deadly resurgence”, we will have lost control entirely. Understanding what real time evaluation can reveal, and acting on the evidence as it emerges, might offer the only way of avoiding this worst-case scenario. We need to move rapidly away from simply collecting anecdotes and social media reporting, to empowering health and social care leaders, including those across the voluntary and community sector, to systematically gather and collate observations and data – both quantitative and qualitative - to capture the full picture of implementation and impact on the ground.