Topic 8.6 — Research & Real-World Evaluation

In health and care, a digital innovation is only as trustworthy as the evidence that it actually works in practice — so you must design that evidence in from the start, at a rigour proportionate to the risk and the claims you make.

Why this matters in health and care#

Every digital product you deploy makes an implicit promise: this will help patients, staff, or the system without causing harm. In most sectors an unproven promise costs a customer or a quarter of revenue. In health and care it can cost lives, widen inequity, or erode the public trust on which the whole system depends. Enthusiasm and a good demo are not evidence, and a plausible mechanism is not proof of benefit.

The history of digital health is full of tools that looked convincing and were adopted at scale before anyone had shown they worked — and some that quietly caused harm, missed diagnoses, or drove clinicians to workarounds. Regulators, commissioners, and clinicians have responded by raising the bar: they now expect credible, proportionate evidence before, during, and after deployment. If you cannot demonstrate that your service delivers the outcomes you claim, you should expect it to be challenged, defunded, or withdrawn — and rightly so.

Evidence is also an equity and safety issue, not a bureaucratic hurdle. A tool that improves average outcomes can still harm a subgroup — the digitally excluded, people with limited English, or a population under-represented in your training data. Only deliberate evaluation reveals this. Generating good evidence is therefore part of your duty of care, your stewardship of public money, and your credibility with the people you serve (see Topic 1.3 — Evidence-Driven Decisions, and Topic 1.6 — Clinical Safety & Risk Management).

Core concepts#

Start with a distinction that governs everything else: the difference between research, service evaluation, and clinical audit. Research aims to generate new, generalizable knowledge — findings intended to be true beyond your local setting. Service evaluation asks how well this service is working here, using the standards the service itself has adopted. Clinical audit compares current practice against an existing, agreed standard and closes the loop with change. The distinction is not academic pedantry: it determines your governance path. Research involving National Health Service (NHS) patients, staff, or data almost always needs formal ethics approval; most service evaluation and audit do not. Misclassifying research as evaluation to dodge approvals is a serious governance failure.

In England, research governance runs through the Health Research Authority (HRA), an arm's-length body of the Department of Health and Social Care. The HRA protects and promotes the interests of participants and coordinates approvals so a study is reviewed once for the whole country. Ethical review is carried out by a Research Ethics Committee (REC) — the UK equivalent of an institutional review board — which judges whether a study is ethical, proportionate, and safe for participants. Applications are submitted through the Integrated Research Application System (IRAS), a single online portal that routes your project to the RECs, HRA Approval, and other permissions it needs. The HRA's online decision tools help you determine whether your project counts as research at all.

Much digital-health evidence is now built from real-world data (RWD) — information gathered outside controlled trials, such as electronic health records, device logs, registries, and claims data — analysed to produce real-world evidence (RWE) about how an intervention performs in ordinary conditions. RWE is powerful because digital tools generate rich usage data as a by-product, and because it reflects the messy reality that trials often exclude. It is also fragile: observational data are riddled with confounding and bias, so a naïve before-and-after comparison can mistake a seasonal trend or a co-occurring change for your effect.

That is why evaluation design matters. The strongest design for causal claims remains the randomized controlled trial (RCT), which reduces bias by allocating participants at random. Digital contexts often favour pragmatic variants — cluster-randomized trials, and the stepped-wedge trial, in which sites cross over from control to intervention in randomized waves, fitting a phased roll-out you were going to do anyway. Where randomization is impractical, quasi-experimental designs help: interrupted time series (ITS) analysis tracks an outcome across many points before and after go-live to detect changes in level and trend, and is often the best feasible design for a system-wide digital change. Simple before-and-after comparison is the weakest and most misleading, because it cannot separate your effect from everything else that changed.

Two further anchors shape the work. The evidence bar for digital health in England is set by the Evidence Standards Framework for Digital Health Technologies, published by the National Institute for Health and Care Excellence (NICE): it grades tools by potential risk and specifies the strength of evidence expected for each tier. And a persistent distortion to guard against is publication bias — the tendency to share positive results and bury negative ones, which flatters the whole field and lets ineffective tools spread.

Best practices#

1. Classify your project before you design it. Use the HRA's decision tools to determine whether your work is research, service evaluation, or audit, and record the reasoning. This single step sets your governance path, your timeline, and your permissions. Get it wrong in the optimistic direction and you risk running unapproved research on patients; get it wrong in the cautious direction and you burden a simple evaluation with months of unnecessary process.

2. Build evaluation in from the start, not as a bolt-on. Decide what "working" means, and how you will measure it, during discovery — before a line of production code (see Topic 3.4 — Discovery Phases). Capture a baseline before go-live, because you can never reconstruct it afterwards, and instrument the product to emit the outcome and usage data your design needs. Retrofitting evaluation onto a live service almost always yields weaker evidence at higher cost, and often no usable baseline at all.

3. Match the rigour to the risk and the claim. A booking tool that reorders a waiting list needs less evidential weight than an algorithm that triages chest pain. Use the NICE Evidence Standards Framework to locate your tool's risk tier, and make claims your evidence can bear: if you have an interrupted time series showing reduced did-not-attend rates, say that — do not imply a mortality benefit. Proportionality protects patients from under-evidenced risk and protects you from wasting effort over-proving low-stakes tools.

4. Choose the strongest design that is feasible, and name its limits. Reach for randomization when you can, and consider a stepped-wedge design when a phased roll-out makes a simultaneous control group impossible. When randomization is genuinely infeasible, prefer interrupted time series over naïve before-and-after, and pre-specify your analysis to resist the temptation to fish for a favourable result. Every design has weaknesses; state yours plainly so readers can weigh the evidence honestly.

5. Use real-world data deliberately, and treat confounding as the enemy. Digital services generate abundant RWD, but abundance is not validity. Plan which data you will use, its quality and completeness, and the biases it carries, and involve an analyst or methodologist early rather than mining logs after the fact. Pre-register your outcomes and analysis where possible so the data cannot be tortured into telling you what you hoped to hear.

6. Partner with people who generate evidence for a living. Universities, the National Institute for Health and Care Research (NIHR), and your regional Health Innovation Network — the bodies formerly known as Academic Health Science Networks (AHSNs) — bring methodological rigour, independence, funding routes, and credibility you cannot manufacture in-house. Independent evaluation carries far more weight with commissioners and clinicians than a supplier's own analysis. Engage them early, because the best designs are set before deployment, not after (see Topic 9.0 — Collaborating with Partner Organizations).

7. Measure outcomes and economics, not just activity. Logins, click-throughs, and satisfaction scores are necessary but not sufficient; commissioners fund outcomes and value. Wherever you can, tie your evaluation to clinical outcomes, safety signals, equity across subgroups, and cost consequences, working with health economists to frame benefit in terms decision-makers use (see Topic 6.1 — Health Economics). Always disaggregate results by population so a tool that helps on average but harms a subgroup cannot hide behind the mean.

8. Publish your results — including the disappointing ones. Commit up front to sharing findings, positive or negative, through a report, a preprint, or a peer-reviewed paper. Publishing negative results is a professional obligation: it spares other organizations from repeating your mistakes and counters the publication bias that lets ineffective tools proliferate. A culture that can say "we evaluated this and it did not work, so we stopped" is more trustworthy, not less.

9. Close the loop and act on what you learn. Evidence that changes no decision is waste. Feed findings back into your roadmap, your business case, and your phasing decisions — scaling what works, fixing what half-works, and decommissioning what does not (see Topic 8.0 — Phasing in Innovation). Then re-evaluate, because a digital tool that worked at pilot scale in one setting can behave differently at national scale or in a different population.

Questions to discuss with your team#

1. Is what we are about to do research, service evaluation, or audit — and how confident are we in that answer? This is the first question because the answer sets your entire governance path, and getting it wrong has serious consequences. Talk through who your data subjects are, whether you are randomizing anything, whether you intend to produce generalizable knowledge, and whether you plan to publish — all signals that push a project towards "research". Resist any pressure to label something as evaluation simply because the approval route is faster; that pressure is exactly where governance failures begin. Use the HRA decision tool together, in the room, and write down your reasoning and who agreed it. A good answer is not just a label but a documented, defensible judgement that your information governance and clinical-safety colleagues have seen. If you are genuinely unsure, the honest move is to ask the HRA or your R&D office rather than guess in the direction that suits your timeline.

2. What would count as convincing evidence that this works — and to whom? Different audiences demand different evidence: a clinician wants safety and outcome data, a commissioner wants cost and value, a regulator wants to see the appropriate NICE tier met, and a patient wants to know it is safe and fair for people like them. Debate what "works" actually means for your tool, and what result would make you stop — because a team that cannot articulate a stopping condition is not really evaluating. Consider which design is feasible given your roll-out plan, your data, and your budget, and be honest about the trade-off between rigour and speed. Discuss whether you need independent evaluation to be believed, and if so, who should do it. A good answer names the decision-makers, the outcomes that would move them, the design that can credibly produce those outcomes, and the threshold at which you would conclude the tool has failed.

3. How will we handle a negative or disappointing result? Ask this before you have the result, because the answer is far more honest in advance. Explore what you would do if the evaluation showed no benefit, or benefit for the majority but harm for an excluded subgroup, and who would need to hear it. Surface the real pressures: sunk cost, supplier relationships, reputational worry, and the career incentives that make people want good news. Discuss your prior commitment to publish regardless of outcome, and what governance would hold you to that promise when the result is embarrassing. A good answer treats a well-run evaluation that produces a negative result as a success of process, not a failure of the team — and has a concrete plan for decommissioning gracefully, communicating candidly, and protecting the people who delivered the honest finding.

4. What must we capture and instrument before go-live — and have we actually built it in? An evaluation is only as good as the baseline and the data it can draw on, and both are far cheaper to secure before launch than after. Work through what "before" state you need — admission rates, waiting times, error rates, current pathway costs — and confirm it is being recorded now, disaggregated by the subgroups you care about, because you can never reconstruct a baseline you failed to capture. Check whether the product itself will emit the outcome and usage data your chosen design needs, or whether that instrumentation is still a backlog wish. Debate the timing tension: pressure to ship often strips out measurement to hit a date, and every such cut quietly weakens the evidence you will later be asked to produce. Involve an analyst or methodologist while the design is still soft, not once the logs are already flowing. A good answer is a concrete list of what is captured, from when, at what granularity, and who has confirmed it is live — not an intention to sort it out after roll-out.

5. How good is the data we intend to rely on, and what could confound our conclusion? Real-world data is abundant but rarely clean, and a plausible chart can hide seasonality, a concurrent service change, regression to the mean, or a coding shift that has nothing to do with your tool. Interrogate the provenance, completeness, and quality of every source you plan to use, and name the biases each one carries — who is missing from electronic records, which fields are unreliable, how consistently a code is applied. Ask what else changed in the same window as your go-live, because an interrupted time series or before-and-after that ignores those events will credit your tool with other people's effects. Decide in advance which comparison or design gives you the fairest counterfactual, and pre-specify the analysis so the data cannot be quietly reshaped to flatter the result. A good answer shows you have mapped the data's weaknesses honestly and chosen a design that can survive a sceptic pulling on each of them.

6. Who should run this evaluation, and how independent does it need to be to be believed? Marking your own homework is the fastest way to have credible findings dismissed, so decide early whether a supplier's or in-house analysis will convince the audience you actually need to move. Weigh the cost and time of engaging an NIHR-supported academic team, a university, or a Health Innovation Network against the reputational and commercial value of independent, publishable evidence. Discuss where the conflicts of interest sit — the people who built or bought the tool are rarely the right people to judge whether it works — and how you will insulate the evaluation from pressure to produce good news. Consider what partners bring beyond independence: methodological rigour, funding routes, and standing with commissioners and clinicians you cannot manufacture in-house. A good answer matches the degree of independence to the stakes and the scepticism of the decision-makers, and secures that partnership before the design is fixed rather than after the results disappoint.

In practice: a health & care example#

An integrated care system (ICS) commissions a remote-monitoring app for patients with chronic heart failure, hoping to cut unplanned admissions. The supplier's slides show a 30% reduction elsewhere, and there is pressure to roll out fast across all twelve boroughs before the winter.

The digital delivery lead pauses to classify the work. Because the ICS wants generalizable findings and intends to publish, and because it will analyse identifiable patient data for a new purpose, this is research, not service evaluation — so an IRAS application and REC review are needed. Rather than treat that as a blocker, the lead approaches the regional Health Innovation Network and a local university's NIHR-supported research team, who help design the study and co-write the application. Instead of switching everything on at once, they adopt a stepped-wedge design: the twelve boroughs go live in randomized waves over nine months, so every borough eventually gets the app while the staggered start provides a fair comparison. A baseline of admission rates, disaggregated by deprivation and ethnicity, is captured before the first wave.

Using the NICE Evidence Standards Framework, the team places the tool in a tier that carries clinical risk and sets the evidence expectation accordingly, pre-registering admission rates, mortality as a safety signal, and cost per admission avoided as outcomes. Nine months on, the interim analysis is sobering: overall admissions fall modestly, but among the most deprived cohort — who use the app least — there is no benefit and a worrying rise in late presentations. The team does not bury this. They report it to the ICS board, pause the roll-out to those cohorts, redesign onboarding and add telephone support for digitally excluded patients, and publish both the shortfall and the redesign. The result is a service that genuinely helps and a decision-making board that trusts the evidence because it has seen the team tell the truth when it hurt.

Sector lenses#

Startup#

A small digital-health company lives or dies by evidence, but rarely has the money or in-house methodology to generate it alone. Your fastest credible route is partnership: an NIHR-funded academic collaborator or a Health Innovation Network real-world evaluation gives you independent findings that investors and NHS buyers actually believe, far more than a self-published white paper. Be disciplined about proportionate claims — over-claiming to win a pilot is the quickest way to fail a later evaluation and lose your reputation. Build data capture into your product architecture from day one, because retrofitting it after your seed round is expensive and yields a poorer evidence base.

Small business#

An established small provider — a GP practice, a community pharmacy, a care home, or a single-site clinic adopting a digital tool — is expected to evaluate what it deploys, yet has no R&D office, no methodologist, and barely any spare hands. Your realistic move is not to run a trial but to run a disciplined service evaluation: capture a simple baseline before you switch on, agree a handful of honest outcome measures, and record what actually changes for your patients. Lean hard on shared infrastructure you do not have to build — your Health Innovation Network, a local university, or a federation or primary care network that can pool data and expertise across several small sites. Be especially wary of accepting a supplier's headline claim at face value when you cannot independently check it, and be clear that classification duties still apply to you even at small scale. Proportionate, well-documented local evidence is both achievable and enough to justify continuing, adapting, or dropping the tool.

Enterprise#

A large NHS trust or ICS has an R&D office, information-governance machinery, and existing academic ties — assets a startup would envy — but also the inertia to deploy tools trust-wide before evaluating them. Your challenge is to make "evaluate before you scale" the default, and to resist the momentum of a signed enterprise contract when early evidence is weak. Use your scale as a methodological advantage: multiple sites make stepped-wedge and cluster-randomized designs feasible in ways a single clinic cannot manage. Beware the internal politics of negative results, where a large investment creates powerful incentives to declare success regardless.

Government#

A national body — NHS England, the Department of Health and Social Care, or a devolved government — sets the rules others follow, so its evidence standards shape the whole market. Its job is to steward the frameworks (the HRA governance system, the NICE Evidence Standards Framework, DTAC) and to fund the infrastructure — NIHR, Health Innovation Networks, trusted research environments — that lets everyone generate good evidence. When government procures at national scale, the evidential stakes are enormous: a poorly evidenced roll-out can entrench harm across millions of citizens and is politically near-impossible to reverse. It should therefore hold itself to the highest bar it asks of others, and publish its own evaluations, including the failures, to keep public trust.

Common failure modes#

• Evidence as afterthought. Deploying first and asking "did it work?" later, with no baseline captured — leaving you unable to make any credible causal claim.
• Misclassifying research as evaluation. Choosing the "evaluation" label to avoid HRA and REC approvals, exposing patients to unapproved research and your organization to serious governance breach.
• Naïve before-and-after. Comparing two snapshots and attributing all change to your tool, ignoring seasonality, concurrent initiatives, and regression to the mean.
• Vanity metrics. Reporting logins, downloads, and satisfaction while staying silent on clinical outcomes, safety, equity, and cost.
• Averages that hide harm. Presenting a headline mean improvement without disaggregating by population, concealing a subgroup the tool fails or harms.
• Marking your own homework. Relying solely on supplier-generated evidence with no independent evaluation, so no sceptical audience believes it.
• Burying bad news. Publishing only positive findings, feeding publication bias and letting ineffective tools spread across the system.
• Over-claiming. Stretching a modest, weak-design result into a strong clinical claim the evidence cannot support.

Maturity model#

Checklist#

• Classify the project as research, service evaluation, or audit using the HRA decision tools, and document the reasoning.
• If it is research, plan the IRAS application and Research Ethics Committee review early.
• Define what "working" means, and the outcomes that would prove it, during discovery.
• Capture a baseline — disaggregated by relevant subgroups — before go-live.
• Locate the tool's risk tier in the NICE Evidence Standards Framework and match rigour to it.
• Choose the strongest feasible design (RCT, stepped-wedge, or interrupted time series) and record its limits.
• Plan your real-world data sources, quality, and biases; pre-specify the analysis.
• Engage an independent partner (NIHR, university, Health Innovation Network) where credibility or funding requires it.
• Include clinical outcomes, safety signals, equity, and cost — not just activity metrics.
• Commit in advance to publishing results, including negative ones, and name where.
• Define a stopping condition and a graceful decommissioning plan before you start.
• Feed findings back into the roadmap, business case, and phasing decisions.

Key sources#

• NICE — Evidence Standards Framework for Digital Health Technologies (ESF).
• Health Research Authority — "Is my study research?" decision tools, HRA Approval, and research ethics guidance.
• Integrated Research Application System (IRAS) — the single portal for UK health and social care research applications.
• National Institute for Health and Care Research (NIHR) — funding, evaluation support, and research infrastructure.
• Health Innovation Network (formerly Academic Health Science Networks) — real-world evaluation and adoption support.
• Medicines and Healthcare products Regulatory Agency (MHRA) — real-world data guidance and software/AI as a medical device.
• NHS England — Digital Technology Assessment Criteria (DTAC).
• MRC/NIHR — Developing and evaluating complex interventions (framework guidance).
• EQUATOR Network — reporting guidelines (CONSORT, STROBE, TRIPOD) for transparent research reporting.

References#

1. Real world evidence — Wikipedia — https://en.wikipedia.org/wiki/Real_world_evidence
2. Real world data — Wikipedia — https://en.wikipedia.org/wiki/Real_world_data
3. Clinical audit — Wikipedia — https://en.wikipedia.org/wiki/Clinical_audit
4. Health Research Authority — Wikipedia — https://en.wikipedia.org/wiki/Health_Research_Authority
5. Institutional review board — Wikipedia — https://en.wikipedia.org/wiki/Institutional_review_board
6. Randomized controlled trial — Wikipedia — https://en.wikipedia.org/wiki/Randomized_controlled_trial
7. Stepped-wedge trial — Wikipedia — https://en.wikipedia.org/wiki/Stepped-wedge_trial
8. Interrupted time series — Wikipedia — https://en.wikipedia.org/wiki/Interrupted_time_series
9. National Institute for Health and Care Research — Wikipedia — https://en.wikipedia.org/wiki/National_Institute_for_Health_and_Care_Research
10. National Institute for Health and Care Excellence — Wikipedia — https://en.wikipedia.org/wiki/National_Institute_for_Health_and_Care_Excellence
11. Publication bias — Wikipedia — https://en.wikipedia.org/wiki/Publication_bias
12. Evidence Standards Framework for Digital Health Technologies — National Institute for Health and Care Excellence (NICE) — https://www.nice.org.uk/about/what-we-do/digital-health/evidence-standards-framework-for-digital-health-technologies
13. Is my study research? — Health Research Authority — https://www.hra.nhs.uk/approvals-amendments/what-approvals-do-i-need/
14. Integrated Research Application System (IRAS) — Health Research Authority — https://www.myresearchproject.org.uk/
15. National Institute for Health and Care Research — NIHR — https://www.nihr.ac.uk/
16. Health Innovation Network — The Health Innovation Network — https://thehealthinnovationnetwork.co.uk/
17. Digital Technology Assessment Criteria (DTAC) — NHS England — https://www.nhs.uk/digital-technology-assessment-criteria-dtac/
18. Framework for developing and evaluating complex interventions — Medical Research Council / NIHR — https://www.bmj.com/content/374/bmj.n2061
19. Enhancing the QUAlity and Transparency Of health Research (EQUATOR) Network — https://www.equator-network.org/