Digital health products, including services, campaigns and other interventions, are becoming part of the norm. Such products also have different ways of providing information, such as through an app or website.
Conducting evaluations for all digital health products is an important consideration for health services and device manufacturers. A framework is necessary to help demonstrate the product’s impact and how its costs compare to its benefits. A set of criteria can also provide the developer with the information needed to make a product or service better, including what works and what does not help to demonstrate the value of the product.
The best approach is to develop digital products iteratively, through the process of testing products repeatedly and improving them throughout the development stages. Tool like Kanban can be useful for the development process and to help a company remain agile. Kanban is a way of visualising and improving your current working practices so that workflows through a system quickly. This can assist in delivering value quickly and predictably and put in place a mechanism to get early feedback to find out whether the digital health product or service is meeting user needs.
However, there are different quality standards against which products are produced. There are also cases where the product has not been developed along the criteria required by the recipient. To help to prevent such misalignment and to assist the developers of digital health products, the UK government has produced a set of resources intended to aid companies in carrying out evaluations.
The guidance includes developers considering when they are developing a product, they should work out how to make it better (what is termed a formative evaluation). Later, when a new product is launched, developers should establish as early as possible out whether the product achieves its aims (a summative evaluation).
In addition, when a developer is scaling up use or using the product in new contexts (the implementation evaluation) it is of importance to check the product still works in practice and has no unintended consequences (post-implementation evaluation).
In some circumstances, this means monitoring its safety for the public (part of the process referred to as post-market surveillance).
There are various challenges associated with the development of digital health products. Three important areas include:
- Digital health products often change rapidly, which can be difficult to align with the time it takes to do some evaluations.
To address this challenge, the speed of digital means that evaluations need to be done more quickly
- Most digital health products are complex, with many different functionalities and features.
This means evaluations may focus on specific features or take a more global approach.
- The regulatory framework for digital health products is evolving: new methods of evaluation are also being developed and tested.
Consequently it is not always clear what current best practice is.
A product model can be useful in fine tuning the design; this represents how a product is expected to work. This can be shown as a diagram to help visualise the different processes and how they fit together.
