There has been an explosion of interest in harnessing digital technology for improved mental health care and support. Previous woodland blogs have reported on a diverse range of approaches including online CBT for panic, the increasing application of virtual reality-based exposure therapy and app-based tools for self-management and symptom monitoring. Digital approaches have the potential to make evidence-based interventions and treatments for mental health widely available at a potentially reduced cost, so it is no surprise that serious money is being invested internationally in their research and development.
A colleague of mine once gave a talk entitled ‘Digital for mental health: panacea or pants?’. Underlying the flippant title was a serious message. Are we at risk of being blind to the shortcomings and limitations due to our enthusiasm for digital approaches? Without a doubt, one of the problems we face with digital interventions is that of engagement. The typical graph of engagement over time has something of the look of a cliff edge. Initial levels of enthusiasm, as people start using new tools, are often only maintained for brief periods, followed by a steep drop-off in use, particularly where interventions are used in the ‘real world,’ i.e. outwith the confines of research (Fleming, 2018). This is a huge problem given most interventions are designed to be effective over a certain time period. If people aren’t using interventions for long enough to get any positive benefit then the balance shifts towards the pants end of things.
Thus, a question that is rightly being given increased attention from researchers in this field is how do we better understand the factors that encourage or block the engagement with digital interventions? Answering this question gives the potential to better target interventions and informs how we can provide appropriate support and encouragement for use. Step forward Chelsea Arnold and colleagues (2019) with their study exploring predictors of engagement with an online intervention for people with experiences of psychosis.
Does our enthusiasm for digital approaches put us at risk of being blind to their shortcomings and limitations?
Methods
This was part of a larger Australian study of a web-based intervention for people with experiences of psychosis, the Self-Management and Recovery Technology (SMART: Thomas et al., 2016). Participants were aged between 18 and 65 with experiences of psychosis (diagnostically including people with schizophrenia, bipolar or major depression with psychosis, with a clinical assessment completed to confirm eligibility). People were recruited via community mental health services in the State of Victoria in Australia. In keeping with the study, consent was sought via the SMART website. Similarly, baseline measures were completed online as was randomisation to SMART website alone or to SMART website plus emails.
In terms of the intervention, the SMART website comprises six recovery-focused modules, drawing on the CHIME conceptual framework (Leamy et al, 2011). However, there is no expectation or requirement that people complete all modules or follow a particular order. Features include video content, exercises, self-management tools, and a peer-moderated forum. People with lived experiences of psychosis informed the development of the website and contributed video content. People in the email top-up group received weekly emails from an online coach over the 12 weeks of the intervention, which were designed to encourage engagement with website content. While these messages followed a template, they were to some extent individually tailored. The main outcome was engagement, which was a synthesis of information automatically logged about individual website activity, with both ‘breadth’ and ‘depth’ of website engagement examined. Basic demographic details and information about the frequency of wider internet usage were collected. Measures of ‘recovery style’ and ‘motivational orientation’ were also included, with the latter examining drivers for using the website.
Results
After assessing for eligibility and removing participants who either failed to complete all baseline measures (n=7) or who did not use the website at all (n=6), analyses were based on 98 participants (58 in the website only arm and 47 in the website plus email group). The mean age of participants was 42 and the vast majority of them already used the internet more than once a day (85%). Participants across arms logged in to the website close to a thousand times, completing an impressive 4,698 activities. In terms of the module content, the recovery session was the most popular but people seemed reluctant to post to the online forum with only 18 comments posted.
Group differences
Between-group comparisons showed that those with email support had a higher number of logins, completed activities, unique activities (i.e. the number of different site features used) and active engagements with the site (e.g. forum posts or activity completion). All of these differences were statistically significant. People in the email group also used the website for two weeks longer on average than the website alone group but this difference was not statistically significant. While usage overtime was better for the email group there was a marked drop off in engagement in both groups after the first two weeks of the study.
Factors predicting engagement
In terms of predictors of engagement across groups, activities completed, described by the authors as representing ‘depth of use,’ was significantly higher for those with tertiary education than for those without one. However, unique activities, described by the researchers as representing ‘breadth of engagement’ was not associated with education, and gender played no role in predicting engagement. Regression models predicting both depth and breadth of use over 12-weeks found significant effects for email support, tertiary education, older age and having less controlled (i.e. externally driven) motivations for use. Both models showed an interaction effect between controlled motivations and email support, i.e. the positive effect of email support on engagement was moderated by controlled motivations. No effects for either recovery style or autonomous motivations were observed in either model. A regression model for active/passive website use showed that only allocation (i.e. website alone or website plus email) and autonomous motivations for use (i.e. internally driven) were significant predictors, with those in the email group four times more likely to be active contributors to the site.
People who received email support were more likely to be active contributors to the site, e.g. higher number of logins, completed activities and active engagement.
Conclusions
Several important predictors of engagement with an online resource for people with experiences of psychosis were identified:
- People receiving email support were more engaged than people who did not receive it
- Other predictors of engagement included:
- Older age
- Having a tertiary education
- Being autonomously motivated.
People with older age, higher education and self-motivation were more likely to engage with the digital intervention.
Strengths and limitations
This is an excellent paper, which builds on evidence in this field showing that engagement with digital interventions is enhanced by some form of human guidance and interaction (Killikelly et al., 2017). Of encouragement from this study is the fact that the additional input was relatively limited, i.e. weekly, template-based emails sent by helpline workers and volunteers. One fear for many is that digital interventions will replace or be used as justification for the reduction of one to one mental health support, an obvious attraction to service planners in the face of increasing demand and tight budgets. However, if human support, as light touch as that used here, can increase engagement in interventions by up to 40% then to some extent concerns on both sides of the debate can be met.
The inclusion of a measure of motivation for treatment was another novel feature of this study. It is perhaps not surprising that people who were autonomously (or self-) motivated were more engaged than those who felt externally motivated (even with email support). However, this is the first attempt I’m aware of in this field to quantify that. When researching interventions it is easy to overlook people’s motivations for being involved, which can be diverse. There is also the very real issue that many people who feel less self-motivated, or who feel pressure from others to try things would not get close to consenting to this type of research meaning that results describe the experiences of a select group of participants. Carefully controlled research studies need therefore to include process evaluations to qualitatively examine people’s experiences. These should also be supplemented with evaluations of real-world implementation of digital interventions to more fully understand who they work for, when, where and why.
The finding that older age predicted engagement is also important in that it confounds many people’s perceptions that this type of thing is ‘just for the young ones.’ It was also notable that ‘recovery style’ played no role in predicting engagement but the authors highlight that there were a small number of participants with what is known as a ‘sealing over’ recovery style, who logically may be less attracted to such a recovery-focused intervention. This is again surely more the case for an increased evaluation of such interventions in the real world to ensure that interventions are not just tested with the subgroup of people most likely to enthusiastically engage with them.
Too many studies of digital interventions use simplistic and overly reductionist means to assess engagement but this paper is notable for its attention to detail and for its recognition that different types of engagement are possible (e.g. active or passive). While the differentiation of depth and breadth of engagement was at times confusing for this reader, it represents a laudable attempt to make a more fine-grained assessment of engagement than is commonly seen in a field where there is an unhelpful heterogeneity in its definition and measurement (Ng et al., 2019).
Findings suggest that older age was a strong predictor of engagement, battling the assumptions that digital interventions are “just for the young ones”.
Implications for practice
Researchers and others with an interest in developing and sharing digital mental health interventions should consider the need to support engagement and also take account of people’s motivation for involvement.
More ‘real world’ evaluation of digital mental health interventions is required to ensure people are not excluded from the opportunities they present.
This study highlights the different types of digital engagement and empowers mental health professionals to actively support those and take into account their patients’ motivation.
Conflicts of interest
Simon Bradstreet previously managed a clinical trial of a digital intervention for people with experiences of psychosis and in that role visited the research group that led this study.
Links
Primary paper
Arnold, C., Villagonzalo, K.-A., Meyer, D., Farhall, J., Foley, F., Kyrios, M. & Thomas, N., 2019. Predicting engagement with an online psychosocial intervention for psychosis: Exploring individual- and intervention-level predictors. Internet Interventions, (July), p.100266.
Other references
Fleming, T., Bavin, L., Lucassen, M., Stasiak, K., Hopkins, S. & Merry, S., 2018. Beyond the trial: Systematic review of real-world uptake and engagement with digital self-help interventions for depression, low mood, or anxiety. Journal of Medical Internet Research, 20(6), pp.1–11. Available at: https://www.jmir.org/2018/6/e199/
Killikelly, C., He, Z., Reeder, C., & Wykes, T. (2017). Improving Adherence to Web-Based and Mobile Technologies for People With Psychosis: Systematic Review of New Potential Predictors of Adherence. JMIR MHealth and UHealth, 5(7), e94. Available at: https://doi.org/10.2196/mhealth.7088
Leamy, M., Bird, V., Le Boutillier, C., Williams, J. & Slade, M., 2011. Conceptual framework for personal recovery in mental health: systematic review and narrative synthesis. The British journal of Psychiatry, 199(6), pp.445–52. Available at: http://www.ncbi.nlm.nih.gov/pubmed/22130746
Ng, M.M., Firth, J., Minen, M. & Torous, J., 2019. User Engagement in Mental Health Apps: A Review of Measurement, Reporting, and Validity. Psychiatric Services, p.appi.ps.2018005. Available at: https://psychiatryonline.org/doi/10.1176/appi.ps.201800519.
Thomas, N., Farhall, J., Foley, F., Rossell, S.L., Castle, D., Ladd, E., Meyer, D., Mihalopoulos, C., Leitan, N., Nunan, C., Frankish, R., Smark, T., Farnan, S., McLeod, B., Sterling, L., Murray, G., Fossey, E., Brophy, L. & Kyrios, M., 2016. Randomised controlled trial of a digitally assisted low intensity intervention to promote personal recovery in persisting psychosis: SMART-Therapy study protocol. BMC Psychiatry, 16(1), pp.1–12. Available at: http://dx.doi.org/10.1186/s12888-016-1024-1.
Photo credits
- Photo by Joshua Fuller on Unsplash
- Photo by Austin Distel on Unsplash
- Photo by Web Hosting on Unsplash
- Photo by NeONBRAND on Unsplash
- Photo by Joseph Chan on Unsplash
- Photo by Content Pixie on Unsplash
