Avid followers of the Mental Elf will be familiar with the lack of evidence about the potential harms of psychotherapy from previous blogs (Langford & Laws 2014; Laws & Huda, 2016). The publication of the Negative Effects Questionnaire (NEQ; Rozental A et al, 2016) signifies a positive development in the adverse effects story, but this story is far from over.

Before we critique this latest episode, let’s first recap some of the key points raised by my fellow elves. Think of it as a [put on deep emotive voiceover accent] ‘Previously, when the Mental Elf discussed adverse effects’:

• Trials of psychological interventions are far less likely to measure adverse effects than trials of drug treatments. Vaughan et al (2014) reported that harms were considered in only 60% of psychotherapy interventions in their sample compared to 100% of medication interventions
• A large scale study of adverse events by Crawford et al (2016) identified that over 5% of adults who had experienced psychological therapy reported ‘lasting bad effects’
• Clinical and demographic factors that are associated with the likelihood of reporting lasting bad effects, such as:
  • Timing and length of therapy,
  • Practical accessibility of therapy appointments,
  • Age: people over age 65 report less negative effects
  • Ethnicity: people from BME groups report more negative effects
  • Sexuality: people who identify as LGB report more negative effects

In summary, according to (Laws & Huda, in their 2016 elf blog)

We need qualitative research to identify the different types of harms experienced as a result of psychotherapy and to try and understand the processes leading to it. We then need the development of research methods such as ways of measuring the frequency and severity of these different types of harm identified by qualitative research. We can then move onto properly assessing the extent of harm caused by psychotherapy and how long it persists.

In their recent Plos ONE article, Rozental et al (2016) refer to similar issues, and describe previous attempts to create a meaningful measure of adverse effects, none of which have become popular with researchers nor therapists. These include:

• Vanderbilt Negative Indicators Scale (VNIS)
• Experiences of Therapy Questionnaire (ETQ)
• Unwanted to Adverse Treatment Reactions (UE-ATR)
• Inventory for the Assessment of Negative Effects of Psychotherapy (INEP).

The overarching critique of these is that these measures were not specifically related to measuring the negative effects of therapy, were not theoretically sound, nor psychometrically valid. The Negative Effects Questionnaire was designed in order to address some of these shortcomings.

Methods

To begin, the individual questions (also know as ‘items’) in the NEQ were chosen based upon theoretical considerations and empirical findings. There were four key sources, which are detailed in other peer reviewed publications:

1. Delphi technique to seek consensus amongst ten experts in the field of internet interventions (Rozental A et al, 2014)
2. Responses to open ended questions included in an evaluation of internet based intervention for social anxiety disorder (Boettcher J et al, 2014)
3. Qualitative analysis of patient responses to open ended questions included in four clinical trials of internet delivered interventions (Rozental A et al, 2015)
4. And a comprehensive literature review.

Sixty items were generated and piloted in a cognitive interview with five people ‘unrelated to the study’ to enhance readability and understanding of the items. The measure involves a contingency question.

Did you experience this [negative effect]?

If yes:

• Here’s how it negatively affected me: Not at all, Slightly, Moderately, Very, OR Extremely
• Probably caused by: The treatment I received OR Other circumstances

The questionnaire was hosted online and participants were recruited from two distinct sources in Sweden.

1. Adults with social anxiety undergoing a telephone-based intervention that was based on a CBT model of psychological intervention (n=189)
2. Individuals recruited via the media who had prior experience of psychological therapy in the previous two years (n=464)

Exclusion criteria from the intervention group included suicidality, ongoing psychological treatment or recent change in psychotropic medication status. No exclusion criteria were noted for the ‘media’ group.

Only items attributable to therapy by participants were analysed, that is, where the response ‘probably caused by the treatment I received’ was chosen. Exploratory factor analysis was carried out to validate and determine the factor structure of the instrument.

Results

Most of the 653 participants who completed the survey were female (77%), in a relationship (60%) and had at least a university degree (62%). The most common psychological disorder reported was anxiety and the most common therapy was CBT, unsurprisingly given the intervention sample. Interestingly CBT incorporated a range of therapeutic modalities; schema, cognitive and acceptance and commitment therapy.

The most commonly endorsed items were selected by approximately a third of participants; ‘Unpleasant memories resurfaced’ (38%), ‘I felt like I was under more stress’ (38%) and ‘I experienced more anxiety’ (37%). The data was deemed to be suitable for exploratory factor analysis and there was no risk of multicollinearity, in other words, each item was deemed to be measuring a distinct phenomenon. Seven factors were identified, however one related specifically to the effects of Internet based psychological treatments. As this factor was deemed to be unrelated to the underlying construct, it was removed and six factors remained. One of these six factors was below the statistical threshold, but was still identified as theoretically valid and was included in the parallel analysis. This analysis tested the validity of the factors and the results indicated that the six factor model based upon 32 individual items was reasonable, with internal consistency.

The factors are:

1. Symptoms
2. Quality
3. Dependency
4. Stigma
5. Hopelessness
6. Failure

Conclusions

The Negative Effects Questionnaire (NEQ) has been translated into 11 languages and is freely available for use in research and clinical practice at neqscale.com.

Strengths and limitations

This study provides the background to the development of a theoretically driven measure of the adverse effects of psychological therapy. It was based upon the literature, qualitative data from individuals undertaking internet based therapy, and a Delphi study with experts in internet interventions.

While it is promising that this measure is internally valid, it’s external validity is perhaps less definite. There is a possibility that the NEQ may not be relevant for psychological therapy that is not online. Even though the factor related only to internet interventions was removed, it is important to note that the items were all generated from experts and participants in internet-based interventions.

Twenty-nine percent of the sample for assessment of the items were enrolled in an internet based intervention. The inclusion of a diverse media group is commendable, however the number of people in the media sample who also had experience of internet based therapy is not clarified, an important factor considering the survey was online. The results of the media group alone would perhaps be more relevant to the generalisation of the NEQ to all modes of psychological therapy.

As the survey was hosted online it again suggests that the NEQ might be over representative of an online therapy sample, or indeed people with experience of psychological therapy who are not online.

Similarly, the sample from whom the items were generated, and tested, were primarily individuals with anxiety. This suggests that this measure may not be generalisable to individuals with other presentations, or indeed, face to face psychotherapy for social anxiety which could, by its nature provoke distinct negative effects when compared with online interventions. The three most commonly endorsed items could be related to anxiety specific intervention, such as exposure, rather than psychological therapy in general.

As a fan of open access, it is great to see that the measure is freely available and is extremely useful as a measure of the adverse effects of CBT based internet interventions for social anxiety, however it is perhaps premature to conclude that the NEQ is generalisable to all modes of psychological therapy, or indeed therapeutic approaches other than CBT for mental health presentations other than anxiety.

As a relatively under-researched area there is a lack of agreement in the literature about the definition of adverse effects, and indeed, the authors highlight this in their discussion of four previously published outcome measures. Asking participants to identify the causality of the adverse effects and excluding those items that were not identified as ‘probably caused by the treatment’ is a somewhat dubious method. There is, as yet, little language and conceptual agreement about what the effects might be, or indeed the mechanisms and so some effects could be missed as a result. There is the concern that asking about adverse effects could increase their incidence (similar to the nocebo effect) and so validating the initial list in an RCT of a psychological intervention, rather than through opinion, would be a more useful method of identifying effects that may be caused by therapy.

Implications for practice

You might have guessed it; further research is required, particularly with an offline sample. Findings from an ‘offline’ sample could support, or refute the NEQ’s external validity as a measure of adverse effects in general.

Luckily, this is currently happening. The APT study (Adverse effects of Psychological Therapy) is currently recruiting individuals with experience of any face to face psychological therapy for any mental health condition to take part in a qualitative survey. The APT study is also recruiting professionals who provide therapy to share their opinions, and aims to seek a consensus between professionals and people with personal experience. This will be done by asking everyone who takes part in the first survey to rate the responses as relevant or not. It will be exciting to see if a diverse group will reach a consensus on what we mean by adverse effects of psychological therapy, before it is tested in an RCT.

We invite you to consider taking part: www.aptmeasureofsafety.wordpress.com.

Conflicts of interest

The author of this blog, Edel Mc Glanaghy, is the lead researcher on the APT study.

Links

Primary paper

Rozental A, Kottorp A, Boettcher J, Andersson G, Carlbring P (2016) Negative Effects of Psychological Treatments: An Exploratory Factor Analysis of the Negative Effects Questionnaire for Monitoring and Reporting Adverse and Unwanted Events. PLoS ONE 11(6): e0157503. doi:10.1371/ journal.pone.0157503

Other references

Rozental A, Andersson G, Boettcher J, Ebert DD, Cuijpers P, Knaevelsrud C, et al. (2014) Consensus statement on defining and measuring negative effects of Internet interventions. Internet Interventions (PDF). 2014;1(1):12–9. doi: 10.1016/j.invent.2014.02.001

Boettcher J, Rozental A, Andersson G, Carlbring P. (2014) Side effects in Internet-based interventions for Social Anxiety Disorder. Internet interventions. 2014; 1(1):3–11. doi: 10.1016/j.invent.2014.02.002

Rozental A, Boettcher J, Andersson G, Schmidt B, Carlbring P. (2015) Negative effects of Internet interventions: A qualitative content analysis of patients’ experiences with treatments delivered online. Cognitive Behaviour Therapy. 2015; 44(3):223–36. doi: 10.1080/16506073.2015.1008033 [abstract]

Vaughan B et al. (2014) Frequency of reporting of adverse events in randomized controlled trials of psychotherapy vs. psychopharmacotherapy. Compr Psychiatry. 2014 Jan 23. pii: S0010-440X(14)00006-6. doi: 10.1016/j.comppsych.2014.01.001. [Epub ahead of print] [PubMed abstract]

Photo credits

• Joe Houghton CC BY 2.0
• Quinn Dombrowski CC BY 2.0