Introduction

There is robust evidence that Cognitive Remediation (CR) interventions are effective (e.g.1), and that four key ingredients2 can boost cognitive and functional outcomes3,4,5. From the users’ point of view, CR is associated with perceived cognitive improvement6,7,8,9 as well as increased confidence and motivation7,8,9. We know little about how to implement this therapy successfully as meta-analyses have not found differences between implementation methods (e.g., Independent, Group and One-to-One). One variable that might tip this balance is the view of the CR users, and particularly their satisfaction with treatment. Satisfaction may affect treatment engagement which has an influence on the critical dose of therapy received and therefore the treatment outcome, and so may even suggest the most efficient way of providing treatment for most service users. Alternatively, the outcome of treatment may affect satisfaction.

Acceptability and satisfaction with treatment are usually assessed indirectly by treatment drop-out and hours of engagement, rarely through direct satisfaction measures5,10,11. Even when a satisfaction measure is used, it usually answers questions posed by clinical researchers and not those generated by the potential therapy users. Our approach was to co-develop a Patient Reported Outcome Measures (PROM) using iterative, participatory methods12,13 focussing on questions important to the participants themselves. This psychometrically sound measure of CR treatment satisfaction (Cognitive Remediation Satisfaction (CRS) scale) was then used to investigate the potential effects of satisfaction on treatment engagement, implementation and the primary treatment outcome, personal recovery goals, from a completed randomised clinical trial (RCT), ECLIPSE14 which showed improvements in recovery (measured by the Goal Attainment Scale15, following CR with CIRCuiTSTM software16.

Method

Design

This is a mixed methods study involving two stages: 1) Development of the psychometrically sound Cognitive Remediation Satisfaction scale (CRS); and 2) applying the CRS to evaluate satisfaction with cognitive remediation (CR) implementation methods and outcomes in a secondary analysis of data from an RCT involving three different CR implementation techniques (one-to-one, group and independent use14,16). These data showed overall functional improvement post-treatment for the one-to-one and group methods but not the independent method. Ethical approval for measure development was obtained from Chelsea Research Ethics Committee (REC; 15/LO/1816) and for the RCT from Camden and Kings Cross REC (15/LO/ 1960).

Participants and procedures

Stage 1: CRS development

Participants aged 18-35, and able to provide informed consent were recruited from early intervention services (EIS) to two focus groups facilitated by service user researchers. After familiarising participants with CR, they discussed the content and format of a new satisfaction scale. The topic guide was based on our existing co-developed CR measure designed for paper-and-pencil CR9,17. The focus groups considered the items from the original scale and adapted and added further items. An initial thematic analysis of the transcribed focus group interviews generated a draft questionnaire that was discussed in the second focus group for respondent validation and analysed thematically using NVIVO12 to produce a revised version. The draft CRS was completed online at the end of therapy (post-treatment) in a randomised controlled trial and participants were offered the opportunity to repeat it with a one-to-two-week interval. The draft CRS was then subjected to psychometric testing to refine the ordinal items and produce the final version (see analysis).

Stage 2: Assessing post-CR satisfaction

Figure 1 shows the completion of the draft CRS within the RCT. Participants were aged 16 to 45, had attended UK NHS EIS for at least three months, had a diagnosis of non-affective psychosis and could provide informed consent. Exclusion criteria were an inability to communicate in English, an underlying organic/neurological condition, and a co-morbid diagnosis of learning disability. Participants were randomised to receive the same number of therapy sessions with the CR computer software, CIRCuiTSTM, but with variation in the amount of therapist support available to an individual participant14,18.

Fig. 1
figure 1

Completion of the draft CRS within the RCT.

Full size image

Assessments

We used the baseline measures from the RCT data to assess the robustness of the CRS data collected in case those that completed the measure were different from the remaining sample by considering: demographic variables, total negative symptoms (CAINS19), and global psychopathology (PANSS total20), and a global cognition composite score derived from a range of measures14.

To examine convergent validity and the exploration of satisfaction associations, we collected the Working Alliance Inventory (WAI21) online. The WAI is a 36-item measure of therapy engagement that assesses different aspects of therapy satisfaction. In addition, we used the total number of CR hours in valid sessions as a measure of treatment engagement and the primary outcome (post-treatment Goal Attainment Scale (GAS) weighted T-score15 that measured improvements in personally defined recovery goals.

In addition, we asked if participants thought they had improved their concentration, memory, being alert, confidence, problem solving, planning, social interaction, setting goals and motivation. All were rated on a scale of 0 = not at all to 4 = A lot and we added the scores to get a total subjective improvement score. This allowed us to consider if different implementation methods produced more subjective improvement.

Implementation methods

The CR adopted is a therapist-supported computerised therapy that embeds metacognitive aspects of learning into the software. CIRCuiTSTM is feasible and acceptable and has had independent evaluations17,18,22,23 Although all groups had access to CR17 for up to 42 sessions, they differed in their access to a therapist. In the independent method, the participant had access to an introductory session and then half an hour a week of therapist contact. For group and one-to-one methods, the therapist was always in contact, but in the group method the therapist’s time was shared with up to four other people.

Lived experience involvement

Lived experience involvement is associated with study success24 and so we consulted people with experience of using mental health services at every stage. The development of the RCT and the study protocol involved service users and we also continued their involvement as advisors (Patient Advisory Board), critical reviewers, and authors of this publication in addition to employing researchers with a background in using mental health services.

Analysis

Qualitative analysis

Data were transcribed after each focus group and analysed independently by two service user researchers, using inductive thematic analysis25 on NVIVO12. This involved familiarisation with the data, generation of codes and then development, refinement, and definition of sub- and over-arching themes. Any differences were resolved through discussion.

We also analysed the themes arising for each implementation method in the same way from answers to any open-ended questions included in the final measure.

Data used and sensitivity analyses

Draft CRS and WAI scores were pro-rated if fewer than 20% item scores were missing. Since the CRS was completed by only a subset of trial participants, logistic regression was used to examine whether missing draft CRS scores were related to CR method, trial site, demographic characteristics, cognition, and symptoms. Any variables that were predictive of missingness were included in subsequent analyses.

Scale refinement and validity

For Likert scale items, three psychometric criteria for item selection from the draft CRS were used. CRS: item-rest correlation (Cronbach’s alpha), item discrimination parameter from a graded membership item-response theory model, and test-retest (kappa; intra-class correlation). The test information curve provided an understanding of measure information across the range of scores. After refining the scale, we explored potential sub-structures with a factor analysis using the item polychoric correlation matrix. Validity was assessed on the final scale through Pearson correlations between the post-treatment refined CRS score and: (i) treatment engagement (therapy hours completed), where it was assumed that those more satisfied would be more likely to take part in more therapy hours during the treatment window, and (ii) the WAI score measures aspects of treatment satisfaction but, before analysis, we removed the four CRS therapist items that directly overlapped with the WAI. Higher total CRS scores reflect better satisfaction so, where necessary, some items were reverse scored. The absence of cultural bias was tested by gender and ethnicity.

We also assessed whether the CRS score provided more information than a single question “Overall were you satisfied with treatment” as a high correlation suggests that a simplified measure would be just as appropriate.

Following refinement, we calculated the mean CRS score for each implementation method.

The relationship between satisfaction, implementation models and trial outcome

We first used an ANOVA model to investigate whether satisfaction scored using the post-treatment refined CRS was related to implementation method. The model included trial arm, site, and variables identified as predictive of missingness.

Analyses of potential models of the relationships between satisfaction, implementation method and outcome used structural equation joint modelling (Stata 17 sem, method mlmv; vce(robust); StataCorp 2019) to account for selective drop-out by using all CR recipients to examine the effect of satisfaction on GAS at 15-weeks post-therapy. In these models we assessed whether any effect of satisfaction on post-therapy outcome was due to either the implementation method or to higher levels of treatment engagement (hours of CR). The models were refitted, using each of the newly defined satisfaction component factors to determine whether either component affected the outcome. Sensitivity analyses included: (i) refitting the model with log transformed treatment engagement as measured using hours of CR to address non-normality of the distribution and (ii) investigating potential effects at a 6-month follow-up rather than post-treatment (known as sleeper effects) by replacing GAS at 15-weeks with GAS at 6-months.

Results

Stage 1: CRS development

Sample characteristics

The same 8 participants took part in the two focus groups. All had a diagnosis of non-affective psychosis and were from a minority background; seven were men and the median age was 28.

Draft CRS content

For item format, focus group participants mentioned difficulties they had with attention, concentration, memory, and planning (the cognitive targets of CR) and said that they preferred closed questions with set response options but with added comment boxes for fuller answers for some questions. The derived draft CRS included 24 items on a Likert scale, 5 binary-categorical and two open-ended questions with optional free text space.

There were four key item themes: therapy hours (understanding and use of different CR components); the therapist (understanding the therapist’s role and levels of support); the therapy effects (applying skills learnt from CR into everyday life and the personal impact of undertaking and ending the therapy); and using the computer (ease of computer use and the CR programme). The 2 open-ended items asked about the most helpful and least helpful aspects of CR. Examples from the thematic analysis, used to develop item content, are provided in the supplement. During the respondent validation process, there was consensus that the draft scale was comprehensive and of an appropriate length and wording.

Data and participants

Table 1 shows demographic and baseline characteristics for the 144 trial participants who received CR via the CIRCuiTSTM programme and completed the draft satisfaction scale at post-treatment. Logistic regression showed that the response rate was only associated with living with a partner and living with a parent. Both variables became non-significant once account was taken of the post-baseline measure of completed CR hours, and therefore were not included in the structural equation models.

Table 1 Demographic and baseline characteristics of participants completing the draft CRS.
Full size table

Psychometric analysis and scale refinement

Both item-rest correlations from a Cronbach analysis and item discrimination estimates from an ordinal IRT model showed that four of the 24 Likert-scale items in the draft CRS performed markedly poorer and were removed (See Table 2) and one further item (Q2) did not provide good test-retest reliability. For the total score we also removed another item - the Overall Satisfaction Score - leaving 18 well-performing Likert Scale items (mean=87.48, SD = 9.05, median=87.94, range= 56-108; Cronbach alpha 0.814; test-retest intraclass correlation 0.763). An IRT test information curve (Figure S1, Supplement) indicates that items provided more precise estimates of satisfaction over the lower two-thirds of the score distribution suggesting that the scale is better at measuring lower than higher satisfaction.

Table 2 Eligible Likert satisfaction items in the draft CRS prior to refinement and excluding the overall satisfaction item.
Full size table

Categorical factor analysis gave eigen-values of 6.04, 2.37 and 1.10 for the first 3 factors. To find the smallest number of interpretable factors that explain the maximum amount of variability in the data, a two-factor solution was chosen which explained 73% of the variability in the 18 items. Table S2 and Figure S2 give factor loadings for the 2-factor solution, suggesting the items divide into therapy engagement and therapy effects. The 18-item total score was correlated with treatment hours (r = 0.22, p = 0.007) completed during the therapy window (r = 0.187, p = 0.025 with log-treatment hours). The correlation with WAI score (r = 0.56, p < 0.001) suggested significant and reasonable convergent validity. CRS total was unrelated to study site (5df p = 0.340), baseline negative symptoms (p = 0.274), baseline PANSS (p = 0.740) and the cognitive composite (p = 0.845). The 18-item total score was correlated, but weakly, with the “Overall satisfaction” item of 0.31 (polyserial, p = 0.001) suggesting that the total score does provide an adequate picture of people’s experiences. Analysis by demography showed a tendency for women to be more satisfied than men (p = 0.021) and some variation by ethnicity (3df chi2 p = 0.062) with higher mean scores among Asian participants (91.42) than White (87.43), Black (86.57), or other ethnicities (84.88).

There were two binary items included in the CRS. The first asked whether CR had been helpful and the other asked whether another implementation method would have been preferable. Not all participants completed this section of the CRS but of those that did a large number (95.7%, N = 111) thought that CR was helpful and only 6 (4.6%, N = 129) would have preferred another method with one person indicating their preference (from group to one-to-one). There were few differences between the implementation methods, although one-to-one therapy was found to be more helpful and preferred (One-to-One 98.6%, 97.5%; Group 92%, 90.9%; Independent 87.55, 94% respectively).

The two open-ended questions followed the binary items and asked participants to expand on their answers to say why they did or did not think CR was helpful and why they preferred or not their method of delivery. The themes from the answers were analysed separately for each implementation method. Participants receiving CR on an independent basis appreciated the flexibility and choice it provided (“It gave me freedom to choose when I did the therapy”; “I was able to focus alone”). However, some found it difficult because of lack of motivation “I wasn’t able to set myself targets to get the tasks done on time”.

Group participants appreciated the opportunity to share experiences and views with their peers, although sometimes there was a lack of group engagement that reduced interaction and attending could be anxiety-provoking. They felt that group delivery had a positive impact on motivation. For instance, one person reported that “We could bounce ideas off each other and we got more input from different people than you would have got in a one-to-one setting” but another said, “In the group we didn’t talk to each other, we got on with our own thing”.

Participants who received CR on a one-to-one basis reported a strong therapeutic relationship and appreciated the support provided, particularly at the outset of therapy (“I needed someone holding my hand at first, especially since the tasks seemed too simple to warrant my attention in the beginning”). Having therapy on this basis helped with focus and level of understanding, as well as reducing anxiety and increasing confidence levels (e.g., Having someone to confide in and to go out and about with helped me with my anxiety).

Is satisfaction related to implementation type or treatment outcome

There was no significant effect of implementation method on satisfaction (means Independent 85.84, Group 88.04, One-to-One 87.56; 2df F-test p = 0.606). Subjective improvement from therapy was significantly correlated with the CRS score (0.55, p < 0.01), but when investigated by each implementation method only the correlations with Group (0.643 p < 0.01) and One-to-One (0.55 p < 0.01) were significant, but the Independent method correlation was low (0.102, n.s.).

Figure 2 and Table S1 shows a series of models and estimated effects examining the role of satisfaction on treatment outcome. The simple model (Fig. 2a) indicates no significant association between satisfaction and GAS improvement. Model 2b shows no significant association in satisfaction between the implementation methods (Independent: Group, p = 0.312; One-to-One: Group, p = 0.847; Independent: One-to-One, p = 0.402), and a non-significant association between satisfaction and GAS improvement (p = 0.311). In Model 2c therapy engagement (therapy hours completed) was added and found to be significantly associated with satisfaction (p = 0.001) and with GAS improvement (p = 0.002), but the direct effect of satisfaction on GAS changed little and remained non-significant (p = 0.373).

Fig. 2: A series of models of the relationship of satisfaction to GAS outcome.
figure 2

a The simple model of a direct association. b The additional effect of implementation methods. c The additional effect of engagement (number of hours of therapy).

Full size image

When the satisfaction component factor scores were used rather than the overall score, neither of the two additional analyses found a significant association between satisfaction and improvement in the GAS outcome (Tables S3 and S4).

None of the sensitivity analyses found any significant relationship between satisfaction and improvement in outcome (p = 0.346) after (i) log transforming therapy engagement (Table S5) or (ii) the GAS outcome measured at 6 months (p = 0.964, Table S6), and at that point the relationship between outcome and therapy engagement was also marginal (p = 0.058).

Discussion

We co-developed and refined a psychometrically robust CR satisfaction measure driven by service user views. The measure was also superior to asking a single question on overall satisfaction. One of the key themes to arise from the qualitative work was the important role of the therapist, which is reflected in four CRS items. The professional and interpersonal skills of the therapist and a strong working alliance are consistently cited as important influences on therapy outcomes in existing literature7,9,26. This is also supported in this study as almost all participants were satisfied with their therapist (95.8%), valued their support (99.3%), and felt they had a good working alliance (Table 2).

Service users in the focus groups prioritised the functional, cognitive, and psychological effects of CR, reaffirming existing literature26,27. Prior research has shown general acceptability across different computerised therapies, although lack of access and skills can impede progress, highlighted also by our focus group participants8,17. The final CRS asks about difficulties using computers or tablets, the computerised CR programme and whether any support is needed. Although the items relating only to computer use were removed from the total score, they are retained in the scale as this was important to service users and may shed light on acceptability as some software may be more complex or easier than the one tested here. A key issue for focus group members is ensuring that the CRS could be completed by people who might be experiencing cognitive difficulties associated with psychosis.

We tested the draft CRS on those who entered a trial and the individuals who completed the measure were representative of those attending UK Early Intervention Services so we expect that these results will generalise to people using those services. We also did not find any significant difference between those who completed the measure and those who received therapy but did not complete the measure.

We did expect that there would be a correlation between the overall subjective improvement from CR and satisfaction with treatment and this was true for the Group and One-to-One methods but not for Independent therapy. It may be that the presence of a therapist allows an individual to develop better awareness of their improvements by supporting metacognitive knowledge via interactions. The results of the open-ended questions suggests that discussions with the therapist were key considerations for both group and one-to-one participants. Of interest is that the engagement with the independent therapy was much lower compared to the other conditions and the participants did report how hard it was to motivate themselves and set goals when receiving CR by this method.

Relationship of Satisfaction to CR implementation and outcomes

Satisfaction, as expected, was related to therapy engagement as measured by hours completed but was not associated with how you completed those therapy hours – independently, in a group or one-to one. However, those receiving CR by the independent method completed on average far fewer therapy hours (14.8 vs 24.0 or 26.0), so despite satisfaction not being related to implementation, there was a relationship of implementation and therapy engagement. In the independent method access to a therapist was limited to an hour every two weeks and this may have affected engagement. Therapist effects on engagement have been emphasised by others28,29. However, although a recent meta-analysis demonstrated that active therapists affect CR benefit significantly5, the same group found, in a further meta-analysis, that there were no independent effects of an active therapist on treatment drop-out10.

While satisfaction was significantly associated with therapy hours, and hours were significantly associated with GAS improvement, the direct effect of satisfaction on GAS improvement was not significant. This is surprising as we would assume that the effects of therapy on outcome would be related to satisfaction with treatment but the association between outcome and satisfaction could be wholly explained through increasing engagement although other possible causal routes are possible. Practice with CR together with progressive achievement of outcome goals and improved performance on the CR tasks might all increase satisfaction as well as increasing confidence and self-esteem. The causal ordering among satisfaction, engagement and goal attainment also cannot be firmly established because they were measured at the same time, but the results suggest that satisfaction may not be central to the causal mechanism of increasing functioning but may better be considered a secondary desirable outcome, relating to treatment engagement.

Strengths and Limitations

This is the largest randomised study of computerised CR in EIS that also included a co-developed satisfaction measure. We used both classical and modern psychometric methods and chose our exploratory analyses methods to understand whether completion rates could affect our data. A larger study would increase our confidence in the factor analysis results. We could not however, implement an instrumental variable approach that would have allowed for post-randomisation confounding which would have permitted a stronger attribution of causality. We only tested implementation method and did not contrast different CR software which is important for services choosing specific therapies especially if it incurs a charge. A comparison study would be the next research step.

Conclusions

Evidence based treatments can only be successfully implemented if they are acceptable to service users30 and satisfaction is a key dimension of acceptability. The CRS development highlighted four elements in two factors - therapy engagement and therapy effects and the quantitative analyses show that satisfaction is related to therapy engagement but not to recovery. Satisfaction is related to engagement which is related to benefits and although we have not detected a causal relationship, measuring satisfaction is still important for engaging service users in health services. It is also a way to detect unwanted effects and potentially to improve service efficiency and effectiveness if this measure can begin to detect those at risk of disengagement. We have produced a measure to detect those potential unwanted effects.