ACORN User Feedback Form: Field trials and preliminary results

Jeb Brown, PhD, Center for Clinical Informatics

Christophe Cazauvieilh, PhD University of Bordeaux, France

Purpose

The purpose of this brief report is to summarize findings from a field trial of the ACORN User Feedback Questionnaire (Version 1). Worthen & Lambert (2007) posit that client feedback would influence outcomes by providing information to clinicians they initially did not pay attention to their real value. This information could complete or replace the information naturally present in clinical conversation (Cazauvieilh, 2018). This questionnaire was designed to elicit information from ACORN Toolkit users regarding how useful they found various elements of Toolkit for clinical feedback, as well as how often of they learned something useful from the client completed questionnaires.

Four additional items inquired about frequency of engaging in activities outside of therapy that are hypothesized to be associated superior outcomes. Some of these activities are associated with the term deliberate practice, which has gained currency in psychotherapy circles. For purposes of this report, these will be referred as Improvement Activities (IA).

This initial field trial was also designed to explore to what extent users finding elements of Toolkit feedback useful correlated with the clinician’s measured outcomes. Likewise, the dataset permitted us to investigate the relationship between the frequency of improvement activities and perceived usefulness of the Toolkit and questionnaires, and ultimately measured outcomes.

Description of measure

Below is the list of items on the questionnaire, along with the variable label (UF1, UF2, etc).  These labels will be useful when examining results at the item level, and investigation of the factor structure of the questionnaires.

User feedback item 1 (UF1): How often do you log into your Toolkit?

            Almost never    Once a month    Every other week or so    Weekly    More than weekly 

How often do you find the following information helpful in your clinical work?

UF2: Total Global Distress Score

            Almost never    Rarely    Sometimes    Frequently    Almost always 

UF3: Self-Harm Score

            Almost never    Rarely    Sometimes    Frequently    Almost always 

UF4: Substance Abuse Score

            Almost never    Rarely    Sometimes    Frequently    Almost always 

UF5: Alliance Score

            Almost never    Rarely    Sometimes    Frequently    Almost always 

UF6: Client graphs of change over time

            Almost never    Rarely    Sometimes    Frequently    Almost always 

UF7: Clinical Messages

            Almost never    Rarely    Sometimes    Frequently    Almost always 

UF8 Your overall Effectiveness Score (Severity Adjusted Effect Size)

            Almost never    Rarely    Sometimes    Frequently    Almost always 

How often do you spend time outside of work...

UF9:...thinking about how to modify your approach with a specific client?

            Almost never    Rarely    Sometimes    Frequently    Almost always 

UF10:...reading about research in your field?

            Almost never    Rarely    Sometimes    Frequently    Almost always 

UF11:...reading books that might help you be a better therapist?

            Almost never    Rarely    Sometimes    Frequently    Almost always 

UF12:...thinking about areas you might need to improve your skills?

            Almost never    Rarely    Sometimes    Frequently    Almost always 

UF14: Please estimate the percentage of your patients where the questionnaires gave you information you might otherwise not have been aware of.

  0 10 20 30 40 50 60 70 80 90 100 

A single item (UF13) was a free form text field asking for any feedback the user cared to share.

Description of sample

The field trial was conducted at a large practice that had 290 clinicians actively submitting data at the time of this study. Emails explaining the purpose of the study and containing links to the questionnaire were sent out to these clinicians in two waves just over a week apart. The result was 88 clinicians (30%) returned completed questionnaires. 

These clinicians were matched with their outcome data, with a minimum sample size of 15 cases beginning treatment in the clinical range with at least two measurement points. This allow us to calculate the average Severity Adjusted Effect Size (SAES) for each clinician while using Hierarchical Linear Modeling (HLM) to account for sample size, as displayed via the ACE Stats report in the Toolkit. For purposes of this report, the HLM calculated SAES will simply be referred to as effect size, or ES.

For a more detailed discussion of the methodology for evaluating therapists’ outcomes see Brown, Simon & Minami, 2015.

Eighty three (83) clinicians met the criteria to have their ES included in the analysis.

Results: Questionnaire psychometric properties

The following table presents the results of the factor analysis (principle components) provides some interesting insights. As is evident in the table, three factors emerge.

Factor 1 might be called Helpfulness of Toolkit feedback (uf2-uf7). All of these items have a factor loading of .5 or higher.

Factor 2 contains the items inquiring about outside improvement activities (uf9-uf11). Interestingly, it also includes the single item (uf8) which inquires about how useful the user finds their overall performance feedback (Improvement Score in the Toolkit). 

A single item inquiring about frequency of Toolkit usage (uf1) forms Factor 3.

Strikingly, the item asking about thinking about ways to improve skills (uf12) had a negative loading on Factor 1 and only a weak loading on Factor 2.

Factor Pattern-Principle Components

Using the results of this analysis, two scales were created reflecting factors 1 and 2. These are labeled ACORN Usefulness (AU) consisting of items uf2-uf8 and Improvement Activities (IA) consisting of items uf8-uf11. Reliability (coefficient alpha) was .78 for the AU scale and .80 for the IA scale. This is within the range of coefficients of reliability to be expected from a brief scale. Other ACORN measures of Global Distress would have similar reliability when limited to 6 or fewer items.

Analysis of individual item response frequency showed relatively little variation, with the modal response being Sometimes or Frequently on both scales. The mean item response fell exactly halfway between Sometimes and Frequently on both scales.

The correlation between these two scales indicates these domains are largely uncorrelated (Pearson r=.12;  p>.2).

Results: Correlations with outcome

The subset of clinicians with at least 15 cases with severity adjusted effect size scores (ES) was used to correlate questionnaire scores with outcomes (n=83).  Overall, the outcomes for this group of ACORN users were exceptionally good, with a mean effect size of .86.  This is well above the benchmark for Highly Effective services (ES => .8), and is at approximately the 70th percentile for all therapists using ACORN.  However, when compared to the subset of WPCS clinicians, these are respondents very close to the 50th percentile.

The AU and IA scales appeared largely uncorrelated with the sample size adjusted SAES (Pearson r<.05). However, item uf14, which asks about the percentage of cases where the questionnaires provided useful information about the client did show a positive relation to measured outcomes. The mean response of this item was 45%.

Those users that indicated that the questionnaires provided information they might not otherwise have on at least 50%  of the clients (n=50; 60% of sample) had a strong trend towards a larger ES than those who indicated the questionnaires were less useful (.06 ES difference; p<.10).  

The user reported frequency of Toolkit login did not correlate with outcomes. The mean response was between every other week or so and weekly. The Toolkit recorded user login count did show a weak correlation with the clinicians’ effect size (Pearson r=.19; p<.1). However, prior analyses of ACORN data indicated that the correlation between recorded login count and measured outcomes is non-linear, meaning that Pearson r may not be the best statistic to use.

Rather, there appears to be a threshold effect, with those clinicians logging in an average of twice a month or more exhibiting better outcomes and greater therapist improvement over time than those who logged in less often (Brown & Cazauvieilh, 2019). In this sample 97% of the users met this threshold over the past year.

Summary and discussion

The sample of clinicians is this study had exceptionally good outcomes when compared to ACORN users as a whole. This may have limited the potential to find correlations between the survey scales and user outcomes. Likewise, the frequency of Toolkit usage is particularly high among this sample. This lack of variability on both the independent measures and dependent measure could make it more difficult to find statistical significance.

One result that is quite striking is the lack of correlation between the Improvement Effort scale and the ACORN Usefulness scale. Likewise, any expected correlation with the clinicians’ outcomes were not evident in this sample. The item asking about the percentage of cases in which the clinicians learned something from the questionnaire that they did not otherwise know shows promise as a predictor and may provide a clue as to how clinicians may improve their results.

The results should be viewed as preliminary due to the relatively small sample size. A larger sample size with greater variation in ES is likely to show somewhat different results. However, the current sample is adequate for an initial field trial to better understand the nature of the measure.

References

Brown, G. S. J., Simon, A., & Minami, T. (2015, April). Are you any good…as a therapist? [Web article]. Retrieved from http://www.societyforpsychotherapy.org/are-you-any-good-as-a-therapist

Brown, J & Cazauvieilh, C (2019). Clinician Engagement in Feedback Informed Treatment (FIT) and Patient Outcomes. Retrieved from https://acorncollaboration.org/blog/2019/2/12/therapist-engagement

Cazauvieilh, C. (2018). Évaluer et améliorer l'efficacité des psychothérapies : adaptation française d'un système de suivi des résultats thérapeutiques et étude des processus participant au changement. Psychologie. Université de Bordeaux. Français. 〈NNT : 2018BORD0261〉〈tel-01959442〉

Worthen, V. E., & Lambert, M. J. (2007). Outcome oriented supervision: Advantages of adding systematic client tracking to supportive consultations. Counselling and Psychotherapy Research, 7, 1, 48-53