“Towards developing a methodology to evaluate outcomes of Art Therapy in Adult Mental Illness.”

Short Paper for Oct 2000 TAoAT Conference

given by
Pauline Mottram SRAsTh,
Manchester Mental Health Partnership,
Manchester, M8 5RL, UK

Abstract:

This paper outlines the overall context in the NHS of Evidence Based Practice. The nature and type of ‘evidence’ is explored and ‘research evidence’ in particular is discussed.

Examples of methodology of research drawn from neighbouring disciplines of psychotherapy, nursing and psychiatry are compared and contrasted. The advantages and disadvantages of RCT’s (Randomised Controlled Trials) as the ‘gold standard’ are discussed within the context of EBP initiatives.

The particular needs of Art Therapy research within an Adult Mental Illness context, are explored and a case is made for a longitudinal naturalistic outcome study that includes user evaluation. This takes into account the particular needs of Art Therapy as a discipline, the needs of this particular client group, and the multi-factorial treatment environment.

Introduction:

Parry and Richardson laid down a challenge in their 1996 ‘Psychotherapy Review’, to investigate in a naturalistic/exploratory fashion the practice and delivery of therapy in routine clinical practice, using objective and standardised measures. (ref Chiesa and Fonagy 1999)

Implicit in this challenge is a shift of focus from evidence based practice to practice based evidence. There is a recognition that there is, to quote Parry (2000, p 35), ‘considerable debate about and distrust of the principles of applying research evidence to change psychotherapy practice.’ Some practitioners hold that applying research evidence to practice could be potentially harmful to practice. This view is counter-balanced by evidence that patients can be damaged by inappropriate or incompetent therapy. (Lambert & Bergin 1994)

Efficacy and effectiveness:

Chiesa and Fonagy (1999) draw a distinction between researching the efficacy of a particular treatment for a specific condition, and researching the effectiveness of treatment as applied in routine clinical practice. They assert that these two different questions require different methodologies.

In ‘Efficacy Research’ the RCT is the gold standard. I have collated the characteristic features of Psychotherapy Efficacy RCT’s as identified separately by Wells (1999, pp5-10) and Seligman (1995):

The trial is conducted in ideal or controlled best practice conditions which optimises the treatment effect.

  • Patients are randomly assigned to treatment or control groups that utilise placebo’s or no treatment.
  • Treatment protocols are standardised through use of manuals and high fidelity is ensured.
  • Patients receive a specified number of sessions.
  • Outcomes are operationalised through symptom evaluation methods and DSM diagnoses.
  • .Patients represent homogenous samples, co-morbidity is screened out.
  • Raters and diagnosticians are blinded to patient group.
  • There is a fixed period follow-up with an assessment battery.

Efficacy Studies have demonstrated, for example, that unipolar disorders respond to cognitive therapy, inter-personal therapy and medication; OCD responds to clomipramine and exposure therapy with the latter having more lasting effects; Panic disorders respond to cognitive therapy; sex offenders are only marginally improved by aversion therapy. etc etc

In Drug Trials the RCT experimental conditions closely approximate clinical reality (although there is apparently a higher attrition rate in routine practice than during a drug trial!) In Psychotherapy RCT’s the conditions created represent a substantial deviation from usual psychotherapy clinical practice conditions.

Parry (2000, p35) discusses the fundamental problems that arise from treating psychological therapies as if they are the same as pharmacological treatments. She states: “.....the ‘drug metaphor’ breaks down at a number of points. By its nature therapist and patient cannot be blinded to the intervention being delivered and there are profound conceptual difficulties with ‘placebo’ treatments in psychological therapy. The intervention can never be entirely specified or standardised, as therapists are responsive to emergent issues, changing what is being delivered throughout the course of treatment.”

Seligman (1995, p966) states that  “Efficacy study is the wrong method for empirically validating psychotherapy as it is actually done because it limits too many crucial elements of what is done in the field.” i.e.:

  • Psychotherapy in practice is not of fixed duration
  • Clinicians in practice use self-correction to change interventions or modality to suit the presenting clinical need of the patient.
  • Psychotherapy patients are usually active in selecting a modality and a therapist, especially in private health care.
  • Most psychotherapy patients present with co-morbidity and therapy aims at addressing parallel and interacting difficulties.
  • In practice there is a concern for improvement in general functioning and quality of life issues, rather than a single focus on symptom reduction of a disorder.

In summary psychotherapy in practice is a treatment of variable duration, with improvised and self correcting features, that aims to improve quality of life as well as symptom relief in patients who are not randomly allocated and who have multiple problems.

‘Effectiveness Studies’ in contrast to Efficacy studies, evaluate the effects of treatment in the usual clinical conditions of  the care setting, as delivered by representative staff to a clinical population. This is practice based evidence.

A Consumer Report conducted in the USA in 1995 surveyed a large number of people who had undergone psychotherapy.  The report was titled : ‘Mental Health: Does therapy help?’ (Consumer Reports 1995, Nov pp734-739) This survey was high in realism. It offers an assessment of psychotherapy as performed in the field with the population that sought it. The survey was extensive and carefully conducted.

I will briefly summarise some of the findings of this report as presented by Seligman (1995):

  • Treatment by Mental Health professionals was usually found to have worked. GP’s did as well in the short term, but not in the long term
  • long term therapy produced more improvement than short term work
  • there was no reported difference between psychotherapy alone and psychotherapy with drug treatment
  • active shoppers and participants in therapy did better than passive recipients
  • no specific modality did better than any other for any problem or disorder
  • patients whose choice was limited by insurance had worse outcomes

Chiesa and Fonagy (1999, p 259)  state that “Well designed field studies may provide informative, relevant and complementary approach to the evaluation of the effectiveness of psychotherapy as delivered in routine clinical practice.”

The validity of an Effectiveness study is contingent upon careful methodological design.

It needs to include a structural description of the service delivery context: the level of training of the staff, the therapeutic orientation, resources and provision available. It needs to also offer a process description: how treatment is delivered; an explication of content of treatment; factors in delivery of treatment. These factors affect outcomes of treatment.

Discussion of methodological design:

In measuring the outcome of treatment Julliard (1998) stresses the need to use a reliable and valid measurement instrument that has a proven track record. Measurements need to be taken at the onset of treatment to establish a patient’s baseline level of functioning. Frequent assessments need to be taken during therapy to ascertain a level of causality between the intervention and the results obtained. Follow-up needs to be conducted at one-year to establish whether the patient has relapsed or whether change has been maintained.

Greenlaugh et al (1998) state that an outcome measure needs to be relevant, practical and feasible to routine care. It must also be valid and reliable. They found that short forms tend to be less reliable, but were more implementable. They hold that a balance needs to be found between good psychometric features and feasibility for use in routine practice. They believe that pertinentness of an instrument offsets the burden of data collection. Ideally an outcome measure should match the outcomes of the user, be responsive to change, provide extra information to clinicians, inform clinical decisions, and be feasible in routine clinical practice.

Julliard (1998, p18) urges Art Therapists to “gather data in ways that allow us to interact with existing outcomes endeavours and to communicate in the language of outcomes movement.” He defines Outcomes research as a ‘systematic, methodologically sound way of measuring short and long term outcomes of care, including objective and subjective characteristics of both patient and provider.......measurement of outcomes is a powerful way of preserving and enhancing quality of care in the face of arbitrary efforts at cost reduction.’(p13)

Kaplan (1998, p95) recognises the ‘daunting task’ that faces art therapists in ‘integrating research from multiple disciplines into their understanding of art therapy.’

Proposal of a methodology appropriate for measuring outcomes in Art Therapy routine practice:

In addressing the effectiveness challenge and in seeking a means of measuring outcomes of the art therapy service that I deliver, I have sought a methodology that can provide a valid and reliable quantitative outcome, whilst still respecting the aesthetic and humanistic nature of art therapy practice. This endeavour is made all the more complex by the fact that art therapy lacks a fully developed theory and it has a fragile research base. Generally it is argued that art therapy is more compatible with qualitative research designs that encompass subjectivity, rather than quantitative objective methods. Kaplan (1998, p95) states that ‘Qualitative is exploratory and theory building. Quantitative  tests hypothesis in order to refine and validate theory.’ She holds that art therapy cannot afford to reject either form of inquiry.

Two British art therapy outcome research studies have influenced my thinking. These are Pamela Perry (1989) unpublished research, and Kevin Jones’ RCT (awaiting publication).

Perry’s research is carefully constructed and she describes in detail how she derived and created evaluation measurement tools pertinent to the presenting difficulties of the client group. Her evaluation measurement tools were designed for self assessment by participants and for assessment by therapists. Her research methodology was integral to the overall treatment process. This is a characteristic that I wished to create in my study.  Jones’ research is impressive in that it uses numerous standardised measures of high psychometric validity. I recognise that this potentially enables his study to interact in the wider realms of psychological therapy outcome research, and I wished ensure a similar feature in my study, but I did not want to use data collection tools that were either separate or ‘stuck on’, or that would be cumbersome to implement.

I made a decision to design a protocol based on the features of good clinical practice. I have formalised my clinical practice, making the implicit explicit, and  synthesised this with the most appropriate general psychotherapy psychometrically validated tool available.

As a standardised and validated quantitative outcome measure I use CORE.

This covers the general domains of Subjective Well-being, Symptoms, Functioning and Risk.

A 34 question form is completed at the beginning of therapy to establish a baseline, and at regular intervals throughout therapy. I administer these myself and whilst initially I did find this to be awkward and uncomfortable, I have found that the benefits outweigh this slight intrusion. In routine practice, the form is easy to interpret and can highlight features that I may have missed or that may have not been apparent during a session. Conversely, it can also offer confirmation of my perceptions and assessment of a patient. It provides a baseline against which to measure change and progress and can facilitate treatment planning and termination.

Gantt (1998)quotes a fellow art therapist Garber as having expressed dismay at the discovery that her understanding based on her subjective observations was sometimes simply wrong when assessed with more objective methods. Over time she learnt to rely on intuition as a means to identify possibilities for exploration, not as an ultimate truth. She states that exclusive reliance on either method of understanding i.e. subjective or objective, the world within or the world without, is untenable.

The objectivity of CORE can be balanced by asking a patients to write down a list of things that are troubling, or causing discomfort. I find that this helps the patient to be actively engaged as a collaborator in the treatment and assessment process. I have found that patients are fascinated to read their accounts after a period of time, and use this as a measure of how they have changed or progressed,

This simple idividualistic subjective evaluative technique has an added therapeutic benefit of fostering in the patient hope for and belief in the possibility of change.

Approximately every twelve weeks I hold a review with the patient:

  • The patient is asked to complete a CORE form.
  • Before the review I subject my process notes from the art therapy sessions to a systematic thematic analysis. A Retrospective viewing of images provides an eloquent permanent record of the sessions. This is a process of inter-subjective co-creation of meaning.
  • At the review I ask the patient to select one image that is of particular significance, and we explore this in detail, as a frame and focus for review and evaluation of progress.

The review is re-presented in a formal report as a systematic record of the outcome of treatment.

The reports over time will constitute a longitudinal naturalistic study that provides both quantitative and qualitative data that can be subjected to systematic scrutiny and analysis.

Conclusion:

Kaplan (1998, p94) suggests that a fascination with ‘mystery’ can cause us ‘to grab too quickly at facile explanations’ or conversely we can be so ‘enthralled by mystery that this becomes an end in itself’. Kaplan quotes D’Andrade ‘Testing and checking are required to keep from falling prey to the greatest weakness of the human mind - believing that one’s beliefs are true.’

There can be a dialogue between processes of constructing and uncovering meaning, between qualitative and quantitative data. I hold that art therapy outcome research, to adequately represent  the richness of the clinical process needs to accommodate both as warp and weft of the overall tapestry of art therapy.

References:

Chiesa,M and Fonagy,P (1999) - ‘From the efficacy to the effectiveness model in Psychotherapy research: the APP multi-centre project’ Jn : Psychoanalytic Psychotherapy vol13 no3 pp259-272

Gantt, L (1998) - ‘A discussion of Art Therapy as a Science.’ Jn: Art Therapy AATA 15(1) pp3-12

Greenlaugh,J, Long, AF, Brettle, AJ, Grant,MJ (1998) - ‘Reviewing and selecting outcome measures for use in routine practice.’ Jn: Evaluation in Clinical Practice 1998 4-4-pp339-350

Julliard, K (1998) - ‘Outcome Research in Health Care: Implications for Art Therapy’ JN: Art Therapy, AATA 15 (1) 13-21

Kaplan, F(1998) - “Scientific Art Therapy: An integrative and Research-based approach.” Jn: Art Therapy AATA 15(2) 93-98, 1988.

Lambert, MJ & Bergin,AE (1994)- ‘The effectiveness of psychotherapy’ in Bergin AE &Garfield,SL eds “Handbook of Psychotherapy and Behaviour change” 4th ed pp141-56, New York Wiley and Sons

Parry, G (2000) - ‘Evidence based Psychotherapy: special case or special pleading?’ Jn: Evidence Based Mental Health, BMJ Publishing vol 3 May 2000 pp35-36

Seligman, MEP (1995) - ‘The Effectiveness of Psychotherapy’ Jn : American Psychologist Dec 1995 vol 50 no12 pp965-974

Wells (1999) - ‘Treatment Research at the Cross-roads: The scientific Interface of Clinical Trial and Effectiveness Research’ Jn: American Jn Psychiatry, 156 : 1 Jan 1999 pp5-10


‘Efficacy Research’

characteristic features of Psychotherapy Efficacy RCT’s

  • The trial is conducted in ideal or controlled best practice conditions which optimises the treatment effect.
  • Patients are randomly assigned to treatment or control groups that utilise placebo’s or no treatment.
  • Treatment protocols are standardised through use of manuals and high fidelity is ensured.
  • Patients receive a specified number of sessions.
  • Outcomes are operationalised through symptom evaluation methods and DSM diagnoses.
  • Patients represent homogenous samples, co-morbidity is screened out.
  • Raters and diagnosticians are blinded to patient group.
  • There is a fixed period follow-up with an assessment battery.

ref:

Seligman, MEP (1995) - ‘The Effectiveness of Psychotherapy’ Jn : American Psychologist Dec 1995 vol 50 no12 pp965-974

Wells (1999) - ‘Treatment Research at the Cross-roads: The scientific Interface of Clinical Trial and Effectiveness Research’ Jn: American Jn Psychiatry, 156 : 1 Jan 1999 pp5-10


Psychotherapy in practice:

  • Psychotherapy in practice is not of fixed duration
  • Clinicians in practice use self-correction to change interventions or modality to suit the presenting clinical need of the patient.
  • Psychotherapy patients are usually active in selecting a modality and a therapist, especially in private health care.
  • Most psychotherapy patients present with co-morbidity and therapy aims at addressing parallel and interacting difficulties.
  • In practice there is a concern for improvement in general functioning and quality of life issues, rather than a single focus on symptom reduction of a disorder.


Consumer Report

  • Treatment by Mental Health professionals was usually found to have worked. GP’s did as well in the short term, but not in the long term
  • long term therapy produced more improvement than short term work
  • there was no reported difference between psychotherapy alone and psychotherapy with drug treatment
  • active shoppers and participants in therapy did better than passive recipients
  • no specific modality did better than any other for any problem or disorder
  • patients whose choice was limited by insurance had worse outcomes

(ref: Consumer Report 1995: Nov pp734-739
‘Mental Health: Does therapy help?’ )


EFFECTIVENESS STUDY

  • Structural description
  • Process description
  • Valid outcome measurement tool


CORE

  • Subjective Well-being - 4 items
  • Symptoms - 12 items
  • Functioning - 12 items
  • Risk - 6 items

ref: CORE Systems Group

The Psychological Therapies Research Centre
17 Blenheim Terrace
University of Leeds, LS2 9JT


http://www.taoat.org/mottram2.html

Back to Pauline Mottram conference page