Cover 1
Half-title 3
Title 5
Copyright 6
Dedication 7
Contents 9
Contributors 11
Preface 15
1 Workplace-based assessments in psychiatry: setting the scene 17
Background: changing socio-political context 17
Rationale for workplace-based assessment 18
Purpose of workplace-based assessments 19
Curriculum development and blueprinting 19
Utility of assessments 20
Reliability 20
Validity 21
Educational impact 21
Acceptability 22
Cost 23
Assessments as components of educational programmes 23
Conclusion 24
REFERENCES 24
2 Workplace-based assessments \u2013 an evidence-based overview 26
Editors\u2019 introduction 26
Classification of workplace-based assessments 26
Case-focused assessments 27
Long case assessment 27
Messages from literature 27
Direct observation of procedural skills (DOPS) 28
Messages from literature 28
Mini-CEX (mini clinical evaluation exercise) 29
Messages from literature 29
Case-based discussion (CbD) 30
Messages from literature 31
Multi-source feedback (MSF) 31
Assessment of other professional activities 33
DONCS (direct observation of non-clinical skills) 34
JCP (journal-club presentations) 34
Assessment of clinical letters 34
Conclusion 34
REFERENCES 35
3 Assessing the patient perspective 38
Editors\u2019 introduction 38
Introduction 38
Why the patients view matters 39
Only patients can judge certain aspects of performance 39
Clinicians care what their patients think about them 40
The hazards of patient ratings in psychiatry 40
Small case-loads may not allow for sufficiently wide sampling 40
Patients may struggle to identify the contribution of each clinician 41
Mental illness may affect the reliability of responses 41
Atypical clinician\u2013patient relationships may distort responses 41
Empirical findings from published data outside psychiatry 41
Patient ratings are feasible in a range of settings 41
Patients can provide reproducible and discriminating ratings 42
The pattern of responses across items suggests that they are meaningful 42
Patients and colleagues independently agree about clinicians\u2019 relational performance 42
Some groups are too idiosyncratic to be reliable 43
Empirical findings from unpublished studies within psychiatry 43
Feasibility 43
Reliability 43
Validity 44
Implications for implementation 45
Implications for research 45
APPENDIX 3.1 PATIENT RATING INSTRUMENT 47
REFERENCES 48
4 Experience of workplace-based assessment in the UK 49
Editors\u2019 introduction 49
Introduction 49
Background 50
The initial field trial 51
The UK national pilot study 52
Description of tools 52
Quantitative aspects 54
Qualitative aspects 56
The live workplace-based assessment system 59
Discussion 60
APPENDIX 4.1 ASSESSMENT SHEET FOR DOPS 62
APPENDIX 4.2 ASSESSMENT SHEET FOR MINI-ACE 63
APPENDIX 4.3 ASSESSMENT SHEET FOR ACE 65
APPENDIX 4.4 ASSESSMENT SHEET FOR CBD 67
APPENDIX 4.5 ASSESSMENT SHEET FOR MINI-PAT 69
APPENDIX 4.6 ASSESSMENT SHEET FOR CASE PRESENTATION 72
APPENDIX 4.7 ASSESSMENT SHEET FOR JCP 73
REFERENCES 74
5 Assessing residents\u2019 competencies: challenges to delivery in the developing world \u2013 need for innovation and change 76
Editors\u2019 introduction 76
Introduction 76
Postgraduate psychiatry training in India 77
Assessment methods required by the Medical Council of India 79
Theory examination 79
Clinical examination 80
Oral or viva voce examination 80
The MD thesis 80
Caveats to current assessment methods 81
The relationship between training goals and assessment 81
Standardization 81
Advantages of current assessment methods 82
Disadvantages of current assessment methods 82
Recommendations for the improvement of assessment methods 82
Internal assessments 83
Some innovations in assessment and feedback methods at NIMHANS 83
Supervision methods 83
Clinical supervision 84
Academic supervision 84
Teaching methods 84
Seminars, journal clubs and case conferences 84
Small-group tutorials on psychopathology and classificatory systems 85
Rating of and feedback from seminars and journal clubs 85
Ethics in clinical care and research 86
Modular programmes 86
Skill-based teaching and assessment 86
Adapting the OSCE to become OSCAF \u2013 objective structured assessment and feedback 86
Emergency psychiatry 87
Ongoing assessment and feedback 88
Thesis evaluation 88
The logbook 88
Supervision and feedback on professional issues 88
Psychotherapy training and assessment 89
Research on postgraduate psychiatric training and assessment 89
Postgraduate training in some other developing countries 90
Conclusions \u2013 how can we improve and standardize workplace-based assessments? 90
APPENDIX 5.1 GUIDELINES FOR ASSESSMENT OF JOURNAL-CLUB PRESENTATION 91
APPENDIX 5.2 SEMINAR ASSESSMENT FORM 92
APPENDIX 5.3 UNIT-LEVEL FEEDBACK FORMS 93
APPENDIX 5.4 OSCE FEEDBACK FORMS 95
REFERENCES 97
6 In-training assessment: the Danish experience 98
Editors\u2019 introduction 98
Introduction 98
Rationale for in-training assessment 99
Challenge of introducing in-training assessment 100
Experiments with constructing in-training assessment programmes 102
The internal rational validation process 102
Characteristics of anaesthesiology 103
Characteristics of internal medicine 103
Characteristics of child and adolescent psychiatry 104
Purpose, content, format and programming of in-training assessment 104
The assessment products 105
Anaesthesiology 108
Internal medicine 114
Child and adolescent psychiatry 115
Lessons learned and perspectives 115
REFERENCES 117
7 The US experience of changing roles 121
Editors\u2019 introduction 121
Introduction 121
The status quo 122
Stimulus for change 124
A new model for clinical skills assessment 124
Response from psychiatry residency training directors 125
New opportunities 127
Challenges 128
Looking forward 129
REFERENCES 130
8 Determining competence of psychiatric residents in the USA 132
Editors\u2019 introduction 132
Introduction 132
Description of the stakeholders 133
Training prior to 2001 133
The beginning of change 135
The second phase of change: the clinical skills verification exams 136
Faculty evaluator training initiative 138
Beginnings of innovation in assessment: suggestions for the future 140
Strategic management simulation (SMS) 140
Simulation of psychiatric emergencies 141
Developing standards and assessments in professionalism in psychiatry 142
APPENDIX 8.1 AADPRT CLINICAL SKILLS VERIFICATION EXAMINATION FORM 143
APPENDIX 8.2 ASSESSMENT FORM FOR \u2018ACUTE SUICIDALITY\u2019 149
APPENDIX 8.3 ASSESSMENT FORM FOR \u2018ACUTE PSYCHOSIS\u2019 150
APPENDIX 8.4 ASSESSMENT FORM FOR \u2018SUBSTANCE INTOXICATION\u2019 151
REFERENCES 152
9 The Australian and New Zealand experience 153
Editors\u2019 introduction 153
Introduction 153
The context for change at the RANZCP 154
The CanMEDS framework for the RANZCP 155
A programme approach to assessment 157
Assessing competent performance 157
Entrustable professional activities, performance assessment and CanMEDS 159
Conclusion 162
REFERENCES 163
10 The Canadian experience 167
Editors\u2019 introduction 167
Introduction 168
CanMEDS 168
Description of current training with emphasis on 2007 changes 169
Evaluation 171
Changes to the final Royal College Examination in Psychiatry 172
Residency training for international medical graduates 174
APPENDIX 10.1 RCPSC ORAL EXAMINATION SCORE SHEET FOR PSYCHIATRY 177
APPENDIX 10.2 CANMEDS COMPETENCIES SELF-REFLECTIVE JOURNAL 184
APPENDIX 10.3 THE CANADIAN OBSERVED COMMUNICATION SKILLS SCALE 185
REFERENCES 190
11 Workplace-based assessment in psychotherapy: a Canadian experience 192
Editors\u2019 introduction 192
Introduction 192
The McMaster psychotherapy programme 194
Assessment of competence in psychotherapy 195
Therapeutic alliance 196
Specific therapies 198
Crisis intervention 198
Emotion-focused therapy (supportive therapy) 198
Cognitive-behavioural therapy for depression 199
Cognitive-behavioural therapy for anxiety 200
Psychodynamic psychotherapy 201
Family therapy 202
Interpersonal therapy (IPT) 202
Group therapy 203
Dialectical behaviour therapy 203
Motivational interviewing 204
Conclusion 204
REFERENCES 205
12 Assessments and their utility: looking to the future 209
Editors\u2019 introduction 209
Introduction 209
Weighing validity against reliability 211
Striving to achieve excellence and not just competence 213
Acknowledging the progression from novice to expert 214
Balancing formative and summative assessment to achieve educational impact 217
Conclusions 219
REFERENCES 220
Index 222