Assessment in Medical Education

Traditionally, there has been a general reliance on high stakes barrier assessments in medical training programs with these assessments, frequently the principal determinant of progression in training. While some providers have implemented increased workplace-based assessment (WBA), this has largely been additional assessment load to barrier examinations, rather than as part of a systematic change to a programmatic assessment approach or ‘program of assessment’. Where more WBA is being used (e.g. Entrustable Professional Activities, portfolios) these are viewed positively. In specialist medical training, the disjunction between progression in training determined by barrier assessments and progression in level of employment in the workplace is problematic for health jurisdictions and the workforce pipeline more generally.

Read more here: Evidence from medical education literature – strengths and weaknesses in assessment

The disruption that the COVID-19 pandemic caused to longstanding assessment practices, and the opportunities arising from necessary changes to these, required ‘new thinking’, agility and resilience of individuals and organisations. Some of the opportunities arising from COVID-19 were innovative, for the education providers implementing them, some were challenging at the scale required, and some were constrained by technology issues.

The COVID-19 pandemic has had a significant impact on education and assessment in Australia and around the world. Most organisations in medical education in Australia and New Zealand have had to adapt their processes. When implementing change to assessment programs, consideration should be given to: 

Changing assessment not all about moving to online assessment – focus on moving towards programs of assessment and authentic assessment.

COVID-19 pandemic caused significant disruption and led to rapid changes in assessment. What are the positive changes that should be taken forward, what changes are not working so well.

The ‘new normal’ will not likely be and should not be the same as the ‘old normal’. Amongst other things, COVID-19 has shown the vulnerabilities of specialist medical program assessment to disruption and highlighted where assessment was out of step with advances internationally and in the literature.

 Designing programs of assessment that balance workplace-based assessments with other methods, are aligned to the medical program curriculum and outcomes, and are accessible, relevant and sustainable.

Visit our resources page for more information

The AMC develops accreditation standards for all phases of medical training and education. The standards follow similar structure and formatting but are customised to the requirements of the stage of training and education. The Medical Board of Australia approves accreditation standards for the medical profession. The accreditation standards and the AMC’s accreditation processes are also relied upon by the Medical Council of New Zealand in relation to primary medical qualifications, specialist medical training, continuing professional development and the assessment of specialist international medical graduates.

The AMC uses accreditation standards to assess medical programs for accreditation and for subsequent monitoring of accredited programs and providers. The accreditation standards can be found on the AMC website.

Assessment is one of the areas of focus in the prevocational, primary medical program and specialist medical program accreditation standards. The key concepts underpinning AMC standards on assessment for medical programs across the continuum are:

 Assessment approach – The assessment program is aligned with learning outcomes, with requirements clearly documented and easily accessible to staff, supervisors and students/trainees/interns. 

Assessment methods – The program contains methods that are fit for purpose, has a blueprint to guide assessment through each stage and uses validated methods of standard setting.

Assessment feedback – The provider/program facilitates regular feedback to students/trainees/interns to guide their learning, gives feedback to supervisors on assessment performance and has processes for underperforming students/trainees/interns and implementing remediation.

Assessment quality – The provider regularly reviews its program of assessment to ensure the validity and reliability and scope of its practices, processes and standards is consistent across teaching sites.

Additionally, the accreditation standards contain requirements about preparing learners for assessment including standards about ensuring learners are informed of all requirements, and ability to appeal assessment outcomes.

AMC accreditation standards related to assessment continue to generate significant numbers of conditions which often prove challenging for education providers to satisfy.

Information from AMC accreditation of specialist medical training in the past 10 years highlights some of the issues and risks associated with the current state of assessment. These include:

low pass rates in examinations with subsequent adverse impact on trainee progression through the training program and into the specialist workforce

impacts on trainee wellbeing and effective workplace learning of preparation for barrier examinations

issues of access and equity for trainees in preparation and attendance for assessments

need for more robust quality assurance of assessments, including ensuring alignment with curriculum, determining standards, training assessors and the provision of feedback to trainees to achieve assessment ‘for learning’

De-identified data has been collected on the number of conditions generated and themes arising from AMC accreditation assessment and monitoring.

AMC assessment data – specialist medical colleges

The 2021 Conducting Assessment in a Changing Environment workshop series aimed to build on the 2020 AMC and specialist colleges meeting on moving to online examinations, and the 2017 AMC workshop on programmatic assessment, to provide education providers with opportunities to engage in effective change to their assessment programs. The themes relating to assessment in specialist medical programs identified from the AMC analysis of accreditation conditions, emerging from the consideration of accreditation reports and progress reports and highlighted by trainee surveys provided current context for the workshops.

The workshop, run over four online sessions, provided opportunities to explore the need for change and some of the barriers, some common challenges experienced in meeting accreditation standards in assessment, balancing program based assessment with other methods, share good practice examples of assessment programs, and implementation considerations.

 

The COVID-19 pandemic has required rapid additional adaption and innovation in specialist medical programs and the specialist medical colleges. One area of change across colleges has been in the regulations, requirements and methods for assessment of specialist medical trainees. Addressing COVID-19 disruptions to specialist medical training assessment practices, particularly large scale barrier examinations, has required new thinking, agility and resilience of individuals and organisations.

In early 2021, the Health Workforce Division of the Australian Government Department of Health contracted the AMC and the Council of Presidents of Medical Colleges (CPMC) to undertake a joint project to contribute to identifying, reviewing and considering more broadly and for the long term, opportunities to improve and enhance the medical training and accreditation system in Australia, reflecting on the changes occurring during the COVID-19 pandemic.

The Changes in assessment in specialist medical programs – opportunities for system improvement report developed as part of this project draws on AMC accreditation findings and AMC monitoring of the significant changes in assessment of specialist medical trainees during COVID-19. It highlights where current common assessment practice and tradition in Australian and New Zealand specialist medical programs are out of step with current best practice, presents the AMC’s findings concerning the need for change in assessment in specialist medical programs and barriers to change, preconditions for change, and opportunities for systems improvement.

In the report, the AMC has found that medical education providers and other stakeholders acknowledge the need for and are open to changes to assessment practice in specialist medical programs. Analysis identified priority issues to be addressed, and opportunities and significant challenges to be considered in developing, implementing and embedding changes in practice. These are:

Priority assessment issues

Achieving alignment of curriculum, training and assessment

Ensuring fairness in assessment

Support for trainees to complete once they are accepted to a specialty training program

Effective supervision

The burden (volume) of assessment

Opportunities for assessment

Involvement of trainees in assessment design

The administration of assessment i.e. reliance on single site assessment Technology

Utilisation of technology in assessment

Increase sector knowledge about best practice in assessment

Challenges

Organisational culture –tradition, custom, and investment in current models

Technology issues

Security concerns

Resources

 

Read the report here: 2021 11 AMC Project Report – Changes in assessment in specialist medical programs