The DWE invites expressions of interest from suitably qualified Fellows who would like to contribute as a DWE Fellow to future examination standard setting activities.
This Expression of Interest will remain open on an ongoing basis. Fellows who meet the relevant requirements may be added to the DWE Fellow Pool and contacted when their expertise is required for a standard setting activity.
What is Standard Setting?
Standard setting is a structured process used to establish an appropriate performance standard for an examination.
For DWE examinations, standard setting is undertaken using the Modified Angoff methodology, a widely used approach to establishing examination standards. The process draws on the professional judgement of a panel of suitably qualified subject matter experts, supported by a structured and facilitated methodology.
During a Modified Angoff standard setting exercise, Fellows review examination questions and consider how a minimally competent candidate would be expected to perform on each item. Fellows make judgements about the likelihood that such a candidate would answer each item correctly.
These judgements are considered collectively and, through the Modified Angoff process, contribute to establishing a recommended performance standard for the examination.
The process is designed to establish a standard based on the level of competence expected of candidates, rather than simply determining a pass rate based on how a particular cohort performs.
Why does DWE undertake Standard Setting?
The DWE is committed to ensuring its examinations are fair, valid, reliable and appropriately aligned with the level of competence candidates are expected to demonstrate.
Standard setting forms part of the DWE’s broader examination quality assurance and psychometric processes. It provides a systematic way of bringing together professional expertise and psychometric principles to support an appropriate and defensible examination standard.
Importantly, the purpose of setting standards is not for individual Fellows to decide which candidates should pass or fail. Rather, Fellows contribute their professional expertise to defining the level of performance that should reasonably be expected of a minimally competent candidate.
When does Standard Setting occur?
Standard setting is undertaken in the lead-up to DWE examinations where it is psychometrically necessary and appropriate.
It's therefore not conducted according to a fixed schedule. The need for a standard setting activity will depend on factors relating to the examination, its development, and the psychometric requirements identified by the DWE.
Fellows in the DWE Fellow Pool will be contacted when an appropriate opportunity arises.
What is involved in being a DWE Standard Setting Fellow?
Fellows participating in a standard setting activity will be asked to contribute their professional knowledge, experience and judgement as part of a facilitated panel.
Depending on the standard setting activity, Fellows may be required to:
- participate in a briefing or orientation on the Modified Angoff method and the purpose of the standard setting exercise
- become familiar with the definition and expected capabilities of the minimally competent candidate relevant to the examination
- independently review examination questions and consider the knowledge, skills and level of competence required to answer each question correctly
- provide an individual judgement about the likelihood that a minimally competent candidate would answer each question correctly
- participate in facilitated panel discussions about examination items and the judgements made by panel members.
- review and, where appropriate, revise individual judgements following panel discussions
- contribute constructively and independently, recognising that differences in professional judgements are an expected part of the process
- maintain the confidentiality, integrity and security of all examination materials and information accessed through the standard setting process
Fellows aren't expected to be experts in psychometrics or the Modified Angoff method. The DWE will provide appropriate instructions and facilitation to support participants through the process. The Fellow’s primary contribution is their subject matter expertise and informed professional judgement.
Who are we looking for?
The DWE seeks to bring together a diverse and representative panel of Fellows for each standard setting activity.
Our Fellows are subject matter experts within their respective sub-specialities. We recognise that valuable expertise and perspectives come from different professional backgrounds, practice settings, locations and stages of a Fellow’s career.
When forming a standard setting panel, we aim to include a diverse range of Fellows:
- from across Aotearoa New Zealand and Australia
- representing different sub-specialities and areas of subject matter expertise
- working across a range of practice settings and professional contexts
- at different stages of their careers, from recently admitted Fellows through to Fellows with many years of experience.
This breadth of experience is important to the Modified Angoff process. Standard setting relies on the informed professional judgement of the panel, and bringing together Fellows with different areas of expertise and experience helps ensure that the discussions reflect the breadth of contemporary practice.
Fellows don't need previous experience in standard setting or psychometrics to participate. The DWE will provide guidance on the Modified Angoff method, the standard setting process and the role of the Fellow.
What we ask Fellows to bring is their subject matter expertise, professional experience and independent judgement, together with a willingness to engage thoughtfully and constructively with the other members of the panel.
Join the DWE Fellow Pool
Fellows who submit an Expression of Interest may be considered for inclusion in the DWE Fellow Pool. Inclusion in the pool does not guarantee participation in a standard setting activity. Fellows will be approached as required, having regard to the examination, the expertise required, panel composition and other relevant considerations.
When a standard setting activity is required, the DWE will provide selected Fellows with further information about the activity, including the expected commitment, timing, format and any preparation required.
If you're interested in contributing your expertise to the quality and integrity of DWE examinations, we invite you to submit an Expression of Interest to join the DWE Fellow Pool.
Download the DWE Fellow Expression of Interest Form (DOC)
To discuss the DWE Fellow Pool, the Modified Angoff Standard Setting Process, or the Divisional Written Examination, please contact Corey Matthews, Executive Officer – Assessment Services at examinatons@racp.edu.au