Case Study: Structuring the Interview at Dartmouth

A group of researchers at Dartmouth were interested in improving the uniformity of content between different faculty interviewers for specialty training applications. This problem had arisen partly due to the inclusion of more interviewers to reduce the burden on individuals.

Standardised questions were introduced as part of the existing process of interviewing. Twenty-five candidates had two 30-minute interviews as one-on-ones with faculty members. Each interview included a unique standardised question, developed as a Past Behaviour Question (PBQ).

The majority of faculty liked asking the question as they enjoyed hearing the answers to the questions, which were often short stories that enhanced the conversation and were also firmly based on appropriate job-related topics. They liked knowing that the other faculty interviews were unified with theirs and felt less pressure to create all the pertinent questions themselves. The grading rubric was liked, as well as the examples of good and bad answers, because it gave them confidence that they were not overscoring or underscoring compared with their peers.

Standardised question examples

Question 1: Can you tell me a time during one of your rotations where you needed to take a leadership role in the case workup or care of the patient? How did this occur and what was the outcome?

Answer: Rate the candidate’s answer 1–4 (1 = poor answer, 4 = best answer).

1. No, I have never taken leadership in a case.
2. Probably once, I do not recall how it happened but it turned out alright.
3. Yes, I took an action (specifics given) that added to the care of the patient/case (specifics given).
4. Multiple times I have worked as a team leader to care for my patients/workup cases (specifics given). As a result, I was recognised in some way (by peer reputation, honoured, awarded etc).

Question 2: Think about one of your mentors/role models (professional or otherwise). What characteristics do/did they have that you believe made them a good mentor/role model?

1. No, I never have viewed anyone as a mentor or role model.
2. "X" was a role model for me because they motivated me through fear, or showed how you can get by without really having to put effort into it, or other questionable value.
3. "X" was my role model because I really liked them a lot. They were very nice.
4. "X" was my role model because their characteristics/values/actions were (specifics given) those that others continue to try to emulate.

Case Study: The Clinical Problem Question

Many interviews for Basic Training positions include one or more questions which ask candidates to respond to a clinical scenario and in so doing demonstrate their clinical knowledge or lack of clinical knowledge. On the surface it is a reasonably valid approach to take. After all, the candidate’s core role will probably be assessing and dealing with clinical problems daily.

There are some problems, however, with using a clinical problem as the basis for an interview question.

Firstly, the extent of required medical and clinical knowledge for training positions is broad and there are better assessments (such as a Multiple-Choice Examination or Clinical Problem-Solving Test) for gaining a more valid assessment of the extent of a candidate’s clinical knowledge.

Secondly, designing clinical problems for interview questions may risk development of a scenario which is very common (so that most candidates can be reasonably deemed to know about such problems), in which case the candidate responses may not be particularly discriminatory. Alternatively, it can risk a scenario which is uncommon, in which case there will be difficulties in discriminating between candidates who have good breadth of knowledge (true positives), ones who have narrow knowledge but happen to know about the particular problem (false positive), candidates who have good breadth of knowledge but did not study the particular problem (false negatives), and those candidates who have generally poor clinical knowledge (true negatives).

Thirdly, to give sufficient information about a clinical problem, it is often required that the information be provided to the candidate in a written format. This can take up precious time in the interview whilst the candidate reads the question. This final issue can be countered by asking candidates to arrive early so that they can read through the problem, but this then raises the issue as to whether candidates then need to be supervised or whether using “Dr Google” is an acceptable tool for dealing with clinical problems. A reading station in a Multiple Mini Interview may provide another alternative for addressing this problem.

Clinical problems can still be of use in an interview so long as what is being assessed is not just clinical knowledge but other abilities such as clinical reasoning skills. They may, for example, provide useful insights as to how candidate’s think or reason.

Another use for a clinical problem may be to provide an example to a candidate to allow them to provide their own example of past behaviours. For example:

“In our hospital it is not unusual to have a scenario where a male patient over 60 presents to the emergency department with unstable diabetes and non-specific chest pain with no obvious changes on their electrocardiogram.

“Can you give us an example of a scenario that sounds like this? It should be something you have dealt with yourself in the past? Describe the scenario and how you managed it?”

Case Study: Adopting the MMI in WA

The development of a new major tertiary hospital and realignment of health services in Perth were the catalyst for Training Program Directors for Adult Basic Physician Training in Western Australia to come together in 2015 to review their networking arrangements and process for recruitment and selection.

Recruitment to Basic Training in Adult Internal Medicine now occurs in one of three networks. Each network is based around a major metropolitan hospital plus a series of secondment hospitals, including rural sites, and takes around 25 new trainees per year (per network). Trainees must first be appointed to a service medical registrar position before being reviewed for a Basic Trainee position.

Selection and recruitment into Basic Training is centralised and involves a two-stage process of application review (including a review of cover letter, CV and references) and interview. Both stages are scored equally and have well-developed scoring criteria. Candidates are encouraged to list recent supervisors as referees. The interview consists of a five-station multiple mini-interview with two assessors on each station. All MMIs are conducted on the one day of the year with ability for those not present in Perth to teleconference in – the teleconference is moved between stations; the candidates stay on the same line while the interviewers move. As seen below, selection criteria have been developed, which are relatable to aspects of the RACP Basic Training curriculum but are also contextualised to the local environment.

Essential criteria

• Demonstrated commitment to training as a specialist physician.
• Demonstrated clinical and cultural competence with sound decision-making appropriate to the applicant’s level of experience.
• Demonstrated professionalism, including ethical behaviour.
• Demonstrated high level communication and interpersonal skills as well as cultural awareness.

Desirable criteria

• Demonstrated commitment to continuing professional development as well as teaching and training support of other staff such as Teaching on the Run program (or similar) or Advanced Life Support (ALS) training.
• Experience in conducting, or active involvement in research activities, audits and/or publications.

The distinction between Essential and Desirable Criteria is becoming less practiced by Human Resources and some jurisdictions may require there to be only one set of selection criteria

There was some concern and confusion the first time the MMI was run (mainly due to this being the inaugural run) but those involved now feel it is a much more rigorous process and improves the quality of selection.

Values-based interviewing

Hospitals and health services have endeavoured to embed values-based criteria in the selection and recruitment process as part of an overall approach to developing workplace culture. Often this is done by a hybrid of specific position-related selection criteria and selection criteria that reflects an expectation around values and attitudes that every employee should be able to demonstrate in the workplace.

Person-Organisation (P-O) fit, also known as 'organisational fit' or 'cultural fit', is the extent to which an individual’s values align with those of an organisation.

To determine P-O fit, the organisation's values and culture must be defined and outlined in official documents, such as strategic plans and organisational capability frameworks. If candidates are to be assessed for their degree of P-O fit, it is important to ensure that the desired organisational values are legitimate and meaningful.

There are no well-recognised tools or robust measures currently available to effectively assess an individual’s degree of P-O fit. Therefore, its key to ask appropriately designed behaviourally oriented interview questions. These questions should aim to probe for examples of behaviours that demonstrate the organisation’s values.

Organisations should also consider being open and transparent about their values by including them in position descriptions and advertisements.

Online interviewing and videoconferencing

Generally, candidates are expected to attend their interview in person. However, this is sometimes unachievable and presents the strong case for provision of facilitating online interviewing and videoconferencing to ensure equality and fairness overall in the interview process for remote candidates wanting to participate.

To compare, internal candidates may only have to block out an hour of time from their work to attend an interview, whereas rural or remote candidates attending the central interview may have to take several days off work. Not only is this a possible inconvenience for the candidate but potentially creates service and rostering disruptions to their current hospital setting.

Online interviewing is being used more across industries as part of the hiring process. Generally, a recruiter will program interview questions into an interview system. Questions can be posed in text, audio or video. Some platforms allow for a mix of responses, for example providing video responses to some questions, text responses to others and document uploads. The candidate sits for the interview remotely, at their convenience, using webcam to record their responses. The recorded interviews are then viewed by the selection committee members at their own time and pace.

Online systems are often used as an initial interview prior to shortlisting the final interview group.

The videoconference interview may be the most underutilised tool given it can decrease costs and streamline the experience of the interview process¹⁰.

Concerns over the use of video interviewing include the qualifications (or lack of qualifications) of the person assessing the candidate and the criteria used for screening out applicants. Another concern is around the time taken by the assessor to review candidates and not, for instance, fast forward through their interview. These concerns give greater opportunity for existing problems and biases with the interviewing process to occur, for example interviewers can form a first impression of the candidate in the first few minutes of the video and find it difficult to move on.