Deciding

References
Referee
types
Decision making
Outcomes and
feedback

References

References are a crucial component of most selection processes. They serve to test the authenticity of a candidate's application and rule out any overstated and misrepresented achievements. References provide valuable third-party insight about the candidate that will help inform the decision of the Selection Committee.

Traditionally, reference checks occur after interviews have been completed and a preferred candidate has been identified. This approach is recommended as a candidate can reasonably expect to be interviewed if the content of their application satisfies the selection criteria.

Conducting reference checks to create a shortlist for interview is useful in certain circumstances, such as shortlisting for interview to avoid candidates incurring expensive travel and time costs. In this instance, you should make candidates aware prior to carrying out of the reference check and the reasons for it.

The effectiveness and usefulness of references can vary significantly. In many countries, including Australia and New Zealand, recruitment processes for medical training positions are generally conducted altogether on an annual basis. The number of positions available and number of applicants received in most settings demands a level of automation to ensure a smooth process. As a result, automated referee reports have become commonplace in many online recruitment systems.

Factors that affect a referee report's usefulness
Increases weight	Decreases weight
References taken verbally	Written references
References taken using selection criteria	Generic reference template used
Verbal references taken using behavioural interview techniques	Informal verbal reference
Criteria for referees	Allowing candidates to choose their own referees
Structured Letters of Reference	Unstructured Letters of Reference
References taken independently by a trained expert	References taken by untrained staff

Automated reference reports

Candidates often apply for multiple positions and certain referees may be popular. Automated referee reports can help to reduce the burden on referees and ensure completion of the referee checks, which is generally required as a compliance measure.

Automated references take the format of a Structured Letter of Reference (SLOR). North American studies conducted into the use of SLORs have shown they are easily implemented, quicker to assess, appear to provide better match selection outcomes and are easily sustained over time¹. However, some caution needs to be exercised as its been found that less experienced referees are more inclined to give a favourable reference than more experienced referees².

An automated reference systems normally deploy a generic reference template, which is designed to suit a range of medical trainee positions. It is often not possible to insert questions that are specific to the role. This may be problematic for a Selection Committee, which for example is considering applications for Basic Training in Paediatrics & Child Health, that wants to know how well the candidate communicates with small children. The committee may not be satisfied with an answer to a question that is only about candidate communication with patients and family.

Verbal references

When verbal references are taken, they should generally conducted by the same person to ensure consistency. In medical selection, this task normally falls to the Chair of the Selection Committee. However, it is possible and sometimes better to have an expert to perform this role independently from the Selection Committee. Many hospital and health service human resource departments have trained personnel in the various roles of recruitment who can take on this function.

Verbal references should be recorded in written form or transcribed. The referee should be informed that there will be a record of their reference, which can be requested by the candidate.

Despite such caveats, verbal referee reports generally yield better information about the strengths and weaknesses of a candidate and, in some cases, may be the only red flag or warning sign that a candidate might not be suitable for the role.

Best practice is to pose questions to referees that are related to the selection criteria, in the same way that interview questions are to be related to the selection criteria. If the interview questions have already been established, it may be possible to use these with the referee as well, particularly if they are of a behavioural nature.

"Can you give me an example of where the candidate identified a problem in the hospital and used their initiative to work with others to bring about a solution?"

This is a suitable question for a referee to ask in order to gain information on a selection criterion that requires evidence of the ability to work in a team and solve problems. The question can also address a selection criterion about demonstrating the ability to be innovative through teamwork and collaboration.

Generally, more questions can be covered in a verbal reference of 15 minutes than a candidate interview of 30 minutes, providing an opportunity to ask more questions of the referee.

Some of the questions can be of high value. It is recommended that these type of questions are asked towards the end of the conversation once a rapport and a referee's willingness to assist is established.

Examples:

"If you were in the position to offer the candidate another job, would you?"
"Are there any reservations you have about the candidate?"
"If there was a new patient referral in the emergency department, at what point would you be happy for the candidate to do the initial assessment and commence management?" (Not at all? While you were watching them? While you were elsewhere in the hospital? While you were at home?)
"Would you be happy for the candidate to treat a member of your family?"
"Hypothetically, the candidate has secured the position. What advice would you give to their Director of Physician Education about the candidate's strengths and weaknesses?"

Tool: District Health Board referee report template

¹ Fraser J, Aguayo P, St Peter S, Ostlie D, Holcomb G, Andrews W et al. Analysis of the pediatric surgery match: factors predicting outcome. Pediatric Surgery International. 2011;27(11):1239-1244. Prager J, Perkins J, McFann K, Myer C, Pensak M, Chan K. Standardized letter of recommendation for pediatric fellowship selection. The Laryngoscope. 2011;122(2):415-424.
² Love J, DeIorio N, Ronan-Bentle S, Howell J, Doty C, Lane D et al. Characterization of the Council of Emergency Medicine Residency Directors' Standardized Letter of Recommendation in 2011-2012. Academic Emergency Medicine. 2013;20(9):926-932.

Referee types

Medical trainees can spend significant time working out who the best people are to put forward as referees. To improve your recruitment outcome, it may be beneficial to give advice to candidates about who makes a good referee.

A candidate's referee selection should reflect recency, relevancy and diversity. Most recruitment processes ask for details of up to three referees, which gives candidates the scope to ensure their referees meet most or all of the above criteria. It is preferable that candidates nominate their current or most recent manager. If a candidate doesn't, there is cause to ask the candidate why this is the case.

For medical trainees at a prevocational level, it can be unclear as to who they should nominate as their current manager as multiple hospital staff are responsible for managing aspects of a prevocational trainee’s duties. A current manager could be their:

Director of Physician or Paediatric Education – who supervises their overall education and progress
Junior Medical Officer (JMO) Manager – who manages their contract, pay, rosters and leave
Current consultant supervisor – who supervises their daily performance
Nurse Unit Manager
Registrar (Senior House Officer)

From this list, candidates are advised to nominate the one person they believe to have the most recent observations and oversight of their performance.

Candidates who are applying for Basic Training positions may hold the understanding that they should list three referees who are Fellows of the RACP, as these referees are most relevant to their application for Basic Training. This is not the case as the RACP doesn't specify a particular type of referee that trainees must nominate.

A candidate should have had interaction with one Fellow of the RACP who they could ask to act as a referee. A nominated referee who is a Fellow of the RACP can offer valuable information in regard to a candidate's readiness to undertake RACP training in a work and training role. Requiring a candidate to provide three references from Fellows could raise fairness issues as not all medical trainees will have had the opportunity to work with three different consultants in their prevocational training period.

Aside from a supervisor or Fellow of the RACP, candidates should be encouraged to nominate other referees who can give different perspectives or observations on their performance. These perspectives can be particularly useful to assess the candidate’s skills and capabilities in communication and teamwork. This referee could be another manager such as a Nurse Unit Manager, who is often a keen observer of how a medical trainee contributes to the overall ward environment. An Advanced Trainee can also act as a referee as they may interact with the candidate daily and can assess the candidate against their recent knowledge of the training program.

Referee reports

Referee reports – verbally, written or online – should be circulated to other members of the Selection Committee along with any other candidate documentation. If a weighting is to be given to the reports, this should be established and agreed ahead of time. When receiving the reports, members of the Selection Committee should be made aware of the risks associated with confirmatory bias. Confirmatory bias occurs where information is sought or interpreted that confirms the outcome of the interview, which may increase the chances of a selection error occurring.

Decision making

When all candidates have been interviewed, references collected and reviewed, and a scoring and weighting system applied if previously agreed upon, it's time to make a final determination about who is to be made an offer of employment.

A best practice process:

Confirm consensus on those candidates who are deemed unsuitable.
This may be reflected in the scoring, if scoring is used. Regardless of scoring, there should also be a process to agree on a candidate’s unsuitability in a case where significant concerns are raised as part of a referee report.
Confirm consensus on those candidates who are clearly suitable and of high merit.
Candidates need to be ranked given that they will all be offered a position in the first round of offers but ordering candidates shouldn't require a significant amount of the Committee's time.
Considered discussions may be required for those uncertain candidates
For candidates whose suitability is not clear, more information such as verbal references may need to be sought to make a final decision about these candidates.

Once a decision has been made about the eligibility and ranking of each candidate, the Selection Committee can issue its final decision on job offers to candidates.

At this stage, its important to collate all candidate scoring sheets or interview notes as they may be required to complete the Selection Committee's report or if an appeal is made to the employing training site.

The process and final decisions should be clearly documented in writing or on the online recruitment system.

Dissent

Dissent can occur among panel members and a clear consensus about a particular candidate, or group of candidates, cannot be reached. To avoid dissent, there should be clear established criteria for determining when a candidate is ineligible. Organisational policies can also provide guidance as to what to do when a Selection Committee is divided in their opinion.

Often candidates can be considering more than one position which leads competition between programs and networks. Candidate preferences remain confidential to ensure fairness in the recruiting process. It is important for Selection Committee members to rank each candidate based on their assessment only and without bias based on the candidate's assumed preferences. Effective Selection Committees are those that continue to review and develop policy supporting this objective approach.

Dissent over rankings may also affect position offers to alternative candidates. Its helpful at this stage to review where all candidates sit in the rankings, including their prospects for being offered a position in a second round of offers, should some candidates not accept following the first round.

Where there is dissent regarding the final decision on ranking, appointment is usually determined by the majority position of the Selection Committee, with the Chair of the Committee having the final say in the decision.

This scenario emphasises how important it is to document the decision. A majority report should be prepared with the list of candidates in preferred order and reasons for their selection. A minority report should also be prepared with a list of candidates in preferred order and reasons for making their selection. The candidate order is then finalised as per the majority report.

If some Selection Committee members remain dissatisfied with the outcome, it is best to seek further advice and input from your local human resources representative or department prior to finalising all reports and proceeding to offers.

The Selection Committee must be aware that the majority and minority reports can be reviewed by an unsuccessful candidate when making an appeal against the decision.

Outcomes and feedback

It is unfair to leave unsuccessful candidates uncertain of their result. Candidates who have not been successful deserve to be notified as early as possible and preferably over the phone or in person rather than by email. However, its acknowledged that bulk or centralised selection for medical trainee recruitment, as practised by some hospitals and networks, will impact the committee's ability to provide timely and personalised feedback. See Advantages and Disadvantages of Centralised Recruitment and Case Study: Timely Feedback.

All candidates should be provided feedback on their performance in selection, with unsuccessful candidates usually the most interested in receiving feedback. For practical reasons, it can be difficult to give personalised feedback to each candidate. Generalised feedback may be easier to give to candidates, for instance 'there were certain distinctions that characterised successful over unsuccessful candidates' or 'there was a particular interview question that candidates struggled with'.

Understandably, trainees may be upset or dissatisfied with the selection process when they are unsuccessful. The provision of feedback, particularly when given promptly and includes advice on how candidates could improve their chances next time, can be well received and reduce any misgivings a candidate may have about the process. When taking into consideration the impact an unsuccessful interview can have on a trainee's trajectory, providing no feedback can viewed as disrespectful and likely to increase a trainee's pessimism towards the selection process.

Seek guidance from your local human resource department when in doubt about how to develop and support the process around providing feedback to unsuccessful candidates.

Local site selection vs centralised recruitment

Local site selection	Centralised recruitment
Advantages
Ensures the local site maintains control over the recruitment process and the terms and conditions of engagement Ensures selected candidates align with the individual needs of the site from a values perspective Enables the site to be more agile in recruitment Allows individual attention to candidates	Less work for the local site Consistent recruitment processes and decisions Lower costs due to more an efficient bulk process
Disadvantages
Inefficient and cumbersome when dealing with large numbers of candidates Can be resource intensive Can become a burden for the local site	Less opportunity for agile and customised solutions for tailored to local site Sourcing and maintaining interviewers with the capacity to interview large cohorts Some loss of personalised contact for candidates

Process review

As part of finalising the Selection Committee report, the Chair should gather feedback from other Selection Committee members, and possibly candidates, about how they felt the process went in order to uncover any opportunity there is to improve the selection process for next time.

Case study: Timely feedback

New South Wales operates the largest annual bulk medical recruitment system in Australia and New Zealand. Each year approximately 45,000 applicants apply for around 3,600 available positions³. This system can only operate effectively with a sound information technology structure and good processes in place. The medical recruitment system relies on those involved in managing the process interacting with the system on a timely basis.

Once selection processes have been determined, job offers are made in a staged manner so that job offers for particular groupings all go out at a designated time. For example, offers of positions for the 11 Adult Internal Medicine Basic Physician Training Networks go out on the same date in September to ensure that individual networks are not disadvantaged and trainees' decision making isn't pressured by receiving offers out of sync.

Unsuccessful candidates are usually put on an eligibility list in the case of first round offers being declined. This makes networks reluctant to let unsuccessful candidates know of their outcome in case they need to make a later offer. If a particular network doesn't have cause to use its eligibility list, candidates may never be informed until the system is reset prior to the next year of medical recruitment. NSW medical trainees have reported being notified as late as March, in the following year, that they were unsuccessful in their applications.

³ IECO Consulting. NSW Junior Medical Officer Recruitment Strategy Review: Phase One Report; 2015.