References are a crucial component of most selection processes. They serve to test the authenticity of a candidate's application and rule out any overstated and misrepresented achievements. References provide valuable third-party insight about the candidate that will help inform the decision of the Selection Committee.
Traditionally, reference checks occur after interviews have been completed and a preferred candidate has been identified. This approach is recommended as a candidate can reasonably expect to be interviewed if the content of their application satisfies the selection criteria.
Conducting reference checks to create a shortlist for interview is useful in certain circumstances, such as shortlisting for interview to avoid candidates incurring expensive travel and time costs. In this instance, you should make candidates aware prior to carrying out of the reference check and the reasons for it.
The effectiveness and usefulness of references can vary significantly. In many countries, including Australia and New Zealand, recruitment processes for medical training positions are generally conducted altogether on an annual basis. The number of positions available and number of applicants received in most settings demands a level of automation to ensure a smooth process. As a result, automated referee reports have become commonplace in many online recruitment systems.
| Factors that affect a referee report's usefulness
|References taken verbally
|References taken using selection criteria
|Generic reference template used
|Verbal references taken using behavioural interview techniques
|Informal verbal reference
|Criteria for referees
|Allowing candidates to choose their own referees
|Structured Letters of Reference
|Unstructured Letters of Reference
|References taken independently by a trained expert
|References taken by untrained staff
Automated reference reports
Candidates often apply for multiple positions and certain referees may be popular. Automated referee reports can help to reduce the burden on referees and ensure completion of the referee checks, which is generally required as a compliance measure.
Automated references take the format of a Structured Letter of Reference (SLOR). North American studies conducted into the use of SLORs have shown they are easily implemented, quicker to assess, appear to provide better match selection outcomes and are easily sustained over time1. However, some caution needs to be exercised as its been found that less experienced referees are more inclined to give a favourable reference than more experienced referees2.
An automated reference systems normally deploy a generic reference template, which is designed to suit a range of medical trainee positions. It is often not possible to insert questions that are specific to the role. This may be problematic for a Selection Committee, which for example is considering applications for Basic Training in Paediatrics & Child Health, that wants to know how well the candidate communicates with small children. The committee may not be satisfied with an answer to a question that is only about candidate communication with patients and family.
When verbal references are taken, they should generally conducted by the same person to ensure consistency. In medical selection, this task normally falls to the Chair of the Selection Committee. However, it is possible and sometimes better to have an expert to perform this role independently from the Selection Committee. Many hospital and health service human resource departments have trained personnel in the various roles of recruitment who can take on this function.
Verbal references should be recorded in written form or transcribed. The referee should be informed that there will be a record of their reference, which can be requested by the candidate.
Despite such caveats, verbal referee reports generally yield better information about the strengths and weaknesses of a candidate and, in some cases, may be the only red flag or warning sign that a candidate might not be suitable for the role.
Best practice is to pose questions to referees that are related to the selection criteria, in the same way that interview questions are to be related to the selection criteria. If the interview questions have already been established, it may be possible to use these with the referee as well, particularly if they are of a behavioural nature.
"Can you give me an example of where the candidate identified a problem in the hospital and used their initiative to work with others to bring about a solution?"
This is a suitable question for a referee to ask in order to gain information on a selection criterion that requires evidence of the ability to work in a team and solve problems. The question can also address a selection criterion about demonstrating the ability to be innovative through teamwork and collaboration.
Generally, more questions can be covered in a verbal reference of 15 minutes than a candidate interview of 30 minutes, providing an opportunity to ask more questions of the referee.
Some of the questions can be of high value. It is recommended that these type of questions are asked towards the end of the conversation once a rapport and a referee's willingness to assist is established.
- "If you were in the position to offer the candidate another job, would you?"
- "Are there any reservations you have about the candidate?"
- "If there was a new patient referral in the emergency department, at what point would you be happy for the candidate to do the initial assessment and commence management?" (Not at all? While you were watching them? While you were elsewhere in the hospital? While you were at home?)
- "Would you be happy for the candidate to treat a member of your family?"
- "Hypothetically, the candidate has secured the position. What advice would you give to their Director of Physician Education about the candidate's strengths and weaknesses?"
1 Fraser J, Aguayo P, St Peter S, Ostlie D, Holcomb G, Andrews W et al. Analysis of the pediatric surgery match: factors predicting outcome. Pediatric Surgery International. 2011;27(11):1239-1244. Prager J, Perkins J, McFann K, Myer C, Pensak M, Chan K. Standardized letter of recommendation for pediatric fellowship selection. The Laryngoscope. 2011;122(2):415-424.
2 Love J, DeIorio N, Ronan-Bentle S, Howell J, Doty C, Lane D et al. Characterization of the Council of Emergency Medicine Residency Directors' Standardized Letter of Recommendation in 2011-2012. Academic Emergency Medicine. 2013;20(9):926-932.