What’s Wrong With Reference Checking? Pretty Much Everything (Parts 1 & 2)

Employment reference checking and background screening should win recognition as the weakest of all corporate HR processes. A validity meta-analysis study conducted by Aamodt & Williams in 2005 found that the corrected validity coefficient for reference recommendations and actual job performance was a staggeringly low .29. Despite the facts, 96% of organizations use reference checks as a screening and selection tool, according to a recent survey by SHRM.

Anyone who’s been in the profession for more than a minute or two knows deep down that references suck as indicators.

That’s why it is a task often assigned to an unknowledgeable coordinator, outsourced, or even turned into a multiple choice online form. While there is a handful of people who swear that they do it right, the vast majority do it simply because it has always been done. Maybe reference checking is/was more accurate in smaller communities where everyone knows everyone, but for many organizations today, new hires are strangers. As we enter an era where organizations have years of data and access to sophisticated tools to better determine what does and doesn’t work as an indicator, recruiting leaders should expect more scrutiny of this process by everyone.

Government agencies and commissions that oversee equal opportunity and business dealings are paying more attention and increasingly holding organizations accountable for the actions of third-party services commissioned to carry out screening activities. If you are in a leadership role or influence someone who is, it is your professional obligation to ensure that what your organization is currently doing and plans to continue doing isn’t stupid!

The following is a long list of things that routinely plague reference checking processes broken into four categories. If you look for these weaknesses in your current approach and address them, just maybe your validity coefficient will improve.

The Top Five Most Serious Reference-checking Problems

• Few provide detailed reference information — because former employers face legal action relating to defamation, many firms have a policy restricting reference conversations to factual information such as job title, dates of employment, and possibly salary history. Questions aimed at identifying if the applicant could be rehired by a former employer often go unanswered. The end result: some candidates may benefit from robust reference profiles, while others may not depending on the stance of their former employers.
• Reference information provided on the wrong person — in larger organizations it is not uncommon for supervisors and coworkers to confuse past employees with the same/similar names, so it is possible the individual being described isn’t the individual in question.
• Historical situations may not align with present conditions — references often focus on generic aspects of job performance specific to a different environment at a different time. Even if you are able to robustly identify how someone previously acted, you can only speculate that they will act the same way today in a completely different environment. Most firms have no hard data proving that in a fast-changing world where skills and knowledge become obsolete much faster, that past performance at another firm is an accurate predictor of future performance at your firm.
• Knowledge of the applicant is limited — for a variety of reasons, the individual providing reference information may have extremely limited knowledge about the individual being referenced. For example, they may be able to confirm job responsibilities, but know nothing of actual performance if they were not the person’s actual supervisor. In large organizations, few in HR would have personal knowledge of the individual’s work history or traits.
• Records are not reviewed — unfortunately, many references (especially telephone references) are provided by former employers who are relying 100% on their memory. You can be almost 100% certain that performance appraisals or attendance records will not be reviewed prior to the reference. Not relying on records and data almost guarantees an inaccurate reference.

Problems Related to a Changing Business Environment

• Bankruptcies, mergers, and layoffs make getting references difficult — because some employers simply no longer exist, it is often impossible to contact previous employers. Large-scale layoffs might also mean that even if you find the firm, the applicant’s supervisor may no longer be there. Layoffs and lean staffs in HR may also slow responsiveness to reference requests.
• Culture and language issues complicate — because different cultures and regions have unique ways of handling references, global consistency is difficult. In addition, language issues can make some reference checking extremely problematic.
• Many have bad credit — firms that used to supplement employment reference checking with credit checks now find that so many people have credit issues stemming from bank actions that relying on past standards will cause you to lose too many candidates.
• Identity theft issues — identity theft has grown beyond credit card fraud and has entered the employment arena. Unless you take and verify fingerprints, you may be checking the references of a real person who unfortunately is not actually the person you interviewed.
• Illegal or private information is viewed — supervisors and managers all too often provide illegal or personal information that may unfortunately end up being used in the hiring decision. Reference checking that includes searching Internet or social network sites also increases the likelihood of you obtaining information that should not be part of an employment reference check.
• Resume lies are more common — in a weakened economy where individuals are desperate to get a job, it is much more likely that they will stretch the truth on their resume, application, or during the reference process. This makes it even more important to get accurate reference information beyond the resume.

Problems Related to the Person Providing The Reference Information

• Bias produces slanted references — a supervisor or coworker with a vendetta may slant the reference so that it comes out overly negative. Supervisors may provide a negative reference on a current employee in order to improve their chances of keeping them.
• A positive slant — the majority of references are positive because people try not to say bad things. But when the reference is or was a close friend or colleague who wants to help you, the odds of finding negatives drops dramatically. Personal references are even more likely to be 100% positive because the individual selects only those individuals who they know will provide a positive reference. Managers may give a glowing reference in order to get rid of a problem employee.
• Self-provided references may mislead or be fraudulent — many firms allow the applicant to provide the names, e-mail addresses, and phone numbers of their references, and it should be obvious that self-selected references are more likely to give positive answers. In addition, unfortunately the names that applicants provide may not have been the person’s actual supervisor. In some cases, the number or e-mail address provided might not even belong to a real firm.
• No one is fired anymore — if you are using references to find out if someone has been previously fired from a job, you need to realize that because of legal issues, these days most individuals are allowed to resign. As a result, their employment record will not show a termination or the person will not divulge that fact.
• Leniency effect — when corporate employees do reference checking, there is a measurable tendency to be “lenient” and to downplay negative information. This leniency is caused in part because finding too much negative information will mean that the reference checker will be asked to delve deeper (more work) or that they must reopen the position search. People providing references can also be overly positive because they know that a bad reference may cost the applicant (who may also be a friend) their next job.
• Competitor firms will not cooperate — in a highly competitive business world, it can be increasingly difficult to get any form of information from a major product or talent competitor.

Administrative Problems With the Reference Checking Process

• No company data to prove they work — because many firms execute their own reference-checking process, it would be a mistake to rely on external research to prove the accuracy of the process. Unfortunately, I have never come across an organization that routinely gathers and analyzes data on the predictive accuracy of their reference process.
• Reliability is low — most corporate reference-checking processes (unlike vendor processes) are loosely designed and poorly controlled. Moreover, when reference checking is decentralized and many untrained individuals are conducting them, the execution (consistency and reliability of the process) slips even further.
• The person doing the checks are weak — many times the employees who are given the assignment are not trained in reference checking or knowledgeable about the organization or job specifics, making it difficult for them to dive deeply into relevant matters.
• References are often not completed until after the time of hire — due to the cost and time involved, some firms now complete reference checks after the individual is already on the job.
• Lawsuits are real — final hiring decisions are made based on reference-checking results, and in a down economy, more rejected candidates resort to legal remedies. Unfortunately, during legal discovery you may find that your references have a sudden memory loss and they will not repeat what they initially said, leaving you in legal jeopardy (which is why written references are superior, even though they are slower). In direct contrast, failing to check references may result in the possibility of negligent hiring claims.
• Reference checking is slow — in order to get documentation, it is important to use written references. However, written references take much longer, and a higher percentage of individuals are reluctant to participate in references when they must provide their signature. Reference checking is often not a high priority among recruiters or support staff even though slow reference checking may cause you to lose top candidates.
• The use of leading questions — a significant percentage of the reference questions that I encounter are binary questions pertaining to positive traits that almost always elicit positive answers. Requiring the individual to force rank or pick from a list of both positive and negative answers is a superior approach. Open-ended questions result in different information, making comparisons between candidates difficult.
• Poor documentation — telephone references often suffer from poor documentation and capture no signatures, which can be a problem should legal issues arise.
• Automated reference-checking limits depth — web-based reference-gathering systems shift the burden of getting references to the candidate, added work that may cause some candidates to drop out. In addition, to make such tools possible, the scope of the reference is limited and the ability to dive deeper into specific responses is virtually non-existent.
• The same process for all jobs — even though some jobs demand much more thorough reference checking (i.e. those who handle money, those with childcare responsibilities, etc.) most corporations use the same exact process for all jobs.
• A checklist is not used — rather than using a checklist of standard questions, many supervisors who check references “wing it” or make up their reference questions as they go, resulting in no consistency. If standard questions are used, they are often not weighted based on their relative importance in predicting future job performance.
• Mixed results make comparisons and conclusions difficult — the more references you check and the further back you go, you increase the likelihood that you will be faced with mixed references. Also, because an individual may receive several glowing and one extremely negative reference, making a decision based on consistent results may be difficult. When candidate slates include those recently out of school with little or no experience, managers are forced to make difficult comparisons between candidates with business experience and those who only have school-related references.
• All references shouldn’t receive equal weight — older references should be given a lower weight because the further back you go in a candidate’s work history, the less likely that the results will be valid predictors.
• No permission signature is obtained — if you are using resumes instead of application forms, you may be checking references without written authorization. If you inadvertently call their current employer, you may get them fired, which could lead to them taking action against you.
• It is not a continuous process — most references are done exclusively during the hiring process, but criminal, motor vehicle, or credit issues could occur well after the individual has been hired.
• Inclusion requires flexibility — if your recruiting or diversity goals include attracting individuals from lower-income groups, it is important to realize that some of these individuals may be more likely to have criminal, driving, credit, or job-jumping issues that can result in weaker overall reference scores.
• Contingent workers are treated differently — some organizations fail to require reference checks or allow vendors to use less stringent processes for temporary, part-time, and contract labor. As contingent workers become a larger percentage of a firm’s total labor, this inconsistency will need to end.

Question 1 — What are the most accurate indicators of past, current and future performance?

Finding accurate real world predictors of future performance is difficult but not impossible. Professional sports teams find that the best predictor of a new hire’s potential performance is their performance on the field in practice and preseason games — i.e., a work sample. Google looks at a multitude of factors that can be combined by an algorithm to successfully predict both future on-the-job performance and retention risk. The U.S. military and numerous firms in industries with extreme operational risk like airlines and chemical production facilities rely on sophisticated simulations to assess how a candidate would react in various situations.

There are literally hundreds of potential tools and approaches that can be used. Unfortunately, the vast majority of research on the subject is questionable at best.

Rarely are academic researchers trusted to manipulate the assessment processes of organizations hiring in significant volume such that the process could provide valid data, and few if any organizations execute consistently without extreme oversight  Commercially sponsored studies are flawed in that they are executed to prove a tool or approach works, not to test the extent to which it works.

That said, most studies generally conclude, and I tend to agree, that work samples are often the best predictors. Based on my observations and learnings, I routinely recommend the following.

Most Accurate Tools for Assessing Current Performance

1. Temporary hiring — The best way to identify top performers is to hire them into the job as temporary workers, contractors, consultants, or interns. Interns often have the highest success rate among college hires because you can rely on their internship track record as a predictor of their ability to perform in your environment.
2. Actual work cases — next to “playing in the game” is assigning them a case challenge based on the real problems that the candidate would face shortly upon accepting an offer. The problem should be a real current issue, so that even if you opt not to hire the candidate, you benefit from their insight into the issue. An alternative approach that has been used by firms like Raytheon is to give them a “broken” or flawed process and ask them to find the errors and source of problems.
3. A verbal simulation — this approach takes advantage of the traditional interview time period and asks candidates to walk you through how they would tackle an actual work problem. Firms using this approach generally use a case that has already been solved (so that the interviewer/assessor is familiar with all of the relevant information the candidate may need to inquire about and knows the likely outcome of each solution step introduced).
4. Samples of current work — in roles that produce physical products, one of the best ways to assess a candidate is to assess samples of their current work. If you want to find out if a chef is any good, you taste their food. In some cases candidates may not be able to provide existing work samples from their current employer due to confidentiality issues.

Most Accurate Tools for Assessing Future Performance

1. Ask them to forecast — if you need to assess candidates’ ability to perform in the future, consider asking them to predict future opportunities and problems your industry or firm will likely face and the skills and competencies that will be required to address them successfully. Consider whether their vision aligns with or is more/less robust than yours.
2. Future work case — give the candidate a problem or opportunity that is projected to occur two years out. Ask them to “walk you through the steps” that they would take and the skills that they would need to resolve it.
3. Demonstrate leading-edge learning — the most important competency in a fast-changing world is the ability to continuously learn and stay on the leading edge of knowledge. Even if candidates and employees are successful performers today, it is unlikely they will remain top performers unless they are continuous self-directed learners. You can assess whether they have that competency by asking them specific questions about what are the leading-edge “next practices” in their area, who are the benchmark firms, the key thought leaders, and what specifically do they do every day to remain on the leading edge of knowledge.

Question 2 — When do you recommend using reference and background checks?

As I said in the first part of this article, it’s okay to use one or more reference and background checking approaches provided that you understand why you are doing so and that you have designed a process that can be consistently executed to limit exposure to the vast majority of limitations found in the typical traditional process. If you have read my stuff for a while, you will know that I look at references as future candidates, so getting current candidates to identify their network through any process is a win, even if it wasn’t the intended focus of the process.

Two good reasons that would make doing references essential:

1. A correlation with quality of hire — any time you have supporting data showing that at your company there is a positive correlation between a candidate’s on-the-job performance and retention after one year (quality of hire) and a scale-based recommendation output by your reference/background checking process.
2. When there is a legal requirement — when you are hiring for roles that require due diligence to confirm proper licensure, certification, education, etc.

Note that 99.9% of all murderers, bank robbers, and major evildoers are first-time offenders, so the fact that they haven’t done anything in their past doesn’t prevent them from future missteps. If you really believe that criminal, driving, credit, licenses, etc. are valid indicators, you need to check these elements periodically post hire to ensure that your employees maintain their initial pristine status.

Four not-so-good but still-common reasons to do them:

1. Covering your butt — the most common reason to use reference checks are to shield yourself from potential verbal lashings (after a new hire turns out to be a real turkey) by being able to say “but I checked their references.” In addition, in the almost miniscule chance that you’ll be called in a negligent hiring suit, having checked the references will get you a handful of minor points from the judge.
2. When you want a particular candidate no matter what — if the recruiter or the manager is dead-set on a particular candidate, I understand the common practice of assigning an inexperienced and positive-thinking person (overly positive people will seek out positive attributes until they find them) to do the reference checking. You can almost guarantee that they will come back with the desired “I found no problems” answer.
3. When you need another excuse to fire a bad performer — if you are reluctant to fire a bad-performing new hire, I have seen many firms purposely redo reference and background checks in the hopes that they can find a lie or omission that could be used as an excuse to terminate.
4. You yourself need job security and are a bit selfish — if your job involves doing reference checks and recruiting in general, it’s in your best interest to continue executing a time-consuming, flawed, yet accepted process. Avoiding more effective candidate assessment methodologies will ensure that a greater percentage of new hires will fail and therefore require replacement recruiting; in other words, provide job security.

Question 3 — Are reference/background checks accurate assessments of past performance?

First of all we must establish that there is little to no consistency in what is considered a reference check. I cannot say that no reference check focuses on past performance, but based on my observation of practices in place at hundreds of organizations, the vast majority come nowhere close to remotely gathering intelligence on past performance. For example, many of the following elements often focused on during a reference/background check are not indicative of performance:

• Job title
• Dates of employment
• Credit scores
• Criminal history
• Personality traits
• Attainment of a degree
• Knowledge of buzzwords
• Information relating to job performance in another era (skills and knowledge grow obsolete too!)

All performance can be quantified, but rarely would a reference checker be granted access to real records of output/productivity from another employer. While candidates could request copies of past performance appraisals, we all know that most view that process as being worse than reference checking with regard to validity!

Question 4 — Are reference checks predictors of present or future performance?

If traditional reference checks are weak ways of assessing performance and especially past performance, do they provide any insight into present or future performance potential? Again, the answer is a sad no. As typically executed the traditional, haphazardly executed reference check is not a valid indicator for the following reasons:

• The past isn’t today or tomorrow — the world has and continues to change, rendering information and skills obsolete at a fastening pace. Behaviors and actions that produced spectacular results yesterday may fail miserably if the environment changes, which we know it does.
• Your culture is different — what makes one successful in one environment will not necessarily do so in another. Top performers can easily become average performers under a new manager, and vice versa. A lot of factors influence a candidate’s ability to perform.
• They may have been bridled — a candidate with a poor “performance related reference” may have everything needed to perform in your environment, but been unable to do so in a previous environment due to lack of professional freedom, proper tools, poor management, etc.
• The individual has changed — skill and motivation levels change over the years, so how someone previously performed in a different state of mind may not be indicative of how they would perform in a different state.

Question 5 — Are the current vendor offerings in the areas of reference and background checking worth examining?

Technologies and service offerings in the HR/talent management space are evolving at a rapid pace. Many emerging offerings are not bound by the traditions and ingrained practices of the corporate world, and bring valid arguments to the table that all professionals should be aware of. The hottest tools under development today, and others that have been around for a short time, focus on providing organizations with a more comprehensive look into a candidate’s professional network. Tools leveraging a 360-type approach increase the size of the assessor pool, and thereby increase the degree of structure afforded the process and the probability that more qualitative data will emerge. I can’t say without firm specific evaluation that any tool would produce valid results in your organization, but some criteria that I would recommend using when assessing top vendors include:

• Proof — do they have proof demonstrating the correlation between reference process outcome and on-the-job performance in real client organizations that can be referenced themselves?
• Integration — do they integrate the many different checks (i.e. credit, work history, personal references, criminal, educational, license, driving, Internet, etc.) into a single system?
• Metrics — do they have access to/knowledge of professional intelligence about the validity of component processes so that they can advise you using data on what practice improve/harm your candidate assessment efforts?
• Global — do they present a solution that can be used around the globe and that would not be limited in adoption by variances in culture or language? (The labor market is quickly becoming more global, so it is increasingly unrealistic to assume than a candidate’s references will come from the same region or cultural background as the candidate.)

Final thoughts

This series has introduced 30+ reasons to be wary of traditional reference checking processes, as well as a host of alternative approaches that could lead to a much more accurate assessment of a candidate’s past, current, and future performance. As the EEOC increases its scrutiny and focuses on this subject, everyone in the field of recruiting should learn to be cynical and test their efforts and solutions being adopted.

As seen on ERE Media on (PART 1) DECEMBER 13, 2010 and (PART 2) DECEMBER 20, 2010.