Testing For Selection: A Cautionary Tale

When helping clients set up their recruitment and selection process, one question will always come up. What do you recommend as a psychometric test? They are often taken aback when I say that I don’t recommend any of the specific psychometric tools. You need to select a tool that relates to the specific attributes for which you are hiring. Finally, I add, please only use it sparingly on the final two or three individuals after they have been through a structured behavioural interview. This cuts against the grain of the many who are engaged in the hiring community. So, let me explain.

First, full disclosure. My background is education. As I studied and practiced in schools for several years I learned that testing is often misleading. How could a star athlete be at academic risk when he or she could successfully master an encyclopedia of playbooks? Testing was always the means to segregate the smart (dare I say ‘gifted’) from the rest of the herd. The result was a movement in the 1960s and 70s for homogeneous classrooms. My own student experience said that homogeneous groupings just don’t work. Students, given the correct support and learning environment, can learn anywhere. Homogeneous classrooms are not the way of building school community and might I add in today’s terminology not the way of building diversity. When intelligence and other tests were used to separate out who belongs in which classroom many highly capable students were lost in the system. Why Test?

The rationale for using psychometric test is to find the best fit to the company. It has become a common practice for many private and public employers with the goal of finding the right fit for their organization and reducing employee turnover and improve time to productivity. But academic research has shown that while the claims are that behavioural assessment tools can add great value to an organization during the hiring process, their low validity (Gilliland, 1995, pg. 48[1]) and unsupported claims of return on investment (supported by third party academic research), call into question their use in the hiring process.

Don’t get me wrong there is a place for such assessments, just not in the hiring or promotional process. They can be used for employee self-awareness, teambuilding, opening awareness to different work and communication styles and perhaps leadership development and should not be used in candidate selection or promotion.

“There is a science to predictive analytics, but the outcomes are not guaranteed” (Roberts, 2014). One of the issues is that the source of the validation is often the company selling the selection assessment tool. That presents an inherit bias. They are selling you something that ultimately benefits them. The sample size and validation results are based on what they say proves the assessment will work. Once you invest significant time and dollars in the process you are predisposed to see the desired results. The elements missing from these assessments are often the individual’s cognitive skills, effective skills, values or experience. What are the concerns:

1. Behavioural assessments are self-reported (completed), so a candidate’s answers can be faked or influenced by events in the day prior to the completion of the instrument[2]. All of which will affect the results. Hiring managers should understand the low validity of using behavioural assessments as the foundation for employee selection, and recognize that there are multiple steps and tools to be used in the hiring process.

2. Why do Human Resource professionals and academics disagree on the effectiveness of behavioural assessment tools? The reason could be the lack of time the HR professional has to read through long and often highly technical and statistical research. As a result, they attend conferences and read professional HR journals where the author of the test is speaking passionately of the results along with a client of the consultant who has recently used the process. But, usually they present only recent case studies; not longitudinal case studies to demonstrate sustained effectiveness. As well, these are anecdotal results that prove nothing.

3. Those who sell the assessment tools will always say they have been designed to prevent people from ‘faking’ the answers and hence manipulating the results. At the 2004 Society for Industrial and Organizational Psychology conference in Chicago, a panel discussion was held to discuss the issue of “faking in personality testing”. This discussion was published as an article in which several issues associated with the use of self-report personality tests as part of the personnel selection process were highlighted. “Faking on self-report personality tests should be expected, and it probably cannot be avoided, although there is some disagreement among the authors on the extent to which faking is problematic” (Morgenson, et. al., 2007, page 720.[3]).

4. One assessment tool that is common because of its ease of completion and purported ‘validity’ is the DiSc. It has been around as a psychometric instrument since the 1950’s. The DiSc Model of Behaviour was first proposed by William Moulton Marston, a psychologist with a Ph.D. from Harvard. His 1928 book, Emotions of Normal People, explains his theory on how normal human emotions lead to behavioural differences among groups of people and how a person’s behaviour might change over time. He never intended to turn the content into an assessment.

In the 1950s a series of industrial psychologists separately used his theory as the basis of a tool for employee selection. Today there are many tests that exist under the name DISc, and Extended DiSc, marketed by various consulting companies for use by human resources departments, each with their own unique twist. There has no new research using a DISc model in academic psychology since the first work on DISC in the 1950s. Today’s issue with using DiSc is there are many versions of DiSc and not all have the same questions, only the same shared a common source.

If you choose to use the DISc please ensure it is used in combination with other tools that test a person’s knowledge and skills in a certain area (a work sample), as well as conducting face-to-face structured behavioural interviews. In short, like the Myers Briggs Type Indicator (MBTI), the DISc should not be used as part of a selection process.

5. The usage of an assessment tool would mean you must believe in a universalistic approach to talent. This concept is endorsed by SHRM. Meaning that there is a set of specific and well defined interrelated practices that characterize achieving competitive success through talent management. Under the universalistic approach there is a well-defined set of behaviours that contribute to the achievement of higher organizational performance, regardless of the organization’s strategy or culture (values). This would mean that whatever assessment tool you are using as your universal foundation it matches the exact behaviours of your culture and values. It means that regardless of organization if you pass this assessment you should be successful in all organizations. Yet, we know that is not going to happen. There is no universal set of success behaviours and anyone who tells you otherwise is selling you something, likely their assessment tool.

6. The inherent issue is that test elements are based on the bias of the author of the test. They are biased on their perception as to the correct attributes for success as an employee to fit and be engaged, productive, and retained. However, if no two organizations are the same than unless the instrument is developed, validated and revalidated within your own organization the usefulness is questionable.

7. Recently I was working in Saudi Arabia and they are using one of the instruments that is very popular in Western nations. It was developed and ‘validated’ in the West. The test was being administered, along with an interview, during the time I was present in Saudi Arabia. It was the same time I was teaching structured behavioural interviewing. A few employees approached me after the assessment tool completion and questioned the validity of even using the test in their location and in their company. The test had several questions regarding the drinking of alcohol in social settings. I wondered out loud if you scored these items differently how would it impact the results of someone who answered the test saying they drank socially and engaged in the social situations described.

It is important the test captures the specific desired attributes or traits that define success in the culture of the firm but is also valid to the local beliefs and customs of the geography.

8. The language was another point that was raised. None of the employees taking the test are native English speakers. While every employee is fluent in English and dare, I say probably use the English language more correctly than many native speakers, English is not their language of first choice. Speaking two languages myself I realize that the translation of a statement from one language to another is based on the level of knowledge and nuance of the translator. It is at best a translation frequently close but not the exact to the original. I would caution using these tests that are translated, with the best of intents, but a full third-party validation study has not been conducted. I simply wish to raise the warning flag on the use of the test in a language other than the original wording of the instrument.

9. Perhaps coming from the cynic in me I have become concerned with the usage of the Internet for the completion of assessments. When the results are the score, the individual is called in for an interview. Only to discover upon, the in-person interview the level of language proficiency does not seem to indicate they would have comprehended the meaning of all the words on the instrument. When asking the candidate for an explanation only to learn that someone else, because of language issues, completed the test on their behalf.

While there are many other issues or concerns about using a behavioural assessment as a means of screening in or out the instruments do have a place in the assessment process. It is my perspective that such a tool is a good instrument if it is aligned specifically to the attributes that define success in your company. That before using the instrument you have ‘tested’ several of your current successful employees to get a company specific indicator of how to use the results internally. It is also important to note the test instrument should come after the structured technical and behavioural interview as not to predispose the hiring manager’s perspective.

Testing might be the quick way and many HR people are under pressure for knee jerk prompt resourcing of talent. But that only leads to problems because HR will be blamed for every bad hire. Slow down, hire correctly, in the long run turnover will be reduced, productivity and retention increased, and the need for hiring replacements for bad hires slowed down. If you buy in to the idea that the assessment instrument will speed up the selection process you are buying into the ‘fast food’ approach to talent management. Fast food might be quick and feel good even comfort food but you are also buying into the hardening of your arteries. Too many quick and easy processes could, like fast food, be deadly. Finally: Test the Test

A successful instrument of assessment of behaviours should predict job-person-match. This gives the company “criteria for correctness” that it can use to measure as its pre-employment screening analyses to predict future on-the-job success. It’s best to think of the process of validation as an internal research endeavor. The hypothesis is that the selection psychometric assessment will predict successful job performance, and with that hypothesis being subject to ongoing empirical internal validation with the potential for disconfirmation. If an assessment doesn’t predict performance over time, stop using it. If the assessment and the structured behavioural interview align and there is not new revelation from the instrument, stop using it. If you buy a manufacturing tool that doesn’t prove to be as useful, over time, it is replaced. Apply the same logic to your testing instrument in talent management. How to integrate assessment tools

1. Review the candidate’s resume/application
2. Conduct a gap analysis to the resume to the skills, knowledge, behaviours, values, education and experiences required to successfully start in the role
3. Short list only those whom meet the criteria or keep looking
4. Conduct a phone interview to clarify any information and ask at least one behavioural question related to fit to the company values.
5. Conduct in-person structured technical skills and behavioural panel interview
6. If the candidate passes these processes facilitate a behavioural assessment tool for only the final two candidates
7. Review the assessment tool results against what you have already learned about the candidate to discover alignment or red-flags
8. If there are any discrepancies conduct a second interview with the candidate focusing on the gaps
9. Members of the panel meet to reach consensus on a final decision based on the information collected but not based on the test alone
10. Validate the candidate’s certifications and credentials by contacting the issuing institution directly. (Never accept them directly from the candidate.)
11. If required, process a criminal background check and reference check
12. Make a job offer

Final Thoughts on Tests and Testing

Often, I have found that HR professionals have misconceptions about both the value of formal assessment tools and rely on them to short list a longer list of applicants. I would suggest this to be the process even if the organization has tailored their assessments to their specific needs. This is because the amount of customization will vary, depending on the needs and budget of an organization. The question would be how many of their current successful employees have been ‘tested’ and an internal norm developed prior to using the customized instrument?

My point of view is that a lot of these standardized testing procedures are being used as weeding out devices and many are without any alignment to the specific behaviours and values of the organization using them. Generally, I have an aversion to any types of standardized tests promoting a panacea for selection of new employees or promotion of existing employees because that doesn’t exist. But we are in a hurry and we are always looking for the magic test. We are under pressure to meet the demands of the line-hiring manager to put a bum in a seat.

On a personal note and ensuring full disclosure. Per standardized testing back in grade 8 I should have gone onto a vocational high school as, according to testing and my grade 8 teachers, I would never succeed in an academic pre-college high school. Because the tests said I would not be successful in a college prep – academic high school program. Truth be told, I did complete a Bachelor of Arts followed by two master degrees, a high school teaching license, a doctorate in Humanistic and Behavioural Studies and written two books. Stereotyping people or pigeon holing them based on test scores is a disservice to you and the individual.

[1] Gilliland, Elaine (1995). Selection Assessment Methods. SHRM Foundation

[2] Research shows that personality, circadian rhythms, and stimulants interact to affect performance. From “Ace the Assessment” Tomas Chamorro-Premuzic • From the Harvard Business Review July–August 2015

[3] Morgenson, Frederick, et. al. (2007). Reconsidering the Use of Personality Tests in Personnel Selection Contexts. Blackwell Publishing, Personnel Psychology.

______________________________________________________________________

David is a globally recognized thought leader in the areas of talent management and corporate culture. If you want to learn more on the structured behavioral interview process a, please read The Talent Edge: A Behavioral Approach to Hiring, Developing, and Keeping Top Performers. If your company is interested in a workshop on how to conducted Structured Behavioral Interview, how to write company specific and authentic behavioural statements, and wish to learn more about an anchored rating scale for more objective and accurate scoring of responses; please contact David directly at david@sagltd.com.

You can read more on David’s perspective on how strong corporate cultures drives passion, productivity, high engagement and pride in his book on values, culture and leadership: Inside the Box.

#StrategicHR #BehaviouralInterviewing