Skip to content
Search
Generic filters
Exact matches only

Assessing Statistical Rigor of Employee Surveys | by Kamil Mysiak | Aug, 2020

The human resources industry relies heavily on a wide range of assessments to support its functions. In fact, to ensure unbiased and fair hiring practices the US department of labor maintains a set of guidelines (Uniform Guidelines) to aid HR professionals in their assessment development ventures.

Personality assessments are often used in selection batteries to determine cultural fit into a company. Cognitive ability (ie. IQ) tests are consistently found to be the best overall predictor of job performance across all types and levels of jobs (Schmidt & Hunter, 1998). Structured interviews are used extensively in hiring decisions as they help to remove bias by standardizing the question and scoring. Performance reviews use rigorous Likert assessments that ask managers and co-workers to rate employees of their performance (ie. behaviorally anchored rating scales). Employee engagement surveys assess the extent employees feel satisfaction, passion, effort, and commitment to their employer and job. Last but not least, employee exit surveys are often employed upon the termination of an employee in order to determine how the employee felt about a range of topics related to the organization.

This extensive use of employee assessment has given rise to a multi-billion dollar industry specializing in the development of a wide range of tests. Let’s focus our attention on employee attitude surveys as they form a very large segment of this industry. If one is to purchase a survey how can you be sure you are getting a quality product? Any reputable developer should supply you with not only a manual but also a validation report which outlines the steps taken to make sure the survey is actually measuring what it’s meant to measure.

In this article, I would like to examine an employee exit survey and determine the quality of the survey based on a selected few metrics. Therefore, when you are handed a validation report from a survey vendor you will know and understand the metrics needed to make an informed purchase.

Before we jump into our metrics and code, let’s take a few mins to review how a statistically rigorous survey is developed and validated.

  • The process begins with a question.
    For example: “Are my employees happy with their jobs and the company?”
  • How do we define “Happiness”? Before we can start writing survey questions we need to operationally define “happiness”. We scour the literature for employee happiness research. Undoubtedly, you will arrive at topics such as employee satisfaction, commitment, and engagement. You will read dozens of studies proposing unique models of employee satisfaction. Hackman & Oldham’s Job Characteristics model is often sited when developing employee satisfaction surveys for its comprehensiveness and statistical validity. Validity in the sense that many researchers have adopted this model in their research and/or assessments and found that it holds true.
  • The selected model will serve as the basis for the questions or items composing the survey. Depending on the length of the survey, we will write 3–10 items for each component (ie. skill variety, task identity, autonomy, etc.) of the model. We write multiple questions to confirm the sentiment of an employee on each model component. We will be focusing on quantitative items that require the employee to select a label on a continuous scale which corresponds with their internal attitude (ie. Likert-Scale).
  • Each question can be validated (content validation) by allowing prominent researchers in the area of study (ie. employee satisfaction) to scrutinize each question. Content validity is very often cited in legal proceedings when an employee is under litigation for improper hiring decisions. Other forms of validation include construct and criterion validity.
  • Upon completion of the first draft, a pilot test is conducted with a sample of employees closely resembling the full employee population. It is important to obtain a large and representative sample in order to be confident in the results.
  • Once the survey has been administered and the data collected, certain psychometric properties need to be examined, namely reliability and validity. Based on the psychometric results the survey is revised until optimal psychometrics can be achieved.

First and foremost, this fictional dataset and its results should be treated as such, fictional. Secondly, termination reason (ie. voluntary, involuntary, retirement, etc.) have been omitted from the loaded dataset as this article will focus exclusively on the psychometric properties on the Likert-scale questions.