Law enforcement organizations are facing an outcry to reform the way they operate. Many agencies have a vision and set of values that could address the calls for reform, but the desired change has not happened. This inability to operate in a way that reflects the values of the agency contributes to tragic events such as the killing of George Floyd and similar incidents that alienate the public and make policing even more challenging.

To enact real change, agencies must hire and promote individuals who have the knowledge, skills, abilities, and other characteristics (KSAOs) needed to execute the values of the department. Thus, developing and administering high-quality selection procedures that identify those KSAOs are essential.

Current circumstances highlight the need to address this issue: Law enforcement agencies need to employ in-house testing professionals or hire qualified testing vendors to assist them in developing and implementing a high-quality selection process. Agencies should consider the vendor as a partner to the process where the agency and the vendor work together to design and implement an assessment program that best fits the needs of the agency. However, selecting a vendor who can serve as an effective partner can be a challenge, particularly if no one on staff is an expert on testing.

What to Expect from a Testing Partner or Vendor

Thorough, Customized Job Analysis

A thorough and updated job analysis is the foundation of any test development process. A job description is not sufficient to serve this purpose. A professional job analysis has a broad and deep scope, as it entails collecting specific evaluations of job tasks and the KSAOs needed to perform those tasks.¹ Without a valid job analysis that focuses on the relevant issues in an organization, the resulting assessments will be ineffective, seen as unfair, and nearly impossible to defend if challenged.

Vendors should be able to provide answers to the following questions about job analysis:

• What steps will you take to ensure that the job analysis captures the values we expect our officers or leaders to embody?
• How will you ensure that the job analysis captures the KSAOs needed to perform in the most challenging situations?
• How will you ensure that the job analysis addresses changes in the job due to changes in technology, tactics, and community expectations?
• How do you ensure that you are adhering to professional and legal guidelines?

Vendors should be able to explain how they will conduct a job analysis that not only describes the job, but also determines how organizational values can be manifested in the position. Thus, if an organization is attempting to change or meet new needs, a new job analysis is essential, and a task and KSAO listing used elsewhere by the vendor is insufficient to serve as the job analysis. One way to ensure that the job analysis taps important job content is to examine on-the job incidents where officer or leader behavior either embodied or conflicted with the values of the organization. Analyses of these incidents can help identify key KSAOs that otherwise would not have been identified. This method, called the critical incident technique, was first used to identify the source of fatal pilot errors during World War II and ever since then has been commonly used to conduct job analysis and improve performance.²

 

Assessment of the “Whole Person”

Professional and legal guidelines make it clear that tests should assess the content “constituting most of the job,” and specifically state that reliance on “a single selection instrument… related to only one [aspect] of job performance will also be subject to close review.”³ Thus, it is unwise to rely on one tool, such as a multiple-choice knowledge exam to identify leaders or a reading comprehension test to identify officers. A high score on a knowledge exam does not clearly reveal if the candidate would be a good officer or a supervisor, let alone whether the individual could successfully execute the values of the organization.

To determine whether vendors will provide or develop assessments of the whole person, an agency could ask the following questions:

• How would you describe your best practice assessment process and why?
• Describe how a multiple-choice knowledge exam can be an efficient method of selection and yet may be an ineffective method to use for selection?
• How do your proposed assessments tap into the abilities and skills needed to embody the values of the organization?
• What steps will you take to incorporate the most important or common scenarios that officers or leaders face on the job?

Vendors should be able to explain how they use multiple assessments to focus on a whole-person approach. For entry-level assessment, this might entail using a battery of assessments that engage the candidate in solving problems, communicating with others, handling challenging interpersonal situations, and correctly implementing rules and procedures. Assessments that demonstrate or simulate the job have the added benefit of ensuring that candidates fully understand the job. For selecting leaders, tools such as video-based simulations, situational judgment tests, and structured interviews are best practices. The evaluation of competencies such as communication and responsiveness, self-awareness, community engagement, problem-solving, and leadership, among others, are possible through these types of techniques.

One excellent way to accomplish whole-person assessments is to use videos or animations to present scenarios that the candidate must respond to. Vendors should be able to develop and present scenarios and pair them with relevant questions and scoring to measure the KSAOs relevant to the job. For example, image 1 was part of a video-based assessment (using green screen technology) where candidates were required to respond verbally regarding how they would react in this situation.

A Highly Structured Assessment Process

Structure, in the world of assessment development and administration, involves targeting the KSAOs identified in the job analysis with clearly defined procedures, applying those procedures consistently, and applying the same scoring process for all candidates. For interviews and other assessments involving ratings, this means ensuring that all candidates receive the same exercises and questions under the same conditions and are evaluated using anchored rating scales that are consistent across candidates.⁴

Research shows that greater structure leads to improved validity and fairness. For example, meta-analytic studies (i.e., aggregations of the results of many studies) demonstrate that interviews with more structure are more effective at predicting job performance and are viewed as more fair as they result in lower differences in interview performance between white interviewees and their black and Latinx counterparts.⁵ These findings suggest that structure, regardless of the assessment type, leads to more effective decision-making.

To understand vendor capabilities and intentions regarding the use of structure in the assessment procedure, the agency might want to ask vendors to provide answers to the following questions:

• What degree of structure do you incorporate into your assessments and why?
• What structured components do your assessments include?

Vendors should describe their experience on previous projects in providing a consistent, repeatable structure in all assessments, including targeting the same KSAOs identified in the job analysis, using consistent instructions, time limits, rating scales, scoring procedures, and decision-making processes. Vendors should also be able to explain their rationale for building in structure and how it contributes to the validity and quality of the outcomes. Some vendors may claim that their approach does not require much structure or suggest that less structure is better to allow decision makers to use a “gut feeling” to determine selections. This is a legally perilous approach to take and can undermine effective, high-quality decision-making, as well as perceptions of fairness and impartiality.

Clear Evidence of Validity

In the context of most law enforcement agency employment testing, test validity is the degree to which accumulated evidence and theory support the way scores are used in the selection procedure.⁶ A valid test should not only lead to higher job performance, it should also be more effective at reinforcing the values of the organization. Test validity evidence will be needed if the selection process is challenged under Title VII of the Civil Rights Act or any other employment law.⁷

One way to demonstrate validity is to build a customized assessment from the ground up that specifically targets the job. Custom assessments have a simple path to validity evidence because, by definition, they target the KSAOs and the relevant tasks, situations, and challenges encountered on the job. This content-oriented approach makes the process of defending the assessments straightforward.

A less expensive alternative to custom assessments is the off-the shelf (OTS) test. OTS law-enforcement assessments are ready-made instruments that may have been built based on the content of jobs similar to the ones in the purchasing organization, saving users the time and expense of developing a custom assessment. OTS assessments must meet the same standards for validity evidence and the guidelines are clear that “under no circumstances will the general reputation of a test or other selection procedures, its author or its publisher, or casual reports of its validity be accepted in lieu of evidence of validity.”⁸

If cost and time considerations restrict the agency to an OTS, be sure to ask the following questions to ensure that the test is appropriate:

• Can we obtain a copy of the technical report that documents the validation process?
• What does your test measure and how do you ensure that the test is valid for predicting job performance?
• What steps have you taken to minimize any adverse impact of your test?

Vendors should provide information that supports the development and validation of their test, including the KSAOs measured, the reliability and variance of the assessments, and statistics on the performance of different subgroups. The goal should be to ensure that the test is measuring the broad range of characteristics, skills, and competencies that are needed to perform the job in accordance with the values of the organization and that it does so reliably. Finally, steps taken to minimize adverse impacts should be clearly outlined and understandable — so the agency can make the appropriate decisions. It is critical to ensure that the test selected meets the values of the agency beyond success in the academy.

Steps to Ensure Fairness and Impartiality

Fairness and impartiality are important values in many law enforcement agencies. They are pillars for procedural justice, which, “when embraced, promotes positive organizational change and bolsters better relationships.”⁹ Fairness in employment testing is important because it affects the composition of the workforce and influences the degree to which officers represent the community that they serve. Simple steps to advance fairness can make a big difference. For example, one study found that when judges assessed musical audition performances through a blind process (i.e., where the musician was hidden from view) more females were selected than when the musicians auditioned in front of the judges.¹⁰ Another study demonstrated that a simple change in response format (open-ended instead of multiple choice questions) led to smaller differences in assessment results for individuals from Western cultures relative to their counterparts from non-Western cultures.¹¹

To assist in evaluating vendor capabilities with regard to ensuring fairness and impartiality, the agency might ask a vendor to describe the specific steps the vendor has taken on previous projects to reduce or minimize adverse impacts.

Vendors should describe their experience using whole-person assessments, removing visual cues of candidate identity, using an open-ended response format (possibly including audio rather than written responses), ensuring that the reading level on the test does not exceed that needed for the job, and employing a sensitivity review process to ensure that the test content is free of bias. Further, for processes that require evaluators, the agency should expect to hear about the steps taken in the training process to reinforce fairness and impartiality.

Helping Vendors Help You

To succeed in selecting a vendor that can help address the values of the organization, an agency should engage in the steps outlined in Table 1. It is important to work closely with the agency’s procurement professional or contracting specialist throughout the acquisition to ensure compliance with acquisition regulations for the jurisdiction.

Table 1. Steps to Procuring Employee Testing Services.
Step	Tips
Define needs	Use the expectations identified to assist in describing your needs.
Develop a clear understanding of the market.	Conduct market research through a request for information process. This might involve providing the statement of work to vendors and asking a few standard questions: • What features do you recommend requesting? • What are some potential risks and pitfalls with developing and administering a selection procedure and how do you mitigate them? The agency can use the responses to inform the request for proposal (RFP).
Ensure sufficient funding.	As part of the market research, ask vendors to indicate their price range for different solutions and services that fit the agency’s needs. Be specific when describing these services.
Ensure sufficient time for development.	Plan. Know that most custom development projects take at least six months depending on whether a recent and thorough job analysis exists.
Be sure the best vendors see it.	Publish the opportunity online in a way that vendors can find it. The contracting office can probably put it on the jurisdiction’s procurement site, and then it will be picked up by aggregation sites like GovWin or BidNet.
Lay out a project map.	It helps to organize the statement of work in terms of the following tasks: • Conduct a job analysis • Develop a test plan • Develop draft tests • Pilot test • Document validity • Administer exam • Report scores
Share information on factors that affect the level of effort and price.	Be clear in the procurement process regarding issues such as the number of persons who need to be assessed, and whether evaluators are provided by the agency.
Ask for a separate estimate of variable costs.	Ask vendors to provide a separate line item for other direct costs such assessor travel and expenses. Do not embed expenses into an overall price. This does not allow for accurate comparisons across proposals.
Make it easy to respond.	To the extent possible, minimize the paperwork required to respond.
Bundle service requests.	Consider combining entry-level testing, and all promotional testing into one procurement or combine the procurement with other agencies within the jurisdiction (e.g., fire and police).

It is time to change the mindset of selection efforts to focus on quality processes that will help organizations select the people best prepared to meet the needs of the communities they serve. Knowing what to expect from a test vendor and effectively selecting a high-quality vendor is a critical first step for agencies seeking to accomplish this change. 🛡

 

Notes:

¹ Paul Sackett et al., Principles for the Validation and Use of Personnel Selection Procedures (Bowling Green, OH: Society for Industrial and Organizational Psychology 2018); Labor, 29 C.F.R. §1607— Uniform Guidelines on Employee Selection Procedures (1978).

² John C. Flanagan, “The Critical Incident Technique,” Psychological Bulletin 51, no. 4 (July 1954): 327–358.

³ 29 C.F.R. §1607.

⁴ Julia Levashina et al., “The Structured Employment Interview: Narrative and Quantitative Review of the Research Literature,” Personnel Psychology 67, no.1 (Spring 2014): 241–293.

⁵ Leveshina et al, “The Structured Employment Interview”; Michael A. McDaniel et al., “The Validity of Employment Interviews: A Comprehensive Review and Meta-Analysis,” Journal of Applied Psychology 79, no. 4 (August 1994): 599–616; Allen I. Huffcutt and Philip L. Roth, “Racial Group Differences in Employment Interview Evaluations,” Journal of Applied Psychology 83, no. 2 (April 1998): 179–189.

⁶ Sackett et al., Principles for the Validation and Use of Personnel Selection Procedures.

⁷ Civil Rights Act of 1991, Pub. L. 102–166.

⁸ 29 C.F.R. §1607.

⁹ U.S. Department of Justice, Office of Community Oriented Policing Services, “Procedural Justice.”

¹⁰ Claudia Goldin and Cecilia Rouse, Orchestrating Impartiality: The Impact of “Blind” Auditions on Female Musicians, NBER Working Paper Series (Cambridge, MA: National Bureau of Economic Research, 1997).

¹¹ Filip Lievens et al., “Constructed Response Formats and Their Effects on Minority-Majority Differences and Validity,” Journal of Applied Psychology 104, no. 5 (May 2019): 715–726.

---

Please cite as

Lance Anderson, Shane Pittman, and Michael McLenagan, “Ensuring That Hiring and Promotional Procedures Support Organizational Values,” Police Chief Online, July 15, 2020.