Adverse Impact Study

Validation

Two Story is committed to diligence and transparency. Our team monitors the reliability and validity of Habit Story with an on-going, statistical analysis of reliability and adverse impact study, analyzing results from individuals that belong to protected categories vs. the majority group categories.

The resources below unpack the results from our last analysis conducted in June 2023.

What is validity?

Validity refers to the accuracy of the assessment. To measure validity, Two Story analyzes criterion validity via concurrent and predictive validity.

Criterion validity explained: assessment results should be predictive or significantly correlated with evidence of current job performance. To establish concurrent validity, you would measure the test scores and the criterion variable simultaneously (e.g., on the same day).

Predictive validity explained: assessment results should be predictive or significantly correlated with evidence of future job performance To establish predictive validity, you would obtain the test scores at one point in time (e.g., today) and the criterion scores at another point in time (e.g., a month from now).

Conducting an internal review of validity (“local validation”) is the most robust approach for demonstrating validity. If you would like to learn more about Two Story can support your effort to establish criterion validity at your organization, please contact our professional services team at [e-mail].

What are my responsibilities as an employer incorporating psychometric assessments into the employee selection process? What is the purpose of the adverse impact study?

Based on their interpretation of the 1964 Civil Rights Act Title VII, the U.S. Equal Employment Opportunity Commission (EEOC) states that an employer may not use an employment or hiring practice that disproportionately impacts members of protected categories, including divisions by gender, race, disability or veteran status (“adverse impact”).

In 1978, five Federal agencies published the Uniform Guidelines on Employee Selection Procedures (UGESP) to provide a uniform set of principles for employee selection decision processes.

These guidelines state that employers should maintain records of employee selection procedures, including the candidate’s sex and ethnic group, and demonstrate that protected categories are hired at a rate that is at least 80% of the majority category (“the four-fifths rule”). In the event there is adverse impact, UGESP recommends analyzing the adverse impact of the assessment.

The adverse impact study is designed to augment and support an organization’s internal review of adverse impact within their employee selection process, analyzing differential effects among non-member and

What are the results of the Habit Story Adverse Impact Study?

In mathematical terms, we are analyzing the assessment results to determine whether significant differential effects are present when we consider how the various groups score on the assessments. To align with best practices, laid out by Jacob Cohen in “Statistical Power Analysis of the Behavioral Sciences”, we measure the overlap between two distributions.

Cohen's Score

The Cohen's d score is calculated with the following formula:

$d = \frac{{M2 - M1}}{{SD_{pooled}}}$

where:

• M1 is the mean of group 1,

• M2 is the mean of group 2,

• SD_{pooled} is the pooled standard deviation for the two groups.

The pooled standard deviation (SD_{pooled}) is calculated as follows:

$SD_{pooled} = \sqrt{\frac{{(n1-1)*SD1^2 + (n2-1)*SD2^2}}{{n1 + n2 - 2}}}$

where

• SD1 is the standard deviation of group 1,

• SD2 is the standard deviation of group 2,

• n1 is the size of group 1,

• n2 is the size of group 2.

This is the formula for an unbiased estimator of the effect size, assuming the two groups have similar variances and the data are normally distributed.

Habits

	Male	Female	Cohen Score
Attend	67.26	55.94	0.59
Enjoy	67.42	60.87	0.39
Resolve	66.13	58.99	0.36
Protect	74.60	69.57	0.29
Deny	37.66	41.88	0.26
Achieve	73.23	69.71	0.23
Respond	55.65	59.13	0.17
Neglect	24.35	27.39	0.16
Dismiss	39.84	37.39	0.15
React	54.84	57.25	0.14
Strain	37.34	39.71	0.11
Blame Shift	40.24	42.61	0.10
Anticipate	61.94	63.62	0.10
Resign	20.00	18.41	0.09
Sabotage	26.29	27.10	0.05
Presume	25.73	26.23	0.03

Response Pattern

	Male	Female	Cohen Score
Persistence	58.71	54.93	0.26
Openness	57.18	56.81	0.02
Decision Speed	73.15	65.80	0.42

Emotional State

	Male	Female	Cohen Score
Emotional Stability	34.27	27.01	0.39
Emotional Intensity	90.44	94.78	0.28
Drain	23.47	30.87	0.30

How can Two Story support my organization in compliance and validation?

The team at Two Story makes compliance and validation simple:

• For clients who elect to meet their compliance needs with “other validity studies” (described in section 60-3.7), the Habit Story documentation is updated regularly with the latest results of our on-going reliability and adverse impact results.

• For clients who elect to conduct an internal review of validity (described in section 60-3.5), the Two Story team developed a tool called Performance Story. The predictive analytics within Performance Story execute the predictive validity study on your behalf. It’s simple and easy to get started.

If you have any questions, we invite you to contact us at [e-mail].