• First, the Part 1 blog was about a technical (math) clarification for calculating validity differences in selection systems. Although Schmidt & Hunter (1998) correctly calculated utility differences between two selection systems as the difference in multiple R values (ignoring the cost of testing, etc.), they did not calculate validity differences correctly. Validity differences have to be calculated as the difference between R² values. You can then report that as ∆R², or take the square root and report ∆R, or do whatever other transformation you would like to make—so long as it is based on this ∆R² value. My post did not intend to take a stand on what you “should” report; instead, the goal was very modest, to focus solely on how to calculate a validity difference correctly.
  
  Let me reinforce the point with a specific numerical example: Say that a selection system has a multiple R of .20, and after adding more predictors, the multiple R goes up to .30. The difference in utility is a function of validity and is indeed simply R = .30 - .20 = .10 (S&H had this correct). But .10 is not the validity difference, where instead you have to subtract the respective R² values: ∆R² = .09 - .04 = .05. If you then want to report the R metric instead, you would take the square root of .05 to get R = .22 (…and not .10).
  
  This isn’t new stuff, by the way! (a) In hierarchical linear regression, if you want to report ∆R, you cannot just subtract multiple Rs from each stage of the regression; you would have to calculate ∆R² and then take the square root. (b) Similarly, if you want the average of two standard deviations, you first must square them (to get variances), then average them, and then take the square root.

• Second, given that the Part 1 blog was so narrow and “mathy” in scope, it was decidedly not a commentary on two recent and related papers by Sackett et al. (2022, 2023)1 that update the data and the statistical approach of S&H. Those two papers generated numerous responses2, and two Sackett et al. replies to those responses3. As far as I can tell, none of this recent work performs the calculation the Part 1 blog focused on. However, given the timing of these Sackett et al. papers, I probably should have stated that I was not focused on them—so I’ll do that now.

OK, whew! Now in my next post, I will turn to Part 2, “More Adventures in Meta-Analysis,” which will be much more expansive (and less “mathy”) than Part 1 was. Yes, here I will refer to the Sackett et al. papers and responses, but in the context of a broader meta-analysis idea. Please stay tuned!

The general conclusion of this article is hard to dispute—so long as the employment tests in question are conceptually and psychometrically well supported. However, I will put myself at variance with S&H in two ways. The first way is narrow and statistical; it is covered here (Part 1). The second way is much broader and will be covered in a forthcoming blog post (Part 2).

At Variance with the Variance

Appealing to fair use2, let’s take a look at the often-referred-to Table 1 (S&H, p. 266):

As my wonderful PowerPoint® flourishes indicate above, the second and third columns of numbers would be correct if these referred to utility — not validity. When estimating utility (e.g., predicting the dollar value of selecting employees), then word on the street since 1949 is that you should directly subtract the validity coefficients (correlations, multiple Rs) of each selection system from one another (Brogden, 1949, p. 183)3:

The left side above is the change in the value of a selection system M, when moving to using selection system a, from selection system b. On the right side, the yellow highlight is the difference in the systems’ respective validities (or multiple Rs). Thus, to obtain R_d(utility) the difference in utility that is due to validity, simply calculate:

But in the context of validity, some different math is needed, contrary to the footnotes in Tables 1 and 2 of S&H, both of which state:

This is incorrect. If you want to express an increase in the validity of a selection system in terms of a multiple R, one cannot simply take the difference between the multiple Rs for each system, as we (correctly) did above for utility estimation. Instead, the difference in R²s for each system (the variance components) need to be subtracted from one another. You could then simply express the difference as an R² as well (similar to the ∆R² in multiple linear regression)—or take the square root to report the difference as a multiple R. The need to focus on variances in validity estimation is why I am ‘at variance’ with S&H. 😎

Where R_d(validity) is the multiple R for the difference in validities for two selection systems a and b, where R²_a > R²_b. Use of this formula results in validity gains that are much higher than those reported by S&H (again, S&H should be reporting their results as utility gains, not validity gains).

Stay tuned for Part 2, the broader reason that I’m ‘at variance’ with S&H (insert suspenseful Halloween music here, for one because it might take at least that long for me to post again…).