Construct Validity: Do You Know What You're Actually Measuring?

Most interview scorecards include dimensions like “communication skills,” “leadership potential,” or “problem-solving ability.” Most organizations have not defined what those phrases mean for their specific jobs.

This is a construct validity problem. And it sits at the foundation of almost every assessment that produces unreliable data.

What a construct is

A construct is a psychological concept that you are trying to measure. “Communication” is a construct. It is not directly observable: you cannot see communication ability the way you can see height. You can only observe its manifestations, specific behaviors that serve as evidence of the underlying construct.

This is important because “communication” means different things in different jobs. A software engineer’s communication competency is mostly about documenting technical decisions clearly and translating requirements across functional lines. An account manager’s communication competency is primarily about managing client expectations and presenting complex information simply. These are related but not identical, and measuring one does not tell you much about the other.

Convergent and discriminant validity evidence

Construct validity is established through two types of evidence:

Convergent validity: Your measure correlates with other measures of the same construct. If you are measuring “analytical reasoning” with a structured interview, candidates who score high should also perform well on other analytical tasks. Convergence across measures is evidence that you are measuring the same underlying thing.

Discriminant validity: Your measure does not correlate strongly with measures of different constructs. If your “analytical reasoning” interview score correlates as strongly with “interpersonal warmth” as it does with problem-solving tasks, you are probably not measuring a distinct analytical construct.

Most organizations never collect this type of evidence, which means they cannot demonstrate that their competency ratings are measuring what they say they measure.

Why this matters in practice

Consider a structured interview that scores candidates on five competencies. If the interrater correlations across competencies are uniformly high, it suggests the interviewers are rating a general impression (a “halo” effect) rather than five distinct dimensions. The five scores are not adding five data points. They are adding one data point, labeled five different ways.

Construct validity problems also appear when:

• Interviewers cannot agree on what “leadership potential” means in this specific role
• The same question is used to score different competencies because “it touches all of them”
• Competency definitions are copied from generic frameworks without adaptation to the job

Building toward construct validity

You do not need a formal psychometric study to improve construct validity in your hiring process. The minimum steps:

1. Define each competency behaviorally for this role, based on job analysis. Not a generic definition: what does strong performance on this competency look like in this job?

2. Assign one or two questions per competency, not one question scored on all five competencies simultaneously.

3. Check for halo effects in your scoring data. If one competency score predicts all the others, you have a construct problem worth investigating.

4. Have SMEs review your competency definitions to confirm they describe relevant behavior for the role, not abstract ideals.

None of these steps requires a research budget. They require time and rigor in the design phase, which is consistently where organizations invest the least.