The Illusion of Learning in a World Obsessed with Metrics

It is only now, well into my career, that I’m realizing that my primary role has essentially always been that of a “data analyst.” Irrespective of the actual position or formal title I’ve held, I’ve been responsible for identifying the most reliable and relevant sources for data, coordinating those data to surface meaningful lessons, and using the resulting conclusions to answer ambiguous questions inform decisions.

Some data sources are directly tied to our goals and relatively easy to interpret. This is particularly true for quantitative data which are typically easier to track, compare, and assign to meaning. Other sources are far more variable and subject to wider interpretation. This is especially true for qualitative data, which are usually more difficult to ascribe to meaning. And yet, this is far more consequential for learning.

I learned this lesson most clearly while writing my dissertation on the costs of institutional accreditation. Because I was studying “cost,” I envisioned a number! I planned a quantitative study and assumed it would be straightforward for Accreditation Liaison Officers to indicate the cost of accreditation-related activity at the institutions. Fortunately, a very wise advisor encouraged me to include space for comment and clarification in the survey. Without that, she warned that the response would be limited and the results misleading.

I was struck by the sheer complexity of what emerged in those comments. The rich qualitative data revealed dimensions of the problem I had not anticipated. Just like that, the project became a true mixed methods study… but the real value lay in the qualitative analysis. The quantitative data were just incomplete. The actual fiscal cost, it turns out, is not high! So why does accreditation feel so burdensome? Because the true “cost” is not financial, but experiential: the time, effort, and energy expended at both the institutional level and by countless individuals. Had I focused on the quantitative cost, I would have missed the most important learning entirely.

That experience was formative, but it also revealed something more persistent and universal. Over time and across many roles, I’ve seen how easily qualitative insight is crowded out by quantitative measures, not because qualitative analysis is less valuable but because there are powerful forces that consistently privilege numbers over interpretation. Two of those forces stand out.

First, watching the numbers is rewarding in and of itself! Even when the numbers are moving in the wrong direction, they provide immediate feedback. Our collective obsession with metrics is understandable, like the dopamine hit of endless scrolling. Tracking movement is easy to mistake for insight. Understanding that movement is not automatic. Watching the numbers can create the illusion of learning. It can make us complacent.

Second, qualitative analysis is hard! The most relevant data sources can be difficult to identify, and once identified the data can be even harder to gather. The data are rarely uniform. The process of compilation and triangulation is often messy. The ensuing interpretation is open to legitimate interrogation and re-interpretation. And yet, this is where precisely where the true learning lives! That ambiguity is the raw material for decision-making.

Quantitative data will almost always function as lagging indicators. Even when decisions are grounded in numbers, they ultimately rely on subjective judgments about interpretation and meaning. The real opportunity for learning and sound decision-making comes from qualitative analysis. The expertise required to make sense of qualitative data is what makes us truly valuable to the organizations we serve.