The Diversity Penalty: Why LLMs Win the Rubric but Human Reasoning Wins the Future

Do we still need to teach foundational cognitive skills when artificial intelligence can instantly generate a competent solution? A massive new randomized controlled trial (RCT) involving 1,053 first-year university students tackles this exact question—and the findings reveal a fascinating tension between what AI produces and what our educational systems actually reward. 

The study, conducted in August 2026, assigned undergraduate economics and management students to one of four conditions: a training game focused on causal reasoning, access to ChatGPT Edu (GPT-4o), a combination of both, or a control group. The students were then tasked with a realistic consultant-style challenge: write a 180-word recommendation to increase alumni awareness and usage of the university's merchandising shop. 

Here is what the data uncovered about the collision between human cognition and large language models (LLMs).

The LLM Advantage: Closing the Novice-Expert Gap When it came to standard evaluation metrics, ChatGPT was the undisputed champion. Evaluators—comprising both master's students and domain experts like a marketing professor and a merchandising manager—consistently scored the LLM-assisted responses higher on awareness and usage metrics. 

The AI didn't just polish the grammar; it fundamentally improved the output in several ways:

• It increases the total number of ideas included in a solution. 

• It improves the coherent logic of the arguments presented. 

• It generates recommendations that closely resemble the solutions provided by domain experts. 

Essentially, for well-defined problems where the criteria are clear, LLMs allow novices to punch above their weight class and mimic expert performance. 

The Causal Reasoning Plot Twist However, the students who received the short, intensive training on causal reasoning—learning to build coherent logic chains, identify mechanisms, and establish falsifiable conditions—experienced a different kind of transformation. 

This causal training significantly changed how the students thought. Relative to the control group, they were much more likely to identify underlying mechanisms and use falsification logic. Most importantly, causal training was the only intervention that meaningfully increased the diversity of ideas, both within a single student's solution and compared to the solutions of their peers. 

But there was a catch: the evaluation rubrics penalized them for it.

Post-double-selection lasso regressions revealed that evaluators actively marked down solutions that emphasized testability (falsifiability) or causal mechanisms, and they penalized ideas that strayed too far from the conventional solution space. The assessment system demanded a standard, competent answer, and the human-generated diverse ideas were effectively penalized for their novelty. 

The Synergy of Skills and Tools The most encouraging finding for the future of education comes from the intersection of these two treatments. When students had both causal training and access to ChatGPT, the LLM did not crowd out their human cognition. The students still applied their newly learned causal reasoning skills, utilizing the AI to enhance the coherent logic and expand the volume of ideas without sacrificing the diversity of thought that the human training provided. 

The ultimate bottleneck is not our ability to teach cognitive skills, nor is it the proliferation of AI tools. If we want to cultivate true diversity of thought and prepare the next generation of experts, we must fundamentally redesign our evaluation systems to demand originality rather than rewarding standard conformity. 

Strategic Insights for Instructional Design

The findings in this RCT carry massive implications for those of us designing learning simulations, K-12 teacher professional development, and assessment frameworks in the era of Human-AI collaboration.

• The Assessment Trap Requires Redesign: The revelation that rubrics actively penalized "Between-solution diversity" and falsification logic exposes a critical flaw in traditional instructional design. When we structure assessments for efficiency and high inter-rater reliability, we inadvertently train students (and raters) to seek the median. To foster genuine AI literacy and critical thinking, instructional strategists must explicitly build "diversity multipliers" into assessment rubrics, rewarding K-12 and higher-ed learners for charting non-obvious pathways rather than just hitting standard competencies. 

• Cognitive Scaffolding Survives AI Automation: One of the greatest anxieties in education is that students will simply outsource their thinking to tools like Gemini or ChatGPT. However, this data proves that an explicitly taught cognitive framework (like causal reasoning) is still exercised by the learner even when an LLM is available. This strongly validates the use of structured prompt engineering frameworks (like Meta-Prompting or Intelligent-TPACK models). When we teach K-12 educators or adult learners a concrete mental model, it acts as a permanent cognitive lens that guides how they wield the AI, rather than being replaced by it. 

• The "Jagged Frontier" of Novice Development: LLMs are exceptional at generating standard solutions because their training data is saturated with them, which perfectly matches what novices need to pass standard exams. But true expertise requires solving problems that haven't been solved yet. If K-20 education relies solely on AI to clear standard academic hurdles, we risk stunting the development of future experts. Training programs must intentionally push learners into ambiguous, unstructured territories where the LLM lacks a reliable prior, forcing the human to lead the causal inquiry. 

Resource article: https://www.nature.com/articles/s41562-024-02077-2