Because NAEP data disaggregates outcomes across student subpopulations, it gives us visibility into how students’ differences in opportunity manifest as differences in outcomes.

The challenge is that when charts show “differences in outcomes,” it makes “differences in opportunities” easy to overlook. And this can lead viewers toward some surprising misjudgments about the data.

Group-level achievement data are often shown using conventional charts like the above. Sometimes they’re bar charts, contrasting average outcomes between groups (left). Other times they’re timeseries, highlighting the contrast between two groups (right).

By fixating viewers on the differences between groups, charts like these can be dangerously misleading, misrepresenting the underlying causes of outcome differences and the student groups being visualized (Holder & Xiong 2022, Gutiérrez 2008). These misbeliefs have troubling implications not only for students, but also in undermining successful education policy (Holder & Blakely, Metzyl 2019).

3iap’s research was actually the first to empirically demonstrate this biased interpretation for data visualizations, which you can read more about here: Unfair Comparisons.

Our research shows how these biases are influenced by design choices. For example, bar charts like the above, which hide variability within groups, tend to make these biases worse. On the other hand, charts like jitter plots and histograms show full distributions of data, emphasizing within-group variability and between group overlap; these more expressive charts can actually correct some of the bias, reducing the risk of misattributions and misrepresentations (Holder & Padilla 2024).

3iap’s collaboration with ETS focused on uncovering new charts, intentionally designed for visualizing group-level education outcomes while minimizing social-cognitive biases.

Specifically, we were exploring alternative designs aligned with the following goals:

1. NAEP Score charts should meet basic standards of clarity and approachability for The Report Card’s original communication goals. They need to clearly present group-level test scores.
2. The charts should minimize misleading biases by applying 3iap and others’ research into clearer social outcome reporting.
3. The charts need to be practically useful for education analysts, school administrators, and education policymakers.

While our research-backed design principles are a great first step, there’s still a big gap between principles and producing viable designs that are ready for a big platform like The Nation’s Report Card.

Resilient Designs: A guided codesign process can be a reliable way to develop designs that not only build on research and best practices, but that also meet practitioners’ other communication goals and adapt to their specific contexts. The key is how this collaborative process helps surface a) a wider set of constraints and considerations, b) from a more diverse set of perspectives, c) as early in the process as possible. Getting as many of these ideas out of people’s heads and onto paper makes it easier to manage critical design tradeoffs and ultimately reach more optimal solutions.

Learning By Doing: In addition to producing better designs, codesign also helps teams learn and adopt more effective design practices. While workshops are helpful for introducing research and design guidelines, there’s nothing like hands-on experience for learning something new. This is particularly true when the material is unintuitive or challenges entrenched conventional wisdom.

Decision Alignment: Codesign also promotes organizational alignment and cohesive decisionmaking. By actively participating in the design process, everyone involved has a more intimate understanding of important tradeoffs, making it easier to ultimately align around important decisions.

3iap partnered with ETS researchers on three main areas:

• Design training and data-design expertise. This ranged from early training sessions for participants, to targeted insights related to specific codesign sessions, to ongoing feedback and consulting on more general best practices for effective data communication.
• Facilitated design exercises. These helped codesign participants to articulate a diverse set of needs, as well as to brainstorm and prioritize viable alternative solutions.
• Design research consulting. As ETS researchers planned further studies, 3iap supported ongoing research efforts, contributing to literature reviews, instrument exploration, and experiment design tradeoffs for empirically validating candidate designs.