16 AI Literacy Scales and None of Them Measure What Matters

The New York Times reported today that AI literacy is trending in schools. Google, Microsoft, and OpenAI are all funding curricula. States are writing standards. Teachers are scrambling to integrate it. But here's the uncomfortable question nobody is asking: what exactly are we measuring?

A 2024 systematic review by Lintner, published in the International Journal of Educational Technology in Higher Education, applied the COSMIN framework — the gold standard for evaluating measurement instruments in health and social sciences — to every AI literacy scale in existence. The result: 16 instruments. Zero measure communicative competency.

The Measurement Paradox

Let that sink in. The field of AI literacy has produced 16 different instruments to measure how well people understand artificial intelligence. They measure knowledge of machine learning concepts. They measure attitudes toward AI. They measure self-efficacy — how confident you feel using AI tools. A few even measure ethical reasoning.

But not a single one measures whether you can communicate effectively about algorithmic systems — to other people, within organizations, across stakeholder groups with fundamentally different mental models of how these systems work.

This isn't a minor oversight. It's a structural blind spot that reveals what the field actually values: individual knowledge over collective capacity.

13 Out of 16 Are Self-Report

The Lintner review surfaces another problem. Of the 16 instruments, 13 are self-report measures. They ask people how much they think they know about AI. Not what they can actually do with that knowledge. Not whether they can translate technical concepts for a non-technical audience. Not whether they can identify when an algorithmic system is shaping their information environment and articulate that to someone who doesn't see it.

Self-report scales measure confidence, not competence. And in AI literacy specifically, this creates a dangerous gap: people who feel literate but can't communicate what they know to anyone outside their bubble.

We are building an entire educational infrastructure around a competency we measure with confidence surveys.

The One Exception That Proves the Rule

There is exactly one scale in the Lintner review that includes anything resembling communicative competency: the ChatGPT Literacy Scale. It has a “communication proficiency” factor. But it's platform-specific (ChatGPT only), it's untheorized (no communication framework underpins it), and it measures communication with the AI, not communication about AI systems between humans.

That single exception makes the gap more visible, not less. Even when researchers intuit that communication matters, they frame it as prompt engineering — talking to the machine — rather than the far harder and more consequential skill of talking to each other about what the machine is doing.

The 62% Floor

Why does this matter? Because of what Gran, Booth & Bucher found in 2021.

Their study of Norwegian internet users — one of the most digitally literate populations on Earth — found that 62% lack basic algorithm awareness. They don't know that the content they see is curated by algorithmic systems. They think their social media feed is chronological, or random, or chosen by the platform's editorial team.

Gran et al. identified six distinct awareness clusters, from the completely unaware to the critically engaged. But here's the key insight: awareness alone doesn't produce agency. Their most aware cluster — people who understood algorithmic curation, personalization, and data extraction — still reported feeling powerless to do anything about it.

Awareness without communication produces cynicism, not capacity.

If 62% of Norwegians — a population with universal broadband, high education levels, and strong digital infrastructure — lack algorithm awareness, that's not the ceiling. That's the floor. And for the 38% who are aware, knowledge without the communicative tools to act on it collectively just produces informed helplessness.

The Collective Resistance Problem

This connects directly to Zhao's 2025 research on algorithmic resistance in China. Studying how users on Douyin, Xiaohongshu, and Weibo collectively push back against algorithmic systems, Zhao found two pathways: within-framework resistance (gaming the algorithm) and beyond-framework resistance (opting out or building alternatives).

But here's what Zhao discovered: when people hold conflicting folk theories about how algorithms work, collective resistance collapses. One group thinks the algorithm prioritizes engagement. Another thinks it prioritizes recency. A third thinks it's random with paid promotion. They can't coordinate because they can't communicate a shared model of what they're resisting.

This is exactly what happens when you have individual AI literacy without communicative competency. Everyone knows something is happening. Nobody can talk about it coherently enough to do anything together.

What Schools Should Actually Measure

So what would a communicative AI literacy measure look like? Here's what's missing from all 16 scales:

• Translation capacity: Can you explain an algorithmic system's behavior to someone with a fundamentally different mental model than yours?
• Folk theory identification: Can you recognize when two people are operating from incompatible assumptions about how a system works?
• Stakeholder bridging: Can you facilitate a conversation between a developer, a user, and a policymaker about the same system without losing all three?
• Collective framing: Can you articulate algorithmic impacts in terms that enable coordinated response rather than individual complaint?
• Cross-context transfer: Can you apply insight from one algorithmic environment (e.g., TikTok's recommendation engine) to another (e.g., a hiring algorithm) in a way that others can follow?

None of these are radical ideas. They're basic communication competencies applied to algorithmic systems. But they require a theoretical framework that the AI literacy field has never adopted: Application Layer Communication (ALC).

The Rush and the Gap

The NYT reports that schools are moving fast. States like California, Virginia, and New York are drafting AI literacy standards. Tech companies are providing free curricula. The U.S. Department of Labor just released a five-competency AI literacy framework.

All of this activity shares the same structural flaw: it treats AI literacy as an individual knowledge problem. Learn what machine learning is. Understand bias. Think critically about AI outputs. These are necessary but woefully insufficient. They produce people who know things but can't do anything with that knowledge collectively.

The Lintner review makes this measurably clear. We have 16 instruments, and the thing they collectively fail to measure — communicative competency — is the thing that converts individual awareness into collective agency.

If you can't measure communicative competency, you can't teach it. If you can't teach it, you're not building AI literacy. You're building AI awareness — and awareness without communication is just anxiety with better vocabulary.

What This Means for Organizations

This isn't just an education problem. Every organization deploying AI tools faces the same gap. Your engineers understand the system. Your users experience the system. Your executives make decisions about the system. And none of them can talk to each other about it coherently.

When your product team says “the algorithm optimizes for engagement,” your marketing team hears “the algorithm makes things go viral,” and your legal team hears “the algorithm creates liability.” Same sentence, three different mental models, zero shared framework for resolving the difference.

That's not an AI literacy problem. It's a communicative competency problem. And until we can measure it, we can't fix it.

References

• Lintner, S. (2024). A systematic review of AI literacy scales. International Journal of Educational Technology in Higher Education. PMC.
• Gran, A.-B., Booth, P., & Bucher, T. (2021). To be or not to be algorithm aware: A question of a new digital divide? Information, Communication & Society, 24(12), 1779–1796.
• Zhao, Y. (2025). Boosting popularity: Folk theories and algorithmic resistance on Chinese social media. Big Data & Society.
• Long, D., & Magerko, B. (2020). What is AI literacy? Competencies and design considerations. CHI '20.
• Singer, N. (2026, February 23). A.I. literacy is trending in schools. The New York Times.

This analysis is part of Topanga Consulting's ongoing research into Application Layer Communication (ALC) — the study of how humans communicate about, through, and around algorithmic systems. If your organization is deploying AI tools and struggling with the gap between technical capability and human understanding, that's the gap we work in.