The AI Transparency Problem: Why We Don't Really Know How People Use ChatGPT and Claude

The AI Transparency Crisis We're Not Talking About

OpenAI and Anthropic regularly publish reports showcasing how millions of people use ChatGPT and Claude. These narratives shape our understanding of AI's role in society—but there's a critical problem: we're only seeing the data these companies want us to see.

According to MIT Technology Review, independent researchers have raised serious concerns about this selective disclosure. As Stanford computer science PhD candidate Anka Reuel points out, "There is no independent source to corroborate it." This lack of verification matters far more than it might seem at first glance.

Why This Matters for AI Users and the Industry

The ability to independently verify how people actually use AI tools isn't just an academic concern—it affects everyone from casual users to enterprise customers making significant investments in AI infrastructure.

What We're Missing

• Real usage patterns: Companies can highlight their most impressive use cases while downplaying problematic trends or limitations
• Bias and fairness insights: Without independent auditing, we can't assess whether these tools perform equally across different demographics
• Security and safety data: Are there edge cases or vulnerabilities that companies aren't disclosing publicly?
• Market dynamics: Which tools are actually winning with real users versus which are winning the PR game?

The Control Factor

When companies control both the product and the narrative around it, they have every incentive to frame the story favorably. A report showing that Claude excels at coding tasks might be factually accurate, but it tells us nothing about relative performance against competitors or real-world failure rates that users experience.

This selective transparency creates what researchers call an "information asymmetry"—AI developers know far more about how their tools perform in the wild than the public does. The companies can adjust messaging, highlight successes, and quietly improve features based on usage data that remains proprietary.

The Broader Implications

This lack of independent verification has consequences rippling through multiple sectors:

• Business decisions: Companies choosing between AI tools don't have reliable, independent benchmarks
• Regulation: Policymakers can't craft informed rules without access to real usage data
• Research: The broader AI research community lacks the empirical foundation needed to understand real-world impacts
• Public trust: When information feels controlled, skepticism grows—even if the underlying data is honest

What Would Real Transparency Look Like?

Independent researchers aren't asking companies to surrender proprietary secrets. Rather, they're advocating for:

• Third-party access to anonymized usage data
• Regular, independent audits of AI performance claims
• Published data on error rates, limitations, and edge cases
• Transparent methodology behind published research

The Bottom Line

As AI tools become increasingly central to how we work and live, we need reliable information about how they actually perform in the real world. Right now, that information is locked behind corporate walls.

For users evaluating AI tools, this means being skeptical of published performance claims and seeking out independent reviews and benchmarks wherever possible. For the industry, it's a reminder that trust—whether with users, regulators, or the broader public—requires transparency that goes beyond curated press releases.

The conversation about AI regulation, safety, and societal impact can't move forward when the fundamental data remains proprietary. Until we solve the transparency problem, we're essentially asking society to make decisions about powerful technology based on incomplete information.