Education Framework Audit Kit

How This Kit Works

There are four layers to auditing a state education framework. Each layer builds on the previous one, but any layer can be done independently. You do not need to complete all four. Even Layer 1 alone — checking your framework against the federal evidence base — will tell you whether your state's guidance matches the research.

1. Layer 1: Evidence-Base Comparison — Does your framework cite the research?
2. Layer 2: Citation Verification — Does the framework accurately represent what the cited papers actually say?
3. Layer 3: Conflict-of-Interest Audit — Who wrote it, and what do they sell?
4. Layer 4: Public Records Requests — What happened inside the process?

Layer 1: Evidence-Base Comparison

This is the fastest, most damning test. The federal Institute of Education Sciences (IES) maintains What Works Clearinghouse practice guides — meta-analyses of the strongest available research on teaching specific math topics. Each guide rates its evidence as "strong," "moderate," or "minimal." These are the gold standard. If your state's framework ignores them, that is a finding.

The Four Practice Guides to Check

Instructions for Your AI Agent

Give your AI agent the following prompt. Replace [STATE] with your state and provide a link to or PDF of your state's math framework.

What the Results Mean

• >50% overlap: Your framework is engaging with the evidence base. Check Layer 2 to see if it's representing the studies accurately.
• 10–50% overlap: Selective citation. Your framework is cherry-picking. Check which recommendations it excludes and whether those exclusions have a pattern.
• <10% overlap: Your framework is functionally ignoring the federal evidence base. This is a finding. California's framework cited 1 of 114 (0.9%).

Layer 2: Citation Verification

This is what Brian Conrad did. It is labor-intensive for a human but straightforward for an AI agent: take each cited paper, read the relevant sections, and check whether the framework's characterization matches what the paper actually says.

Instructions for Your AI Agent

Red Flags to Watch For

• Self-citation concentration: Count how many times each writing team member's own work is cited. In California, one author was cited 76 times out of ~400 total references. Any single author exceeding 10% of total citations in a document they helped write is a red flag.
• Conference paper substitution: Citing a conference presentation or working paper when a peer-reviewed study on the same topic reached a different conclusion.
• Limitations laundering: Citing a study's findings while omitting the study's own stated limitations (e.g., "cannot be generalized").
• Temporal mismatch: Citing decades-old studies for claims about current practice when more recent studies exist on the same topic.
• "Studies show" with no citation: Framework text asserting research support without any citation at all.

Layer 3: Conflict-of-Interest Audit

Who wrote your framework, and what do they sell?

Instructions for Your AI Agent

Layer 4: Public Records Requests

Every state has a public records law equivalent to California's Public Records Act. The requests below are adapted from the ones we filed with the California Department of Education. Modify the agency names, statutory citations, and date ranges for your state.

Request 1: Writing Team Selection & Conflict of Interest

What this surfaces: Whether any COI policy existed. Whether commercial interests were disclosed. Whether the selection process considered disciplinary diversity (mathematicians vs. education professors).

Request 2: Internal Response to External Criticism

What this surfaces: Whether criticism was taken seriously or dismissed. Whether authors participated in deciding how to handle criticism of their own work. Whether accuracy concerns were systematically deprioritized.

Request 3: Comment Period and Vote Timeline

What this surfaces: Whether the compressed timeline was deliberate. What board members were actually told before voting. Whether staff summaries mentioned citation accuracy concerns.

Request 4: Financial Relationships

What this surfaces: Whether the department had financial relationships with writing team members before, during, or after the framework was developed. Whether the framework creates commercial benefit for its own authors.

State-by-State Filing Guide

Every state has a public records law. The request format is nearly identical — change the statutory citation and the agency name. Your AI agent can look up the correct statute and mailing address for your state's department of education.

What to Do with the Results

If You Find Problems

1. Document everything. Screenshots, downloaded PDFs of cited papers, side-by-side comparisons of framework claims vs. paper findings. Your AI agent can generate these comparison documents.
2. Write it up. A clear, evidence-based summary with specific page references. Not "the framework is bad" — rather "on page 47, the framework claims Smith (2019) found X, but Smith's paper states the opposite on page 12."
3. Send it to your school board. Most school board members have never read the framework, let alone the cited papers. A 2-page summary with 5 specific misrepresentations is more effective than a 50-page complaint.
4. File it as a public comment during any open comment period. Your state likely has periodic review cycles for instructional frameworks.
5. Send it to your state legislators. The structural reforms — independent citation review, COI disclosure, minimum comment periods — require legislation. Give them the evidence.
6. Share your findings. Post your audit results publicly. Tag us. We'll link successful audits from this page. The more states that get audited, the harder it becomes for any single state to ignore the pattern.

Building Better Incentives for Teachers

Auditing frameworks is defensive — catching bad policy. The harder, more important work is building better incentives for the teachers who actually have to implement whatever the framework says. Here are concrete asks that come out of this analysis:

• Fund implementation. California passed a 1,000-page framework with zero dollars for teacher training. Any framework without an implementation budget is performative. Demand that your state appropriate specific funding for PD tied to the framework's recommendations.
• Pay for evidence-based PD, not branded workshops. When framework authors sell PD workshops aligned with the framework they wrote, the incentive structure is circular. Demand that state-funded PD be provided by organizations with no financial ties to the framework authors.
• Protect teachers who deviate. A framework is guidance, not law. But teachers who ignore it risk poor evaluations, especially if their district adopted it. Push your district to explicitly affirm that teachers can use evidence-based practices even when those practices diverge from the framework — particularly on fluency, standard algorithms, and timed retrieval practice.
• Reward outcomes, not compliance. Framework adoption rates measure process. NAEP scores measure results. Push your state to evaluate math instruction by student proficiency growth, not by how closely teachers follow a document that may misrepresent its own sources.

International Benchmarks

When your AI agent audits your framework, have it compare pacing against these international standards. If your state teaches a concept later than all three of these systems, that is a finding.

* Common Core specifies these as "culminating standards" — the endpoint of a progression that should begin earlier. California's framework reinterpreted them as first-exposure grades.

License

This toolkit is released under CC0 1.0 (Public Domain). Copy it, modify it, translate it, republish it, sell it. No permission needed. No attribution required, though we'd appreciate a link back.

If you use this kit to audit your state's framework, we want to hear about it. Email findings to [email protected] or tag us when you publish. We'll maintain a running index of completed state audits on this page.

Completed Audits