EdReports.org's Methodology: Curriculum Reviews for K-8 Mathematics
We at EdReports.org believe all students and teachers in the United States must have access to the highest quality materials that will help improve student-learning outcomes. We support educators across the country as they seek, develop and demand high-quality instructional materials.
We trust and rely on educators to define quality, review materials and share their findings.
Instructional Material Selection
For the inaugural review process that began in 2014, EdReports.org created an inventory of more than 60 K-8 mathematics instructional materials from 30 publishers that cover at least a year's worth of instruction.
EdReports.org reviews only the materials that comprise the core components of a program, typically the student and teacher editions for each grade. Although supplementary materials are sold for each series (like homework packets or extra assessment items), these materials are not reviewed. It is the best way to ensure that the same baseline criteria was applied across curricula and so that our reviews would support districts and teachers who may not have the means to purchase supplementary materials.
For the first round of reviews, EdReports.org prioritized those instructional series that were used by the largest number of educators and students and marketed as aligned to the Common Core State Standards. We selected our first set of materials by using the following criteria:
- The materials have at least 10 percent market share for print (any market share for digital); OR
- The materials have been recommended or approved through at least two state reviews (for both print and digital materials).
Based on these criteria, EdReports.org has reviewed 20 K-5 and 6-8 mathematics instructional series.
High Standard: Creating the EdReports.org Quality Instructional Materials Tool
To create the EdReports.org Quality Instructional Materials Tool, we conducted research on existing instructional materials evaluation tools, led a national listening and learning tour to gather input from educators and convened an Anchor Educator Working Group.
The tool's creation began by researching existing tools and by listening to educators. We heard from district and state administrators who make purchasing and development decisions; teachers and principals who make instructional decisions and conduct daily searches to curate their own materials; and rubric creators, policymakers, researchers, and mathematicians. We gathered a diverse range of perspectives (across grade levels, districts and states, representing rural, urban, and suburban communities, including diverse student populations). Knowing which instructional materials educators use, what they value when selecting materials, and how they use materials informed the design of EdReports.org's evaluation tool and process.
The Anchor Educators created the EdReports.org tool based on essential elements and best practices from existing tools used in the field. The tool measures focus, rigor, coherence, the mathematical practices and additional indicators of quality. This tool guides the entire evaluation process to provide educators with credible, reliable and useful information about instructional materials and their alignment with the standards. For more information about some of the materials consulted by the tool's developers and review teams, please see:
- Common Core State Standards for Mathematics
- Standards for Mathematical Practice
- Progressions Documents for the Common Core State Standards for Mathematics
- K-8 Publishers' Criteria for the Common Core State Standards for Mathematics
- Instructional Shifts for Mathematics
EdReports.org content reviewers are teachers, principals and instructional coaches in schools and districts. Reviewers produce thoughtful, rigorous and honest reviews that provide credible and useful information for consumers as well as feedback on materials. They commit to EdReports.org’s mission and vision and work approximately 5 hours a week over 4 months to review materials, collect evidence, discuss findings and prepare reports. Our reviewers hail from 19 states and work in urban, suburban, mid-size and rural schools and districts around the country.
EdReports.org selects its reviewers after a rigorous interview process that includes an application, interview and performance task. Each step of the interview process measures candidates' knowledge of the Core State Standards for Mathematics, their experience as educators and their ability to work collaboratively in addition to key EdReports.org values. Reviewers sign a code of ethics that requires they fully disclose any relationship with publishers and certify that:
- Reviewers have not accepted compensation from any publishing company within the past seven years.
- Reviewers have not authored or co-authored any year-long curriculum currently on the market.
- Reviewers maintain independence from publishing companies or other nonprofit or for-profit curriculum developers for the duration of participation in EdReports.org reviews.
All content reviewers received more than 20 hours of EdReports.org training to apply the EdReports.org Quality Instructional Materials Tool. EdReports.org immersed its reviewers in the K-8 mathematics standards, the EdReports.org review process,a nd the essential work of reviewing instructional materials for quality.
A Rigorous Process: How EdReports.org reviewers examine instructional materials
The EdReports.org's tool supports a sequential review process through three gateways that reflect the importance of alignment to the fundamental design elements of the standards—focus and coherence, rigor and mathematical practices—and then considers other high-quality attributes of curriculum as recommended by educators. Reviewers evaluate at least a year's worth of core instructional materials.
A lot of research and effort goes into the creation of textbooks and digital products. EdReports.org's tool evaluates these products from the end user's perspective. We know when materials enter a classroom, teachers are looking for two key features:
First, is the instructional material aligned to the standards? Access to high-quality instructional materials helps students meet higher standards, enhances educator learning and saves teachers time. Aligned materials allow students to engage in problems and assessments that build logically from year to year so that they graduate college- and career- ready. Such materials help educators teach the deep mathematical concepts expected of students.
Second, are the instructional materials usable for students and educators? Materials must be well designed to facilitate student learning and enhance a teacher's ability to differentiate and build knowledge within the classroom. It also means that the materials offer assessments that help educators and parents focus on mathematical strengths and areas of growth, and incorporate technology effectively as a means to helping students learn.
Our review tool reflects this prioritization. Gateway 1 concentrates on two design elements of the standards: Focus and Coherence. Is appropriate grade level work addressed in the curriculum? Does the material make strong connections between the mathematical content as opposed to teaching skills or concepts in isolation? For Gateway 2, the reviewers considered the rigor of the materials and evaluated connections to the mathematical practices. Finally, when the first two gateways have been met, the materials were reviewed for how well they support teachers in reaching all students and their ease of use in the classroom. Only materials that meet or partially meet expectations for Gateway 1 will be reviewed for Gateway 2, and only materials that meet expectations Gateways 1 and 2 will be reviewed for Gateway 3.
EdReports.org trained educators to apply the evaluation tool, collect evidence, record ratings and calibrate their reviews and evidence with their teams according to a step-by-step process.
Reviewers joined professional learning committees (PLCs) that collaboratively review sets of instructional materials. Each PLC was assigned materials to review and received an orientation to the materials from publishers. Each publisher was invited to conduct an hour-long orientation for the reviewers.
Using the Quality Instructional Materials Tool as their guide, reviewers first analyze curriculum on their own and then identify evidence from the materials, including from activities, lessons, units and chapters. Then the team comes together each week to discuss the evidence for each indicator, norm around what constitutes appropriate evidence, and come to consensus on ratings for each indicator. The facilitator and team then selects which evidence to include in a final grade-level report. This process repeats for each grade level within a grade band (Grades K-2, 3-5, 6-8). After all grade-level reports are completed, the review team compiles the scores and evidence to create a grade-band summary report.
The Review Process at a glance
EdReports.org assembles review teams and assigns instructional materials
Diverse teams of 3-5 reviewers are assigned an instructional series based on their experience and the materials to be reviewed.
A lead reviewer is selected from each team to facilitate weekly calls with the review team and to participate in cross-team weekly calibration PLC.
Reviewers examine instructional materials independently
Reviewers use the EdReports.org evaluation tool independently to review instructional materials, collect evidence and determine ratings.
Once all reviewers have rated instructional materials on their own, the lead reviewer convenes the team to discuss findings.
Reviewers meet to share evidence and discuss ratings
Facilitated by the team's lead reviewer, reviewers meet weekly in small teams to discuss their ratings and evidence.
Each team must reach consensus on an overall rating.
Team calibrates with other teams, finalizes evidence and scores and writes report
Once teams reach a consensus rating for a series, the lead reviewer from each group shares the evidence and ratings with the lead reviewers of the other teams during a weekly calibration meeting.
Lead reviewers scrutinize the evidence and ratings to check to make sure the evaluation tool's criteria is being applied consistently among review teams.
The amount of time and hours all of the reviewers spent looking at the alignment, assessments, rigor, Mathematical Practice Standards, etc. was unbelievable. I am so thankful that I now have a resource to help my district make a solid purchase that will benefit both our teachers and more importantly our students in preparing them for college and careers. –Kristen H., 23-year veteran educator
KEY TERMS THROUGOUT THE EVALUATION TOOL
Indicator: Specific item that reviewers look for in materials.
Criterion: Combination of all of the individual indicators for a single focus area (focus, coherence, rigor, etc.).
Gateway: Organizing feature of the evaluation tool that combines criteria and prioritizes order for sequential review.
Alignment: Degree to which materials are consistent with the standards in focus on the major work of the grade, coherence, rigor and attention to the mathematical practices.
Usability: Degree to which materials are consistent with effective practices (as outlined in the evaluation tool) for use and design, teacher planning and learning, assessment, differentiated instruction and effective technology use.
Scoring and Ratings: Grade-Level Reviews
Reviewers take four steps to determine whether the instructional materials reviewed at each individual grade level meet the expectations to proceed to the next gateway.
Review and collect evidence for each indicator, beginning with Gateway 1.
Gateway 1 consists of three criteria and six indicators. Reviewers begin on a single grade level and find evidence and then determine a score for each indicator in Gateway 1.
Rate each indicator according to the evidence collected. Do not proceed to Gateway 2 unless the instructional materials meet or partially meet the expectations for Gateway 1.
Reviewers rate each indicator using the following rating scale*:
- 2 - A rating of 2 means that the materials meet the full intention of the indicator.
- 1 - A rating of 1 means that the materials partially meet the intention of the indicator.
- 0 - A rating of 0 means that the materials do not meet the intention of the indicator.
* EdReports.org's emphasis on the importance of Focus is evident in the scoring of the first two indicators of Gateway 1 (1a and 1b). Materials cannot receive a “partial” rating for these two important indicators.
Add the total points earned of each indicator within the gateway.
Each gateway within the evaluation tool organizes several criteria and indicators. This organization allows for granular reporting (by indicator or criterion) as well as overall reporting (by gateway). Reviewers add the total points earned for each criteria to determine a gateway score.
Determine whether enough points were earned to proceed to the next gateway.
Use the cut scores provided in the EdReports.org tool to determine if a grade-level set of materials moves on to the next gateway.
Determining an Alignment Rating for Grade-Level Reviews
The chart below illustrates how an overall alignment rating is determined for individual grade-level sets of instructional material. In order to be reviewed and attain a rating for usability (Gateway 3), the instructional materials must first meet the expectations for alignment (Gateways 1 and 2).
|Step 1||Step 2||Result: Alignment Rating|
|The reviewed materials Meet Expectations for Gateway 1||AND Meet Expectations for
|BUT either Partially Meet or Do Not Meet the expectations for Gateway 2||Partially Meet Expectations|
|The reviewed materials Partially Meet Expectations for Gateway 1||AND Meet Expectations for Gateway 2||Partially Meet Expectations|
|BUT either Partially Meet or Do Not Meet the expectations for Gateway 2||Do Not Meet Expectations|
|The reviewed materials Do Not Meet the expectations for Gateway 1||If it Does Not Meet Expectations for Gateway 1 the material is not reviewed for Gateway 2||Do Not Meet Expectations|
A Closer Look at the Gateways in the EdReports.org Quality Instructional Materials Tool
Gateway 1: Focus and Coherence
Gateway 2: Rigor and Mathematical Practices
Gateway 3: Usability