The pandemic profoundly slowed students’ learning in schools, with kids missing out on academics, social life and other important developments. Have things improved since then? It’s hard to know, based on the longitudinal data states present on pandemic learning loss.

When it comes to presenting data about the pandemic’s impacts on learning, Rhode Island and 12 other states are flunking, according to a report from the Center on Reinventing Public Education (CRPE) released Thursday. The center is based at Arizona State University’s Mary Lou Fulton Teachers College. 

Statewide data systems on public schools’ performance metrics — like graduation rates, attendance, student test scores, and per-pupil expenditures — are comparable to the report cards students receive. Publicly accessible, these report cards are meant to keep schools accountable in their delivery of a quality education, and might provide a fuller picture of the pandemic’s long-term ramifications for learning. The researchers specifically look at longitudinal data, or changes over time.   

“How easy would it be for an interested parent or advocate to find key performance indicators and compare them pre- and post-COVID?” the researchers asked.

The accountability report card system maintained by the Rhode Island Department of Education (RIDE) received an F grade because it was hard to find and compare performance metrics from before 2020 and in recent years, according to the report’s lead author.

“Reviewers found it very hard to find prior years of data [for Rhode Island],” said Morgan Polikoff, a professor at University of Southern California’s Rossier School of Education who led the six-person research team. “Since our report was mainly focused on the availability of longitudinal data, the state got an F for that reason.”

“But there were things they liked, like the ‘at a glance’ landing page for each school and the little qualitative summaries that seemed to be written by school principals,” Polikoff said.

More context wanted, but still some positive notes

Polikoff said the research team would have liked more context on the summary page, like answering if a school is on track or not, but they also made positive notes about the presentation of data on student subgroups and the school narrative section. The narratives, usually written by principals, “could be a model for other states to consider” as Polikoff said.

RIDE updated the Report Card layout in the last year, emphasizing achievement and proficiency numbers front and center on individual school, district and statewide profiles. Statewide proficiency in English Language Arts was at 35.1% in the 2022-2023 school year, for example — a number prominent on the report card’s first page.

But the first page of the 2018-2019 report card is formatted differently and doesn’t immediately share this figure. That makes comparing pre- and post-COVID data tricky for those unacquainted with RIDE’s site. 

Longitudinal data may be easier to understand when condensed by organizations who already comb through state data, like the Annie E. Casey Foundation, which makes academic progress over time easily deciphered in its KIDS COUNT Factbook.

While academic achievements have seen somewhat sluggish recovery, chronic absenteeism has more vigorously pursued a corrective course, with RIDE attributing the positive change partly to its absenteeism dashboard, which has seen praise and national attention in recent months. (The dashboard is run separately from the overall report card system.) Last month, the education department also shared results of a Harvard Graduate School of Education study that showed Providence schools are doing a little better post-pandemic than comparable districts in other New England states.

“As the report notes, this is a challenge for most states,” Victor Morente, RIDE spokesperson, said in an email Thursday evening. “RIDE is working to better inform all education stakeholders including students, families, and educators about how their school community is doing including identified areas of strength and of needed improvement in the wake of the pandemic.” 

Morente added that the agency is “committed to continuous improvement and is currently undertaking changes to its accountability system,” which includes amending the state’s Every Student Succeeds Act State Plan. Public comments on the amendment were accepted through Sept. 2 and are currently being reviewed. 

How states were evaluated

The six-person research team evaluated each state’s public school report card system on seven performance metrics. The researchers then assigned points based on how easy it was to find longitudinal data on the metrics — or, in other words, how easy it was to compare the metrics from before, during and after the pandemic. The scores, with 21 points the maximum possible, were then converted to letter grades. Rhode Island scored six points overall, which converted to an F grade.

The seven metrics included:

• Achievement levels in English Language Arts and mathematics
• Growth in in English Language Arts and mathematics achievements
• Proficiency in science
• Proficiency in social studies
• Chronic absenteeism or other attendance indicators
• High school graduation rate
• English language learner proficiency or growth

An additional qualitative rating was assigned to each state for their report card’s usability. Rhode Island received a “fair” rating in this additional category, but researchers’ main gripe with RIDE’s report card was the difficulty involved in finding data — they found pre- and post-COVID data on six metrics only “with too much difficulty,” and one metric (social studies) was not found at all.

Citing nationwide, post-pandemic trends in declining academic achievements, attendance and equity for marginalized students, the report argues that Rhode Island is one of the states that doesn’t effectively share data about changes in public school learning since the pandemic.

“How transparent are these trends to parents or other interested parties?” the report’s six authors asked. “We have lots of suggestive evidence that parents don’t understand the magnitude of the COVID-19 downturns in achievement or attendance, or at least aren’t as concerned as experts think they should be. Is that because school report cards aren’t leveling with parents about how these outcomes have changed since before the pandemic?”

Maine, Vermont also receive F

In the new CRPE study, Connecticut fared the best of all New England states, with an A grade for its report card data on COVID, while New Hampshire was second highest with a C grade. Massachusetts received a D, and Maine and Vermont joined Rhode Island in the F group. Maine was one of three states to score zero points overall — “not necessarily because these states have terrible report cards…[but because] these states’ report cards do not make longitudinal comparisons back to pre-COVID possible for the average user,” according to the report.

Alas, Rhode Island was not alone in its at-times confusing presentation of pandemic-related data.

“Some sites featured attractive visuals that we thought a parent would be able to interpret. In contrast, other sites bombarded the user with mountains of disaggregated data that would be very difficult, if not impossible, for an average viewer without a Ph.D. in data science to understand,” the authors state. “On some sites, the menus for searching and selecting schools were easy to use, while on others, they were sources of maddening frustration.”

Among the most vexing was Vermont, which the report used as an example of poor user-oriented design.

“Vermont offers a series of dashboards that are incredibly challenging to operate or interpret. They are slow, they don’t seem to allow for obvious data export, and the figures and tables they produce are hard to understand,” the report concluded.