Why Do They Keep Trying to “Validate” Restorative Practices with Lousy (or Worse) Data?
More Proof that Schools Need to Avoid Restorative (Justice) Programs and Practices
Educators need to be wary when people, organizations, or agencies intent on promoting their beliefs or rationalizing their existence inaccurately or deceptively disseminate data, information, and/or Reports to “validate” their presence, power, or positions.
Dear Colleagues,
Introduction
Over the years, I have clearly debunked—from both research and practice perspectives—the use of (especially packaged) Restorative (Justice/Practice) Programs in schools.
Regardless of their implicit or explicit desire to “cure” disproportionate disciplinary referrals, actions, or suspensions for students of color, they do not work. Indeed:
- No single Restorative (Justice/Practice) Program has ever been objectively validated using sound research methods in a school setting; and
- The most methodologically-sound research studies report that Restorative (Justice/Practice) Programs have actually produced negative results relative to student behavior and academic achievement, and teacher/staff morale, feelings of administrative support, and classroom management success.
And while there is a place for more specific, individual Restorative Practices—that are not layered into an “all-knowing, all-purpose, too-good-to-be-true” packaged program—many educators do not fully understand the science-to-practice needed to make them successful.
Specifically, Restorative Practices do not effectively change students’ inappropriate or antisocial behavior when:
- The behaviors are intense, extreme, dangerous, or physically harmful, abusive, or destructive;
- Students do not have the social, emotional, or behavioral skills to behave appropriately after the apology, remediation, restoration, or compensation has occurred; and/or
- The Practices are used simplistically and in isolation with students whose inappropriate behaviors have complex and interdependent root causes that involve home, school, and/or peer or other factors.
But the biggest problem with individual Restorative Practices is that there is no empirical research that identifies exactly which practices will successfully change what social, emotional, or behavioral student problems.
But Reports Inaccurately Touting Restorative Programs and Practices Continue to be Published
Despite these well-documented results and issues, popular press stories (testimonials) and published technical reports continue to—inaccurately—tout some Restorative Programs and practices.
The biggest problems with the latter are that some of these Reports:
- Are published by National Technical Assistance Centers or other organizations that educators believe that they can trust;
- Do not (fully) report their methodological flaws, and/or how they are subjectively representing their results; and/or
- Are published to promote a political, organizational, or personal agenda to “validate” the Center or organization’s presence, power, or positions (see the Quote headlining this Blog).
A case in point is the July 2021 Research Brief from WestEd, “Can Restorative Practices Bridge Racial Disparities in Schools?” which was recently re-promoted in a national press release.
According to the Brief, “WestEd is a nonpartisan, nonprofit agency that conducts and applies research, develops evidence-based solutions, and provides services and resources in education, human development, and related fields, with the goal of improving outcomes and ensuring equity for individuals from infancy through adulthood.”
WestEd makes millions of dollars per year through its consultation contracts. These contracts include (again) millions of dollars per year in U.S. Department of Education and other federally-funded grants.
Indeed, WestEd currently runs at least the following taxpayer-funded National Support Networks for the U.S. Department of Education: (a) the Regional Education Laboratory-West; (b) Regional Comprehensive Centers in at least 18 states or U.S. territories; (c) The Center for IDEA Fiscal Reporting; (d) the National Center for Systemic (Special Education) Improvement; (e) the Center to Improve Social and Emotional Learning and School Safety; and (f) The National Research and Development Center to Improve Education for Secondary English Learners.
WestEd’s restorative justice work most often is affiliated with the WestEd Justice & Prevention Research Center which also has some funding from the Robert Wood Johnson Foundation.
_ _ _ _ _
WestEd’s Research Brief reports the results of some sophisticated statistical analyses from 838,166 California middle and high school students who completed the California Healthy Kids Survey (CHKS) between the 2013 to 2014 and 2018 to 2019 school years.
Significantly, WestEd has a contract from the California Department of Education both to administer the CHKS to all students across California and to provide CHKS training and technical assistance to school districts statewide.
Relative to the CHKS, the Brief explicitly states that,
“The CHKS includes eight questions about students’ experiences with restorative practices. We used students’ answers to these questions to create a measure, ranging from 1 to 5, that indicates how much exposure each student had to restorative practices.”
The two results from the study were (in bold, large font, highlighted headlines):
- Restorative Practices May Bridge Discipline Disparities and Improve Academic Achievement for All. . . Students with higher levels of exposure to restorative practices evidenced smaller Black-White discipline disparities; and
- Exposure to restorative practices was associated with a higher GPA for all students, Black students, and White students.
[NOTE, above, the use of the conditional words “may bridge” and “was associated with.”]
_ _ _ _ _
The Problem?
Educators wanting or needing to close the discipline and achievement gap between students of color and White students, who are impressed by complex statistical analyses, and/or who trust WestEd and its association with the U.S. Department of Education, may possibly embrace and implement a Restorative Practice Program or the restorative practices cited in the Brief. . . . without reading or fully understanding the flaws and weaknesses of the study, and the deceptiveness of the headlines above.
This—supported by a review of the current Restorative Practice studies—may result not just in a school’s waste of time, resources, commitment, and staff and student involvement, but in the negative outcomes discussed above.
But let’s look at the facts.
As a first step in critiquing the WestEd research, let’s listen to what the authors of the Brief said in the piece:
Limitations. It is important to note that our models are not designed to estimate “causal” effects. In other words, we cannot glean from these models whether exposure to restorative practices causes fewer suspensions or improved GPAs. Instead, we can only say that restorative practices are associated with the aforementioned positive outcomes.
That is, student exposure to greater levels of restorative practices tended to coincide with less discipline exposure, smaller racial discipline disparities, and better academic achievement.
While we controlled for a range of student, parent, and district factors, there were many factors that we did not control for because they were not available in our data.
Thus, based on the data available to us, while it is possible that student exposure to restorative practices does indeed abridge discipline disparities and improve academic achievement, it is also possible that unobserved student-, school-, or community-level characteristics drove both student exposure to restorative practices and student outcomes. Additional research is thus warranted to estimate the causal effect of these practices.
Another critical facet of this research is that we were (intentionally) not identifying the impact of restorative programming (i.e., teachers receiving professional development in restorative practices). Instead, we were attempting to evaluate restorative practices (i.e., students being exposed to teachers who actually, for example, help resolve conflicts or inculcate conflict resolution skills).
Given this “confession” by the authors (with more to come below), there are three primary reasons why this Brief should never have been published and why it cannot be used to validate the utility of the Restorative practices “investigated.”
[In fact—parenthetically—given my 40 years of services on six Editorial Boards for national journals in School Psychology, I can guarantee that this study would never have been published by any of these journals.]
- Research Rejection #1: “Restorative Practices” data. This was NOT a study of Restorative practices; it was a study of what the authors subjectively decided were Restorative practices.
As acknowledged by the authors, all they did was review the 132 and 139 questions, respectively, in current the Middle and High School California Health Kids Survey In-School Core Modules, and pull out eight questions that they decided had a relationship to “Restorative practices.”
Critically, (a) the CHKS did not include these questions because of their relationship to Restorative practices, (b) the authors never objectively validated this relationship, and (c) most educators—if presented these questions without any context or pretense—would simply say that they generically describe the characteristics of a positive and supportive school climate where students and staff get along with each other.
The eight CHKS items evaluated in the WestEd Brief are:
* This school encourages students to feel responsible for how they act.
* This school encourages students to understand how others think and feel.
* This school encourages students to care about how others feel.
* Students are taught that they can control their own behavior.
* This school helps students solve conflicts with one another.
* If I tell a teacher that someone is bullying me, the teacher will do something.
* Teachers show it is important for students of different races to get along.
* The adults in this school respect differences in students.
_ _ _ _ _
- Research Rejection #2: Student Achievement Data. ALL of the data in this study involved student survey responses. Thus, this was a study evaluating students’ perceptions.
No objective data was collected, and none of the students’ perspectives or perceptions were validated using cumulative record, teacher, administrator, or other data.
Indeed, in a footnote in the Brief, the authors conceded,
The CHKS does not ask students to report their grade point average (GPA) directly. Instead, it asks students to indicate which of eight options best fits the grades they received in the last 12 months. Categories include options such as “mostly F’s” and “A’s and B’s.” We adapted these options to create a measure of estimated GPA ranging from 0 (low GPA) to 4 (high GPA).
Given this, the academic achievement results reported in the Brief need to be summarily rejected.
_ _ _ _ _
- Research Rejection #3: Student Suspension Data. Similar to the achievement data, the authors did not analyze objective student suspension data. . . they analyzed student survey reports of suspensions that were never validated.
Indeed, in another footnote, the authors noted:
Suspension rate is captured in the CHKS data by a question that asks whether students have missed school in the past 30 days due to being suspended (0 = “no,” 1 = “yes”). This question only appeared in the 2013/14, 2014/15, 2015/16, and 2016/17 CHKS data.
_ _ _ _ _
While I have no explanation for the authors’ decision to conduct, publish, and disseminate this Brief, I do know that it should be retracted immediately given its flaws, subjectivity, inaccuracy, and potential to do harm in our schools.
And yet, I know that the needed retraction will not occur, and that unsuspecting educators (or those wanting “research” to validate their Restorative Program or practices beliefs) might use this Brief to justify beginning (or maintaining) one or more restorative initiatives.
This clearly will not serve our schools or—especially—our students of color well.
But unfortunately, I have some more bad news. . .
But first, I want to make sure that this entire discussion is focused on what Educators really searching for when they read a Restorative Justice or Program headline or study.
They want:
A field-tested and research-to-practice blueprint on how to teach and increase students’ social, emotional, and behavioral interpersonal, social problem-solving, conflict prevention and resolution, and emotional control, communication, and coping skills
while nurturing, reinforcing, or improving teachers’ (and administrators’) student relationship, classroom management, and challenging behavior response skills and interactions
so that
disproportionate disciplinary actions for students of color (and with disabilities) are eliminated, and students’ challenging behaviors are similarly eliminated and replaced with appropriate, prosocial interactions.
Isn’t this the goal of virtually any Restorative Program/Practices approach?
But—as above—isn’t it curious that some Restorative Program/Practice researchers feel the need to artificially create a restorative practice study out of eight random items on a student survey designed for a totally different purpose?
When Restorative Program Data May Be No Data at All
While the WestEd Brief discussed above was—purportedly—about validating the utility of Restorative practices, I want to revisit one of the most methodologically-sound investigations attempting to validate Restorative Justice Programs.
Conducted between June 2015 and June 2017, this study involved the implementation of the International Institute for Restorative Practices’ (IIRP) SaferSanerSchools Whole-School Change restorative practices program in 22 randomly-selected elementary and middle schools in the Pittsburgh (PA) School District. The implementation schools were matched with 22 randomly-selected Pittsburgh School District schools, and the efficacy of the Restorative Practices Program was independently evaluated by the Rand Corporation.
The SaferSanerSchools Program consisted of the following elements: Affective statements, Restorative questions, Small impromptu conferences, Proactive circles, Responsive circles, Restorative conferences, Fair process, Reintegrative management of shame, Restorative staff community, Restorative approach with families, and the Fundamental hypothesis that human beings are happiest, healthiest, and most likely to make positive changes in their behavior when those in authority do things with them rather than to them or for them.
While the IIRP and Rand Corporation focused on the positive outcomes of the study in the latter’s 2018 report. . . and this was predictably highlighted by the popular press and those looking for these results. . . other more objective organizations noted—after reading and analyzing the report—that the results were mixed.
Indeed, in a January 26, 2019 Blog, I identified some of the most critical negative results reported in the study, stating:
The results of the study indicated that, while the District’s suspension rates had been declining prior to the implementation of the study, the suspension rates in the Restorative Practices schools declined even more than the rates in the Control schools.
In addition, in the Restorative Practices schools (a) alternative school placements decreased; (b) students were less likely to be suspended multiple times; (c) disparities in suspension rates between African-American (vs. Caucasian), and low-income (vs. higher-income) students, respectively, decreased; and (d) suspension rates for female students declined.
However, more in-depth analyses revealed that:
- While suspension rates in the Restorative Practices schools declined by 36% during the two-year study, suspension rates in the Control schools also declined 18% during the same time period.
- The overall suspension results were driven by lower rates in the Restorative Practices elementary schools.
- Fewer suspensions were not found in the Restorative Practices Middle schools (Grades 6 to 8).
- Fewer suspensions were not found for male students or students with disabilities.
- There were no reductions in student arrests, or for incidents of violence or weapons violations.
- In the Restorative Practices Middle schools, academic outcomes actually worsened when compared with the Control schools.
- Survey results from staff in the Restorative Practices schools indicated that they did not think the IIRP program was affecting student behavior. They did, however, report that their relationships with students had improved because of program involvement.
- The Program was supported by outside grant funding, its implementation involved significant amounts of time, resources, and staff involvement, and there were concerns about the Program’s ability to be sustained once the funding had lapsed.
“New Rand Corporation Study Finds Restorative Practices Produce Mixed and Underwhelming Results: But Some Publications are “Spinning” the Outcomes and Twisting these Results”]
_ _ _ _ _
So. . . beyond the mixed results, where is the “bad news”?
Last month (January 18, 2022), the Pennsylvania Chapter of the American Civil Liberties Union (ACLU-P) published a research report, “Student Arrests in Allegheny County Schools: The Need for Transparency and Accountability.”
Analyzing 2017-2018 school year data reported by Allegheny County’s (PA) 43 public school districts (which includes the Pittsburgh School District), ACLU-P staff examined student arrest patterns, referrals to police, juvenile justice involvement and the use of “summary citations” (tickets issued to students by police) when students were attending school. The Report also compared juvenile justice system data for Allegheny County to the data that schools provided to federal and state education departments.
In general, the Report found that student arrests or referrals to police occurred more often than were documented by schools. The report also found that students in Allegheny County were more likely to be arrested at school than students elsewhere in Pennsylvania. Critically, the pattern of arrests and referrals fell disproportionately on Black students and students with disabilities.
Relative to the first statement above, the Report specifically stated that,
Juvenile arrest data suggested that some districts in the county underreported student arrests. For example, Pittsburgh Public Schools (PPS) incorrectly reported zero arrests to the U.S. Department of Education for the 2017-18 school year. PPS subsequently produced non-zero arrest data for that year, but it has not corrected publicly posted data. Even the revised figures undercount the true number of student arrests. And PPS is not alone in undercounting arrests.
NOTE: The Pittsburgh Public School data are from the school year immediately after the completion of the IIRP and Rand Corporation’s Restorative Program study.
Mark Keierleber from the 74 Million (January 19, 2022) commented on this news as follows:
Zero. That’s how many Pittsburgh students were arrested at school during the 2017-18 academic year, according to the most recent federal education data. Certainly that’d be something for the 20,000-student district to celebrate, but there’s just one problem.
It isn’t true.
In fact, county juvenile court data tell a completely different story — one in which police actually carried out nearly 500 arrests in Pittsburgh schools that year, disproportionately against Black students and children with disabilities, often for minor offenses. That’s a key revelation in a study released Wednesday by the American Civil Liberties Union of Pennsylvania. . .
“The conversation is really about the harms to Black children” Jordan said, and while the Pittsburgh district “does not want to be seen as anti-Black or insensitive to the concerns of Black parents,” leaders have failed to adopt sufficient student interventions that don’t involve the criminal justice system, he said. . .
The Pittsburgh district attributed the underreporting of its data to the federal government to an error. After taking heat for racial disparities in arrests, school leaders hired a consulting group in 2020 to study arrest data and created a task force focused on improving school safety. . .
By underreporting campus arrests, however, districts could give parents an inaccurate picture of campus safety and the effects of school-based police on the young people who interact with them. . .
“The harms of having police in schools are much more widespread than districts report,” said Jordan, who called Allegheny County a “hot spot” nationally for youth arrests. During the 2018-19 school year, Pittsburgh students were arrested at nearly eight times the state rate, the ACLU found...
Local activists have been sounding the alarm for several years. In a 2020 report, the local Black Girls Equity Alliance found Pittsburgh school district police were the single largest source of juvenile justice referrals for Black girls, accounting for a third of all referrals countywide. Black girls in Allegheny County were referred to the juvenile justice system at a rate 10 times higher than white girls, researchers found.
“These really high rates of referrals of Black youth are not because there’s a problem with young people. It’s that there’s a problem with the adults who are responding to them and with the systems we have in place,” she said...
During the 2017-18 school year, the Pittsburgh district reported 86 arrests and 395 law enforcement referrals to the state education department. That same school year, the district reported zero arrests to the U.S. Department of Education while the county juvenile court tallied 499 school-related arrests.
“For a district in which the arrest rates have been high for a very long time, why should they be so inaccurate,” Jordan asked. “I can’t speak to intentionality, but they are in the position to know that what they have put out to the public is inaccurate. They are well aware of that.”
“The numbers speak from themselves,” Dempsey said. “There’s obviously bias in decision-making from people in power who have the ability to decide whether to either charge these individuals or not.”
_ _ _ _ _
So what are the implications here?
At least two implications come to mind:
- If the ACLU-P found data glitches in the Pittsburgh School District’s 2017-2018 discipline data, how can we be sure that there were not similar errors in the IIRP/Rand Corporation’s Restorative Justice Program data from June 2015 through June 2017?
Moreover, if the Restorative Justice Program data are inaccurate and if they under-reported the number of discipline events involving students of color, then how many of the “positive” program results found in the Rand Corporation report are now null and void?
_ _ _ _ _
- Even if the mixed results from the Rand Corporation report stand, it would appear that any impact from the IIRP’s SaferSanerSchools Restorative Justice Program in the Pittsburgh School District virtually disappeared during the Summer of 2017. . . as the alarming student arrest figures can attest.
This does not speak well for the long-term impact of the SaferSanerSchools Program, the sustainability of the Program, and/or the Program’s effects on—as cited by Keierleber—the “obvious bias in decision-making from people in power who have the ability to decide whether to either charge these individuals or not.”
Summary
One of the suggested primary themes to guide educators’ practice in their schools—relative to maximizing the academic, social, emotional, and behavioral outcomes and proficiencies of all students—is that:
We need to be guided by scientifically-proven multi-tiered PRACTICES (services, supports, strategies, and/or interventions) that are matched to the specific demographics, conditions, and circumstances of our students, staff, and schools
and NOT by PROGRAMS that lack these characteristics, that cannot demonstrate their causal efficacy, and that cannot be sustained.
In this context, and from the first part of this Blog, we need to be wary of over-zealous (or mis-guided) researchers who seem to make more out of some good (CHKS) survey data than the data allow.
Moreover, from the second part of this Blog, we need to be confident that the data reported by researchers, program evaluators, and schools are consistently well-chosen, well-collected, well-analyzed, and that they are reliable and valid.
There are no Restorative Justice Programs that have been found—in one or more well-designed, methodologically-sound longitudinal studies—to produce the results that the program developers purport or desire.
Moreover, beyond the fact that many restorative practices are simply effective school and schooling strategies, no research is available to tell us which practices work, with what student behavior problems, in what combinations or sequences, and why.
And yet, we are still getting reports like the two detailed in this Blog.
And worse, we still have thousands of schools (often encouraged by their State Departments of Education) initiating Restorative Justice Programs or implementing restorative practices that will likely fail, and may actually—as in Pittsburgh—make the problems worse.
_ _ _ _ _
As always, I appreciate those of you who read these Blogs, and I hope they are useful to you.
If you want to read some research-to-practice solutions for the disproportionate disciplinary actions taken against students of color and with disabilities, go to our three-part Blog Series that began on July 31, 2021.
The Critical Common Sense Components Needed to Eliminate Disproportionate School Discipline Referrals and Suspensions for Students of Color: This is NOT About Critical Race Theory (But We Discuss It)]
Meanwhile, if I can help you, your colleagues, your school, your district, or those in your professional setting to address your students’ social, emotional, or behavioral challenges, send me an email and let’s set up a time to talk. This first consultation hour is on the house.
Best,
