Committing to Educational Excellence by Learning from Hattie’s and SEL’s Limitations
Dear Colleagues,
Introduction
The holidays. . . the New Year. . . a time of reflection. . . a time of hope and joy and renewal.
I would love to say I am feeling nostalgic. But. . . I’m not.
On a professional level, I’m dismayed. I’m disappointed. I’m determined. And I know there is a lot of work to do to improve our schools in 2019.
_ _ _ _ _
In preparing this piece, I read through all of the Blogs that I wrote this year. I did this to “Review 2018” because—when you are preparing and writing two major messages each month, while maintaining a national consulting business (with almost 200 days per year “on the road”)—you tend to lose sight of what happened in January. . . never mind September or October.
My Blog review revealed the following themes:
* Theme 1: Choosing High-Success Initiatives. Here, we discussed the importance of schools doing their own science-to-research “due diligence” so that they adopt and implement defensible and high-probability-of-success initiatives and programs on behalf of their students and staff.
We also critically reviewed the research of John Hattie—detailing the strengths and limitations of meta-analytic studies, and emphasizing that schools cannot take Hattie’s effect sizes and move directly to implementation. Indeed, because meta-analysis pools many separate research studies together, these studies often have different methods, procedures, strategies, and implementation sequences.
Thus, schools would not know exactly what to implement without critically evaluating the separate studies.
* Theme 2: The Selling of Social-Emotional Learning (SEL). As a specific example of the Theme above, we encouraged schools to critically look at the history and foundation of the Social-Emotional Learning (SEL) “movement” (including “mindfulness” practices) so that they understand its political history and motivation, recognize the flaws in its research and practice, and “step back” to reassess how to effectively improve students’ social, emotional, and behavioral skills and self-management abilities.
* Theme 3: Preventing School Shootings. Here, we suggested that schools need to go “Back to the Future” by reviewing past recommendations from previous years’ school shooting analyses. . . when re-evaluating their current school safety systems and approaches. Clearly, this is especially important given the rash of school shootings during 2018.
* Theme 4: School Discipline and Disproportionality. Here, we reviewed the importance of proactive, scientifically-based, and multi-tiered school discipline approaches, as well as how to realistically, comprehensively, and pragmatically address the issue of disproportionality. . . especially with minority students and students with disabilities.
_ _ _ _ _ _ _ _ _ _
Theme 1: Choosing High-Success School Initiatives
My very first Blog this year (January 13, 2018) focused on what we know about school improvement—based on evaluations from the “No Child Left Behind” (NCLB) years, and what districts and schools need to know about school improvement—from the science-to-practice approaches embedded in strategic planning.
From an NCLB perspective, published studies consistently conclude that there are lots of school improvement strategies, but most educators do not know how to comprehensively analyze their school’s current strengths, weaknesses, and gaps so that they can strategically and systematically implement the most effective and efficient strategies that will build their school’s capacity—resulting in sustained student outcomes.
We added a critical point to this first conclusion—that school improvement is contextual.
That is, some schools want to go from “great to greater.” Some schools from “good to great.” And some schools need to go from a Targeted or Comprehensive Support and Improvement level, respectfully, to a point where they are simply providing a consistent, foundationally sound level of good instruction.
In addition, we emphasized that, in order for continuous school improvement and (especially) school turn-around to succeed, it needs to be done at each involved school and district site using coordinated and sustained activities that include:
- Ongoing local needs assessments and strategic planning science-to-practice processes;
- Local resource analyses and capacity-strengthening science-to-practice processes; and
- Local and on-site organizational, staff development, consultation, and technical assistance science-to-practice processes.
These “keys to success” clearly require professionals both employed at each school site, and in- or out-of-district consultants—all with the shared ability to use the strategic planning processes cited above to select the best services, supports, strategies, and interventions at the district, school, staff, and student levels to facilitate ongoing and sustained success.
_ _ _ _ _
Below are the 2018 Blogs written in this theme area. . . with their titles, dates of publication, and web-links to the original message.
[CLICK on the Date below to link to the Original Blog]
January 13, 2018 Every School is in “School Improvement” Every Year: Preparing for ESEA/ESSA–What Effective Schools Do to Continuously Improve . . . and What Ineffective Schools Need to do to Significantly Improve [Part I of II]
January 28, 2018 How Strategic Planning and Organizational Development is Done by Every School . . . Every Year: An Introduction to Successful School-based Strategic Planning Science-to-Practice [Part II of II]
June 26, 2018 Learning from Another Gates Failure: It’s Not Just the Money–It’s What You Accomplish with It. How to Spend ESEA’s Title IV Money Wisely
July 21, 2018 Hattie Haters and Lovers: Both Still Miss the Effective Implementation that Practitioners Need. Critical Questions to Ask your “Hattie Consultant” Before You Sign the Contract
August 4, 2018 School Improvement, Strategic Planning, ESEA, and Multi-Tiered Services: An Anthology of Previous Blogs. Integrating Successful Research-to-Practice Strategies into the New School Year (Part I of II)
November 25, 2018 It’s Not Too Late to Change: The School Year’s Not Even Half Over. Why Schools Fail to Act When their Students Fail
December 8, 2018 Reconsidering What Effective High Schools Do, and What Failing High Schools Miss: Credit Recovery Programs Should be Strategic, Selective, Student-Focused, and Not the Only Game in Town
_ _ _ _ _
The Take-Aways
Relative to continuous school improvement and—especially—improvement at the Targeted or Comprehensive Support and Improvement levels, respectfully, our Blogs looked at recent national reports and other valid and previously-established science-to-practice strategies that create a blueprint for school planning and effectiveness.
Our School Improvement Blueprint included the following components:
School Vision
Establish and Communicate a Clear Vision
Help Staff Understand and Embrace the Need for Change
Improvement Goals
Prioritize Goals and Focus Areas
Make Action Plans Based on Data
Identify and Achieve a Few Early Wins
Reduce Time Focused on Nonessentials
Data-based Decision-Making
Establish the Expectations for a Data Culture
Adjust Instructional Practice through Visible Data
Use Data Continually to Solve Problems
Establishing a Culture of Change
Focus on Successful Tactics, Discontinue Unsuccessful Ones
Break Rules and Norms, Take New Action
Change Systems and Structures
Effective Teachers and Leaders
Make Necessary Replacements
Attract, Select, and Retain Top Talent
Build and Lead a Team of Leaders
Ensure Ongoing Professional Growth Opportunities
Instructional Excellence
Align Instruction to Assessments and Standards
Monitor and Improve Instructional Quality
Develop and Deploy a Team of Instructional Leaders
Strategic Partnerships
Gain Support of Key Influencers
Enlist Partner Organizations
_ _ _ _ _
Critically, and as emphasized above, this blueprint should not be used as a static, one-size-fits-all menu. Instead, needs and status assessments, resource analyses and coordination, and strategic planning and organizational development strategies are required to individualize the process for each district and school.
For districts or schools in significant need of improvement, two questions are essential here:
- With all that a school in improvement status needs to do, which of the possible strategies are the immediate, high-hit strategies that will begin the improvement process in a timely way?
- Once these high-hit strategies are identified; exactly what is the training, who and where are the targets; and what are the resources, implementation steps, and short- and long-term outcomes needed such that improvement begins, is established, and can be maintained over time?
_ _ _ _ _
In the end in this area, here are the Blog take-aways:
Take-Away #1. The Elementary and Secondary Education Act (ESEA/ESSA) has virtually replaced the No Child Left Behind term “scientifically-based” with the term “evidence-based”—providing a specific statutory definition.
Take-Away #2. Educators need to recognize that they must independently validate any program, intervention, or strategy that claims it is “research-based”—as the research could be sound, unsound, or non-existent.
Take-Away #3. Even when research validly supports a specific program, intervention, or strategy, educators still need to validate that (a) it is applicable to the students, staff, schools, or situations that they want to change/affect, and (b) it can be realistically implemented “in the real world” (as opposed to a controlled or “laboratory” setting).
Take-Away #4. As but one example: John Hattie’s research significantly contributes to educational decision-making. . . but educators need to fully understand the decision rules and outcomes inherent in his meta-meta-analytic methods and outcomes.
[See an entire Blog message devoted to Hattie’s research, and how to evaluate and use it pragmatically in a school or district: CLICK HERE.]
Take-Away #5. Even when Hattie’s research provides a programmatic, intervention, or strategy-related “recommendation,” educators need to understand that (a) meta-analytic research often pools research focusing on the same approach, but using different methodologies; and (b) it is effective methodology, implemented with fidelity, that ultimately determines student, staff, and/or situational success.
Take-Away #6. Meta-analytic research typically follows some common steps. These involve:
- Identifying the program, strategy, or intervention to be studied
- Completing a literature search of relevant research studies
- Deciding on the selection criteria that will be used to include an individual study’s empirical results
- Pulling out the relevant data from each study, and running the statistical analyses
- Reporting and interpreting the meta-analytic results
As with all research, there are a number of subjective decisions embedded in meta-analytic research, and thus, there are good and bad meta-analytic studies.
Take-Away #7. Educational leaders cannot assume that “all research is good because it is published,” and they cannot assume that even “good” meta-analytic research is applicable to their communities, schools, staff, and students.
And so, educational leaders need to independently evaluate the results of any reported meta-analytic research—including research discussed by Hattie—before accepting the results.
Take-Away #8. Among the questions that leaders should ask when reviewing (or when told about the results from) meta-analytic studies are the following:
- Do the programs, strategies, or interventions chosen for investigation use similar implementation steps or protocols?
- Are the variables investigated, by a meta-analytic study, variables that are causally- versus correlationally-related to student learning, and can they be taught to a parent, teacher, or administrator?
- In conducting the literature review, did the researchers consider (and control for) the potential of a “publication bias?”
- What were the selection criteria used by the author of the meta-analysis to determine which individual studies would be included in the analysis, and were these criteria reliably and validly applied?
- Were the best statistical methods used in the meta-analysis? Did one or two large-scale or large-effect studies outweigh the results of other small-scale, small-participant studies that also were included? Did the researcher’s conclusions match the actual statistical results from the meta-analysis?
_ _ _ _ _ _ _ _ _ _
Theme 2: The Selling of Social-Emotional Learning
One of the most notable examples of Theme #1 above is the SEL (Social-Emotional Learning) movement as politically powered by CASEL (the Collaborative for Academic, Social, and Emotional Learning).
While recognizing that most schools nationwide are doing “something” that they call “SEL,” this year’s Blog messages provided extensive information on (a) CASEL’s political and foundation-driven agenda, (b) the flaws and limitations in the research that it uses as a rationale for that agenda, and (c) the research-to-practice components of an SEL model that is focused on measurable and developmentally-sensitive multi-tiered social, emotional, and behavioral student skills.
In the context of Theme #1, districts and schools are encouraged to take a “step back” off the SEL bandwagon, to critically review the research-to-practice multi-tiered components, and to reconfigure the strategies, resources, timelines, and training needed to effectively improve their student, staff, and school “return-on-investment.”
Below are the 2018 Blogs written in this theme area. . . with their titles, dates of publication, and web-links to the original message.
[CLICK on the Date below to link to the Original Blog]
February 10, 2018 The Folly and Frustration of Evaluating Schools and Staff Based on the Progress of Students with Significant Social, Emotional, and Behavioral Challenges: Understanding the Student, Home, and Community Factors that Impact Challenging Students
June 2, 2018 Making Mountains Out of Molehills: Mindfulness and Growth Mindsets. Critical Research Questions the Impact of Both
October 13, 2018 Social-Emotional Learning: Education’s Newest Bandwagon. . . and the History of How We Got There (Part I). Why Most Schools are not Implementing Scientifically-Sound Practices—Wasting Time and Resources
November 10, 2018 The SEL-ing of Social-Emotional Learning: Education’s Newest Bandwagon. . . Science-to-Practice Goals, Flaws, and Cautions (Part II). Why Schools Need to Re-Think, Re-Evaluate, Re-Load, and Re-Boot
_ _ _ _ _
The Take-Aways
All students need to learn and demonstrate—at an appropriate developmental level—effective interpersonal, social problem-solving, conflict prevention and resolution, and emotional control and coping skills. In the classroom, these skills are essential to maximizing their academic engagement and achievement, as well as their ability to collaborate and learn in cooperative and project-based learning groups.
The “Good News” is that this is increasingly recognized across our educational communities.
The “Bad News” is that many schools are targeting (often due to CASEL’s advocacy), SEL goals and targets that involve constructs (instead of skills and behaviors) that are open to interpretation (hence, they are unreliable) and, hence, that cannot be measured or measured validly.
The additional “Bad News” is that “SEL” has been “validated” by the popular press . . . using testimonials, “research” that would be rejected by the Editorial Board of virtually any professional publication, and data that will never demonstrate a causal relationship between school-based activities and student-based outcomes.
Here are the Blog take-aways:
Take-Away #1. Beyond the points immediately above, most of the popular press is “positive press” because “Why would a school or district send out (and why would a press outlet publish) a press release announcing that its SEL program has failed?”
Take-Away #2. Many districts and schools are implementing or purchasing “SEL programs and curricula” without independently and objectively evaluating (a) their research to determine if they are “ready” for field-based implementation; (b) whether they “fit” the demographics, students, and needs of their schools; and (c) whether they have a high probability of positively impacting the social, emotional, and behavioral student outcomes that they seek.
Take-Aways #3. The biggest concerns with CASEL’s approach to SEL includes:
- Saying that districts and schools should have broad discretion in deciding what their SEL initiative will target and look like.
[This runs the risk that schools will implement unproven or the easiest-to-employ practices that are wasteful, ineffective, or counterproductive.]
_ _ _ _ _
- Missing the importance of adapting SEL initiatives for students from different cultural, racial, language, socio-economic, or family constellation backgrounds; with different gender or psychosexual orientations; or with one or more disabilities.
[This may result in maintaining or increasing the SEL skill gaps between these students and others who may already have these essential skills.]
_ _ _ _ _
- Missing the need for multi-tiered services, supports, strategies, and interventions for at-risk, underachieving, underperforming, unresponsive, and unsuccessful students, as well as those with significant social, emotional, behavioral, and/or mental health challenges.
[This potentially denies these students the opportunities to access the SEL instruction and, thus, to successfully learn and master the embedded skills.]
_ _ _ _ _
- The three foundational SEL meta-analyses, published by CASEL principals and used to support its movement in the schools, have significant methodological and empirical flaws. For example, the meta-analyses did not use objective criteria to choose the individual studies included in the meta-analyses, and a significant number of studies were not published in refereed journals (e.g., research reports and dissertations), and/or did not involve schools in the United States (creating possible cross-cultural confusion).
_ _ _ _ _
- In addition, CASEL has never acknowledged that it is virtually impossible to apply meta-analytic results to effective field-based practice. This is because, for example, the different studies in their meta-analyses used different SEL programs, approaches, and/or strategies. Thus, there is no way to know exactly which programs, approaches, and/or strategies are the “right” ones for a specific district, school, or set of student conditions.
_ _ _ _ _
- CASEL’s “School Theory of Action framework” is actually no different than any sound strategic planning approach applied to school and schooling processes.
_ _ _ _ _
- CASEL’s just-released document on SEL Competency Assessments recognizes that “few SEL assessments have gone through the validation process typical of most large-scale academic assessments” . . . that “most SEL assessments were not specifically developed for the purpose of comparing schools, and (that) little research exists to determine whether currently available assessments have the precision necessary to make such comparisons.”
This document does little to close this assessment gap, but it does call into question (once again) its meta-analyses (see above) that may have included studies that used unreliable or invalid assessment tools—resulting in unreliable or invalid results and conclusions.
_ _ _ _ _
Take-Away #4. If you want to teach social-emotional learning skills, you need to teach specific behaviorally-based interpersonal, social problem-solving, conflict prevention and resolution, and emotional control and coping skills.
This instruction should use the social learning theory steps of (a) teach the skills and scripts, (b) model, (c) role play, (d) provide performance feedback, and (e) transfer the training. This instruction also should teach students how to perform these skills under conditions of emotionality, which maintaining positive expectations for success.
Finally, as part of the instructional process, students need to learn, master, and be able to apply these skills in a timely way to different situations.
_ _ _ _ _
In summary, relative to SEL, the Blog messages emphasized the fact that students need to learn and demonstrate:
Self-control—when experiencing emotional conditions;
Cognitive, or attributional, control—so that their thoughts, beliefs, attitudes, and expectations support and motivate prosocial behavior; and the
Verbal, non-verbal, and physical behaviors needed to demonstrate their interpersonal, social problem-solving, conflict prevention and resolution, and emotional control and coping skills.
The science-to-practice learning process should result in students’ social, emotional, cognitive, and behavioral self-management and self-efficacy.
These, once again, are the outcomes that every school-based Social-Emotional Learning (SEL) initiative and/or program in this country should target for all students.
But these initiatives or programs also need to recognize that not all students learn the same way, and that social, emotional, and behavior instruction needs to be adapted for students (a) from different cultural, racial, language, socio-economic, or family constellation backgrounds; (b) with different gender or psychosexual orientations; or (c) with one or more of the thirteen different disabilities recognized in federal law (i.e., IDEA).
And these initiatives or programs especially need to consciously identify and integrate the multi-tiered services, supports, strategies, and interventions required by students who are at-risk, underachieving, underperforming, unresponsive, and unsuccessful. . . in addition to the students who are demonstrating frequent or intense social, emotional, or behavioral challenges.
_ _ _ _ _ _ _ _ _ _
Summary
Obviously, my primary goal in writing these Blogs is to help districts and schools to maximize the academic and social, emotional, and behavioral skills and competencies of all students.
In a multi-tiered context, this means that some students will need remediation, accommodation, and/or modification services, supports, and strategies when struggling academically or presenting with behavioral challenges. In addition, other students will need strategic or intensive interventions as identified through data-based functional assessment problem-solving processes.
But another goal is to add a science-to-practice perspective to some of the national reports, approaches, and beliefs that are published and accepted by others. . . sometimes without a full understanding of their history or implications, and sometimes based simply on the perceived “expertise” of the author or the organization sponsoring the work.
Thus, a final goal is to help educators to “stop and think” and “take a step back” from the premature acceptance of a framework or program that either will not work with their students or will not work with any students.
Time and resources are precious commodities. When it comes to our students, staff, and schools, we all need to make sure that these commodities are used well, and that they have a high “return on investment.” This means that—before implementation—we have validated that they have a high probability of success, that they can be and are implemented with integrity and the correct intensity, and that we are sensitively evaluating their short- and long-term outcomes.
_ _ _ _ _
In Part II (coming in approximately two weeks), we will discuss and analyze the second set of 2018 themes:
* Theme 3: Preventing School Shootings. Here, we will encourage schools to go “Back to the Future” by reviewing past recommendations from previous years’ school shooting analyses when re-evaluating their current school safety systems and approaches. Clearly, this is especially important given the rash of school shootings during 2018.
This discussion also will critically review—in the most depoliticized way possible—the Federal Commission on School Safety’s Final Report released less than four weeks ago on December 18, 2018.
* Theme 4: School Discipline and Disproportionality. Here, we will review the importance of proactive, scientifically-based, and multi-tiered school discipline approaches, as well as how to realistically, comprehensively, and pragmatically address the issue of disproportionality. . . especially with minority students and students with disabilities.
This theme will discuss the implications of the U.S. Department of Education’s December 21, 2018 rescission of the Obama-era guidance aimed at reducing racial discrimination when students are disciplined. This was done officially by Secretary DeVos just three days after the release of the Federal Commission on School Safety’s Final Report which included this in its recommendations.
Meanwhile, I hope that this information is useful to you. Believe it or not, if you would like to discuss anything on an individual district, school, or agency level, I am (still) providing free one-hour conference calls even during this holiday season.
Speaking of which, I hope that your Holidays were filled with happiness and joy. Please accept my best wishes for the upcoming New Year !!!
Best,
Howie
