Posts tagged accountability
Warnings on SEL and Accountability

Social Emotional Learning (SEL) continues to spread as we deepen our understanding of what really contributes to youth success. As it grows, there are more data collection tools and reports. That’s great for sparking conversations about how we can support students, BUT, there are reasons to be cautious:

  • The most ubiquitous measure of SEL is currently self-reported surveys. Surveys have 99 problems. Including:

    • Reference bias. With surveys, we’re trusting young people to grade themselves in their own mindsets and beliefs. Like this:

      Survey: Do you care about other people?

      Student: Sure do!

      Actual Person: Can you hold the door for me?

      Student: Hold it yourself!

      Would we also ask them to grade themselves in Math? No, because a student’s perception of their own skills is subject to reference bias. A student might report being good at math because he is the best in his class, but in another school he would be below average. I worked with a group of programs that found students who were lowest performing academically actually rated themselves highly in Academic Self-Efficacy. The experts agree that reference bias can “produce results opposite of the truth.” The Learning Policy Institute recommends NOT holding schools accountable for self-report survey data due to reference bias and because SEL surveys are relatively new and not designed for accountability purposes.

    • Surveys are administered and taken by humans. And humans are subject to a lot of factors that might impact results: weather, literacy, mood, relationships, understanding of the survey purpose, etc. In my experience working with dozens of programs on survey administration, the results don’t always turn out the way you expected, and it’s not necessarily a reflection of your work. Which is why I always hear skepticism from staff when looking at survey results.

      • Some issues highlighted by the Student Success Network:

        • How the survey is framed: is it personal, genuine, and does it make sense? Students may not understand or pay attention to the questions.

        • How the survey is perceived based on norms and culture. Students may give the right answer rather than the honest answer.

    • Other administration considerations:

      • The survey is not given to students who don’t show up (skewing the data)

      • The survey is given right before a state math test vs. during a happy graduation ceremony.

  • Will accountability help? Some organizations are in the early stages of learning about and institutionalizing SEL. Before we grade staff on their ability to move the needle, do they know what they’re being graded on? Have they been trained? Does the organization have supports in place for them to develop practices? If we use the data to identify weak spots - how will that help?

I believe non-profits need to be held accountable, and SEL is a key focus of our work. Surveys can provide thought-provoking data, and we’ve seen important connections between SEL growth and achievement. We still need to be very careful about the messaging and administration plan before holding people accountable for survey results. I’ll try to have a growth mindset about it, but we’re not there…yet. Our measures are not sharp enough to say to funders: Don’t give us money if we fail to move the needle on this survey.

Dear NYC Transfer School Principals

Dear Principals of NYC Transfer High Schools,

I know it has been a difficult year or two trying to figure out what new state accountability standards would mean for your school. You’ve engaged parents and students, written letters, attended meetings - all trying to make the case that you INTENTIONALLY serve students who are unlikely to graduate, and maybe whether or not 67% of them graduate (YES or NO) in 6 years is not the best measure of your school’s ability to serve them.

Well, now we have some answers from the state. One fifth of transfer schools need improvement. Students fell off track at other schools, and now you have ‘em, so there’s your problem. You can see clearly from the state grad rates that don’t match the city grad rates at all, that you performed better or worse than you thought maybe you did for some cohort. And now we’ve also learned how we are going to get off or stay off the state’s list:

1) Don’t accept students you were designed to accept. If you accept a student who only has a few credits and didn’t pass any Regents in their first or second year, well, they might not graduate. They probably have some challenges in their lives, and they will hurt your numbers. If they have financial or family responsibilities, have an difficulty learning in a traditional setting, have children, were suspended in the past, have mental health issues, are homeless, or experienced any kind of trauma, they are what we now call “TOO risky” and you should refer them to the nearest dropout center.

2) Test test test. If a student took the ELA Regents six times - failing each time until they finally reached a 65 - make them take it again and again and again until they get a Level 4 score. That will help with your Auto Appeal. What does that mean? It’s confusing. But higher is better for keeping your school open. Better yet, refer to my first tip by accepting students who already passed the Regents with high scores, because whether they took the test at your school or at their previous schools is irrelevant to holding your school accountable.

3) Get that paper! At the end of the day, the diploma is what matters. Unless a student graduates in their seventh year of high school - then it doesn’t matter at all. As research shows, if they graduate high school (in 6 years), they are basically all set for life. So, if you were looking at other outcomes, like growth in reading and math skills, increased engagement and attendance in school, improved social or emotional mindsets and behaviors, or credit accumulation (other silly ideas here), don’t bother. Make it rain credits.

If you have additional questions about your data, see the link here. Oh, sorry, the state forgot to create a link to explain all this. However, if you read Greek, try the State’s Educator Guide.

Hope this was helpful!


P.S. This letter was my therapeutic response of turning rage into humor. Transfer school educators - you are my heroes, and I hope you can stay focused on doing what you do best.