Warnings on SEL and Accountability

Social Emotional Learning (SEL) continues to spread as we deepen our understanding of what really contributes to youth success. As it grows, there are more data collection tools and reports. That’s great for sparking conversations about how we can support students, BUT, there are reasons to be cautious:

  • The most ubiquitous measure of SEL is currently self-reported surveys. Surveys have 99 problems. Including:

    • Reference bias. With surveys, we’re trusting young people to grade themselves in their own mindsets and beliefs. Like this:

      Survey: Do you care about other people?

      Student: Sure do!

      Actual Person: Can you hold the door for me?

      Student: Hold it yourself!

      Would we also ask them to grade themselves in Math? No, because a student’s perception of their own skills is subject to reference bias. A student might report being good at math because he is the best in his class, but in another school he would be below average. I worked with a group of programs that found students who were lowest performing academically actually rated themselves highly in Academic Self-Efficacy. The experts agree that reference bias can “produce results opposite of the truth.” The Learning Policy Institute recommends NOT holding schools accountable for self-report survey data due to reference bias and because SEL surveys are relatively new and not designed for accountability purposes.

    • Surveys are administered and taken by humans. And humans are subject to a lot of factors that might impact results: weather, literacy, mood, relationships, understanding of the survey purpose, etc. In my experience working with dozens of programs on survey administration, the results don’t always turn out the way you expected, and it’s not necessarily a reflection of your work. Which is why I always hear skepticism from staff when looking at survey results.

      • Some issues highlighted by the Student Success Network:

        • How the survey is framed: is it personal, genuine, and does it make sense? Students may not understand or pay attention to the questions.

        • How the survey is perceived based on norms and culture. Students may give the right answer rather than the honest answer.

    • Other administration considerations:

      • The survey is not given to students who don’t show up (skewing the data)

      • The survey is given right before a state math test vs. during a happy graduation ceremony.

  • Will accountability help? Some organizations are in the early stages of learning about and institutionalizing SEL. Before we grade staff on their ability to move the needle, do they know what they’re being graded on? Have they been trained? Does the organization have supports in place for them to develop practices? If we use the data to identify weak spots - how will that help?

I believe non-profits need to be held accountable, and SEL is a key focus of our work. Surveys can provide thought-provoking data, and we’ve seen important connections between SEL growth and achievement. We still need to be very careful about the messaging and administration plan before holding people accountable for survey results. I’ll try to have a growth mindset about it, but we’re not there…yet. Our measures are not sharp enough to say to funders: Don’t give us money if we fail to move the needle on this survey.