How Data Helps You Know More About The Kids You Know

Why would staff in a small school or program do a needs and assets assessment? A counselor already feels like they know their young people better than any survey or database. They are probably right. Here’s why we should do it anyway:

  • Prioritizing Resources and Advocating: In a program that has 160 students on four caseloads, each counselor may have a few students with mental health needs. They are addressing the needs (or not) on an individual basis. However, if an assessment reveals that a few kids on each caseload is actually 10% of the program, there is now motivation to identify partners, train staff, talk to funders, etc. One alternative school I worked with administered a survey that revealed ONE THIRD of students reported being sexually assaulted. While I may not recommend giving a survey that asks such intimate questions, WOW does that give some insight into what supports students need.

  • Programming to Build Upon Strengths and Interests: A key piece of any youth development program is getting students to learn more about themselves. As they share these revelations with us, where does it go? Despite the lack of advanced degrees, young people are most qualified to design programs that work for them. And we should collect data on their dreams, learning goals, desired experiences, strengths and growth areas, hobbies, etc. to inform program design. Don’t forget the assets, even if they seem less urgent. The assets give us the best clues as to how we can engage students.

I’m still trying to figure out needs and assets assessments, but here’s what I’m learning:

  • They can be ongoing. You don’t need to know all the needs and assets from a survey administered on Day 1. And sometimes, we must build relationships first to understand what questions to ask. If you are planning trips or resource fairs do it collaboratively with youth who can best identify their interests and needs. Photo voice is a great activity for this. Focus groups can be excellent. Community schools in NYC also have family forums each spring where they incorporate activities designed to get feedback into fun barbecues or carnivals. Assessments should be important, cyclical moments where we collect data on the bigger picture, but the assessment mindset can be applied daily.

  • They can be separate. In one school, we had counselors fill out a basic spreadsheet with each student listed and 12 columns of “Needs” because we realized we didn’t have a high-level view of many needs we had identified at the student-level (substance abuse, work conflicts, family responsibilities, etc.). With a “Yes”, “No”, or “Unknown” response, we got to see that over 20% of students (as identified by their counselor) have a mental health issue. By looking at the needs along with academic data, we found that our attendance problem was also a WORK CONFLICT problem. All 18 students with a work conflict were severely chronically absent. How can the attendance team address THAT?

  • They can be irresponsible. If we ask students and families what they need, and they tell us, we have a responsibility to respond to the best of our ability. And, we should be cognizant of how someone might feel when answering questions about his or her home, family, or trauma. I work with a program where we’ve heard anecdotally about youth experiencing trauma, and we considered administering the Adverse Childhood Experiences (ACES) Questionnaire to learn more and help advocate for resources. However, without mental health specialists on site and a comprehensive plan to avoid triggering young people, we decided to hold off.

Overall, just because staff are empathetic, curious, active listeners, does not necessarily translate to program design. The better we know our young people, at every level of an organization, the better they are served.

A few resources:

Alison HolsteinComment
Graduation AND Chronic Absenteeism Up in NYC Traditional High Schools

How does a high school have an 82% 4-year grad rate, but 70% of students are chronically absent (missing 10% of school days)?

While digging around NYC’s school quality data recently, I saw many schools that had high graduation rates AND a high rate of chronic absenteesim. Of course, schools with 100% grad rates are generally doing better with attendance than schools with 70% grad rates, but the data’s all over the place: see scatter plot below with the traditional high schools with 70%+ grad rates. On average, 4-year grad rates for traditional high schools are up 3 percentage points in the past two years, and chronic absenteeism is also up slightly at just over 1 percentage point. Of the schools who improved their grad rates, 56% of them have HIGHER rates of chronic absenteeism, and 10% have the same rate, compared to the previous year. So, improved attendance is not the reason for that growth. My optimistic hypothesis: more graduation pathways, more support for absent students, inaccurate data. My investigative journalist hypothesis: lower and varied standards across schools.

This is what normal people do for fun. Right, guys?

Source: NYC DOE School Quality Reports 2017-18

Source: NYC DOE School Quality Reports 2017-18

Alison HolsteinComment
Before You Collect Data, Ask Some Questions:

We take for granted now that schools and non-profits run on data. You’ll never hear a non-profit leader say “we have too much data”, or “we shouldn’t be collecting this or that”. However, before you say “let’s collect that”, ask yourself:

1) What’s the point? You might not hear it easily, but within each data request there is a subtle distinction between “Show me data so I can learn what works” and “Show me data so I can prove it works.” In one case, the purpose is learning and improvement, and the other it’s probably funding. Both purposes serve an organization’s mission. When bringing data in to a conversation, make sure you’ve got data that responds to the right question. For example, you may set a shoo-in goal for your funder that 75% of participants will achieve the outcome. When you meet with your internal leadership team or staff, let’s celebrate that outcome, but then look at the participants who did NOT achieve the outcome. In an improvement culture, we’re combing the data for problems: any participant who the program didn’t work for or who may need more interventions. In funder reports, we’re highlighting success and growth.

2) Is the juice worth the squeeze? I was chatting with a high school counselor recently who told me he spends three half-days a week inputting data, mostly case notes. I must have misheard. That’s a full day and a half each week! Why does he do this? Well, the boss needs it to keep the organization running. While I appreciate his commitment to the organization and data collection, that’s too much “squeeze” for something the funders aren’t actually asking for. When considering whether to collect data on something, know your plan for what you’re going to do with it. It’s not enough to say “It would be cool if we knew…”. We have to think about the time staff takes collecting data as a monetary cost to the organization. How much would you pay? You should pay for data that benefits participants because it will: (1) help fund your organization, (2) help staff do their jobs more effectively or efficiently (even considering that data collection), (3) play a key role in leadership’s decision-making process.

3) Are we going to look at the data? If there’s no plan and no team to review the data, and the purpose is learning and improvement (not funder-driven), don’t waste your time. Let everyone leave a bit early today. OR, consider how the data will inform key services before you collect it. If half of your organization’s work is helping young people plan for college, what do counselors need to know, and when? If the transcript report takes 5 minutes to run every time they meet with a student, the data can be used for reflection, but it’s not a tool. Do leaders and supervisors know how to talk about data, or value it? If not, get feedback to inform capacity building.

3 bullets is pretty standard. I’ll stop there. Anything I missed?

Dear NYC Transfer School Principals

Dear Principals of NYC Transfer High Schools,

I know it has been a difficult year or two trying to figure out what new state accountability standards would mean for your school. You’ve engaged parents and students, written letters, attended meetings - all trying to make the case that you INTENTIONALLY serve students who are unlikely to graduate, and maybe whether or not 67% of them graduate (YES or NO) in 6 years is not the best measure of your school’s ability to serve them.

Well, now we have some answers from the state. One fifth of transfer schools need improvement. Students fell off track at other schools, and now you have ‘em, so there’s your problem. You can see clearly from the state grad rates that don’t match the city grad rates at all, that you performed better or worse than you thought maybe you did for some cohort. And now we’ve also learned how we are going to get off or stay off the state’s list:

1) Don’t accept students you were designed to accept. If you accept a student who only has a few credits and didn’t pass any Regents in their first or second year, well, they might not graduate. They probably have some challenges in their lives, and they will hurt your numbers. If they have financial or family responsibilities, have an difficulty learning in a traditional setting, have children, were suspended in the past, have mental health issues, are homeless, or experienced any kind of trauma, they are what we now call “TOO risky” and you should refer them to the nearest dropout center.

2) Test test test. If a student took the ELA Regents six times - failing each time until they finally reached a 65 - make them take it again and again and again until they get a Level 4 score. That will help with your Auto Appeal. What does that mean? It’s confusing. But higher is better for keeping your school open. Better yet, refer to my first tip by accepting students who already passed the Regents with high scores, because whether they took the test at your school or at their previous schools is irrelevant to holding your school accountable.

3) Get that paper! At the end of the day, the diploma is what matters. Unless a student graduates in their seventh year of high school - then it doesn’t matter at all. As research shows, if they graduate high school (in 6 years), they are basically all set for life. So, if you were looking at other outcomes, like growth in reading and math skills, increased engagement and attendance in school, improved social or emotional mindsets and behaviors, or credit accumulation (other silly ideas here), don’t bother. Make it rain credits.

If you have additional questions about your data, see the link here. Oh, sorry, the state forgot to create a link to explain all this. However, if you read Greek, try the State’s Educator Guide.

Hope this was helpful!

Ali

P.S. This letter was my therapeutic response of turning rage into humor. Transfer school educators - you are my heroes, and I hope you can stay focused on doing what you do best.