Before You Collect Data, Ask Some Questions:

We take for granted now that schools and non-profits run on data. You’ll never hear a non-profit leader say “we have too much data”, or “we shouldn’t be collecting this or that”. However, before you say “let’s collect that”, ask yourself:

1) What’s the point? You might not hear it easily, but within each data request there is a subtle distinction between “Show me data so I can learn what works” and “Show me data so I can prove it works.” In one case, the purpose is learning and improvement, and the other it’s probably funding. Both purposes serve an organization’s mission. When bringing data in to a conversation, make sure you’ve got data that responds to the right question. For example, you may set a shoo-in goal for your funder that 75% of participants will achieve the outcome. When you meet with your internal leadership team or staff, let’s celebrate that outcome, but then look at the participants who did NOT achieve the outcome. In an improvement culture, we’re combing the data for problems: any participant who the program didn’t work for or who may need more interventions. In funder reports, we’re highlighting success and growth.

2) Is the juice worth the squeeze? I was chatting with a high school counselor recently who told me he spends three half-days a week inputting data, mostly case notes. I must have misheard. That’s a full day and a half each week! Why does he do this? Well, the boss needs it to keep the organization running. While I appreciate his commitment to the organization and data collection, that’s too much “squeeze” for something the funders aren’t actually asking for. When considering whether to collect data on something, know your plan for what you’re going to do with it. It’s not enough to say “It would be cool if we knew…”. We have to think about the time staff takes collecting data as a monetary cost to the organization. How much would you pay? You should pay for data that benefits participants because it will: (1) help fund your organization, (2) help staff do their jobs more effectively or efficiently (even considering that data collection), (3) play a key role in leadership’s decision-making process.

3) Are we going to look at the data? If there’s no plan and no team to review the data, and the purpose is learning and improvement (not funder-driven), don’t waste your time. Let everyone leave a bit early today. OR, consider how the data will inform key services before you collect it. If half of your organization’s work is helping young people plan for college, what do counselors need to know, and when? If the transcript report takes 5 minutes to run every time they meet with a student, the data can be used for reflection, but it’s not a tool. Do leaders and supervisors know how to talk about data, or value it? If not, get feedback to inform capacity building.

3 bullets is pretty standard. I’ll stop there. Anything I missed?

Dear NYC Transfer School Principals

Dear Principals of NYC Transfer High Schools,

I know it has been a difficult year or two trying to figure out what new state accountability standards would mean for your school. You’ve engaged parents and students, written letters, attended meetings - all trying to make the case that you INTENTIONALLY serve students who are unlikely to graduate, and maybe whether or not 67% of them graduate (YES or NO) in 6 years is not the best measure of your school’s ability to serve them.

Well, now we have some answers from the state. One fifth of transfer schools need improvement. Students fell off track at other schools, and now you have ‘em, so there’s your problem. You can see clearly from the state grad rates that don’t match the city grad rates at all, that you performed better or worse than you thought maybe you did for some cohort. And now we’ve also learned how we are going to get off or stay off the state’s list:

1) Don’t accept students you were designed to accept. If you accept a student who only has a few credits and didn’t pass any Regents in their first or second year, well, they might not graduate. They probably have some challenges in their lives, and they will hurt your numbers. If they have financial or family responsibilities, have an difficulty learning in a traditional setting, have children, were suspended in the past, have mental health issues, are homeless, or experienced any kind of trauma, they are what we now call “TOO risky” and you should refer them to the nearest dropout center.

2) Test test test. If a student took the ELA Regents six times - failing each time until they finally reached a 65 - make them take it again and again and again until they get a Level 4 score. That will help with your Auto Appeal. What does that mean? It’s confusing. But higher is better for keeping your school open. Better yet, refer to my first tip by accepting students who already passed the Regents with high scores, because whether they took the test at your school or at their previous schools is irrelevant to holding your school accountable.

3) Get that paper! At the end of the day, the diploma is what matters. Unless a student graduates in their seventh year of high school - then it doesn’t matter at all. As research shows, if they graduate high school (in 6 years), they are basically all set for life. So, if you were looking at other outcomes, like growth in reading and math skills, increased engagement and attendance in school, improved social or emotional mindsets and behaviors, or credit accumulation (other silly ideas here), don’t bother. Make it rain credits.

If you have additional questions about your data, see the link here. Oh, sorry, the state forgot to create a link to explain all this. However, if you read Greek, try the State’s Educator Guide.

Hope this was helpful!


P.S. This letter was my therapeutic response of turning rage into humor. Transfer school educators - you are my heroes, and I hope you can stay focused on doing what you do best.

What Happens When High Schools Become Accountable for Graduate Earnings?

I read an interesting statement today from Chiefs for Change, calling for public institutions to look at employment and wage data - connecting it to high schools and disaggregating by “key demographics.” So now I’m wondering what would that look like if we evaluated high schools based on whether their alumni are employed and earning family-sustaining wages?

What’s taking so long? It has been about a decade since districts have added college enrollment to school report cards. It takes just a few clicks to see what percent of students from ABC High School went on to postsecondary enrollment in 6 or 18 months post-graduation. There may even be data on military status post-graduation. However, for a couple reasons, I think we haven’t seen employment data: (1) We can’t tell if employment after high school is a good thing or a bad thing. In a wealthy community, college may be the only acceptable outcome (or a gap year), but in a poor community, is earning a living wage after high school an indicator of economic stability or a choice that will limit future mobility? I’ve had some tough conversations with schools where I shock them with the fact that only about 1/5 of students graduate from the community colleges they are sending their students to. (2) Technically, social security numbers are needed to link to wage data at this point - something districts don’t collect (and maybe shouldn’t). In schools that serve a high number of undocumented students, data would be less than complete.

My predictions. So what has been the impact of stretching the horizon of secondary outcomes to college, and how might that be a predictor of this new proposal? At the school-level, I’ve seen a mix of attitudes, from “We don’t even care about the diploma - it’s all about getting young people ready for college and career” to “How can we be held accountable for something that happens without us, and requires a lot of support?” The former mindset has led to new standards, pumped up college readiness/exposure activities, funding, and charter school buildings that are decoupaged with colorful college flags. The latter mindset reflects the reality that schools, especially those who serve students faced with more barriers, see that the task of getting kids through college requires postsecondary supports and resources (like CUNY ASAP’s wildly successful financial, social, and academic supports).

Overall, I say bring on the data! Starting small might be a fun idea that rarely happens in education. Start with a district or two and see what we learn. Hopefully, we’ll see an increased focus on connecting students to strong vocational programs and credit-bearing work-based learning in high school. We might also find that when we are focused on employment outcomes, we’re speaking the same language as young people.

Alison HolsteinComment
Resolution to Learn and Share

Hello Internet! (Or more likely, my dad and a friend or two),

This is my first blog post, and a challenge to myself to start taking stock of what I’m learning from my short experience as Ali Holstein LLC. Just like any good diet, I figured making this learning public would help hold me accountable for continuous learning and reflection. As I resolve to start sharing lessons and resources, I’m also looking back on what I’ve learned in 2018 - a year in which I made a major pivot from working for large organizations to becoming a team of 1. So here are some things I’ve been thinking about that will hopefully be valuable to someone else out there working with schools and non-profits:

  • Data is not a bucket. Data is a mindset and a culture. A data-driven organization does not hire a “data person” to “do the data.” A data-driven organization has leaders and staff who know they need information and evidence to make decisions, and they have a clear idea of what they are trying to achieve. They are able to look at data not just for what they want to share with funders or governing bodies, but for the growth areas (or holes) that must be addressed in order to reach their goals. They know they will never figure it out, but rather, will always be working to make things better.

  • Consulting is more than patching a hole in the ceiling. It’s figuring out what caused the hole, making a plan to prevent future holes, and making sure your client is more prepared, confident, and motivated to NOT hire expensive consultants to fix holes. Not sure how to do all this yet. Stay tuned.

  • Giving myself a “title” in an emerging role was a struggle. My working title is “Youth Program Consultant and Data Translator” to be a little specific about which organizations I work with. “Data Translator” is a term I made up (that already existed) to highlight a problem I’ve found in the field - a deep commitment to collect and report data (to funders) but a need for good structures for learning and growing from the data.

  • Relationships. All work with young people revolves around relationships. I kinda knew this already, but I’m seeing it more in the data (higher attendance when students feel they belong). Glad we’re starting to measure the right things.

Excited for the journey ahead. Please leave a comment, Dad!

Happy New Year!