Abstract:
Throughout this webinar, you will have the opportunity to learn about some of the scariest assessment stories out there. The near 0% response rate on a survey. A 20-page report poured over for months that no one ever touches again. Do you feel the chills of data past upon you yet? Join us for one hour of learning how to navigate 13 assessment horror stories.
Outcomes:
Participants will be able to:
- Deconstruct some common place Student Affairs assessment practices that are not in alignment with data best practices.
- Learn how to apply data story telling best practices.
- Identify when an assessment horror story is starting to happen in their own work.
Presenter:
- Natasha J. G. Monteith (she/her), Leadership Development Program Evaluation Manager
Date Of Recording: 10/13/26
Music: Life Style EDM by Black Rhomb, Monetization ID NUZNJIQKEIGZX73L
Watch the Video:
Links and References:
Transcript
Amanda Knerr:
So now I’m really super excited to present to you today’s featured speaker, Natasha Monteith, who’s a leadership development program evaluation manager and current Roompact fellow. She’s going to be leading today’s session, which I’m really excited about, Scary Assessment Stories to Tell in the Dark, where you will learn how to navigate the worst assessment stories so you can deliver a successful assessment project. Natasha, thank you so much for joining us and sharing your expertise with the Roompact community. The floor is all yours.
Natasha J. G. Monteith:
Thank you so much. And hi everybody. I’m really happy that you’re able to be here today live. And for those of you watching the recording, hello. I’m also really happy that you’re deciding to spend some time digging into assessment and starting to do this work. As Amanda shared, I do serve as a leadership development program evaluation manager right now. So doing assessment work and doing evaluation and data work is my area of expertise and it’s the thing I love to do.
So what we’re going to be doing today is we’re going to be deconstructing, there we go, some commonplace student affairs assessment practices that are not really in alignment with data best practices. We’re also going to identify when an assessment horror story is starting to happen in your own work and thinking about some of the sentences you might hear people saying that go, “Hmm, we need to pause and rethink this.” And then we’re also going to talk through how to apply some data storytelling best practices into your work.
And so the way that this session is set up, there’s going to be 13 scary stories because it is October 13th. It is spooky season. We are leaning into it. If you are watching this in December, my apologies, but it’s spooky season all year round.
And so to get us started, I want us to be thinking about formal collections above everything else. Something I heard a lot about when I was in student affairs is that we really look at our formal collections like a survey, a formal interview, a formal focus group as the end-all be-all of data. The thing I want to tell you is that surveys in these formal collections don’t automatically beat out all other data points. There are so many forms of data that exist in the work that you’re doing that are just as relevant and helpful as formal collections that just don’t go through the same process.
And so you might hear people saying things like, “I hear what you’re saying when you’re giving a piece of feedback, but the survey told us…” And that tells us, hey, maybe we need to be paying attention and having a conversation about what we look to as our sources of truth. What I’d encourage you to do is consider other sources you already have access to that could also help answer these questions.
So you might be running a survey around student engagement, but you also have a bunch of student engagement data that supplements that survey. You might have Roompact forms, you might have Maxient conduct situations. All of these are additional data. But the other thing I really want to highlight is feedback from staff. Your feedback from staff are important points of data that exist outside of formal collections that we also need to be listening to and attuning to. Just because they aren’t formally collected does not mean we need to ignore them.
Another story that I heard often is let’s just run another survey. So we know the problem we’re facing, but we want more data before making a decision about how we’re going to move forward. This is something I saw happen year over year. And so you might hear people say something like, “I don’t think we really understand the problem. Can we add a few more questions to the next survey?”
And the idea here being that if we get more information, if we get more data from a survey, then we’ll feel more confident about the decision we’re making. And if you’re at this point, what I will tell you is you should be skipping that survey. The survey’s not going to do anything but raise up more and more and more questions because it’s not a two-way lane for conversation. You need to be talking with people at this point.
You could be bringing your ideas to a focus group or be doing some interviews with folks to generate ideas, but ultimately just running a survey isn’t going to be able to help you feel confident in the decision you’re making. The other thing to know is if you’re not actually sure what your problem is, these kinds of conversations are also really, really helpful to have. Ultimately, our surveys are good at surfacing patterns, but they’re not good at helping us find solutions.
Another practice I see often is what I call the Frankenstein survey. And so this is a survey that has just been growing year over year, a little of this, a little of that. We decided back in 2020 to add these questions, although the question’s been there since ’96. And ultimately, the thing you’ll hear when you start having a Frankenstein survey is, “Can we add a few more questions to that survey before it launches?” When you start hearing that the purpose of the survey is getting lost and we’re entering Frankenstein territory.
What’s important to do in these situations is to pause and ask ourselves with every question we have on that collection form, “How do we use the answers to this question? What are we going to do in terms of making a decision, in terms of deciding whether or not a program’s working? What is the use for that question?”
And if we don’t have an answer for that, we should be cutting it. Because ultimately every time we’re asking somebody for information in a survey, we’re asking them to share something of themselves. And in terms of appropriateness, we don’t want to keep asking and extracting from people, especially students and staff, if we’re not going to be doing anything with that. It’ll help make sure that we don’t have survey fatigue happening and ultimately we’ll likely drive up some response rates if you can get your surveys to be hyper-specific with a use case.
The other thing I’d encourage you to do is to document how that data is being used and then making the cuts to the information like I talked about. The big thing is documentation here. And that we know in housing and residence life, we see a lot of turnover in terms of staff. And so having documentation of how we use data helps us be able to make sure and remember five years from now why that question exists the way it does and to help us make a good decision about whether or not it actually needs to change.
Another thing that I think is a scary story for so many different reasons is thinking about what did ChatGPT say? What did this AI model say? And I think we need to be thinking really critically about what you put into generative AI models, especially as it relates to assessment work.
You could hear people saying things like, “We could probably use AI for this.” And what I want to say is I am not anti-AI, I am pro thinking critically about what we’re doing and making sure we’re doing it ethically. One of the big rules I have is that we should never be putting names into generative AI software. So we’re never uploading and dumping a whole data set into that. That is not a choice that we should be making because it does not protect our survey takers. I also highly discourage letting any generative AI decide what analysis you’re going to run, deciding what you choose to report on, and ultimately deciding what you’re going to value within your data sets.
All of those things need to have a human touch to them because there are implications that go beyond just what lives in a report. I also run off the rule of thumb of if I couldn’t do it myself, I’m not going to ask AI to do it because I can’t then go back in and check for whether or not this was accurate and appropriate. And again, in full transparency, I use this sparingly in my own data work, and I do that for two reasons. The reason I do it so little is because of the ethical concerns about how inputs are being used, meaning not just the training of models, which is a problem I do have, but also thinking about whether or not the people who gave you this data gave you this information consented to this being put into an AI model.
There is an ethical question there, and it’s one that I stand on the side of, unless I’m being explicit to somebody in a collection process that this is going to be added into an AI model, I don’t do it because it is their information and they should be able to protect it in the ways that they see fit. So I avoid using AI in those ways.
Ways that I do use AI in my work, I have it help me draft the email campaigns for different collections. I have it help me draft up the equations that I use in Google Sheets because I can get it 90% of the way there, but sometimes I just need a little bit of extra help. Those kinds of methods I think are really helpful and really ethical in terms of the data scope. I also completely understand for anybody who chooses not to use AI in this work. And I don’t think that you have to use AI in order to do assessment practice.
Another thing that pops up often is when we are looking at whether it’s a form that’s in Roompact, like your duty logs or intentional interactions or a more formal survey, we often have to ask ourselves who picked these questions? Who made the choice of what’s going to be involved in this collection? And what I would encourage anybody to do that I didn’t see happening as a standard practice is bringing student voice into your collection processes.
So how you might know you need to be bringing student voice into your processes is you might hear people saying things like, “This survey data feels so useless. They don’t really seem to understand what an intentional conversation even is.” And that makes sense. We might put language into our collections that just don’t make sense to students because they’re not sitting and steeped in the same context we have.
And what I recommend doing is adding RAs onto your department committees. If you have a department level assessment committee or curriculum committee, having an RA representative in those spaces is so useful. While I was at UNCG my second year, we were able to bring in an RA to our curriculum committee and it changed the way we thought about form creation, it changed the way that we thought about even what we were asking in intentional conversations, and ultimately it changed the game in the best way.
Another practice I’d encourage anybody on this call to think about is actually testing a collection with students before you decide to roll it out wider. While I was at Georgia Southern, we decided to run this whole new centralized budgeting process. But we knew if we didn’t test that process in the summer with our summer RAs before the fall RAs got there, there were going to be issues in that form, there were going to be problems in the process that we just didn’t flag because it was just the three of us in a room looking and thinking about it.
So before you decide to change an intentional interaction form, before you decide to update your duty log, I’d recommend bringing it to the people who are going to be filling it out and having them go through and name the assumptions they’re making about the language you have, having them work through it with you and talk out loud. It’s called a cognitive interview, if you want to go look that up. But it’s a practice I’m very grateful to be able to bring into processes because it makes it where at the end you really feel confident in the data you have and that people are doing it appropriately.
Another thing to think about, which often was not though about is what happens when somebody skips a question? And so you are filling out this survey, people skip a question because we don’t require every single survey question to have a response. If we’re requiring a response, we need to have a really good reason for why.
But we need to be thinking about before survey’s even administered, how do we handle missing answers? How do we handle a skipped question? And how I know this isn’t something people are thinking about is when I hear people say something along the lines of, “Oh, we don’t really have a plan for how we analyze our data until we have it.” Which as a whole, I recommend always having an analysis plan for your data. An analysis plan is basically the questions of your survey and then how you plan to analyze and use that data afterwards.
It helps you understand how you’re going to be running analysis so that way you aren’t trying to figure that out while also handling the data itself. One of the ways you can do this is you can view it as a zero. I will say this is very uncommon. We don’t usually handle a skipped question as a zero. We more often view it as like an N/A, meaning it’s not applicable and we’re not actually going to count it in to our math.
So what that would look like is somebody skipped a question about how they felt around your training last week. Instead of being like, “Oh, we’re going to just say they scored a zero,” we’re just going to take them out of the equation. We’re going to change the number on the bottom of everything to be one less than them. I’m trying to explain it right now and it’s not coming across extremely clearly. But ultimately it changes your denominator and we pretend like that survey doesn’t exist when we look at that question.
And ultimately, regardless of which way you choose, you need to document why you picked what you picked and ultimately what you picked. Again, all of this comes into that analysis plan.
Another thing I see happening often is when there’s a limited amount of folks that you’re working with or surveying and those folks have certain demographics, their answers can become known really quickly when sharing out findings if we’re not careful about protecting people’s information. And so this could be something like people saying, “We always share out findings by the building, gender, and race.
I think back to one of the buildings I oversaw that had six RAs in it. So those six RAs already, if we’re breaking down by building, don’t necessarily have the most anonymity in their responses. If we now start breaking down that building of six people by race and gender on top of it, I know exactly whose responses are whose based on how they identify. And so we should always be setting a limit of how big a group needs to be for us to share out a response.
I tend to use four to five as my limit given the sizes of the different programs I work with. Ultimately, this helps the survey takers know that their answers are going to be anonymous. If you’re not planning for your survey collection to be anonymous, you need to make that really, really clear to the survey takers so that way they’re aware of who’s going to know information, whether or not their name’s going to be attached to it, but ultimately people assume a level of anonymity in their responses to you.
Something else that I encourage everybody to engage with is knowing that there’s going to be times where you look at a finding or you look at a process and you just go, “That does not make sense.” And so when you’re seeing something in your data that makes you go, “Huh,” or, “That’s weird,” that should be a siren for you that something’s going on here and we need to do some kind of extra digging to figure out what that means.
I’ll give the example that over the summer, I partnered and helped out with a fellowship program that ended with everyone in Washington DC doing a hill day where they went out and talked with Congress folks. Part of what we were looking at with that is did people’s agency grow throughout the course of the fellowship? When I did the analysis the first time around, I was like, I thought this was going to be closer to 40, 50% and we’re looking at 95% plus.
And that made me wonder. What I expected to find was this 40%. Whenever we’re doing extra digging, we need to start by naming what we expected to find so that way we can name the bias that we might be bringing into the process as we keep going. If we don’t know what we’re expecting to find or pretend like there’s not something we were looking for, it ends up creating more bias.
What ended up happening is after I was like, “Wow, there’s a bigger jump in advocacy than I expected.” It was time for me to develop some hypotheses or some guesses as to what this might’ve been about. What I decided was my biggest wondering and why was everyone I talked to when I was in DC told me they weren’t feeling really confident about going to talk with the Congress people.
I didn’t get to talk with anybody after they went to see the Congress people and then they took their survey. I’m guessing that actually going and talking to their Congress folks helped them feel like advocates. They felt like they had that high sense of agency. And ultimately, after having other people check this math, talking through what other people thought the hypothesis was, that’s what we decided is from what we were seeing and hearing at the convening all the way till we had the data, the most logical story was during the fellowship, people learned, but they didn’t feel like they were actually advocates until they did it.
And so when we find things that don’t fit our narrative, don’t fit what we’re expecting, when we get to dig deeper, we actually get to find out more. We get to figure out why, and those are the fun and interesting and exciting data stories that are out there that oftentimes we’re told to ignore. But anytime that gut just goes, this feels funny, we should dig into it because there’s probably something juicy there.
Another thing that I would encourage y’all to think about is people beyond your department might be interested in your data. We know data silos are a huge problem in student affairs work and that we need to be actively thinking about how we break those down. So this might sound like something to be like, “Oh, I wish you would’ve told us.” What I recommend doing is creating a sharing plan for any collection you’re doing before you’ve even started the collection.
Things to think about are who’s going to find this information helpful? And also what is important to them? And this can be a conversation you actually have. This can be something where maybe you go and schedule some time with somebody in the conduct office to understand what might be important to them from what you’re looking at. It might be folks over in health and human services who might be able to help you out, but we need to stop just thinking about assessment for housing as just assessment for housing and realize that there’s a lot of information we have that can help other functional areas also do their work better and more effectively. So how can we get what they need to them in order for them to do their jobs?
And this is where I want to say the gut reaction might be, okay, cool. We’re just going to share out a slide deck where we report to everybody. And there’s this idea that one size fits all for reporting gets the job done. This might sound like somebody saying, “Let’s make a slide deck to share.” That’s not the best way to be thinking about data sharing because we need to consider how much time can the folks we’re sharing data with commit to reviewing it?
If one department is like, “We would love for you all to come in and run a session for us. We’d love for you to take over a space in our meeting.” That shows a level of commitment to reviewing the data that sounds like a beneficial partnership to explore. Whereas other folks might be like, “Hey, you can send it to us. We can’t promise we’re going to be looking at it.” You know not to be spending more time and energy into that route.
You also have to ask yourself, what’s the goal in sharing this with them? Is it to check a box and just say, “Hey, we did it so this way we don’t have as much of a silo.” Or is it to build something collaboratively together and make meaning and take action? If it’s the latter that’s going to mean more individualization, but ultimately it means the data’s being used in the ways that you’re hoping it’s going to be used.
Something I also ask myself often is how can I make this as easy as possible? So sometimes our gut reaction with creating slide decks or reports is I want to put every single thing I have out there for you.
And from my experience, that just leads people feeling lost. We often collect a lot of data that’s not relevant to every single person that we’re partnering with. So instead, I sit back and I go, “Okay, I’m bringing this over to conduct. Conduct’s not going to care about A, B, and C, so I’m not even going to include that in my report for them because ultimately they’re not going to use it. And all that’s going to do is take up more cognitive strength from them to be able to make it through the data I’m sharing.” For us to make data use easier, we need to cut down the steps we’re asking somebody to take in order to work with data.
Another big practice I would love to see just happening across student affairs as a whole is closing the loop. So ultimately, we should be sharing back to those we’re collecting data from. So this could be students, this could be RAs, this could be professional staff. If we’re asking for them to share their opinions, their thoughts, their reactions with us, we should be coming back to them and letting them know what we did with that.
So how I know closing the loop isn’t happening is usually people saying something like, “We keep having fewer and fewer students who take our surveys.” That tells me the buy-in from students isn’t there in the same way. What I’d encourage you to do is let them know what you learned and your next steps. If you know that you had this round of intentional interactions happen, and it sounds like everybody was saying, “Hey, we need help and resources to learn how to study,” you learned that.
And now your next step is during the 8:00 PM to 10:00 PM duty round section, RAs are going to be hosting study hours in the study hall. That’s what you’re going to do. Let them know, “Hey, we heard you and here’s the action we’re going to take.” When people feel valued and feel heard, they’re more likely to continue engaging in the data process with you. When we don’t tell them what we’re doing, all it entails them is I’m kind of spitting out into a black void and I’m not getting anything back. We need to make it clear to people how we’re using their information.
Another practice that, don’t get me wrong, I love a bar graph. I’m a big fan of a bar graph. But oftentimes when I look through student affairs reports, I just see bar graph after bar graph after bar graph after bar graph. And I just want to name, there are so many ways to visualize data that will catch your audience’s eye and make them pay attention. And so how I know we might be caught in the process of just doing it the way that we’ve always done it is when people tell me, “Hey, we’re just going to copy how we did it last year.” The information that we had last year and the findings we had last year might’ve been appropriate for the data visualizations we picked. We have to look at what we have this year and make decisions about how do we share this out and visualize this in a way that makes sense to everybody.
And I’m actually going to share a couple resources here. This is the Data to Viz website and then Stephanie Evergreen’s Data Visualization Checklist. I’m dropping them in the chat for y’all now, but these two resources were the ones that I go back to and consistently lean on during my own data visualization journeys. So this is the From Data to Viz website. It’s pretty easy to navigate, but they have this explore section that actually comes down and gives you all of the different types of data that exist. And then thinking about how you can visualize it based on what you have.
They also have, down here in data story, all of these different ways you can visualize data. And I love it. It’s so good and it’s so helpful because outside of just saying, “Hey, you should do it this way,” you also can come in and underneath here there’s stories where you can actually go through and it gives you a real life scenario to understand why you would maybe choose to run your data viz the way you are. So you’re going to use this and you’re going to figure out exactly what kind of data visualization you need. You’re going to build it. It’s going to be beautiful.
Then I recommend always bringing it to this Rate My Visualization. Evergreen Data is run by Stephanie Evergreen, who is a data visualization expert. She is phenomenal. If you are looking to try to get your reports or your data visualization game up, I highly recommend signing up for her blog and also her email list. It is the only one that when I get it, I look at it every single time and I read everything she publishes.
What you do is you end up uploading a picture of your visualization and then you go through and actually make sure that you’re hitting these best practices for data viz. Talking about whether or not your title is a descriptive sentence, it’s left-justified and it’s in the upper left, whether or not you have your subtitles, your text hierarchy, all of these steps are the nitty-gritty and how do you actually build an effective data visualization that is accessible to people.
So I recommend always checking that out, especially when you’re trying out a new type of visualization that maybe you’re not used to using. It helps make sure that we’re still following best practices. And I now need to get y’all back over to the other presentation. One second.
Perfect. All right. So we’ve learned how in the world we make sure things just aren’t another bar graph. And then the scariest of stories to me, from where I sit professionally, is staff training. So oftentimes I see staff training be the thing that we deprioritize the most as it comes to assessment work. Please make sure you’re building training sessions that cover the purpose of your different forms for the folks who are filling them out.
Saying something like, “This is something each RD can coach on during their one-on-ones,” is a surefire way to get yourself in a situation where we don’t know how much we can actually trust the data you have. Instead, we should really be giving staff the ability to actually practice submitting forms and getting feedback from their supervisors. So this could look like having space in your training for folks to actually fill out a duty log.
It could be time in your training for folks to actually fill out an intentional interaction form. It could be time in your training for folks to fill out a program request form. These forms are going to be something you look back on to be able to make sense of your year and to be able to make sense of experiences. And ultimately, we can’t trust the data coming in. If we can’t trust that, you can’t make good decisions.
And the problem is if we don’t systemically train staff, every single staff member could be told something different about how they should be approaching and thinking about these forms. RD1 might tell them, “Hey, you can round up. It’s fine.” And RD2 tells them, “No, you need to put a specific number.” And now you’re getting to the end of the year and you’re trying to do a count and you don’t know what you can believe. So ultimately, systemic-based training that everyone gets with the ability to practice is the expectation I would put to anybody I’m partnering with around how to be doing staff training.
Those are 13 quick, but in my opinion, rather horrifying stories. And so the big takeaways that go across to all of these is that we need to have a plan. Data work happens most effectively when we have a plan in place, when we know how we’re going to use the data we have, when we know how we’re going to analyze the data we have, and when we have some documentation for the logic and reasoning behind those things. We also do best when we know the why. Where are we going to? What is the intention behind our processes? And ultimately, what is underlying these choices in terms of our values?
We also need to be open and considering new ways to do this work. Assessment work is changing, in my opinion, almost every day now. With the way that AI is coming in, we need to stay on our toes and we need to be aware of all the changes happening around us.
It’s also critical to be asking questions. So especially if you’re on this call and maybe you’re in your first year, second year, third year in a role, now’s the time to ask really good questions about why things are done the way they are, so that way you can better understand how to be working in the system you’re already in. But if we have a plan, if we understand our why, if we’re open to new ways of doing this work and we ask good questions, I think these four things will help make your data and assessment work easier and more effective.
Now, from all of y’all, I’m curious, are there any scary stories that you want to talk through, questions you might have at this point in time. I’m pulling y’all over so I can actually see you.
Olivia Olson:
Hello, my name’s Olivia Olson. I currently work at the University of Nebraska in Omaha and recently came from the East Coast from a school over in Virginia. And my scary story is over the last three years, we do one survey in the fall semester and one for the spring, and it’s essentially the same questions, RA specific related questions, especially to see programming stuff, and then some security related questions for students to see if they’re locking their doors, this, that, and the other.
My two scary stories is we started to implement a charge for door locking because students weren’t locking their doors. It was almost close to 60% of students were not locking their doors. And then over the last three years, our survey has gone down from I think close to 900 to the next year was 400 people responded to the survey.
Natasha J. G. Monteith:
Yeah, so that’s a deep cut.
Olivia Olson:
Very.
Natasha J. G. Monteith:
Yeah.
Olivia Olson:
Yeah. And we usually use it in our RA assessments as well for the RA section. And this last spring we couldn’t even use it in their assessments and stuff because we just didn’t get any responses.
Natasha J. G. Monteith:
Yeah. What kind of changes are you all thinking on your horizon right now around that?
Olivia Olson:
Well, before I left, we talked about the potential trying to split the two from having it be this huge long one survey to maybe a couple of different surveys that went out. But that was at the time they said no because there’s other surveys that go out that students see. But I still think it’s important because nowadays students, just people in general, they don’t pay attention to things as fast.
We have the whole people call about brain rot and everything and the short version of TikToks, Instagram Reels for a reason. And I think that surveys are just as important to be slimmed down because nobody wants to go through a 35 question survey.
Natasha J. G. Monteith:
100%. Right now at work, we’re trying to get our questions down to under 10. That’s the new target we’ve set for ourselves is 10 or less if we can do it. And it’s because, like you’re saying, what we noticed is people were willing to open it, people were willing to start it, but then people weren’t finishing. And so that’s also a part of it too, is it’s like where in your process are people falling off? Because if everyone’s opening your email, cool, you know your marketing’s working. If people are starting it, you also know, okay, we are not running into a problem in terms of technology. But when people aren’t finishing, that says a lot about the length.
Olivia Olson:
Yeah. I think that also the long form answers really deter people. Those are the ones that I see skipped the absolute most.
Natasha J. G. Monteith:
Oh yeah. I also, I didn’t put this in here today. If anybody’s curious, I would put money on it. 75% of long form answers that exist today could easily be switched over to a checkbox. It’s that we get in our own heads sometimes of being like, “Well, we don’t want to cut out anything.” And it’s you always have the option to list out the five things you really think are going to come out and then have an other box where people can add in other things if they want to.
But when we take away some of that cognitive load of having to decide, how do I word this feedback? I know what I need resources for potentially, but I don’t even know, I don’t want to spend five minutes writing it out. And it could just be check, check, check, check, check. It also ends up saving so much time on the back end with analysis.
Those open-ended questions are intense from an analysis standpoint. Are you seeing similar patterns at your new school right now in terms of buy-in and finishing up surveys or is that all feeling really unique to that old school?
Olivia Olson:
We haven’t done a survey yet that I know of. But personally, if I could, I would get rid of an RA specific survey just because I feel like sometimes people just go through and just click either yes to everything or no to everything. Or if you have a resident who’s not happy with you while the survey’s out, there’s a really good chance that you’re just going to get a really bad survey because that person had an unlucky week with you and they just want to put it out on you. So I don’t know. Those surveys I’m still, I think that security surveys are really important to see maintenance-wise and things that really need changes. But in terms of the RA, I feel like having a survey that goes into your evaluations at the end of the semester at the end of the year just feel not as good.
Natasha J. G. Monteith:
No, I agree. And especially when I hear about student staff surveys in that way too, the question that always comes up for me is how far up the chain does this go? Is this something we only do to RAs? Do RDs get it? Do ADs get it? Do directors get it? Is it all the way up? Or is this something that we do uniquely for our student staff?
Olivia Olson:
Yeah, we actually implemented the 360 surveys. So RAs would fill out for each other, for us, for the staff in between the SRAs, the grad assistants, but then it’s how far does the 360 go up? Do we get to have a survey for our supervisors? Do their people or does their directors and bosses get to see that? And usually it just stops right at the entry level.
Natasha J. G. Monteith:
Yeah. And it gets funky too, because it’s like HR gets involved and then it’s not just housing who gets to make the choice and then it just complicates itself and almost snowballs in some ways. But I do wonder too, not so much wonder, but wanted to name that you were like, if somebody’s angry at them, it comes out in the survey. And I think across surveys, that’s also a pattern I’m seeing right now is when we think of a standard distribution, you’ve got people over here who are maybe really happy, people over here are really angry, and then you’ve got the people in the middle that are supposed to make up the biggest chunk of folks.
What I’ve noticed more and more and more is the people in the middle don’t take surveys anymore. You’re only seeing people who are like, “I hate everything, burn it all down.” Or who are like, “I love this so much, please don’t change anything.” And it’s becoming really, really difficult to figure out how to get to those folks in the middle.
The folks who are on that more neutral space or just in a, “Yeah, I liked it, but am I going to sit down and do a 20-minute survey about this?” Probably not. And I think it gets back to what we were talking about earlier of trying to narrow down questions so much so that way maybe we can get more of those folks who are in the middle. Because when we have these long surveys, we only get those two sides and we lose so much story.
We just end up also having a really inaccurate picture of what’s going on too. So even when you said that it was the number dropped down to 400, I was like, I wonder what happened with any questions that were the middle and if you just started seeing everybody on those far sides, because that’s what I’ve been noticing, at least in my own work.
Olivia Olson:
Yeah. I honestly couldn’t tell you other than people just not wanting to do it because some of these people, some of the residents, they’re in their fourth year of doing the same survey with very minimal changes or… Yeah, I was trying to think of what else.
Natasha J. G. Monteith:
And they probably also don’t hear about what happened with it. So it’s like, “I’ve taken this for four years, what happens?” I think about that safety survey you brought up and it’s like, how cool would it be if after somebody did that, you all sat down with campus safety, you figured out the steps you’re going to take. And then y’all just told them where it’s like, “Hey, you told us you really hate how dark this exit is. We’re adding a new light. Thank you for your feedback.” But those closing of loops don’t really happen. And so it doesn’t surprise me that you’d have folks in their fourth year being like, “I’m not doing this again.”
Olivia Olson:
Yeah. And we usually, we have our RAs try and do a lot of outreach and reminders to the students. One thing that I also wasn’t the biggest fan of is being told in a, “Oh, when you do your evaluation, if you have an RA that doesn’t have a lot of people who filled it out, you have to say, is it because you don’t have good connections with your residents?” And I’m like, “I don’t think that a survey being filled out should be equated to the connections that they have with their residents at all.”
Natasha J. G. Monteith:
I think about, I had athletes in a couple of my buildings, and I know that they had connections with their RAs, but we would never get a survey from them. It wasn’t a priority. And so it’s is it a question of a connection or is it a question of priorities? And at least for me, I was joking with Amanda before y’all got in here that I love a lot of my work, but I realize more and more my work is marketing. It wasn’t meant to be marketing. I’m not a trained marketer. But so much of what I do is trying to market the value of like, “Hey, this is what we’re going to do with this information when you give it to us.” And then having to come back and be like, “Thank you so much. Let’s close this loop. Let’s tell you all these things that are changing. Let’s drive buy-in with these folks over here.”
I wish I knew more about marketing because I think it would completely unlock so much as it relates to assessment work. I’ve started working more and more with the marketing team that I partner with, and I’ve learned so much and it’s changed the game. I remember at one point they were like, “Stop saying this word advocacy.” They’re like, “Nobody other than you uses that word. Just say they grew as leaders.”
And I was like, “I don’t hate it. It’s getting to the same heart, and now you don’t have to have all this specialty language in your head about what I’m saying and what it means.” But yeah, I think all of the things you name are extremely scary for the spooky season. And it sounds like there were also some really good lessons learned, and it helped you figured out a little bit more about where you stand and how this work should be happening, which is also part of what happens in a scary story is we end up having heroines come out at the end who show us how we should be doing this work.
But I also know, Amanda, you might’ve had a couple of things to share at the end, maybe, maybe not. Cool. So I’m going to just click really fast. If you are a Roompact school, which I think everybody on this for sure right now is, and maybe some of you in the recordings are too, I wanted to name, you have access to one hour virtual consults with the fellows. I am one of them. So if you want to sit and talk about surveys, talk about focus groups, talk about interviews, talk about your Roompact forms, please talk to me.
This is what I love to do. And if you’re going to be at the R2 conference next week, we’re all going to be there too, and you can come and have chats with us there. We’re happy to do that. But here’s my contact info. I’m happy to have chats with folks.
Olivia, thank you so much for coming off mute and sharing a little bit about your story. I’m really grateful for it. But this is my LinkedIn, which is the most accessible I am. I am a Capricorn and I’m on LinkedIn often. So if you need to get ahold of me, that’s probably the easiest way to do so. But Amanda, I’m going to hand it back off to you.
Amanda Knerr:
Excellent. Thank you so much. What a wonderful session. And for me, taking away that, oh, I have an assessment plan and what’s my why, and I need to think about how my work is changing and how assessment is changing as I’m thinking about developing my assessment projects. Thank you again so much for being our featured guest today, helping us think through how to avoid that scary assessment story so we can have a successful assessment project.
I appreciate your time, your insights, and looking forward to connecting with you next week out in Anaheim. Also, I want to thank all of you who joined with us today. We’re so glad that you were part of the conversation. You will be getting a follow-up email in the next few days with access to the recording. I’ll make sure the two website links that we went through today are in that email as well.
And then you’ll also be able to access it on our YouTube channel in about a week or so, or you can send your friends and colleagues there for additional resources. Before you go, I would appreciate it if you take just a couple of minutes and I just put it in the chat and do a brief feedback survey that really helps us continue to improve and offer sessions that are interesting to you and to the work that you’re doing in your day-to-day job.
Just as a reminder, we have a couple of webinars coming up operating as a learning organization on Wednesday, October 18th, which is at 3:00 Eastern Standard Time. That’s with Steven Herndon out of Syracuse University. It’ll be a great session. He’s a very engaging, great facilitator. Encourage you to be here for that. And then our very own Paul, will be doing Roompact 201 staffing on Monday, November 23rd, right before Thanksgiving holiday at 4:00 PM Eastern Standard Time.
Thank you again for your time today. I hope to see you in a future session. Have a wonderful afternoon and have a great day. Take care.
