Bringing Culturally Responsive Approaches to Your Residence Life Assessment (Part 3)

Other Posts in This Series: Part 1 | Part 2 | Part 3 | Part 4

Within Student Affairs, assessment is often an expectation of everyone’s work. Whether you are someone in an entry level role or have been around the block for 20 academic years, assessment likely comes up. I believe it is critical within our assessment efforts that we center humanizing values and perspectives. I believe that applying the Culturally Responsive Evaluation (CRE) framework to Student Affairs assessment is the best next step in using humanizing values and perspectives to reimagine assessment. Through this series, we will explore what the CRE framework is and how Student Affairs practitioners can apply the CRE framework to their assessment through a case example. In Part 1, an introduction to culturally responsive evaluation and working through step 1 and step 2 of the CRE framework was shared. In Part 2, we worked through step 3, step 4, and step 5 of the CRE framework with our case study focused around the design process. In this part of the series, we will work through part 6 and part 7 which are focused on creating instruments and collecting data. As a reminder, below is the most often conceptualization of CRE which showcases that cultural competence is centered in every step of the assessment process:

Culturally Responsive Evaluation by Hood, Hopson, and Kirkhart (2015)

How To Apply CRE to Student Affairs Assessment

For this section, we will use the following case study created by ChatGPT. For the sake of this case example, let’s assume that you are a member of the Harmony University Housing team and have been involved for the entire history of the STEM Living and Learning Community. I am coming in as your CRE consultant and will be sharing information and posing important questions to consider within each step based on what has already been shared in the case study.

In the heart of a bustling urban campus, Harmony University prides itself on fostering a vibrant and inclusive community. The Residence Life department plays a pivotal role in shaping this community by providing students with a supportive living environment. However, recent changes in the university’s programming prompted the Residence Life team to initiate a comprehensive assessment of the Living Learning Communities (LLCs) within the residential halls.

As part of an innovative initiative, the university introduced specialized LLCs tailored to students pursuing STEM disciplines seven years ago. Recognizing the unique needs and challenges faced by students in these fields, the Residence Life team sought to evaluate the effectiveness of the STEM-focused LLCs. The decision to conduct an assessment arose from the desire to ensure that these communities not only met the academic needs of residents but also contributed positively to their overall college experience.

Feedback from students offered valuable insights into the impact of the STEM LLCs. One student remarked, “Being part of the STEM LLC has been a game-changer for me. The collaborative atmosphere and access to academic resources have greatly enhanced my learning.” Conversely, another student expressed concerns, stating, “While the focus on academics is commendable, the social aspect feels a bit lacking. It would be great to have more events that foster a sense of community outside the classroom.” These perspectives underscored the importance of a nuanced assessment to refine and tailor the STEM LLC program to better meet the holistic needs of its residents.

Step 6 – Select and Adapt Instrumentation

Now that you have decided the best methodologies, data sources, and timeframe, it is time to look at your instruments themselves. When I say instruments, this is the thing that you will use to apply the design you selected. So in our example, this would be the content that we actually put into the surveys and the interviews or focus groups. We could look at instruments that already exist (hello crowdsourcing from other campuses!), however we will need to critically look at those instruments for cultural bias for specific campuses. That is to say that we should never take a pre-existing instrument and use it because another campus does. As we look consider our instruments, here are some questions I would ask for our team to consider:

What specific language in this instrument does not translate to our campus?
Does this instrument actually measure what we are interested in measuring from the STEM LLC?
What aspects of our specific assessment questions and success criteria are missing from this instrument?
What pieces of this instrument do not align to what we care to explore? Does removing that content make this instrument no longer work?
Are there specific subsets of our population that will not trust the questions in this instrument?
Do we need to translate this instrument into different languages?

At this phase, we have to engage in critical reflection because the instruments we are using need to be valid in our cultural-context. This is a great time to include your cultural guides and to bring in multiple perspectives to this process, since the goal is to be as culturally relevant as possible. (PS if you realize that translation of an instrument is necessary, forward translation is not sufficient. Instead forward/backward translation, multiple forward translation, or translation by committee is recommended within the CRE approach. Tap into some experts on your campus around translation for assistance here!)

Perhaps after reviewing other instruments you realize that no instrument actually meets your needs or cannot be adjusted to meet your specific cultural context. This is super common in a CRE approach, developing original instruments for your CRE assessment is often the best choice. If it is decided that an original instrument is necessary, ensure that you account for this time in your assessment timeline and that you add to your assessment team as necessary for absent skills in instrument creation. For example, if you have never created a survey before, reach out on your campus to find out who has that skill set and ask them to join your assessment team. Whether you build your own instrument or adapt a previous instrument, I would highly encourage you to pilot the surveys and interviews with the appropriate population before finalizing your design to ensure cultural relevance and understanding. While this may not happen in this case study (since an LLC is usually a really small group), if I was doing a survey that was going to be sent to the entire campus I would do a pilot first of maybe the RAs first to get feedback.

Step 7 – Collect Data

Outside the instruments, deciding the procedures for the actual data collection is the next phase of the CRE framework. These need to be responsive to your specific program and campus cultural context, this is relevant whether you are collecting quantitative or qualitative data. For example, if the STEM LLC had a high population of students with jobs, trying to host interviews during strictly regular business hours may mean that you miss out on a section of your population. Beyond the data collection process itself, we also need to consider ourselves-as-instrument in this phase. Questions to consider in preparing for data collection include:

Who holds the power to decide who is included in data collection?
For interviews or focus groups, who will participants feel most comfortable sharing information with?
For surveys, how can we honor our students’ humanity within the collection process?
How will we ensure that participants know that this process is voluntary?
What experiences from my life affect how I interpret what I see and hear?
What shared lived experiences are necessary to develop trust with participants?

This phase also includes knowing who on your campus (or department) holds the power to grant access to specific information. This could be specific individuals or groups. Being aware of who they are and how to respectfully gather this information should be included in your culturally relevant protocols. I would encourage you to consider how you could bring students into the data collection process, specifically I would advocate for trained students to perform the interviews within this assessment of the STEM LLC and also consider how you could partner with the STEM LLC RAs to potentially host a survey completion event as a way to encourage students to complete the survey and draw attention to it.

Wrapping Up

Only one more section left, you are so close to having the entire Culturally Responsive approach under your belt! Make sure to stop back next month where we will wrap up the rest of this case example by working through our data collection instruments and the data collection process through the Culturally Responsive Evaluation framework.