Mariah Olson

Mariah Olson

Project title:  Facilitating Metacognitive Reflection in Computer Science Courses

Project description:  In this project we aim to better understand how students self-reflect on their learning process and the potential benefits in having students do so. We created an interactive, web-based activity that allows students to self-reflect on the cognitive processes they use in their coursework, then designed a study to look at the effects of including this reflective activity in an introductory Computer Science course.

Blog:

July 28

This summer was a really cool opportunity to try new things and get out of my comfort zone. I say “opportunity” because I think the REU really ends up being whatever you choose to make out of it. While my faculty mentor was always there to help guide me in the right direction, she was still a mentor rather than a manager. She didn’t have time to monitor everything I was working on. I realized that if I wanted to make the most of my experience I needed to be the one to always set high expectations and standards for my work. I think most mentors are really appreciative when you're proactive about finding ways to contribute to the project, as long as you’re genuinely open for feedback. Put your ideas out there and volunteer to try doing things that you’ve never done before.

Also, remember the REU is designed to be a learning environment, so if you’re going to make mistakes then this is the time to do it. Keep up the mindset that you’re here to learn, and have fun!

July 21

It’s really hard to choose just one favorite thing about this summer, but a definite highlight was going to Carowinds with some of the other REU students. It was just a great trip overall. Very few lines. Very much live music, funnel cake, and good company.
 
I think the hardest thing I did this summer was fixing our IRB while my faculty weren’t in the country, because I had to figure out how to be independent and take initiative while working on something that was completely unfamiliar to me. Our original IRB was returned with eight stipulations, so I had to figure out how to respond to each of the stipulations while bothering my faculty members as little as possible. It stretched my comfort zone and helped me find a good workflow for drafting things and getting feedback from my faculty members.

July 14

This week was a reminder that research is not a clearcut process. Just when I was starting to feel like we had a pretty definite direction with the Bloom’s taxonomy project, a lot of new questions popped up. Based on the results from my preliminary study about the Bloom’s Taxonomy matrix activity, we decided to include mouse-over hints for each box in the matrix to help students understand how the cognitive process and knowledge dimensions interact. As we started coming up with hints, however, we realized that there were a lot of scenarios where we ourselves weren’t sure how the cognitive process and knowledge dimensions interact. This realization brought us back to some of the most basic questions in our project: how applicable is Bloom’s taxonomy to computer science? Is there a “right” way to interpret Bloom’s Taxonomy for computer science, and if so, what is it?

At first it was a little frustrating to go back to the very fundamental parts of our project just as we were fleshing out the final details of our study design. As I kept researching, though, I started to realize that even if our study design isn’t perfectly infallible, our real goal is to gain a better, more nuanced understanding of our research question. Even the struggles we face in designing our study using Bloom’s Taxonomy will be useful information for future researchers.

July 7

Before this week, most of my time in lab was filled with very specific projects: getting familiar with previous research so I knew what was going on, filling out our IRB, scanning sketchbooks, getting materials ready for the Bloom's study and conducting the preliminary interviews. This week was a nice change because I was able to focus more on processing all the information I’ve been absorbing since the beginning of the summer to figure out exactly how things relate to our research questions.

To start, I analyzed the interviews from the preliminary study and summarized them in a report for my research team. I tried to pick up on patterns and themes by listening back to the interviews and reflecting on my notes. In between sessions of working on my interview report, I looked at papers related to our project and built up a LaTeX bibliography for our project. It was exciting to start to see how the research findings in the papers I read related to the trends I was noticing in the interviews.

Looking back, I think this was one of my favorite weeks so far because I got to step back and look at how my project relates to the bigger picture. Everything that we’re doing is built upon people’s previous work, and hopefully what we do will inspire other people’s future work. I think that would be really rewarding to see.

June 30

When I was designing the Bloom’s Taxonomy worksheet for my preliminary study last week, I had one of the REU students fill out the worksheet and give me her feedback–a pre-preliminary study of sorts. Based on her feedback, I decided to change the worksheet a little bit by defining some terms that weren’t previously defined. It seemed harmless enough, so I didn’t run the revision past anyone and jumped right into the study. As the study went on I realized that participants felt tied to using the exact definitions I had written on the worksheet, even if they were mostly familiar with the term already. Because the written definitions were too simple/abstract, participants ended up getting seriously confused.

 
Looking back I realize that I definitely should have taken the time to run my revisions past anyone before doing the preliminary study, but my mistake helped me realize that if a definition of a term is provided then people tend to take it very literally. For a word like “factual,” which most people have some familiarity with, providing a definition seemed to make people forget the meaning that the word previously held in their mind. I realized how important it is to choose the right wording for the definitions we include in the final study, and that we should individually test all the definitions that users will see to make sure they’re clear. It seems like a little thing, but I think it could actually be pretty important to minimize our participants' frustration and get accurate results.

June 23

Today I finally got to submit my project’s IRB for approval. It was a very happy day. In between making the final revisions to the IRB this last week, I designed a preliminary study involving some of the other REU students to see if there were any major issues or things to watch out for in our Bloom’s Taxonomy study. I had four different students fill out the Bloom’s Taxonomy activity and then interviewed them each about their experience. It was pretty stressful to do, especially when I realized that I had underestimated the amount of time the interviews should take when I was trying to recruit people. I felt really bad that my interviewees were staying up to ten minutes longer than they originally committed to staying. Despite many cringeworthy mistakes, I think the input we got from the study will be very useful in going forward with the project. I also learned a lot about the interview process itself–as I’ve listened back on the interviews I’ve been able to think through what I would do differently next time. I know user experience studies are key to the HCI field so I’m glad I was able to learn about what’s involved in conducting them.

June 16

The biggest challenge I’ve been faced with so far this summer has definitely been getting the Bloom’s Taxonomy IRB ready for approval. Our study design isn’t perfectly fleshed out yet, so it’s been a struggle to find the right balance between being specific enough to get the IRB quickly approved and being vague enough that we have freedom to hone the details of our study design. I also needed to draft a lot of different types of technical documents that I've never dealt with before–interview and recruitment scripts, a study termination form and two different consent forms.

Last night I emailed all my drafts for the IRB to the rest of the research team so they can review everything. Getting that email sent off was definitely a good feeling. I’m really looking forward to the next stage of the project: doing a preliminary study where we'll have some people (probably some fellow REU students) try the activity we’re researching and get their feedback so we can hone our study design. I’ve never facilitated user experience testing before, so I think it will be another challenge but also a cool learning experience.

June 9

This week had it’s highs and lows. In both of the projects I’ve been working on, there are a lot of decisions to be made and a lot of unknowns. All the research I’ve done in the past was in my own way and on my own time, which I’ve realized has a completely different workflow than collaborating with faculty on a project that’s already been started. Before, if I got an idea for a new direction for my research I might just grab some coffee and spend all night exploring that line of research. The next day I’d have a completely different hypothesis… no-one knew and no-one cared. I’ve had to learn that in an actual, academic research project, there are deadlines to meet, meetings to plan, emails to write, and plenty of potential toes to step on. While it’s been an adjustment, I’m really thankful to be working with such an awesome group. I think things will get easier as our research questions are solidified, as well.

It’s been a good week, overall. Lot’s of micro-adventures playing pool, taking random walks, watching movies, meeting up with old friends and getting to know my new friends better. I’m really looking forward to going to the whitewater rafting center tomorrow.

June 2

My project investigates how we can encourage CS students to reflect on their learning process by interacting with a Bloom’s taxonomy matrix. Dr. Celine and Stephen already worked on a program that allows students to highlight the boxes of a Bloom’s taxonomy matrix representing the cognitive skills they utilized most in a given assignment. Right now, we are trying to design a study that makes use of this tool to encourage student’s self-reflection as well as to see if exposing students to each other’s reflections may have some benefits. One obstacle that we are facing is that the level of cognitive load required for this type of reflection to be significant may be too high for most students to willingly bear, which could potentially result in random selections and insignificant outcomes. Over the weekend, I want to research some possible ways to minimize this cognitive load and maximize students engagement with the Bloom’s taxonomy matrix.

This summer, I hope to learn how to review, analyze and write papers as well as how to design and conduct studies with proper methodology. I also hope to have some good times with the other REU students because it’s my first time living on campus so I can join in a lot more activities than I usually can as a commuter.