The following section provides an overview of the statistical analyses common for the NSF Required Indicators, and a brief introduction to software packages available for conducting analyses. In many cases, CISE REU PIs do not have background in quantitative social sciences or other fields that use multivariate statistics. Consulting with other faculty or professionals on your campus who have expertise in multivariate statistics is recommended. For detailed information on statistical analyses in behavioral sciences and program evaluation, we recommend you consult the following resources:
UCLA’s Academic Technology Services (Choosing appropriate analysis)
Sample Size Calculator (determines the number of individuals needed for statistical power in your analysis)
- The Web Center for Social Research Methods: by Bill Trochim, Cornell University
- Tabachnick, B. G. & Fidell, L.S. (2007). Using Multivariate Statistics (5th Ed.). Boston, M.A.: Pearson Education, Inc.
The recommended analyses are based upon the indicators central to CISE REU sites. The CISE REU indicators and analyses are as follows.
- What is the demographic picture of REU participants?
Describe student participants by frequency of gender, ethnicity, year in school, and majors. Cross tabulations are suggested. At the final year of your site, provide a summary table of each year, along with a sum total for all three cohorts.
Class Standing Race Gender '06 '07 '08 '06 '07 '08 '06 '07 '08 Rising: African American 4 4 2 Males 6 7 8 Sophomores 4 5 7 Caucasian/not Hispanic 8 8 11 Females 7 6 6 Juniors 5 4 4 Hispanic/Latino 1 Seniors 4 4 3 Asian 1 Native American 1
- What are participant attitudes toward the REU experience, and to what degree do they attribute the research experience to future plans in CISE areas?
Describe students’ attitudes about the experience, and its impact on their plans by summarizing student survey responses to questions related to their experience overall. Means and/or percentages of responses to items should be included.
An encouraging finding was that 70% of the students indicated an interest in pursuing a doctoral degree in computing at the end of the program, up from only 46% at the start.
For REU sites who implement a pre and post assessment: In addition to the above analyses, conduct a paired-sample t test to determine whether or not there was a significant change in student attitudes between the start and end of the REU experience.*
- To what degree do REU participants:
- Acquire research knowledge?
- Acquire computing knowledge?
- Develop self-efficacy?
- Plan to remain in CISE majors?
- Develop intent to attend graduate school in CISE areas?
Describe students’ research and computing knowledge exposure and acquisition, level of self-efficacy, intent to continue in CISE at the undergraduate level, and to continue on to graduate school by summarizing survey responses to questions related to these constructs using means and/or percentages. Conduct a multiple regression analysis to determine if these factors predict commitment to computing. For example, does research knowledge acquisition predict plans to stay in a CISE major; does overall program satisfaction level predict plans to continue into graduate school?
Self Efficacy & Help Seeking Behaviors: Confidence in ability to design a computing research study significantly increased from slightly agree (M=3.1) at pretest to moderately agree (M=2.3) at post test (p<.01). Confidence in knowledge of computing research methods also significantly rose from slightly confident to moderately confident (M=3.0; M= 2.2; p<.01). All students reported they feel comfortable seeking assistance from professors.
For REU sites who implement a pre and post assessment: In addition to the above analyses, conduct a paired-sample t test to determine whether or not there was a significant change in student attitudes and knowledge gains between the start and end of the REU experience.*
*Note: The small number of site participants will result in lower statistical power than is standard in social science research. However, significant findings with moderate power can provide useful insights into REU outcomes. See: [link] for a statistical power analysis calculator; also refer to [link] for an overview of statistical power.
The most popular software packages for social science data analysis are Statistical Package for Social Sciences (SPSS) and Statistical Analysis Software (SAS). Both packages provide comprehensive statistical analysis procedures. SPSS is specifically tailored for use in social science research, whereas SAS is broadly applicable in business and marketing research. Both SPSS and SAS are designed for quantitative data analysis only. For qualitative research analysis, NVIVO is widely used. There are many informative sites that detail using statistical analysis software. The resources below provide clear and useful instructions that will provide guidance in conducting the common analyses noted in the above section.