Common Metrics & Outcomes

In evaluation of CISE REU, goals and objectives inform the purpose of the program and expectations for participants (more specifically than the Recommended Indicators). These are measured by pre-defining the theoretical constructs and metrics to be employed. A construct is the subjective conceptual element, or variable that is being measured, for example, computing self-efficacy. A metric is the standard of measurement used to gauge the construct, such as a battery of survey items designed to measure computing self-efficacy. Simply put, a construct is a variable, i.e. something with differing value. Constructs are linked to outcomes in that outcomes (e.g. increased interest in graduate school) are measured by constructs (e.g. interest in graduate school scale).


Common CISE REU Constructs

Research Exposure: reading and discussing research, understanding research process, statistical analysis, ethics, presentation of research

  • Knowledge acquisition – self report of research content knowledge gain
  • Attitudes toward – like/dislike of research activities
  • Activity (experience) - research projects undertaken, any planned outcomes as a result, like posters, products, papers

Computing: programming, math, problem solving, and interest in computing majors and computing research

  • Knowledge acquisition - self report of computing content knowledge gain
  • Attitudes toward - like/dislike of computing activities
  • Commitment - student self reported intent to remain in major, plans for graduate research in computing, plans for computing career

Self-Efficacy: confidence in one’s ability

  • Research Confidence level – belief in ability to conduct and engage in research activities
  • Confidence level - belief in ability to conduct and engage in computing activities

Site Administration: overall site operations, including auxiliary activities

  • Participant demographics - gender, ethnicity and underrepresented minority participant levels, as defined by glossary descriptions
  • Participant satisfaction levels - enjoyment in program activities beyond research and computing, e.g. social or soft skill development activities
  • Attribution - to what degree participants attribute the research experience to future plans
  • Faculty & peer support – satisfaction with faculty assistance and group work

Additional Indicators are Encouraged:

  • Help Seeking & Coping Behaviors (see Sample Surveys & Instruments)
  • Faculty Relationships (see Sample Surveys & Instruments)
  • Achievement Measures of Research and Computing Domains (see Sample Surveys & Instruments)
  • Use evaluative faculty surveys of student ability in addition to self report survey
  • Use knowledge tests of student ability in addition to self report survey
  • Longitudinal Indicators: (see Longitudinal Guidelines & Recommendations)
  • GPA
  • GRE Score
  • Graduation in Computing
  • Plans/Enrollment in Computing Graduate School
  • Career Placement in Computing
  • To what degree do participants attribute REU to their future plans

 

Chart of Common Constructs, Metrics & Outcomes

 

Construct Sample Metrics Outcomes

Research Exposure
   Research Activities Survey items* Students will engage in computing research
   Research Knowledge Survey items; final student presentations; interviews; assessment (test/faculty) Student knowledge will increase
   Research Attitudes Survey items; interviews* Student attitudes will be positive

Computing
   Computing Knowledge Survey items; final student presentations; interviews; assessment (test/faculty) Student knowledge will increase
   Computing Attitudes Survey items; interviews Student attitudes will be positive
   Commitment Survey items; interviews Students will be committed to computing majors and interested in graduate school in CISE fields

Self Efficacy
   Research Confidence Survey items; interviews Student confidence level in research & computing areas will increase
   Computing Confidence Survey items; interviews

Site Administration
   Faculty Support Survey items; interviews Students will perceive faculty support
   Peer Relationships Survey items; interviews Students will develop peer relationships and report team work
   Overall Satisfaction Survey items; interviews Students will report satisfaction with experience & also provide formative feedback for future
   Demographics Gender; Ethnicity; Academic Level; GPA Descriptive data will reveal trends
   Attribution Survey items; interviews Students will attribute REU as important factor in computing commitment

*Survey items are being analyzed for scale development; references to specific scales and/or items will be forthcoming. Interview protocols are available in the Qualitative Methods section of the toolkit.