Glossary of Terms

Unless otherwise noted, these definitions originated from the NSF’s 2002 User-Friendly Handbook for Project Evaluation [pdf]. REU specifications have been incorporated by the CISE REU Assessment Work Group.

  • Accuracy: The extent to which an evaluation is truthful or valid in what it says about a program, project, or material.
  • Achievement: Performance as determined by some type of assessment or testing.
  • Activities: The distinct education exercises or functions that take place throughout the REU experience, e.g. training workshops, lectures, social endeavors, that are designed to compliment and facilitate research projects [CISE REU Assessment Work Group]
  • Affective: Consists of emotions, feelings, and attitudes.
  • Anonymity (provision for): Evaluator action to ensure that the identity of subjects cannot be ascertained during the course of a study, in study reports, or in any other way.
  • Assessment: Often used as a synonym for evaluation. The term is sometimes recommended for restriction to processes that are focused on quantitative and/or testing approaches. SEE “The Distinction between Assessment and Evaluation.”
  • Assumptions: The beliefs we have about the program, the people involved, and the context and the way we think the program will work.
  • Attitude: A person’s opinion about another person, thing, or state.
  • Attrition: Loss of subjects from the defined sample during the course of data collection, particularly in longitudinal evaluation of REU students.
  • Audience(s): Consumers of the evaluation; those who will or should read or hear of the evaluation, either during or at the end of the evaluation process. Includes those persons who will be guided by the evaluation in making decisions and all others who have a stake in the evaluation (see stakeholders).
  • Background: Information that describes the project, including its goals, objectives, context, and stakeholders.
  • Baseline: Facts about the condition or performance of subjects prior to treatment or intervention (i.e. the REU experience).
  • Behavioral objectives: Measurable changes in behavior that are targeted by a project (e.g. publications, poster sessions, conference attendance, etc.)
  • Bias: A point of view that inhibits objectivity.
  • Case study: An intensive, detailed description and analysis of a single project, program, or instructional material in the context of its environment.
  • Categorical scale: A scale that distinguishes among individuals by putting them into a limited number of groups or categories (e.g. undergraduate vs graduate student)
  • Checklist approach: The principal instrument for practical evaluation, especially for investigating the thoroughness of implementation.
  • Coding: To translate a given set of data or items into descriptive or analytic categories to be used for data labeling and retrieval. For example, categorical data (i.e. class standing) as a number (e.g. freshman=1, sophomore=2) for analysis; or open-ended survey responses placed into themes (e.g. initial interest in research)
  • Cohort: A term used to designate one group among many in a study. For example, “the first cohort” may be the first group to have participated in the REU program at a given institution.
  • Component: A physically or temporally discrete part of a whole. It is any segment that can be combined with others to make a whole, such as a component of the evaluation.
  • Conceptual scheme: A set of concepts that generate hypotheses and simplify description, through the classification and categorization of phenomena, and the identification of relationships among them.
  • Conclusions (of an evaluation): Final judgments and recommendations.
  • Content analysis: A process using a parsimonious classification system to determine the characteristics of a body of material or practices, such as student journals.
  • Context (of an evaluation): The combination of factors accompanying the study that may have influenced its results, including geographic location, timing, political and social climate, economic conditions, and other relevant professional activities in progress at the same time.
  • Continuous scale: A scale containing a large, perhaps infinite, number of intervals. Units on a continuous scale do not have a minimum size but rather can be broken down into smaller and smaller parts. For example, grade point average (GPA) is measured on a continuous scale, a student can have a GPA or 3, 3.5, 3.51, etc. (See categorical scale.)
  • Criterion, criteria: A criterion (variable) is whatever is used to measure a successful or unsuccessful outcome, e.g., grade point average.
  • Criterion-referenced test: Test whose scores are interpreted by referral to well-defined domains of content or behaviors, rather than by referral to the performance of some comparable group of people.
  • Cross-case analysis: Grouping data from different persons to common questions or analyzing different perspectives on issues under study. (see also Qualitative evaluation)
  • Cross-sectional study: A cross-section is a random sample of a population, and a cross-sectional study examines this sample at one point in time. Successive cross-sectional studies can be used as a substitute for a longitudinal study. For example, examining today’s first year students and today’s graduating seniors may enable the evaluator to infer that the college experience has produced or can be expected to accompany the difference between them. The cross-sectional study substitutes today’s seniors for a population that cannot be studied until 4 years later.
  • Demographic Indicators: Common demographic indicators for REU are gender (male, female), ethnicity (African American, Hispanic/Latino, Native American/American Indian, Pacific Islander)
  • Descriptive data: Information and findings expressed in words, unlike statistical data, which are expressed in numbers (see Qualitative evaluation)
  • Design: The process of stipulating the investigatory procedures to be followed in doing a specific evaluation.
  • Dissemination: The process of communicating information to specific audiences for the purpose of extending knowledge and, in some cases, with a view to modifying policies and practices.
  • Document: Any written or recorded material not specifically prepared for the evaluation.
  • Effectiveness: Refers to the worth of a project in achieving formative or summative objectives. “Success” is its rough equivalent.
  • Elite interviewers: Well-qualified and especially trained persons who can successfully interact with high-level interviewees and are knowledgeable about the issues included in the evaluation.
  • Ethnography: Descriptive anthropology. Ethnographic program evaluation methods often focus on a program’s culture.
  • Evaluation: The systematic determination of merit, worth, and significance of something or someone. See “The Distinction between Assessment and Evaluation.” (Source: Wikipedia, accessed online April 24, 2006 at [link]); [from NSF ADVANCE Handbook, [link]]
  • External evaluation: Evaluation conducted by an evaluator outside the organization within which the project is housed; and typically cost inhibitive for the size and scope of CISE REU site evaluations. Internal Evaluation: Evaluation is conducted by an evaluator inside the organization within which the project is based.
  • Executive summary: A non-technical summary statement designed to provide a quick overview of the full-length report on which it is based.
  • External evaluation: Evaluation conducted by an evaluator outside the organization within which the project is housed.
  • External Factors: the environment in which the program exists includes a variety of external factors that interact with and influence the program action.
  • Faculty: the scholarly staff who teach at colleges or universities. [from NSF ADVANCE Handbook, [link]]
  • Field notes: Observer’s detailed description of what has been observed, used in qualitative research methods.
  • Focus group: A group selected for its relevance to an evaluation that is engaged by a trained facilitator in a series of discussions designed for sharing insights, ideas, and observations on a topic of concern to the evaluation. Focus groups typically are comprised of 8-10 participants.
  • Formative evaluation: Evaluation designed and used to improve an intervention (e.g. REU), especially when it is still being developed.
  • Generalizability: the extent to which the research findings can be generalized to larger populations and applied to different settings (Frankfort-Nachmias and Nachmias (pg. 518). [from NSF ADVANCE Handbook, [link]]
  • Goal: A broad-based description of an intended outcome.
  • Hypothesis testing: The standard model of the classical approach to scientific research in which a hypothesis is formulated before the experiment to test its truth.
  • Impact evaluation: An evaluation focused on outcomes or payoff of a project.
  • Implementation evaluation: Assessing program delivery (a subset of formative evaluation).
  • Indepth interview: A guided conversation between a skilled interviewer and an interviewee that seeks to maximize opportunities for the expression of a respondent’s feelings and ideas through the use of open-ended questions and a loosely structured interview guide.
  • Informed consent: Agreement by the participants in an evaluation to the use, in specified ways for stated purposes, of their names and/or confidential information they supplied.
  • Institutional Review Board (IRB): an independent body constituted of medical, scientific, and non-scientific members of a particular university or college who ensure that the rights, safety, and well-being of animal and human subjects are protected in the conduct of research. [from NSF ADVANCE Handbook, [pdf]
  • Instrument: An assessment device (test, questionnaire, protocol, etc.) adopted, adapted, or constructed for the purpose of the evaluation.
  • Internal evaluator: A staff member or unit from the organization within which the project is housed.
  • Inter-rater reliability: A measure of the extent to which different raters score an event or response in the same way.
  • Intervention: Project feature or innovation subject to evaluation; could be the REU experience itself, and/or specific features within it, such as mentoring.
  • Intra-case analysis: Writing a case study for each person or unit studied.
  • Logic model: defines a research situation and priorities as well as the inputs, outputs (i.e., activities, participants, etc.), outcomes and anticipated impacts (short, medium, and long-range) of a program as well as the assumptions and external factors associated with the plan and context. The logic model graphically shows the chain of connections of how a program is expected to work to achieve the desired results. (see Evaluation Key & Handbook); [from NSF ADVANCE Handbook, [link]]
  • Longitudinal study: An investigation or study in which a particular individual or group of individuals is followed over a substantial period of time to discover changes that may be attributable to the influence of the treatment, or to maturation, or the environment. (See also cross-sectional study.)
  • Matrix: An arrangement of rows and columns used to display multi-dimensional information.
  • Measurement: Determination of the magnitude of a quantity.
  • Meta-evaluation: Evaluation of the merit of the evaluation itself.
  • Mixed-method evaluation: An evaluation for which the design includes the use of both quantitative and qualitative methods for data collection and data analysis.
  • Moderator: Focus group leader; often called a facilitator.
  • Nonparticipant observer: A person whose role is clearly defined to project participants and project personnel as an outside observer or onlooker.
  • Norm-referenced tests: Tests that measure the relative performance of the individual or group by comparison with the performance of other individuals or groups taking the same test.
  • Objective: A specific description of an intended outcome.
  • Observation: The process of direct sensory inspection involving trained observers.
  • Ordered data: Nonnumeric data in ordered categories (for example, students’ performance categorized as excellent, good, adequate, and poor).
  • Outcome: Post-treatment or post-intervention effects, such as number of participants entering graduate programs in computing.
  • Paradigm: A general conception, model, or “worldview” that may be influential in shaping the development of a discipline or subdiscipline (for example, “the classical, positivist social science paradigm in evaluation”).
  • Participants: Those individuals who are directly involved in a project, including the faculty and students.
  • Participant observer: An evaluator who participates in the project (as participant or staff) in order to gain a fuller understanding of the setting and issues.
  • Performance evaluation: A method of assessing what skills students or other project participants have acquired by examining how they accomplish complex tasks or the quality of the products they have created (e.g. poster evaluations, final presentations).
  • Population: All persons in a particular group, such as an entire REU cohort, or the sum total of all REU participants.
  • Prompt: Reminder used by interviewers to obtain complete answers.
  • Purposive sampling: Creating samples by selecting information-rich cases from which one can learn a great deal about issues of central importance to the purpose of the evaluation.
  • Qualitative evaluation: The approach to evaluation that is primarily descriptive and interpretative, by using open-ended items, interviews, and focus groups.
  • Quantitative evaluation: The approach to evaluation involving the use of numerical measurement and data analysis based on statistical methods.
  • Random sampling: Drawing a number of items of any sort from a larger group or population so that every individual item has a specified probability of being chosen.
  • Recommendations: Suggestions for specific actions derived from evidence-based conclusions.
  • Recruitment: an activity, or set of activities, in which the organization attempts to identify and attract candidates to meet the requirements of the REU program at a particular institution. [from NSF ADVANCE Handbook, [link]]
  • Reliability: the consistency of a measuring instrument, that is, the extent to which a measuring instrument exhibits variable error (Frankfort-Nachmias and Nachmias (pg. 524). [from NSF ADVANCE Handbook, [link]]
  • Research Questions: The overall questions that guide the study of the program, and determine the types of assessment and analyses to be conducted; these are established before beginning data collection. [CISE REU Assessment Work Group]
  • Sample: A part of a population, e.g. students within a computing discipline.
  • Secondary data analysis: A reanalysis of data using the same or other appropriate procedures to verify the accuracy of the results of the initial analysis or for answering different questions.
  • Self-administered instrument: A questionnaire or report completed by a study participant without the assistance of an interviewer.
  • Stakeholder: One who has credibility, power, or other capital invested in a project and thus can be held to be to some degree at risk with it.
  • Standardized tests: Tests that have standardized instructions for administration, use, scoring, and interpretation with standard printed forms and content. They are usually norm-referenced tests but can also be criterion referenced.
  • Strategy: A systematic plan of action to reach predefined goals.
  • Structured interview: An interview in which the interviewer asks questions from a detailed guide that contains the questions to be asked and the specific areas for probing.
  • Summary: A short restatement of the main points of a report.
  • Summative evaluation: Evaluation designed to present conclusions about the merit or worth of an intervention and recommendations about whether it should be retained, altered, or eliminated.
  • Sustainability: A systemic concept, relating to the continuity of economic, social, institutional and environmental aspects of human society. It is intended to be a means of configuring civilization and human activity so that society, its members and its economies are able to meet their needs and express their greatest potential in the present, while preserving biodiversity and natural ecosystems, and planning and acting for the ability to maintain these ideals in a very long term. Sustainability affects every level of organization, from the local neighborhood to the entire planet. (Wikipedia accessed online at [link] April 24, 2006). [from NSF ADVANCE Handbook, [link]]
  • Transportable: An intervention that can be replicated in a different site.
  • Triangulation: In an evaluation, an attempt to get corroboration on a phenomenon or measurement by approaching it by several (three or more) independent routes. This effort provides confirmatory measurement. For example, use of student surveys, student interviews and faculty assessment of student outcomes.
  • Underrepresented minorities: The term "minority" refers to all groups other than white, "underrepresented minorities" includes three groups whose representation in science and engineering is less than their representation in the population: African Americans, Hispanics, and American Indians [NSF; ACM; IEEE]; CRAW includes Pacific Islanders; Women and persons with disabilities are frequently the target of BPC initiatives in computing, as they are disproportionately represented in computing fields, although they are not included in the definition of the term “underrepresented minorities”
  • Utility: The extent to which an evaluation produces and disseminates reports that inform relevant audiences and have beneficial impact on their work.
  • Utilization of (evaluations): Use and impact are terms used as substitutes for utilization. Sometimes seen as the equivalent of implementation, but this applies only to evaluations that contain recommendations.
  • Variable: A measurable characteristic or value that can be changed, e.g. intent to attend graduate school, self-efficacy, computing knowledge [CISE REU Assessment Work Group]
  • Validity: The soundness of the inferences made from the results of a data-gathering process.
  • Verification: Revisiting the data as many times as necessary to cross-check or confirm the conclusions that were drawn.
     

The Distinction between Evaluation and Assessment
In the field of evaluation, there is some degree of disagreement in the distinctions often made between the terms 'evaluation' and 'assessment.' Some practitioners would consider these terms to be interchangeable, while others contend that evaluation is broader than assessment and involves making judgments about the merit or worth of something (an evaluand) or someone (an evaluee). When such a distinction is made, 'assessment' is said to primarily involve characterizations – objective descriptions, while 'evaluation' is said to involve characterizations and appraisals – determinations of merit and/or worth. Merit involves judgments about generalized value. Worth involves judgments about instrumental value. For example, a history and a mathematics teacher may have equal merit in terms of mastery of their respective disciplines, but the math teacher may have greater worth because of the higher demand and lower supply of qualified mathematics teachers. A further degree of complexity is introduced to this argument when working in different languages, where the terms 'evaluation' and 'assessment' may be variously translated, with terms being used that convey differing connotations related to conducting characterizations and appraisals. (Source: Wikipedia, accessed online April 24, 2006 at [link]) [from NSF ADVANCE Handbook, [pdf]