TECAID Project Evaluation

TECAID’s External Evaluator, the University of Washington Center for Research & Evaluation for STEM Equity (UW CERSE), undertook ongoing evaluation of the TECAID project using quantitative and qualitative survey analysis, interviews and focus groups, and observation of TECAID activities.

TECAID Evaluation Outcomes Executive Summary (Years 1-3) Read more….

At the beginning of the project, the TECAID PI Team and CERSE articulated four areas of expected outcomes: Individual, Team, Department, and Broader Community. These four areas informed CERSE’s evaluation questions.

Evaluation Questions:

• To what extent did TECAID participants change their interactions with colleagues?
• How, if at all, did the TECAID participants use the information, tools, and strategies they learned to change their leadership, behaviors, and plans?

• What specific actions have TECAID participants taken?  To what extent were these actions motivated by TECAID participation?

• To what extent did participation in TECAID improve participating mechanical engineering (ME) department teams’ awareness about issues of culture change, diversity and bias in their departments?
• In what ways is the participation of the TECAID team members having an impact in the department?

• How are they reaching out to their colleagues? What is the level of interest among others in their mechanical engineering (ME) departments?

• In what ways could TECAID and participating ME department teams contribute to the establishment of inclusive cultures in ME departments nationally?

Specific Evaluation Activities

CERSE conducted data collection for TECAID from 2014 to 2017, as shown below. Please note, the evaluation timeline here does not include observation of Virtual Learning Community meetings; those meetings occurred on a monthly basis.

Longitudinal Surveys (pre, during, and post design) were used to assess individual, departmental, and team changes in readiness, awareness, and actions taken to impact mechanical engineering department cultures in the areas of diversity, equity, and inclusion.

Department Chair Interviews were conducted in Year 1 of the TECAID project. These interviews enabled the evaluators to gather baseline qualitative data on the diversity and climate in each participating TECAID department to more fully assess the complex nature of departmental transformation.

TECAID Team Focus Groups were conducted after Workshop 4 to gain more understanding about how the TECAID experience impacted participants as individuals, teams, and departments.

Observation of TECAID Workshops 1, 2, 3, and 4 provided additional formative feedback to TECAID leaders. This process was guided by an observation protocol that highlighted active engagement of participants, perceived usefulness of knowledge, and skills learned.

Workshop Surveys 1, 2, 3, and 4 were fielded and analyzed. The survey results and reports from each workshop enabled the TECAID project team members to better understand each workshop’s impact and helped improve the project overall.

Observation of Monthly Virtual Learning Community calls offered insight on individual, department level, and broader awareness building—as well as inter-team learning dynamics.

Consulting about the TECAID Model took place in early 2017, after the TECAID workshops were completed. CERSE shared insights from TECAID’s formative and summative survey outcomes to help guide the formation of the TECAID Model for Preparing Engineering Faculty to Lead Department Change in Diversity, Equity, and Inclusion.

Team Accomplishment Summaries were created to document TECAID team progress, and analyze similarities and differences across projects.

A Summative Report of the impacts of the TECAID project on individuals, teams, and broader community levels will be compiled by December, 2017.

Anticipated Outcomes

Individual

• Knowledge, understanding, skills, comfort level increases (on PD topics)
• Interactions improve
• Actions taken
• Participants assume responsibility for ME culture change
• Participants establish departmental practices for faculty accountability

Teams

• Actions taken/attempted
• Share information with dept.
• Generate concrete plans for leading department culture changes
• Begin implementation of plans

Department

• Departments assess policies, events for micro-inequities
• Policy, practice changes are made
• Culture change in dept. (long-term)

Broader Community

• Learning community accelerates change, promotes sharing
• Subset to advocate more broadly to ASME community
• ASME champions further change

‍