VANESSA SANCHEZ
STUDENT HAI RESEARCHER @ UT AUSTIN
Human performance with emotion AI in hiring
HCI/AI RESEARCH
UX RESEARCH
PROJECT MANAGEMENT
OBJECTIVE
What is the impact of AI-driven facial emotion recognition on people in remote job interviews?
Problem
AI’s challenges with transparency and explainability have become engrained through all stages of the hiring process in the last decade. Job applicants who don't fit the benchmark data may experience encoded bias at scale and companies may lose out on candidates, reducing diversity in the workplace.
In this pilot study, we designed a mock interview experiment to examine the impact of AI-driven facial emotion recognition on interviewees.
Research Questions
-
How does emotion-tracking/EAI make participants feel?
-
What information do participants want to see in their EAI results?
-
What is the best visualization of emotion-tracking reports?
Desired Outcomes:
-
Job Applicants: Promote emotion-tracking AI tools to enhance self-awareness, improve video interview performance, and potentially challenge biased systems.
-
Hiring Companies: Support the adoption of AI tools to increase transparency, reduce unconscious bias, and promote diversity in hiring.
-
AI Tool Creators: Advocate for the use of diverse datasets in AI facial analysis tools to ensure fairness and accuracy for a broad user base.
WORK CONTEXT
UX researcher, designer, and leader in cross-disciplinary team
-
Team:
-
1 UX researcher/designer and project manager (me, MS student)
-
1 AI technology researcher and app developer (MS student)
-
1 data scientist and app developer (MS student)
-
1 literature researcher for critical analysis (PhD student)
-
-
What I Did: Developed research methodology, managed the project timeline and defined deliverables, conducted interviews, designed customized interactive reports for each participant and project presentations
-
Timeline: Jan - April 2023
-
Tools: Figma, Google Survey, Google Docs, Google Sheets, Zoom
-
Skills: UX, prototyping, survey design and analysis, remote interviews, remote interviews, project management, presentation design, academic writing, and IRB human subjects research training.
-
Timeline: 3 months
Video is shown with participant's permission.
INSIGHTS
AI vs. human: Who controls the narrative?
-
"It's not just my expression that matters; what about my voice, my body language, etc.?"
-
"[The AI] made me curious and made me wonder why it's answering the way it is."
-
"I feel I have to exaggerate facial expressions to convey positive emotions. It's not natural."
-
"I'd rather start my own business than use AI to become someone I'm not."
-
"I wasn't really thinking about it."
​
These quotes reveal participants' concerns about how AI evaluates their authenticity, with particular emphasis on non-verbal cues like facial expressions, body language, and voice. They highlight a sense of discomfort with having to exaggerate expressions to fit AI's interpretation, curiosity about the AI's decision-making, and a desire to stay true to their identity, even in the face of potentially manipulative systems. Some participants would rather reject AI's influence altogether than compromise their natural self-presentation.
OUTCOME
Positive sentiments toward EAI reports, but concerns over using EAI in hiring decisions
"I recommended...IRB training...drafted a comprehensive research plan...performed data cleaning...[and] designed individualized analytic reports."
METHODS
Ethical human subjects research, mixed methods
-
Ethics in research: I recommended we complete IRB training to help us identify risks to participants, although this was not a class requirement.
-
Research plan: I drafted a comprehensive research plan to align the team and we revised it together.
-
Screener design: I designed a screener survey with 12 questions to obtain a relevant and equitable sampling to meet our participant quota (from which we collected 27 viable respondents)
-
Data cleaning and selection: I performed data cleaning of survey responses and prioritized demographics, which resulted in 12 short-listed participants. We had 9 total participants that we interviewed via Zoom in 1 week.​
-
Script design: We followed a script with 3 behavioral questions designed to elicit a neutral baseline, confidence, and stress.
-
Post-interview: Then we asked 3 post-interview questions to learn more about the participants' perception of their interview experience.
-
​Analysis: 2 team members analyzed the answers using an open source Facial Expression Recognition (FER) model on Python used for sentiment analysis of images and videos.
-
UX design: I designed individualized analytic reports for follow-up sessions so we could get participants' reactions to their results compared to aggregated results broken down by demographics.
"5/9 participants agreed with the EAI analysis while many were surprised."
Results
-
Sentiment: Overall positive response to reports
-
Ground Truth: 5/9 participants agreed with the EAI analysis while many were surprised
-
Concerns: Most participants were concerned about usage of EAI analysis in future interviews, especially if it makes final hiring decision
-
Satisfaction: 7/9 participants would use EAI tool again for interview prep​​​​​
​​​​Findings
We found that while facial recognition adds complexity and stress in interview settings, emotion-tracking outputs can be used for increased self awareness in behavioral interviews. We hope to empower people interviewed with AI and encourage transparency and helpful feedback loops from AI interview-prep companies.​​
​
​​​Future work​
-
Emotional analysis as prep tool: Can emotional analysis be used with NLP analysis for better interview preparation?
-
Bias study: Who is most affected by AI emotional analysis in interviews?
-
Connect emotions to performance: Which emotions are best for job offers?
REFLECTION
Maintain rigor but be nimble
Process Insights
-
By beginning with a detailed research plan, I was able to streamline process, keep the team aligned, and deliver on goals.
-
Align on goals and protocols so everyone is rowing in the same direction and findings are defensible.
-
Reality will force plans to shift, so stay nimble, have some backup plans, and discuss as a team how this impacts the study.
-
Meet people where they're at to enable fruitful discussions, both within the team and with research participants.​​
​​​Design insights
-
Provide a timebar on video playback and AI analysis chart so user can see the correlation
-
Users would like to connect data performance to behavioral insights and suggestions for improvement
-
Users would like to evaluated holistically, not just by facial expression
-
Practicing with a real person via zoom vs. practicing alone with an AI interface may yield different results
-
Benchmarking performance against other users interested participants but also caused some discomfort​​​​
Researching RAI practices through remote participatory workshops
Student RAI Researcher @ UT Austin
RESPONSIBLE AI
HUMAN-AI INTERACTION
HCI RESEARCH
