top of page

Sentiments on Emotion AI in Hiring

The University of Texas at Austin // 2023

Co Workers
SUMMARY

Can emotion AI objectively deduce behaviors from emotion-tracking data? Users are doubtful.

AI’s challenges with transparency and explainability have become engrained through all stages of the hiring process in the last decade. In this pilot study, we designed a mock interview experiment to quantify the impact of AI-driven facial emotion recognition. 

 

We conducted 9 remote mock interviews and analyzed the answers using an open source Facial Expression Recognition (FER) model on Python used for sentiment analysis of images and videos. We curated individualized analytics to understand the impact of AI emotion-tracking on video interviews and how such tools can be used for effective mock video interview preparation.

 

We found that while facial recognition adds complexity and stress in interview settings, emotion-tracking outputs can be used for increased self awareness in behavioral interviews. We hope to empower people interviewed with AI and encourage transparency and helpful feedback loops from AI interview-prep companies.

EAI-Prototype.png
PROBLEM + GOALS

"Job applicants who don't fit the benchmark data may experience encoded bias at scale..."

Research Questions
  1. How does emotion-tracking/EAI make participants feel?
  2. What information do participants want to see in their EAI results?
  3. What is the best visualization of emotion-tracking reports?

Objectives
  • Improve awareness of the increasing prevalence of AI in hiring processes and how AI works in this context
  • Enable participants to identify what kind of visual feedback they find most useful
  • Invite participants to express their opinions about the use of AI in hiring practices
Intended Outcomes for Stakeholders
  • Job Applicants: We advocate for the use of emotion-tracking AI training tools to increase self-awareness and improve performance in video interviews. This may also potentially challenge biased automated systems.
  • Hiring Companies: We encourage the adoption of AI tools that enhance transparency and foster an inclusive hiring process. This helps in recognizing and reducing unconscious biases, avoiding discriminatory practices and broadening the diversity of the workforce.
  • Creators of AI Tools: We promote the use of diverse data sets in the development of AI-based facial analysis tools, aiming to improve the accuracy and fairness of these products. This is crucial in ensuring that AI technologies serve a broad and diverse user base effectively.
CONTEXT

Contributing UX and leadership

What I Did
  • Developed research methodology, managed the project timeline and defined deliverables
  • Conduct interviews
  • Designed customized interactive reports for each participant and project presentations
  • Skills included UX, prototyping, survey design and analysis, remote interviews, project management, presentation design, academic writing, and IRB human subjects research training.

Tools
Figma, Google Survey, Google Docs, Google Sheets, Zoom

Timeline
3 Months

Providing Project Management and UX on a Balanced Team
Vanessa Sanchez, HCI & Responsible AI MSc Student: Project Management, UX Research and Design
Kyle Soares, Computer Science MSc Student: Application Research and Development
Dhanny Indrakusuma, Data Science MSc Student: Data Analysis and Data Visualization
Silvia Dalben Furtado, AI in Journalism PhD Candidate: Literary Research and Critical Analysis
RESULTS

5 out of 9 participants agreed with the EAI analysis while many were surprised and wanted to know more

Video is shown with participant's permission.

"It's not just my expression that matters; what about my voice, my body language, etc.?"

Participant Insights
  • Sentiment: Overall positive response to reports
  • Ground Truth: 5/9 participants agreed with the EAI analysis while many were surprised
  • Concerns: Most participants were concerned about usage of EAI analysis in future interviews, especially if it makes final hiring decision
  • Satisfaction: 7/9 participants would use EAI tool again for interview prep​
Successful performance against objectives
Our team not only met all objectives we set for ourselves--we went above and beyond, leveraging our unique skill sets to achieve something none of us could have done on our own. Overall, we collaborated successfully, hit milestones and had a successful outcome.
Factors that may have impacted our results:
  • Limited access to controlled interview environmentsResults might be affected by camera angle or lighting
  • Difficult to manufacture realistic behaviors in participants for "fake interview"
  • Participants may have had different reactions based on who was interviewing them; although our team members had scripts to follow and were paired with strangers, some of us deviated from the script, some of us chose to be more personable, and some of us were intentionally flat.
  • Limitations of small sample size and broad range of participants prevented us from achieving saturation in this pilot study.
REFLECTION

Research is iterative; Stay nimble yet aligned

Process Insights
  • By beginning with a detailed research plan, I was able to streamline process, keep the team aligned, and deliver on goals.
  • Align on goals and protocols so everyone is rowing in the same direction and findings are defensible.
  • Reality will force plans to shift, so stay nimble, have some backup plans, and discuss as a team how this impacts the study.
  • Meet people where they're at to enable fruitful discussions, both within the team and with research participants.
Design insights
  • Provide a timebar on video playback and AI analysis chart so user can see the correlation
  • Users would like to connect data performance to behavioral insights and suggestions for improvement
  • Users would like to evaluated holistically, not just by facial expression
  • Practicing with a real person via zoom vs. practicing alone with an AI interface may yield different results
  • Benchmarking performance against other users interested participants but also caused some discomfort​
Future work
  • Emotional analysis as prep tool: Can emotional analysis be used with NLP analysis for better interview preparation?
  • Bias study: Who is most affected by AI emotional analysis in interviews?
  • Connect emotions to performance: Which emotions are best for job offers?
bottom of page