Emotion AI in Hiring | Vanessa Sanchez UX

Sentiments on Emotion AI in Hiring

The University of Texas at Austin // 2023

SUMMARY

Can emotion AI objectively deduce behaviors from emotion-tracking data? Users are doubtful.

AI’s challenges with transparency and explainability have become engrained through all stages of the hiring process in the last decade. In this pilot study, we designed a mock interview experiment to quantify the impact of AI-driven facial emotion recognition.

We conducted 9 remote mock interviews and analyzed the answers using an open source Facial Expression Recognition (FER) model on Python used for sentiment analysis of images and videos. We curated individualized analytics to understand the impact of AI emotion-tracking on video interviews and how such tools can be used for effective mock video interview preparation.

We found that while facial recognition adds complexity and stress in interview settings, emotion-tracking outputs can be used for increased self awareness in behavioral interviews. We hope to empower people interviewed with AI and encourage transparency and helpful feedback loops from AI interview-prep companies.

DOWNLOAD PAPER

PROBLEM + GOALS

"Job applicants who don't fit the benchmark data may experience encoded bias at scale..."

Research Questions

How does emotion-tracking/EAI make participants feel?
What information do participants want to see in their EAI results?
What is the best visualization of emotion-tracking reports?

Objectives

Improve awareness of the increasing prevalence of AI in hiring processes and how AI works in this context
Enable participants to identify what kind of visual feedback they find most useful
Invite participants to express their opinions about the use of AI in hiring practices

Intended Outcomes for Stakeholders

Job Applicants: We advocate for the use of emotion-tracking AI training tools to increase self-awareness and improve performance in video interviews. This may also potentially challenge biased automated systems.
Hiring Companies: We encourage the adoption of AI tools that enhance transparency and foster an inclusive hiring process. This helps in recognizing and reducing unconscious biases, avoiding discriminatory practices and broadening the diversity of the workforce.
Creators of AI Tools: We promote the use of diverse data sets in the development of AI-based facial analysis tools, aiming to improve the accuracy and fairness of these products. This is crucial in ensuring that AI technologies serve a broad and diverse user base effectively.

CONTEXT

Contributing UX and leadership

What I Did

Developed research methodology, managed the project timeline and defined deliverables
Conduct interviews
Designed customized interactive reports for each participant and project presentations
Skills included UX, prototyping, survey design and analysis, remote interviews, project management, presentation design, academic writing, and IRB human subjects research training.

Tools
Figma, Google Survey, Google Docs, Google Sheets, Zoom

Timeline
3 Months

Providing Project Management and UX on a Balanced Team
Vanessa Sanchez, HCI & Responsible AI MSc Student: Project Management, UX Research and Design
Kyle Soares, Computer Science MSc Student: Application Research and Development
Dhanny Indrakusuma, Data Science MSc Student: Data Analysis and Data Visualization
Silvia Dalben Furtado, AI in Journalism PhD Candidate: Literary Research and Critical Analysis

RESULTS

5 out of 9 participants agreed with the EAI analysis while many were surprised and wanted to know more

Video is shown with participant's permission.

"It's not just my expression that matters; what about my voice, my body language, etc.?"

Participant Insights

Sentiment: Overall positive response to reports
Ground Truth: 5/9 participants agreed with the EAI analysis while many were surprised
Concerns: Most participants were concerned about usage of EAI analysis in future interviews, especially if it makes final hiring decision
Satisfaction: 7/9 participants would use EAI tool again for interview prep

Successful performance against objectives
Our team not only met all objectives we set for ourselves--we went above and beyond, leveraging our unique skill sets to achieve something none of us could have done on our own. Overall, we collaborated successfully, hit milestones and had a successful outcome.

Factors that may have impacted our results:

Limited access to controlled interview environments; Results might be affected by camera angle or lighting
Difficult to manufacture realistic behaviors in participants for "fake interview"
Participants may have had different reactions based on who was interviewing them; although our team members had scripts to follow and were paired with strangers, some of us deviated from the script, some of us chose to be more personable, and some of us were intentionally flat.
Limitations of small sample size and broad range of participants prevented us from achieving saturation in this pilot study.

REFLECTION

Research is iterative; Stay nimble yet aligned

Process Insights

By beginning with a detailed research plan, I was able to streamline process, keep the team aligned, and deliver on goals.
Align on goals and protocols so everyone is rowing in the same direction and findings are defensible.
Reality will force plans to shift, so stay nimble, have some backup plans, and discuss as a team how this impacts the study.
Meet people where they're at to enable fruitful discussions, both within the team and with research participants.

Design insights

Provide a timebar on video playback and AI analysis chart so user can see the correlation
Users would like to connect data performance to behavioral insights and suggestions for improvement
Users would like to evaluated holistically, not just by facial expression
Practicing with a real person via zoom vs. practicing alone with an AI interface may yield different results
Benchmarking performance against other users interested participants but also caused some discomfort

Future work

Emotional analysis as prep tool: Can emotional analysis be used with NLP analysis for better interview preparation?
Bias study: Who is most affected by AI emotional analysis in interviews?
Connect emotions to performance: Which emotions are best for job offers?

VANESSA SANCHEZ

Sentiments on Emotion AI in Hiring

SUMMARY

Can emotion AI objectively deduce behaviors from emotion-tracking data? Users are doubtful.

PROBLEM + GOALS

"Job applicants who don't fit the benchmark data may experience encoded bias at scale..."

Research Questions

How does emotion-tracking/EAI make participants feel?

What information do participants want to see in their EAI results?

What is the best visualization of emotion-tracking reports?

Objectives

Improve awareness of the increasing prevalence of AI in hiring processes and how AI works in this context

Enable participants to identify what kind of visual feedback they find most useful

Invite participants to express their opinions about the use of AI in hiring practices

​​

Intended Outcomes for Stakeholders

Job Applicants: We advocate for the use of emotion-tracking AI training tools to increase self-awareness and improve performance in video interviews. This may also potentially challenge biased automated systems.

Hiring Companies: We encourage the adoption of AI tools that enhance transparency and foster an inclusive hiring process. This helps in recognizing and reducing unconscious biases, avoiding discriminatory practices and broadening the diversity of the workforce.

Creators of AI Tools: We promote the use of diverse data sets in the development of AI-based facial analysis tools, aiming to improve the accuracy and fairness of these products. This is crucial in ensuring that AI technologies serve a broad and diverse user base effectively.

CONTEXT

Contributing UX and leadership

What I Did

Developed research methodology, managed the project timeline and defined deliverables

Conduct interviews

Designed customized interactive reports for each participant and project presentations

​Skills included UX, prototyping, survey design and analysis, remote interviews, project management, presentation design, academic writing, and IRB human subjects research training.

Tools Figma, Google Survey, Google Docs, Google Sheets, Zoom Timeline 3 Months​

RESULTS

5 out of 9 participants agreed with the EAI analysis while many were surprised and wanted to know more

Video is shown with participant's permission.

"It's not just my expression that matters; what about my voice, my body language, etc.?"

Participant Insights

Sentiment: Overall positive response to reports

Ground Truth: 5/9 participants agreed with the EAI analysis while many were surprised

Concerns: Most participants were concerned about usage of EAI analysis in future interviews, especially if it makes final hiring decision

Satisfaction: 7/9 participants would use EAI tool again for interview prep​

​

​

Factors that may have impacted our results:​

Limited access to controlled interview environments; Results might be affected by camera angle or lighting

Difficult to manufacture realistic behaviors in participants for "fake interview"

Participants may have had different reactions based on who was interviewing them; although our team members had scripts to follow and were paired with strangers, some of us deviated from the script, some of us chose to be more personable, and some of us were intentionally flat.

Limitations of small sample size and broad range of participants prevented us from achieving saturation in this pilot study.

REFLECTION

Research is iterative; Stay nimble yet aligned

Process Insights

By beginning with a detailed research plan, I was able to streamline process, keep the team aligned, and deliver on goals.

Align on goals and protocols so everyone is rowing in the same direction and findings are defensible.

Reality will force plans to shift, so stay nimble, have some backup plans, and discuss as a team how this impacts the study.

Meet people where they're at to enable fruitful discussions, both within the team and with research participants.

​

Design insights

Provide a timebar on video playback and AI analysis chart so user can see the correlation

Users would like to connect data performance to behavioral insights and suggestions for improvement

Users would like to evaluated holistically, not just by facial expression

Practicing with a real person via zoom vs. practicing alone with an AI interface may yield different results

Benchmarking performance against other users interested participants but also caused some discomfort​

​

Future work​

Emotional analysis as prep tool: Can emotional analysis be used with NLP analysis for better interview preparation?

Bias study: Who is most affected by AI emotional analysis in interviews?

Connect emotions to performance: Which emotions are best for job offers?

Skills included UX, prototyping, survey design and analysis, remote interviews, project management, presentation design, academic writing, and IRB human subjects research training.

Tools
Figma, Google Survey, Google Docs, Google Sheets, Zoom

Timeline
3 Months

Satisfaction: 7/9 participants would use EAI tool again for interview prep

Factors that may have impacted our results:

Benchmarking performance against other users interested participants but also caused some discomfort

Future work