Moderated AI Trust Study

Customer Payment Predictions

Payment Predictions in D365 Finance. Copyright property of Microsoft

Study Objective

The goal of this study is to track, benchmark, and report AI trust and related metrics over time to examine impact of research/design on this AI-related feature and functionality.

Methodology

Moderated, 1:1 60-minutes session with high fidelity, interactive prototype

Persona / Participant Profile

6 Collection agents or collection clerks. Screener questions used to recruit participants were defined by department, finance specialty area, job level, etc.

Study Details

1. Introduction to participants - message providing context, instructions, and objectives to participants

2. Pre-Study Brand Perception - Survey measuring brand perception of company and competitors.

3. Scenario - contextual situation provided to participants to better understand the purpose of the test tasks

4. Tasks - 3 singular tasks. Each task is followed up with follow-up questions to gather specific feedback about UI elements, UX processes, etc.

5. Post-Study Metrics - Survey measuring AI Confidence, AI Work Usefulness, AI SAT using Forms.

6. Post-Study Brand Perception - Question asked out-loud directly to participants.

Results & Impact

• AI Trust Scorecard was the main output in the final report and illustrated how the tested solution measures.

• These findings contributed to the overall AI research for the department across apps and solutions

• Important UX and AI-related issues discovered. Design recommendations provided to address these issues.

• Additional UX such as CSAT were very positive.

• Brand perception were favorable.