TLDR: A new research paper introduces a 16-item trust measurement instrument designed for laypersons to evaluate their trust attitudes towards AI systems, particularly in medical support contexts. Developed through a six-stage psychometric process, the instrument assesses eight key trust factors like technical competence, understandability, and institutional credibility. It was found to be empirically reliable and valid, providing a much-needed standardized tool for researchers to understand and build trustworthy AI.
As Artificial Intelligence (AI) continues to integrate into various aspects of our lives, particularly in critical fields like healthcare, the concept of trust becomes paramount. How do ordinary people, those without expert knowledge of AI, perceive and trust these advanced systems? This question has been a significant challenge for researchers, as there hasn’t been a standardized and robust way to measure human trust in AI from a non-expert perspective.
The increasing presence of AI has highlighted the need for trustworthy systems. Incidents like AI misclassifying images or chatbots turning problematic have fueled public skepticism. In healthcare, controversies such as the alleged illegal sharing of patient records or inaccuracies in AI diagnosis have further eroded public confidence. Yet, paradoxically, some studies show that frequent AI users tend to over-trust algorithmic outputs, even when the systems malfunction – a phenomenon known as automation bias.
A major hurdle in advancing research on trustworthy AI has been the lack of a consistent method for evaluating trust. Different studies often use varying definitions of trust, leading to diverse metrics and making meaningful comparisons difficult. Existing measurement tools, often adapted from human-robot or human-automation trust research, are frequently context-specific or lack comprehensive validation.
Addressing this critical gap, a recent research paper titled “Human and AI Trust: Trust Attitude Measurement Instrument” by Retno Larasatia, Anna De Liddo, and Enrico Motta introduces a new, rigorously developed and validated instrument to measure trust attitudes towards AI systems. This instrument is specifically designed for laypersons (non-experts) and was developed using the context of AI medical support systems, such as cancer/health prediction tools.
Developing a Robust Trust Scale
The development of this trust measurement instrument followed established psychometric principles, involving six meticulous stages:
- Item Development: Researchers identified key trust factors through a combination of literature review (deductive approach) and exploratory research using online surveys and focus groups (inductive approach). This helped uncover factors not immediately apparent from existing theories and contextualize them for healthcare scenarios.
- Item Evaluation: An initial set of items was reviewed by experts from AI, medicine, and scale development fields to ensure content validity, clarity, and relevance. Cognitive interviews with the target population (general public) further refined the items to ensure they were understandable and accurately reflected the intended concepts.
- Survey Administration: The refined instrument was then administered to a large sample of 300 participants via an online survey, after they viewed videos of AI medical support systems.
- Test of Dimensionality: Confirmatory Factor Analysis (CFA) was used to confirm that the proposed eight trust factors (dimensions) fit the collected data well.
- Test of Reliability: Both internal consistency (using Cronbach’s alpha and omega coefficients) and test-retest reliability were assessed. The instrument demonstrated high consistency and stability over time.
- Test of Validity: Content validity (assessed by experts and cognitive interviews) and construct validity (convergent and discriminant validity) were established. The instrument also showed concurrent validity with subjective trust levels, meaning its factors correlated positively with how much people generally trusted AI. However, predictive validity for trust levels was not fully supported, suggesting a more complex, possibly non-linear, relationship between trust factors and overall trust.
The Eight Pillars of Trust
The final instrument consists of 16 items, measuring eight distinct trust factors, each with two statements:
- Perceived Technical Competence: The AI system’s ability to perform tasks accurately and correctly.
- Perceived Understandability: The user’s ability to form a mental model of how the AI works and predict its future behavior.
- Perceived Reliability: The system’s consistent and predictable functioning over time.
- Perceived Helpfulness: The system’s capacity to provide adequate, effective, and responsive assistance.
- Personal Attachment: The user finding the system agreeable, preferable, and suiting their personal taste.
- User Autonomy: The user’s feeling of control over the AI system’s functions and features.
- Faith: The user’s confidence in the AI system’s future ability, even in untried situations.
- Institutional Credibility: Belief that the AI system has been tested or certified by reputable overseeing bodies.
This comprehensive approach ensures that the instrument captures various facets of trust, from rational assessments of performance to more emotional and institutional considerations. While some overlap was noted between certain trust factors, the overall instrument demonstrated strong reliability and validity.
Also Read:
- Navigating the Nuances of AI Trustworthiness: A Review of Leading Ethics Research
- Navigating Trust: When Should We Defer to AI’s Expertise?
Implications for Trustworthy AI
This research provides a significant contribution to the field of human-AI interaction. By offering a methodologically sound and validated instrument, future researchers can now systematically measure and compare non-experts’ trust in AI systems, particularly in sensitive areas like medical support. This will be crucial for designing, developing, and deploying AI systems that are not only technologically advanced but also genuinely trustworthy and accepted by the public. The paper encourages the replication and adaptation of this instrument to further solidify its validity and generality across different AI applications. You can read the full paper here: Human and AI Trust: Trust Attitude Measurement Instrument.
