TLDR: A study evaluated a vision language model (InternVL2) for generating image catalogue descriptions for archives. It found that while people could generally distinguish AI-generated from human-written descriptions, higher quality AI outputs were harder to spot. Experts showed less willingness to adopt AI due to concerns about accuracy and preservation responsibility. The research emphasizes the need for human oversight, domain-specific AI fine-tuning, and transparent AI systems to build trust for successful integration in archival workflows.
The accelerating growth of photographic collections in archives and museums presents a significant challenge for manual cataloguing. This has spurred interest in using advanced artificial intelligence, specifically vision language models (VLMs), to automate the creation of descriptive metadata for these vast visual records. A recent study, titled ArchiveGPT: A human-centered evaluation of using a vision language model for image cataloguing, delves into whether AI-generated descriptions can achieve the quality of human-written ones and how generative AI might be integrated into existing cataloguing practices within archival and museum settings.
The research employed InternVL2, a VLM, to produce catalogue descriptions for archaeological photographic prints. These AI-generated descriptions were then put to the test, evaluated by a diverse group including archive and archaeology experts, as well as non-experts. Participants were tasked with identifying whether a description was AI-generated or human-written, rating its overall quality, and expressing their willingness to use and trust AI tools in their work.
A notable finding was that participants, across both expert and non-expert groups, were able to differentiate between AI-generated and human-written descriptions at a rate better than chance. Interestingly, both groups underestimated their own ability to make this distinction. This suggests that the VLM, used without specific fine-tuning on archival data, did not fully achieve human-level indistinguishability, often referred to as passing the “Turing test.” The study observed that human-written descriptions were more frequently and correctly identified as such compared to their AI-generated counterparts.
The perceived quality of the AI-generated descriptions was impacted by issues such as errors in optical character recognition (OCR) and instances of “hallucinations,” where the AI produced factually incorrect or fabricated information. However, the study also revealed a fascinating paradox: descriptions that were rated higher in terms of accuracy and usefulness were paradoxically more difficult for participants to classify as AI-generated. This underscores a critical point: human review remains indispensable to ensure the accuracy and quality of catalogue descriptions generated by AI, particularly in highly specialized domains like archaeological cataloguing.
Experts, in comparison to non-experts, demonstrated a lower willingness to adopt AI tools. Their primary concerns revolved around the profound responsibility of preserving accurate and reliable information, rather than solely focusing on the technical performance metrics of the AI. This highlights a crucial need for a collaborative framework where AI serves as a supportive tool for drafting initial descriptions, but human verification and oversight remain paramount. This ensures that core curatorial values, such as the provenance (origin and history) of an item and transparency in its description, are rigorously maintained.
The study concludes that the successful integration of AI into archival practices hinges not only on technical advancements, such as fine-tuning AI models with domain-specific archival data, but perhaps even more significantly on establishing trust among professionals. A transparent and explainable AI system, which allows users to understand the reasoning behind the AI’s outputs, could be a key factor in fostering this trust.
In terms of perceived quality, there was a consistent trend: human-written descriptions were rated as more accurate and useful than those generated by AI. For experts, the correlation between accuracy and usefulness ratings was particularly strong, suggesting that for professionals, these two aspects are intrinsically linked and equally vital when creating comprehensive and reliable descriptions.
Furthermore, the study observed a decline in participants’ general trust in AI tools and their willingness to use them after direct exposure to the AI model’s performance. This indicates that hands-on experience can temper initial optimism, reinforcing existing reservations, especially among experts who are acutely aware of the potential risks associated with inaccurate metadata.
Looking to the future, the researchers propose a dual approach focusing on technical advancements and trust-building measures. Technical advancements include further fine-tuning of VLMs on specific archival datasets to minimize errors and hallucinations. Trust-building measures involve providing targeted training and education for archivists and collection managers, enabling them to critically engage with AI tools and balance efficiency gains with their curatorial responsibilities for preservation. The ultimate vision is an “explainable AI pipeline” that offers transparency, auditability, and reversibility, allowing users to understand and verify the AI’s output, thereby building greater confidence and facilitating a more effective human-AI collaboration.
Also Read:
- AI Models Streamline Metadata Creation for Data Catalogs
- Navigating AI Transparency: Digital Humanists Weigh In on Disclosure Practices
Conclusion
This research emphasizes that while AI holds immense promise for automating cataloguing processes, its successful adoption in cultural heritage institutions necessitates a human-centered approach that prioritizes accuracy, transparency, and professional trust. It advocates for a future where AI acts as a trusted collaborator, enhancing archival work while upholding the integrity and evidential value of historical collections.
