TLDR: A study investigates the presence of religious language in climate discourse from secular and religious NGOs using a rule-based model and large language models (LLMs). Analyzing over 880,000 sentences, the research found that the rule-based method identified more religious content. LLMs, particularly GPT-4o mini and Llama 3.3 70B, showed inconsistencies in detecting implicit religious terms like “Mother Earth” and “sacred earth,” often prioritizing explicit religious markers. The findings highlight the challenges of computationally defining religious language and suggest that LLMs reflect existing cultural biases in their interpretation.
Religious language, often thought to belong to sacred spaces, is surprisingly prevalent in contemporary discussions, even in seemingly secular areas like environmental activism and climate change debates. A recent study delves into this phenomenon, investigating how both overt and subtle forms of religious language appear in climate-related texts produced by various non-governmental organizations (NGOs), both secular and religious.
The research, titled Detecting Religious Language in Climate Discourse, introduces a novel dual approach to identify this language. One method is a rule-based model, which uses a carefully constructed hierarchical tree of religious terms drawn from ecotheology literature. The other method employs large language models (LLMs), specifically GPT-4o mini and Llama 3.3 70B, operating in a zero-shot setting, meaning they were not specifically trained for this task beforehand but relied on their general understanding of language.
The study utilized a massive dataset comprising over 880,000 sentences. These sentences were scraped from the websites of nine environmental NGOs over the past decade. The NGOs included secular groups like Greenpeace and Extinction Rebellion, as well as religious organizations such as Christian Climate Action and GreenFaith. By analyzing this extensive text, the researchers aimed to compare how the rule-based method and the LLMs detected religious language and to pinpoint areas where their interpretations agreed or diverged.
Key Findings and Methodological Insights
The results revealed several interesting patterns. The rule-based method consistently labeled more sentences as religious compared to the LLMs. This might seem counterintuitive, as LLMs are known for their advanced contextual understanding. However, the LLMs often interpreted terms identified by the rule-based tree as merely descriptive rather than conveying explicit religious meaning. This highlights a core challenge: should religious language be defined solely by its vocabulary, or by its contextual meaning and intent?
The LLMs themselves showed some inconsistencies. For instance, terms like “Mother Earth” or “sacred earth,” which carry significant spiritual connotations in various traditions, were not always consistently classified as religious by the LLMs. While GPT-4o mini classified about a third of sentences mentioning “Mother Earth” as religious, Llama 3.3 70B was slightly more inclined, labeling around 37% as such. For “sacred earth,” the difference was more pronounced, with Llama identifying 75% as religious compared to GPT’s 11.5%. This suggests that LLMs, in their default zero-shot mode, tend to prioritize explicit, conventional religious markers (like references to God or scripture) over more subtle, implicit forms of spiritual expression.
Another notable difference was Llama’s greater sensitivity to biblical references. In cases of disagreement between the two LLMs, Llama more frequently identified specific biblical passages or allusions, whereas GPT often did not. This could be due to differences in their training data or architectural designs.
Also Read:
- Unmasking Corporate Narratives: A New Benchmark for Detecting Greenwashing in Oil & Gas Video Ads
- Unpacking Financial Narratives: How AI Detects Stance on Debt, EPS, and Sales
Implications for Understanding Religious Language and AI
The study underscores the methodological complexities of computationally detecting religious language. It reveals that LLMs, while powerful, often reflect the prevailing cultural and linguistic biases embedded in their vast training data. Their tendency to equate “religious” with “explicitly religious” aligns more with conventional understandings of religion rather than broader, post-secular interpretations that acknowledge implicit religiosity in secular contexts.
Ultimately, the research concludes that LLMs, in their current zero-shot application, cannot be considered objective classifiers for religious language. Instead, they serve as tools that reproduce the ambiguities and biases present in their training data and the prompts they are given. This study significantly contributes to digital methods in religious studies, offering insights into both the potential and limitations of using advanced AI to analyze how the sacred continues to manifest in contemporary climate discourse.
