TLDR: This paper introduces the Collective Predictive Coding (CPC) hypothesis, proposing that memory, attention, and language emergence extend beyond individual cognition to a collective level. It suggests language acts as a collectively formed external representation, with distributional semantics arising from group interactions. CPC affirms next-word prediction as crucial for learning a collective world model and views a language-coupled group as a “System 3” super-cognitive entity with collective memory and attention.
In a thought-provoking commentary, Tadahiro Taniguchi from Kyoto University extends the conversation around memory and attention, moving beyond individual cognitive systems to explore how these faculties, along with the emergence of language, operate at a group level. This perspective is framed by the Collective Predictive Coding (CPC) hypothesis, a framework that offers new insights into the intricate relationship between shared linguistic structures and collective cognition.
The paper, titled “Beyond Individuals: Collective Predictive Coding for Memory, Attention, and the Emergence of Language,” introduces a compelling idea: language, with its inherent distributional semantics, functions as a collectively formed external representation. This means that the very fabric of language, including its semantic and syntactic structures, isn’t just a product of individual minds but emerges from the interactions and shared experiences of a group. CPC essentially takes the familiar concepts of individual memory and attention and scales them up to a collective level, suggesting that a group can exhibit these cognitive functions in a shared manner.
A central question addressed is who embeds the “retrievable” information and structures into the sequential data of language that gives rise to distributional semantics. Taniguchi argues that it’s not solely individual humans but humans as a collective. This collective effort is crucial for understanding language emergence, which is deeply intertwined with individual cognitive abilities like memory and attention.
The CPC hypothesis proposes a hierarchical structure where individuals learn internal representations (building world models), while simultaneously, the collective learns external representations, which manifest as language or symbols. This entire process can be understood as a form of decentralized Bayesian inference, with external representation learning approximated through “language games” like the Metropolis-Hastings Naming Game.
From this viewpoint, a group of agents, connected by their emergent language, can be seen as a single entity engaging in active inference or predictive coding. This concept has even been applied to model scientific inquiry as a form of Collective Predictive Coding, demonstrating its consistency with individual agent’s free energy minimization. Furthermore, it’s hypothesized that this emergent language can exert a top-down influence on individual cognition, potentially shaping consciousness itself.
The paper also tackles a key question posed by other researchers: “Is next-word prediction the most effective way to learn language?” Taniguchi’s CPC perspective offers an affirmative answer. If language, from a collective viewpoint, integrates observations from a group of agents and structurally represents the probability distribution of those observations, then modeling this distribution through next-token prediction becomes the very essence of human language learning and, by extension, the learning of a collective world model.
This framework extends to the idea of a “System 3,” a super-cognitive system that builds upon the well-known System 1/2 framework. From this elevated perspective, memory and attention transcend individual boundaries. For instance, written language allows society to store information externally, creating a collective memory. Individuals can then access these shared memories through communication. Similarly, the attentional mechanisms within this super-cognitive system, which is the subject of group-level active inference, might be factorized into individual beings.
Also Read:
- CommonKV: A Training-Free Approach to Efficient LLM Memory Management
- Unpacking AI’s Grasp of Human Reasoning Styles in Social Games
In conclusion, this commentary highlights that exploring memory and attention through generative models, active inference, and collective predictive coding opens up new avenues for research beyond individual cognitive science. These collective inquiries promise to illuminate the profound mystery of language, revealing its origins as a group-level phenomenon. You can read the full paper here: Beyond Individuals: Collective Predictive Coding for Memory, Attention, and the Emergence of Language.
