TLDR: Hierarchical Conformal Classification (HCC) is a new method that improves standard Conformal Prediction by incorporating class hierarchies into prediction sets. Instead of flat labels, HCC provides compact and specific predictions by leveraging relationships between categories, ensuring the true label is covered while making results more intuitive. Evaluated on text, image, and audio datasets, HCC produces smaller, more meaningful prediction sets and was preferred by users over traditional methods.
In the world of machine learning, especially when classifying data, understanding uncertainty is crucial. Conformal Prediction (CP) is a powerful method that helps quantify this uncertainty by providing a set of possible labels for a given input, guaranteeing that the true label is included with a high probability. However, traditional CP often treats all categories as equal, ignoring any natural relationships or hierarchical structures that might exist between them.
Imagine classifying an image of an animal. Standard CP might give you a long list of specific animal breeds. But what if the model is less certain about the exact breed? It would be more helpful to know it’s definitely a ‘domestic animal’ or even just an ‘animal’ rather than a long, confusing list of possibilities. This is where the concept of hierarchical classification comes into play.
Researchers Floris den Hengst, In`es Blin, Majid Mohammadi, Syed Ihtesham Hussain Shah, and Taraneh Younesian from Vrije Universiteit Amsterdam and Sony Computer Science Laboratories – Paris have introduced a new approach called Hierarchical Conformal Classification (HCC). This method extends the capabilities of standard Conformal Prediction by integrating the existing hierarchical relationships among class labels directly into the prediction process. Instead of flat, unstructured labels, HCC uses a given class taxonomy, often modeled as a Directed Acyclic Graph (DAG), to create prediction sets that can include labels from different levels of the hierarchy – from very specific ‘leaf’ categories to more general ‘ancestor’ nodes.
The core idea behind HCC is to find prediction sets that are both compact and specific, while still maintaining the crucial coverage guarantees of conformal prediction. This means the prediction set should be small and easy to understand, but also provide meaningful information by potentially grouping similar fine-grained categories under a broader, more general parent category when appropriate. For example, instead of listing ‘Egyptian cat’, ‘tabby’, and ‘tiger cat’, HCC might simply predict ‘domestic cat’ if it’s sufficiently confident at that higher level.
To achieve this, HCC formulates the problem as a constrained optimization, aiming to minimize the size of the prediction set and the number of leaves it covers, while ensuring the true label is still included. A key innovation is how they tackle the complex, combinatorial nature of searching through all possible hierarchical subsets. They’ve shown that by focusing on a much smaller, well-structured subset of candidates, called ‘Non-Overlapping Leaf Covers’ (NOL-covers), they can efficiently find optimal solutions without compromising the coverage guarantee.
The process involves two main phases: calibration and inference. During calibration, the system propagates prediction scores and ground-truth labels up the hierarchy to train multiple conformal classifiers, each tailored to a specific NOL-cover. At inference time, for a new input, these calibrated classifiers are used to generate candidate prediction sets, and the one that best balances compactness and specificity is chosen. The paper also discusses methods to adjust for multiple comparisons during inference, such as the Bonferroni correction, to maintain robust coverage guarantees.
The effectiveness of HCC was demonstrated through empirical evaluations on three diverse benchmark datasets: DBpedia14 (text classification), ImageNet1K (image classification), and GTZAN Genre Collection (audio classification). These datasets were chosen because they naturally possess hierarchical structures. The results showed that HCC consistently met its coverage requirements and produced significantly smaller prediction sets compared to standard CP and other baselines, while still providing meaningful and actionable predictions. Furthermore, a user study revealed a significant preference for HCC’s hierarchical prediction sets over flat ones, highlighting its practical utility and user-friendliness.
Also Read:
- New Framework Improves Industrial Fault Diagnosis with Hierarchical Knowledge
- A New Framework for Evaluating AI Training Data Trustworthiness
This research marks a significant step towards bridging structured prediction and uncertainty quantification. By incorporating domain knowledge in the form of class hierarchies, HCC offers a foundation for more reliable, compact, and user-aligned classification systems across various domains. For more technical details, you can refer to the full research paper: Hierarchical Conformal Classification.
