TLDR: SelfJudge is a novel method that accelerates large language model (LLM) inference by improving speculative decoding. It introduces a self-supervised approach to train a ‘judge verifier’ that assesses whether candidate tokens preserve semantic meaning, using the target LLM’s own likelihoods. This eliminates the need for human annotations or task-specific ground truths, allowing for automatic verifier training across diverse NLP tasks like math, coding, QA, and summarization. SelfJudge achieves superior inference speed-ups with minimal accuracy degradation and demonstrates consistent generalization across various domains, making LLM inference faster and more broadly applicable.
Large Language Models (LLMs) have transformed natural language processing, achieving impressive results across many tasks. However, their power comes with a cost: high inference latency. Generating text with LLMs is a sequential process, where each new word or ‘token’ requires significant computation, creating a bottleneck that slows down applications.
To tackle this, a technique called Speculative Decoding (SD) emerged. SD works by using a smaller, faster ‘draft’ model to propose several candidate tokens. These candidates are then quickly checked by the larger, more accurate ‘target’ model. If the target model agrees with the draft, the tokens are accepted in parallel, speeding up the generation process. Standard SD, however, is quite strict. It often rejects draft tokens even if they only have minor differences from what the target model would produce, as long as the meaning remains the same. For example, it might reject ‘It’s a strong instrument’ if the target model would have said ‘It is a strong instrument’. This conservative approach limits the potential speedup.
Introducing Judge Decoding
To overcome this limitation, ‘Judge Decoding’ was proposed. This method relaxes the strict token-by-token alignment by using a ‘judge verifier’ to evaluate the semantic compatibility of draft tokens. Essentially, it asks: ‘Does this proposed token change the meaning of the sentence?’ If not, it can be accepted, leading to faster inference. While promising, existing judge decoding methods have faced significant hurdles in training these verifiers. They typically rely on human annotations, which are subjective and time-consuming, or on tasks with clear, verifiable ground truths, like math problems or code generation. This reliance severely restricts their applicability to a wide range of NLP tasks, especially open-ended ones where a single ‘correct’ answer isn’t available.
SelfJudge: A Self-Supervised Solution
A new approach, called SelfJudge, addresses these challenges by introducing a novel self-supervised method for training judge verifiers. Instead of external supervision, SelfJudge leverages the target LLM’s own understanding of semantics to generate training data. The core idea is to measure ‘semantic preservation’ – how much the meaning of a response is maintained when a token is replaced by an alternative from the draft model.
Here’s how it works: SelfJudge identifies tokens where the draft and target models disagree. For each mismatched token, it calculates a ‘semantic preservation score’. This score quantifies the likelihood difference between the original response and a version where the token is substituted. If replacing a token minimally affects the target model’s overall response confidence and future predictions, it’s considered semantically acceptable. This innovative method allows for automatic generation of high-quality training data across diverse NLP domains, including open-ended question answering and text summarization, where traditional ground truths are absent.
Also Read:
- DiffuSpec: Accelerating LLM Inference with Diffusion Language Models
- ChunkLLM: A New Approach to Faster and More Efficient Large Language Model Inference
Performance and Generalization
Experiments demonstrate that SelfJudge achieves superior inference-accuracy trade-offs compared to existing methods like AutoJudge and standard speculative decoding. For instance, while prior judge verification methods might accept more tokens per cycle but suffer significant accuracy drops, SelfJudge delivers enhanced token acceptance with minimal performance degradation. This indicates SelfJudge’s superior applicability to general-purpose NLP scenarios, maintaining task quality while enabling efficient speculative decoding across diverse domains.
A key advantage of SelfJudge is its consistent generalization across all tasks. While other baselines show significant performance loss on certain tasks (e.g., AutoJudge struggling with general knowledge or code generation), SelfJudge maintains speedups with minimal performance degradation across mathematical reasoning (GSM8K, MATH-500), code generation (LiveCodeBench), summarization (CNN/DailyMail), and general knowledge (MMLU).
Qualitative analysis further reveals that SelfJudge intelligently relaxes verification criteria. It frequently accepts minor token transformations that preserve semantic meaning, such as changes in formatting (e.g., comma to period, single newline to double newline), synonymous word substitutions (e.g., ‘solve’ to ‘find’ in math, ‘number’ to ‘element’ in coding), and variations in sentence starters. These findings confirm that SelfJudge achieves greater inference efficiency by accepting tokens with minor discrepancies that do not alter the core meaning or functionality of the generated text.
In conclusion, SelfJudge represents a significant advancement in judge verification for speculative decoding. By pioneering a self-supervised approach that leverages the target model’s own semantic understanding, it moves beyond the limitations of human annotations and task-specific ground truths. This framework provides a practical and scalable foundation for faster and more efficient large language model inference across a broad spectrum of NLP applications. You can read the full research paper here.
