Paper2Rebuttal:
A Multi-Agent Framework for Transparent Author Response Assistance

Overview

We introduce REBUTTALAGENT, the first multi-agent framework that reframes rebuttal generation as an evidence-centric planning task. This approach addresses the limitations of current direct-to-text methods, which often lead to hallucination, overlooked critiques, and a lack of verifiable grounding. Our system decomposes complex feedback into atomic concerns, dynamically constructs hybrid contexts by synthesizing compressed summaries with high-fidelity text, and integrates an autonomous external search module to resolve concerns requiring outside literature. Crucially, REBUTTALAGENT generates an inspectable response plan before drafting, ensuring every argument is explicitly anchored in internal or external evidence. This validation process, performed on our proposed REBUTTALBENCH, demonstrates that REBUTTALAGENT outperforms strong baselines in coverage, faithfulness, and strategic coherence, offering a transparent and controllable assistant for the peer review process. Our work is summarized in the following figure, contrasting our approach with previous methods.

RebuttalAgent

REBUTTALAGENT operates as a multi-agent framework designed to transform the rebuttal process into a structured and inspectable workflow. Our system generates evidence-linked intermediate artifacts before drafting the final text to ensure grounded and controllable output. The architecture, as shown in the figure below, decomposes complex reasoning into specialized agents paired with lightweight checkers. This design exposes critical decision points, allowing authors to retain responsibility for strategic stance and final wording. The pipeline begins by distilling the manuscript into a structured summary and extracting atomic reviewer concerns for stable long-context reasoning. Guided by these concerns, the system constructs evidence bundles by retrieving high-fidelity excerpts from the manuscript and augmenting them with verifiable external literature via web search. The workflow concludes by synthesizing an explicit response plan outlining arguments and evidence links, which authors can refine through a human-in-the-loop mechanism before the system produces a formal rebuttal letter.

RebuttalBench

We introduce REBUTTALBENCH, a benchmark built from real OpenReview discussions to fix the gap between rebuttal quality and generic text metrics. It curates dense review-response dyads and evaluates what matters most for rebuttals: coverage of atomic concerns and evidence-grounded arguments, not just surface fluency.

The dataset is derived from ICLR OpenReview forums. Each item pairs the initial review with the author rebuttal and the reviewer follow-up, which provides a supervision signal for positive/negative outcomes. This yields REBUTTALBENCH-CORPUS with 9.3K review-rebuttal pairs; a harder REBUTTALBENCH-CHALLENGE subset selects the top 20 papers with over 100 reviewers to maximize concern diversity.

We evaluate responses using an LLM-as-judge rubric on a 0–5 scale across Relevance (R-Score), Argumentation Quality (A-Score), and Communication Quality (C-Score). The word cloud and top-word histogram below highlight recurring reviewer focuses like clarity, novelty, and reproducibility, which the rubric explicitly targets.

Experimental Results

We assessed REBUTTALAGENT's efficacy by comparing it with strong closed-source LLM baselines and by ablating key system components. All experiments were conducted in a fully automated mode without human intervention, providing a conservative lower bound on performance. We used four SOTA LLMs: GPT-5-mini, Grok-4.1-fast, Gemini-3-Flash, and DeepSeekV3.2 as baselines, and adopted Gemini-3-Flash as a unified LLM judge for evaluation.

REBUTTALAGENT consistently outperforms strong closed-source LLMs across all evaluation dimensions on REBUTTALBENCH when sharing the same base models. The largest gains are observed in Relevance and Argumentation Quality, with improvements in coverage up to +0.78 (DeepSeekV3.2) and specificity up to +1.33 (GPT5-mini). Argumentation quality is strengthened with up to +0.63 higher rebuttal quality. This indicates that performance improvements stem from task decomposition and structured intermediate reasoning, not merely stronger generative capacity. The benefit of REBUTTALAGENT is also larger for weaker base models, suggesting that explicit concern structuring, evidence construction, and response planning can partially compensate for limited base-model capability.

Our ablation studies reveal that Evidence Construction is the most critical intermediate artifact. Removing external evidence briefs leads to the largest and most consistent degradation across dimensions, with clear drops in Relevance and Communication Quality. Input Structuring and Checkers also contribute measurably, indicating that the gains of REBUTTALAGENT arise from the combination of complementary modules. Evidence-centered artifacts are the primary driver, while structuring and verification provide crucial guardrails.

Case studies demonstrate that REBUTTALAGENT's approach of producing an inspectable plan, separating interpretative defense from necessary intervention, effectively reduces hallucination and over-commitment. It outputs concrete deliverables and a scoped to-do list when new experiments or analyses are needed, supporting author decision-making by making the reasoning path and required work explicit before drafting.

Conclusion

We proposed REBUTTALAGENT, a multi-agent framework for transparent rebuttal assistance. Our system constructs structured, evidence-linked intermediate artifacts before drafting text, decomposing rebuttal writing into concern structuring, query-conditioned context building, on-demand external evidence synthesis, and response planning. This decomposition significantly improves traceability and cross-point coherence, while ensuring authors remain responsible for final decisions and wording. Through an author-centric benchmark and a rubric-based evaluation that measures relevance, global coherence, and argumentation quality beyond mere text fluency, our experimental results demonstrate that REBUTTALAGENT effectively improves the key requirements of reliable rebuttal assistance. This highlights the benefits of a transparent “verify-then-write” workflow, which reduces cognitive burden while empowering authors with ultimate control over the final wording.

BibTeX

@misc{ma2026paper2rebuttal,
    title={Paper2Rebuttal: A Multi-Agent Framework for Transparent Author Response Assistance}, 
    author={Qianli Ma and Chang Guo and Zhiheng Tian and Siyu Wang and Jipeng Xiao and Yuanhao Yue and Zhipeng Zhang},
    year={2026},
    eprint={2601.14171},
    archivePrefix={arXiv},
    primaryClass={cs.AI},
    url={https://arxiv.org/abs/2601.14171}, 
}