journal article·2026-05-14Open access

When Agents Disagree: The Selection Bottleneck in Multi-Agent LLM Pipelines

Artem Maryanskyy, Dmitry Budnikov, Alibek T. Kaliyev

Applied Sciences· Volume 16· Issue 10

Abstract

Multi-agent LLM pipelines produce contradictory evidence on whether team diversity improves output quality: heterogeneous Mixture-of-Agents teams outperform single models, yet homogeneous Self-MoA teams consistently win under synthesis-based aggregation. We propose a resolution by identifying the selection bottleneck—a crossover threshold in aggregation quality that determines whether diversity helps or hurts. Under this model, we obtain a closed-form crossover threshold s∗ (Proposition 1) that separates the regimes where diversity helps and hurts. In a targeted experiment spanning 42 tasks across seven categories (N=210), a diverse team with judge-based selection achieves a win rate of 0.810 against a single-model baseline, while a homogeneous team scores 0.512—near chance (Glass’s Δ=2.07). Judge-based selection outperforms MoA-style synthesis by ΔWR=+0.631—the synthesis approach is preferred over the baseline in zero of 42 tasks by the judge panel. A decoupled evaluation with fully independent judges confirms all directional findings (Spearman ρ=0.90); win rates attenuate by 53–67% under independent evaluation relative to the primary estimates, consistent with partial measurement circularity in the judge-based cells. Exploratory evidence suggests that including a weaker model improves performance while reducing cost (p<10−4, not pre-registered). Our results suggest that selector quality may be a more impactful design lever than generator diversity in single-round generate-then-select pipelines, with a specific Opus-only singleton as the baseline.
Source: crossref · Quality: baseline

Research context

OpenAlex topics: Mobile Crowdsensing and Crowdsourcing · Ethics and Social Impacts of AI · Team Dynamics and Performance

SciVX discovery tags are terminology-based. OpenAlex topics are source-provided classifications.

Affiliations

Uber Technologies, Sunnyvale, CA 94085, USA

Laboratories of Electrical, Thermal Technologies and Energy Saving, Federal Scientific Agroengineering Center VIM, 109428 Moscow, Russia

Department of Computer Science, University of Texas at Austin, Austin, TX 78712, USA

References (25)

Show source-provided references
1. Groups of Diverse Problem Solvers Can Outperform Groups of High-Ability Problem Solvers10.1073/pnas.0403723101 ↗
2. Wang, J., Wang, J., Athiwaratkun, B., Zhang, C., and Zou, J. (2024). Mixture-of-Agents Yields State-of-the-Art on AlpacaEval 2.0, MT-Bench, and FLASK. arXiv.
3. Li, X., Zhang, L., Zhang, Z., Yang, Y., and Wang, Z. (2025). More Agents Is All You Need: Self-MoA Outperforms Mixed-MoA. arXiv.
4. Wu, Q., Bansal, G., Zhang, J., Wu, Y., Li, B., Zhu, E., Jiang, L., Zhang, X., Zhang, S., and Liu, J. (2023). AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. arXiv.
5. Li, G., Hammoud, H.A.A.K., Itani, H., Khizbullin, D., and Ghanem, B. (2023). CAMEL: Communicative Agents for “Mind” Exploration of Large Language Model Society. arXiv.
6. Hong, S., Zheng, X., Chen, J., Cheng, Y., Wang, J., Zhang, C., Wang, Z., Yau, S.K.S., Lin, Z., and Zhou, L. (2023). MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework. arXiv.
7. Du, Y., Li, S., Torralba, A., Tenenbaum, J.B., and Mordatch, I. (2023, January 23–29). Improving Factuality and Reasoning in Language Models through Multiagent Debate. Proceedings of the 40th International Conference on Machine Learning (ICML), Honolulu, HI, USA.
8. Smit, K., Keane, I., and Mao, W. (2023). When Models Think Alike: The Limits of Multi-Agent Debate. arXiv.
9. Choi, J., Lee, S., and Ok, J. (2025). Identity Bias in Large Language Model Debate. arXiv.
10. Jiang, D., Ren, X., and Lin, B.Y. (2023, January 9–14). LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusion. Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (ACL), Toronto, ON, Canada.10.18653/v1/2023.acl-long.792 ↗
11. Chen, J., Wang, X., Xu, R., Yuan, S., Chen, L., and Xiao, Y. (2025). LLMSelector: Selecting the Right LLM for Any Task. arXiv.
12. Stiennon, N., Ouyang, L., Wu, J., Ziegler, D.M., Lowe, R., Voss, C., Radford, A., Amodei, D., and Christiano, P. (2020, January 6–12). Learning to Summarize from Human Feedback. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), Virtual.
13. Snell, C., Lee, J., Xu, K., and Kumar, A. (2024). Scaling LLM Test-Time Compute Optimally Can Be More Effective than Scaling Model Parameters. arXiv.
14. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A., and Zhou, D. (2023, January 1–5). Self-Consistency Improves Chain of Thought Reasoning in Language Models. Proceedings of the International Conference on Learning Representations (ICLR), Kigali, Rwanda.
15. Tang, Z., Kharlapenko, D., Li, B., Li, F., Chen, H., and Cai, D. (2026). The Value of Diversity in Multi-Agent Systems. arXiv.
16. de Condorcet, M. (1785). Essai sur l’Application de l’Analyse à la Probabilité des Décisions Rendues à la Pluralité des Voix, Royale.
17. The Condorcet Jury Theorem, Free Speech, and Correlated Votes10.2307/2111584 ↗
18. Li, J., Zhang, Q., Yu, Y., Fu, Q., and Ye, D. (2024). More Agents Is All You Need. arXiv.
19. Zheng, L., Chiang, W.L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., and Xing, E.P. (2023, January 10–16). Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. Proceedings of the Advances in Neural Information Processing Systems (NeurIPS), New Orleans, LA, USA.10.52202/075280-2020 ↗
20. Stureborg, R., Alikaniotis, D., and Suhara, Y. (2024). Large Language Models are Inconsistent and Biased Evaluators. arXiv.
21. Panickssery, N., Bowman, S.R., and Feng, S. (2024). LLM Evaluators Recognize and Favor Their Own Generations. arXiv.
22. Verga, P., Hofstätter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., and Lewis, P. (2024). Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models. arXiv.
23. Chiang, W.L., Zheng, L., Sheng, Y., Angelopoulos, A.N., Li, T., Li, D., Zhang, H., Zhu, B., Jordan, M., and Gonzalez, J.E. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. arXiv.
24. Rank Analysis of Incomplete Block Designs: I. The Method of Paired Comparisons
25. Random Effects Structure for Confirmatory Hypothesis Testing: Keep It Maximal10.1016/j.jml.2012.11.001 ↗