publications

2025

1. Logical Reasoning in Large Language Models: A Survey

   Hanmeng Liu, Zhizhang Fu, Mengru Ding, and 4 more authors

   2025

2024

1. 大模型逻辑推理研究综述(Survey on Logical Reasoning of Large Pre-trained Language Models)

   Liu Hanmeng and Zhang Yue

   In Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 2: Frontier Forum), Jul 2024

   Abs Bib PDF

   “理解自然语言的逻辑结构和关系是机器理解的核心任务,也是人工智能领域的关键研究议题。随着大数据和计算能力的提升,预训练语言模型在逻辑推理方面取得了显著进展,使得大规模模型的逻辑推理能力成为研究的新焦点。本综述旨在全面梳理大模型在逻辑推理领域的研究进展,探讨其对人工智能系统智能水平评估的重要性及其在推动人工智能发展中的作用。 本文首先界定了大模型逻辑推理能力的研究范畴,系统性地讨论了逻辑推理的类型和 特点,并回顾了相关理论的发展,为研究提供了清晰的框架。接着,从任务形式和数 据基准的角度,详细介绍了逻辑推理研究的基础工作,为理解大模型的性能提供了基 准。进一步,本文深入分析了大模型在逻辑推理能力上的现状,通过不同推理类型的 案例研究,展示了大模型的能力表现。同时,本文还探讨了提升大模型逻辑推理能力 的方法,包括预训练、指令微调、解码策略和神经符号混合方法,并对这些方法进行 了比较分析。最后,本文提出了对未来研究方向的展望,旨在激发更多的学术讨论和 探索,推动逻辑推理能力研究的进一步发展。”

   @inproceedings{hanmeng-yue-2024-da,
     title = {大模型逻辑推理研究综述(Survey on Logical Reasoning of Large Pre-trained Language Models)},
     author = {Hanmeng, Liu and Yue, Zhang},
     editor = {Zhao, Xin},
     booktitle = {Proceedings of the 23rd Chinese National Conference on Computational Linguistics (Volume 2: Frontier Forum)},
     month = jul,
     year = {2024},
     address = {Taiyuan, China},
     publisher = {Chinese Information Processing Society of China},
     url = {https://aclanthology.org/2024.ccl-2.3/},
     pages = {48--62},
     language = {zho},
   }

2. Break the Chain: Large Language Models Can be Shortcut Reasoners

   Mengru Ding, Hanmeng Liu, Zhizhang Fu, and 3 more authors

   Jul 2024

2023

1. GLUE-X: Evaluating Natural Language Understanding Models from an Out-of-Distribution Generalization Perspective

   Linyi Yang, Shuibai Zhang, Libo Qin, and 6 more authors

   In Findings of the Association for Computational Linguistics: ACL 2023, Jul 2023

   Abs DOI

   Pre-trained language models (PLMs) are known to improve the generalization performance of natural language understanding models by leveraging large amounts of data during the pre-training phase. However, the out-of-distribution (OOD) generalization problem remains a challenge in many NLP tasks, limiting the real-world deployment of these methods. This paper presents the first attempt at creating a unified benchmark named GLUE-X for evaluating OOD robustness in NLP models, highlighting the importance of OOD robustness and providing insights on how to measure the robustness of a model and how to improve it. The benchmark includes 13 publicly available datasets for OOD testing, and evaluations are conducted on 8 classic NLP tasks over 21 popularly used PLMs. Our findings confirm the need for improved OOD accuracy in NLP tasks, as significant performance degradation was observed in all settings compared to in-distribution (ID) accuracy.

2. Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4

   Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, and 3 more authors

   In , Jul 2023
3. GLoRE: Evaluating Logical Reasoning of Large Language Models

   Hanmeng Liu, Zhiyang Teng, Ruoxi Ning, and 4 more authors

   In , Oct 2023

4. 

   LogiCoT: Logical Chain-of-Thought Instruction Tuning

   Hanmeng Liu, Zhiyang Teng, Leyang Cui, and 3 more authors

   In Findings of the Association for Computational Linguistics: EMNLP 2023, Oct 2023

   Abs Bib HTML PDF Supp Website

   Generative Pre-trained Transformer 4 (GPT-4) demonstrates impressive chain-of-thought reasoning ability. Recent work on self-instruction tuning, such as Alpaca, has focused on enhancing the general proficiency of models. These instructions enable the model to achieve performance comparable to GPT-3.5 on general tasks like open-domain text generation and paraphrasing. However, they fall short of helping the model handle complex reasoning tasks. To bridge the gap, this paper presents LogiCoT, a new instruction-tuning dataset for Logical Chain-of-Thought reasoning with GPT-4. We elaborate on the process of harvesting instructions for prompting GPT-4 to generate chain-of-thought rationales. LogiCoT serves as an instruction set for teaching models of logical reasoning and elicits general reasoning skills.

   @inproceedings{liu2023logicot,
     title = {LogiCoT: Logical Chain-of-Thought Instruction Tuning},
     author = {Liu, Hanmeng and Teng, Zhiyang and Cui, Leyang and Zhang, Chaoli and Zhou, Qiji and Zhang, Yue},
     booktitle = {Findings of the Association for Computational Linguistics: EMNLP 2023},
     pages = {2908--2921},
     year = {2023},
   }

2021

1. Natural Language Inference in Context - Investigating Contextual Reasoning over Long Texts

   Hanmeng Liu, Leyang Cui, Jian Liu, and 1 more author

   Proceedings of the AAAI Conference on Artificial Intelligence, May 2021

   Abs Bib PDF

   Natural language inference (NLI) is a fundamental NLP task, investigating the entailment relationship between two texts. Popular NLI datasets present the task at sentence-level. While adequate for testing semantic representations, they fall short for testing contextual reasoning over long texts, which is a natural part of the human inference process. We introduce ConTRoL, a new dataset for ConTextual Reasoning over Long texts. Consisting of 8,325 expert-designed" context-hypothesis" pairs with gold labels, ConTRoL is a passage-level NLI dataset with a focus on complex contextual reasoning types such as logical reasoning. It is derived from competitive selection and recruitment test (verbal reasoning test) for police recruitment, with expert level quality. Compared with previous NLI benchmarks, the materials in ConTRoL are much more challenging, involving a range of reasoning types. Empirical results show that state-of-the-art language models perform by far worse than educated humans. Our dataset can also serve as a testing-set for downstream tasks like checking the factual correctness of summaries.

   @article{Liu_Cui_Liu_Zhang_2021,
     title = {Natural Language Inference in Context - Investigating Contextual Reasoning over Long Texts},
     volume = {35},
     number = {15},
     journal = {Proceedings of the AAAI Conference on Artificial Intelligence},
     author = {Liu, Hanmeng and Cui, Leyang and Liu, Jian and Zhang, Yue},
     year = {2021},
     month = may,
     pages = {13388-13396},
   }

2. Solving Aspect Category Sentiment Analysis as a Text Generation Task

   Jian Liu, Zhiyang Teng, Leyang Cui, and 2 more authors

   In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Nov 2021

   Bib

   @inproceedings{liu-etal-2021-solving,
     title = {Solving Aspect Category Sentiment Analysis as a Text Generation Task},
     author = {Liu, Jian and Teng, Zhiyang and Cui, Leyang and Liu, Hanmeng and Zhang, Yue},
     booktitle = {Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing},
     month = nov,
     year = {2021},
     publisher = {Association for Computational Linguistics (EMNLP)},
     pages = {4406--4416},
   }