Wargame SITREP 25-02 ~ LLMs and wargaming

Japanese wargamer and X user @ShunGumma tried an interesting experiment:

For those of you who have (or can get) access to IEEE articles I direct your attention to “Large Language Models in Wargaming: Methodology, Application, and Robustness” published in the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW).¹ Here is the abstract:

Traditional artificial intelligence (AI) has contributed strategic enhancements to wargaming but often encounters difficulties in dynamically complex environments and in adapting to unforeseen developments. In contrast, Large Language Models (LLMs) offer advanced natural language processing, analytical capabilities, and intuitive decision-making communication. LLMs excel in rapidly analyzing voluminous textual data, identifying patterns, and generating insights for strategic planning, thereby addressing the critical demand for anticipatory strategy and creative solution development in wargaming. Nonetheless, deploying LLMs in this context introduces potential robustness challenges, particularly their vulnerability to adversarial prompts. Our experimental investigations reveal LLMs’ susceptibility to misleading or hostile inputs, underscoring the imperative for implementing robustness measures to safeguard their operational integrity and reliability in strategic applications. Our pioneering research, through targeted experiments within a commercial wargaming, demonstrates the feasibility and potential of LLMs to significantly improve outcomes in representative scenarios. This work not only evidences the significant impact of LLMs on the decision-making landscape in wargaming but also establishes a foundation for future research and the practical implementation of LLMs in advanced decision support systems.

The authors of the IEEE paper developed a wargaming scenario built using Command Modern Operations (CMO) and then fed GPT-4 with descriptions of the game mechanisms, the scenario objective, the condition of the blue-side strike aircraft, and their ammunition status. The objective was for blue to strike a red target. The simulation started with a screenshot of the initial state after which GPT-4 was asked, “What is the next move?” The LLM also was told to provide a detailed explanation of its decision process at each step.

Interestingly, while the authors of the IEEE study claim their work, “constitutes a pioneering exploration of integrating LLMs into the realm of wargaming,” they go on to assert their paper “aims to showcase the broad potential of LLMs in various non-military domains, leveraging their exceptional analytical and explainability capacities” (Chan & Chu, 2901). In their final statement they try to stay away from military uses of LLMs:

We categorically state that the objective of integrating LLMs into wargaming environments is to highlight their versatility and impact in improving complex problem-solving and strategic planning in civilian contexts, thereby contributing positively to societal, economic, and technological advancements without supporting or enhancing military capabilities.

Chen & Chu, 2901

---

1. Y. Chen and S. Chu, “Large Language Models in Wargaming: Methodology, Application, and Robustness,” 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), Seattle, WA, USA, 2024, pp. 2894-2903, doi: 10.1109/CVPRW63382.2024.00295. keywords: {Navigation; Large language models; Decision making; Strategic planning; Solids; Robustness; Natural language processing}, ↩︎

Feature image courtesy cnblogs.com (screenshot NOT related to the IEEE experiment discussed)

The opinions and views expressed in this blog are those of the author alone and are presented in a personal capacity. They do not necessarily represent the views of U.S. Navy or any other U.S. government Department, Agency, Office, or employer.

RockyMountainNavy.com © 2007-2025 by Ian B is licensed under CC BY-SA 4.0