{"status":"ok","message-type":"work","message-version":"1.0.0","message":{"indexed":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T19:47:34Z","timestamp":1782848854810,"version":"3.54.5"},"reference-count":93,"publisher":"Association for Computing Machinery (ACM)","issue":"FSE","license":[{"start":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T00:00:00Z","timestamp":1782777600000},"content-version":"vor","delay-in-days":0,"URL":"https:\/\/creativecommons.org\/licenses\/by\/4.0\/legalcode"}],"content-domain":{"domain":[],"crossmark-restriction":false},"short-container-title":["Proc. ACM Softw. Eng."],"published-print":{"date-parts":[[2026,6,30]]},"abstract":"<jats:p>Infrastructure-as-Code (IaC) generation holds significant promise for automating the provisioning of cloud infrastructure. Recent advances in Large Language Models (LLMs) present a promising opportunity to democratize IaC development by generating deployable infrastructure templates from natural language descriptions. However, current evaluation focuses on syntactic correctness while ignoring deployability, the critical measure of the utility of IaC configuration files. Six state-of-the-art LLMs performed poorly on deployability, achieving only 20.8\u223c30.2% deployment success rate on the first attempt. In this paper, we construct DPIaC-Eval, the first deployability-centric IaC template benchmark consisting of 153 real-world scenarios across 58 unique services. Also, we propose an LLM-based deployability-centric framework, dubbed IaCGen, that uses an iterative feedback mechanism encompassing format verification, syntax checking, and live deployment stages, thereby closely mirroring the real DevOps workflows. Results show that IaCGen can make 54.6\u223c91.6% generated IaC templates from all evaluated models deployable in the first 10 iterations. Additionally, human-in-the-loop feedback that provide direct guidance for the deployability errors, can further boost the performance to over 90% passItr@25 on all evaluated LLMs. Furthermore, we explore the trustworthiness of the generated IaC templates on user intent alignment and security compliance. The poor performance (25.2% user requirement coverage and 8.4% security compliance rate) indicates a critical need for continued research in this domain.<\/jats:p>","DOI":"10.1145\/3797142","type":"journal-article","created":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T17:06:14Z","timestamp":1782839174000},"page":"321-343","source":"Crossref","is-referenced-by-count":0,"title":["Deployability-Centric Infrastructure-as-Code Generation: Fail, Learn, Refine, and Succeed through LLM-Empowered DevOps Simulation"],"prefix":"10.1145","volume":"3","author":[{"ORCID":"https:\/\/orcid.org\/0009-0008-9459-8504","authenticated-orcid":false,"given":"Tianyi","family":"Zhang","sequence":"first","affiliation":[{"name":"The Australian National University, Canberra, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0002-2162-0407","authenticated-orcid":false,"given":"Shidong","family":"Pan","sequence":"additional","affiliation":[{"name":"New York University, New York, USA"},{"name":"Columbia University, New York, USA"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0009-0006-6212-6313","authenticated-orcid":false,"given":"Zejun","family":"Zhang","sequence":"additional","affiliation":[{"name":"Nanyang Technological University, Singapore, Singapore"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7663-1421","authenticated-orcid":false,"given":"Zhenchang","family":"Xing","sequence":"additional","affiliation":[{"name":"CSIRO\u2019s Data61, Canberra, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]},{"ORCID":"https:\/\/orcid.org\/0000-0001-7434-0452","authenticated-orcid":false,"given":"Xiaoyu","family":"Sun","sequence":"additional","affiliation":[{"name":"The Australian National University, Canberra, Australia"}],"role":[{"vocabulary":"crossref","role":"author"}]}],"member":"320","published-online":{"date-parts":[[2026,6,30]]},"reference":[{"key":"e_1_2_1_1_1","doi-asserted-by":"publisher","unstructured":"Jacob Austin Augustus Odena Maxwell Nye Maarten Bosma Henryk Michalewski David Dohan Ellen Jiang Carrie Cai Michael Terry Quoc Le et al. 2021. Program synthesis with large language models. arXiv preprint arXiv:2108.07732 (2021). doi:10.48550\/arXiv.2108.07732 10.48550\/arXiv.2108.07732","DOI":"10.48550\/arXiv.2108.07732"},{"key":"e_1_2_1_2_1","unstructured":"AWS. 2025. AWS CloudFormation User Guide. https:\/\/docs.aws.amazon.com\/AWSCloudFormation\/latest\/UserGuide\/ Welcome.html"},{"key":"e_1_2_1_3_1","unstructured":"AWS. 2025. AWS Prescriptive Guidance. https:\/\/docs.aws.amazon.com\/prescriptive-guidance\/latest\/iac-edp-comboapproach\/best-practices.html"},{"key":"e_1_2_1_4_1","unstructured":"AWS. 2025. Managing AWS resources as a single unit with AWS CloudFormation stacks. https:\/\/docs.aws.amazon. com\/AWSCloudFormation\/latest\/UserGuide\/stacks.html"},{"key":"e_1_2_1_5_1","unstructured":"AWS. n.d.. CreateDBInstanceReadReplica. https:\/\/docs.aws.amazon.com\/AmazonRDS\/latest\/APIReference\/API_ CreateDBInstanceReadReplica.html"},{"key":"e_1_2_1_6_1","unstructured":"AWS. n.d.. General purpose bucket naming rules. https:\/\/docs.aws.amazon.com\/AmazonS3\/latest\/userguide\/ bucketnamingrules.html"},{"key":"e_1_2_1_7_1","unstructured":"Microsoft Azure. 2023. Architecture strategies for using infrastructure as code. https:\/\/learn.microsoft.com\/enus\/azure\/well-architected\/operational-excellence\/infrastructure-as-code-design"},{"key":"e_1_2_1_8_1","unstructured":"Mohamed Basher. 2019. DevOps: An explorative case study on the challenges and opportunities in implementing Infrastructure as code. https:\/\/api.semanticscholar.org\/CorpusID:281306300"},{"key":"e_1_2_1_9_1","unstructured":"Eric Z. Beard. 2024. AWS CloudFormation Sample Templates. https:\/\/github.com\/aws-cloudformation\/awscloudformation-templates\/blob\/main\/SNS\/SNSTopic.yaml"},{"key":"e_1_2_1_10_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.findings-acl.138"},{"key":"e_1_2_1_11_1","doi-asserted-by":"publisher","DOI":"10.1145\/3728922"},{"key":"e_1_2_1_12_1","unstructured":"Jonathan Carpenter. 2024. Why Ignoring Low-Severity Vulnerabilities Can Cost You: A Risk-Based Patch Management Guide: Part Four. https:\/\/anchorcybersecurity.com\/blog\/2024-10-25-Cost-of-Ignoring"},{"key":"e_1_2_1_13_1","unstructured":"Luis Casas. 2025. aws projects. https:\/\/github.com\/LSCasas\/aws_projects\/blob\/ a19f570a682028186078ff7396e58a2e9f6c612e\/ecs-fargate\/CloudFormation_Template\/.yaml#L113"},{"key":"e_1_2_1_14_1","unstructured":"Checkov. n.d.. Custom Policies Overview. https:\/\/www.checkov.io\/3.Custom%20Policies\/Custom%20Policies% 20Overview.html"},{"key":"e_1_2_1_15_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2107.03374"},{"key":"e_1_2_1_16_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2304.05128"},{"key":"e_1_2_1_17_1","doi-asserted-by":"publisher","DOI":"10.21275\/sr24829170834"},{"key":"e_1_2_1_18_1","volume-title":"Mining for Cost Awareness in Cloud Computing: A Study of AWS CloudFormation and Developer Practices. Ph. D. Dissertation","author":"Cirlan Alexandru-Nicolae","unstructured":"Alexandru-Nicolae Cirlan. 2024. Mining for Cost Awareness in Cloud Computing: A Study of AWS CloudFormation and Developer Practices. Ph. D. Dissertation. University of Groningen. https:\/\/fse.studenttheses.ub.rug.nl\/id\/eprint\/33994"},{"key":"e_1_2_1_19_1","unstructured":"AWS cloud. 2025. aws-cloudformation-templates. https:\/\/github.com\/aws-cloudformation\/aws-cloudformationtemplates."},{"key":"e_1_2_1_20_1","unstructured":"Prisma Cloud. 2025. AWS security groups allow ingress from 0.0.0.0\/0 to port 80. https:\/\/docs.prismacloud.io\/en\/ enterprise-edition\/policy-reference\/aws-policies\/aws-networking-policies\/ensure-aws-security-groups-do-notallow-ingress-from-00000-to-port-80"},{"key":"e_1_2_1_21_1","unstructured":"Prisma Cloud. 2025. AWS SNS topic has SSE disabled. https:\/\/docs.prismacloud.io\/en\/enterprise-edition\/policyreference\/aws-policies\/aws-general-policies\/general-15"},{"key":"e_1_2_1_22_1","unstructured":"Prisma Cloud. n.d.. Prisma Cloud Website. https:\/\/www.paloaltonetworks.com\/prisma\/cloud"},{"key":"e_1_2_1_23_1","doi-asserted-by":"publisher","DOI":"10.1145\/3661167.3661221"},{"key":"e_1_2_1_24_1","doi-asserted-by":"publisher","DOI":"10.1145\/1721654.1721667"},{"key":"e_1_2_1_25_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.entcom.2020.100398"},{"key":"e_1_2_1_26_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2024.emnlp-main.64"},{"key":"e_1_2_1_27_1","doi-asserted-by":"publisher","DOI":"10.1145\/3689799"},{"key":"e_1_2_1_28_1","unstructured":"Adrien Verg\u00e9 et al. 2023. yamllint. https:\/\/github.com\/adrienverge\/yamllint"},{"key":"e_1_2_1_29_1","unstructured":"Center for Internet Security. 2025. CIS AWS Foundations Benchmark v5.0.0. https:\/\/www.cisecurity.org\/benchmark\/ amazon_web_services"},{"key":"e_1_2_1_30_1","unstructured":"Center for Internet Security. n.d.. CIS Benchmarks for Cloud Providers. https:\/\/www.cisecurity.org\/cis-benchmarks"},{"key":"e_1_2_1_31_1","unstructured":"GeeksForGeeks. 2024. Difference Between CloudFormation VS Terraform. https:\/\/www.geeksforgeeks.org\/differencebetween-cloudformation-vs-terraform\/"},{"key":"e_1_2_1_32_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2509.22900"},{"key":"e_1_2_1_33_1","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3617850"},{"key":"e_1_2_1_34_1","doi-asserted-by":"publisher","DOI":"10.1109\/icsme.2019.00092"},{"key":"e_1_2_1_35_1","doi-asserted-by":"publisher","unstructured":"Daya Guo Qihao Zhu Dejian Yang Zhenda Xie Kai Dong Wentao Zhang Guanting Chen Xiao Bi Yu Wu YK Li et al. 2024. DeepSeek-Coder: When the Large Language Model Meets Programming-The Rise of Code Intelligence. arXiv preprint arXiv:2401.14196 (2024). doi:10.48550\/arXiv.2401.14196 10.48550\/arXiv.2401.14196","DOI":"10.48550\/arXiv.2401.14196"},{"key":"e_1_2_1_36_1","doi-asserted-by":"publisher","DOI":"10.1109\/tse.2024.3523284"},{"key":"e_1_2_1_37_1","doi-asserted-by":"publisher","DOI":"10.1145\/3416504.3424334"},{"key":"e_1_2_1_38_1","unstructured":"HashiCorp. 2023. Apply previous state file or undo plan. https:\/\/discuss.hashicorp.com\/t\/apply-previous-state-file-orundo-plan\/57833"},{"key":"e_1_2_1_39_1","unstructured":"Hashicorp. 2023. Hashicorp 2023 state of cloud strategy survey. https:\/\/www.hashicorp.com\/en\/state-of-the-cloud"},{"key":"e_1_2_1_40_1","unstructured":"Hashicorp. 2024. Terraform versus CloudFormation Heat and other infrastructure as code tools. https:\/\/developer. hashicorp.com\/terraform\/intro\/vs\/cloudformation"},{"key":"e_1_2_1_41_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.67"},{"key":"e_1_2_1_42_1","unstructured":"Vahid Iranpour. 2024. Mastering Infrastructure as Code (IaC): Best Practices and Real-World Examples. https:\/\/medium.com\/@community.vahid\/mastering-infrastructure-as-code-iac-best-practices-and-real-worldexamples-df0f3c90c560"},{"key":"e_1_2_1_43_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2025.findings-naacl.285"},{"key":"e_1_2_1_44_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2210.07259"},{"key":"e_1_2_1_45_1","unstructured":"DeJong Kevin Myron Pat van Boven Frank et al. 2025. AWS CloudFormation Linter. https:\/\/github.com\/awscloudformation\/cfn-lint"},{"key":"e_1_2_1_46_1","doi-asserted-by":"publisher","DOI":"10.52202\/079017-4273"},{"key":"e_1_2_1_47_1","doi-asserted-by":"publisher","DOI":"10.1145\/3643795.3648384"},{"key":"e_1_2_1_48_1","unstructured":"Vijay Kuma. 2024. S3 Bucket Naming Conventions: Best Practices to Avoid Unexpected Costs. https:\/\/tech.vians.org\/ posts\/s3-bucket-naming-conventions\/"},{"key":"e_1_2_1_49_1","unstructured":"Naveen Kumar. 2025. DeepSeek AI Statistics of 2025 (Users and Revenue). https:\/\/www.demandsage.com\/deepseekstatistics\/"},{"key":"e_1_2_1_50_1","doi-asserted-by":"publisher","DOI":"10.1016\/j.infsof.2021.106593"},{"key":"e_1_2_1_51_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2006.03511"},{"key":"e_1_2_1_52_1","doi-asserted-by":"publisher","DOI":"10.1145\/3691620.3695494"},{"key":"e_1_2_1_53_1","doi-asserted-by":"publisher","DOI":"10.1109\/tse.2024.3486195"},{"key":"e_1_2_1_54_1","doi-asserted-by":"publisher","DOI":"10.1109\/ase63991.2025.00063"},{"key":"e_1_2_1_55_1","doi-asserted-by":"publisher","DOI":"10.52202\/075280-0943"},{"key":"e_1_2_1_56_1","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3561341"},{"key":"e_1_2_1_57_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv"},{"key":"e_1_2_1_58_1","doi-asserted-by":"publisher","DOI":"10.52202\/075280-2019"},{"key":"e_1_2_1_59_1","doi-asserted-by":"publisher","DOI":"10.1109\/iccst.2014.7045193"},{"key":"e_1_2_1_60_1","doi-asserted-by":"publisher","DOI":"10.1007\/s10664-025-10688-0"},{"key":"e_1_2_1_61_1","doi-asserted-by":"publisher","DOI":"10.54660\/.jfmr.2022.3.1.656-669"},{"key":"e_1_2_1_62_1","doi-asserted-by":"publisher","DOI":"10.1007\/978-3-031-72578-4_5"},{"key":"e_1_2_1_63_1","unstructured":"Kor Nimrod Gr\u00fcbel Anton Woolfenden James et al. 2025. Checkov. https:\/\/github.com\/bridgecrewio\/checkov"},{"key":"e_1_2_1_64_1","unstructured":"Stack Overflow. n.d.. Stack Overflow website. https:\/\/stackoverflow.com\/questions"},{"key":"e_1_2_1_65_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2411.19043"},{"key":"e_1_2_1_66_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2406.17431"},{"key":"e_1_2_1_67_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2409.13343"},{"key":"e_1_2_1_68_1","doi-asserted-by":"publisher","DOI":"10.1109\/DAC56929"},{"key":"e_1_2_1_69_1","unstructured":"Mengkorn Pum and Amelia George. 2024. CloudFormation vs. Terraform: A Comparative Study. (2024). https: \/\/www.researchgate.net\/publication\/391016595"},{"key":"e_1_2_1_70_1","doi-asserted-by":"publisher","DOI":"10.1145\/3626111.3628206"},{"key":"e_1_2_1_71_1","unstructured":"Rackspace. 2025. The 2025 State of Cloud Report. https:\/\/www.rackspace.com\/blog\/2025-state-cloud-report"},{"key":"e_1_2_1_72_1","doi-asserted-by":"publisher","DOI":"10.1109\/iceecit63698.2024.10859822"},{"key":"e_1_2_1_73_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2110.06773"},{"key":"e_1_2_1_74_1","unstructured":"James saloman. 2023. Debugging 101: How to Find and Fix Programming Errors. https:\/\/dev.to\/saloman_james\/ debugging-101-how-to-find-and-fix-programming-errors-1d14"},{"key":"e_1_2_1_75_1","unstructured":"AWS Samples. 2025. AWS Samples GitHub Repository. https:\/\/github.com\/aws-samples."},{"key":"e_1_2_1_76_1","unstructured":"Sean. 2022. How Terraform manage apply failure. https:\/\/stackoverflow.com\/questions\/72070394\/how-terraformmanage-apply-failure"},{"key":"e_1_2_1_77_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2407.15407"},{"key":"e_1_2_1_78_1","doi-asserted-by":"publisher","DOI":"10.1080\/08839514.2024.2439610"},{"key":"e_1_2_1_79_1","doi-asserted-by":"publisher","DOI":"10.55640\/ijns-05-01-10"},{"key":"e_1_2_1_80_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2404.00227"},{"key":"e_1_2_1_81_1","doi-asserted-by":"publisher","DOI":"10.1145\/3458336.3465301"},{"key":"e_1_2_1_82_1","unstructured":"Styra. 2023. AI-Generated Infrastructure-as-Code: The Good the Bad and the Ugly. https:\/\/direct-blog-url-here.com."},{"key":"e_1_2_1_83_1","doi-asserted-by":"publisher","DOI":"10.1145\/3611643.3613098"},{"key":"e_1_2_1_84_1","doi-asserted-by":"publisher","DOI":"10.1109\/tse.2023.3266324"},{"key":"e_1_2_1_85_1","doi-asserted-by":"publisher","DOI":"10.1145\/3551349.3561151"},{"key":"e_1_2_1_86_1","doi-asserted-by":"publisher","DOI":"10.1145\/3440033"},{"key":"e_1_2_1_87_1","doi-asserted-by":"publisher","DOI":"10.56553\/popets-2025-0136"},{"key":"e_1_2_1_88_1","doi-asserted-by":"publisher","DOI":"10.48550\/arXiv.2312.01797"},{"key":"e_1_2_1_89_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2022"},{"key":"e_1_2_1_90_1","unstructured":"Martin Yip. 2018. Query for the latest Amazon Linux AMI IDs using AWS Systems Manager Parameter Store. https:\/\/aws.amazon.com\/blogs\/compute\/query-for-the-latest-amazon-linux-ami-ids-using-aws-systemsmanager-parameter-store\/"},{"key":"e_1_2_1_91_1","doi-asserted-by":"publisher","DOI":"10.1613\/jair.1.11345"},{"key":"e_1_2_1_92_1","doi-asserted-by":"publisher","DOI":"10.18653\/v1\/2023.emnlp-main.151"},{"key":"e_1_2_1_93_1","doi-asserted-by":"publisher","DOI":"10.1145\/3715726"}],"container-title":["Proceedings of the ACM on Software Engineering"],"original-title":[],"language":"en","link":[{"URL":"https:\/\/dl.acm.org\/doi\/pdf\/10.1145\/3797142","content-type":"unspecified","content-version":"vor","intended-application":"similarity-checking"}],"deposited":{"date-parts":[[2026,6,30]],"date-time":"2026-06-30T18:56:34Z","timestamp":1782845794000},"score":1,"resource":{"primary":{"URL":"https:\/\/dl.acm.org\/doi\/10.1145\/3797142"}},"subtitle":[],"short-title":[],"issued":{"date-parts":[[2026,6,30]]},"references-count":93,"journal-issue":{"issue":"FSE","published-print":{"date-parts":[[2026,6,30]]}},"alternative-id":["10.1145\/3797142"],"URL":"https:\/\/doi.org\/10.1145\/3797142","relation":{},"ISSN":["2994-970X"],"issn-type":[{"value":"2994-970X","type":"electronic"}],"subject":[],"published":{"date-parts":[[2026,6,30]]}}}