Chandra Shekhar Dhungana
School of IT and Engineering, Melbourne Institute of Technology
Supervisor: Dr Farhana Afroz · Cluster lead: Dr Mahdi Saki
Cluster: AI-Driven Secure and Intelligent Networked Systems
Network configuration is a fundamental part of network management, but translating operational requirements into correct device-level configurations remains difficult because configurations are precise, vendor-specific and sensitive to small errors. Large Language Models (LLMs) provide a promising way to translate natural-language intents into configurations, yet existing studies use different models, datasets, vendors and evaluation methods, making it difficult to determine which techniques actually improve accuracy and whether their benefits justify the computational cost.
This research investigates how an open-weight LLM-based pipeline can generate syntactically and logically accurate Cisco IOS and Juniper Junos configurations while remaining computationally efficient. The study uses a controlled experimental design to evaluate fine-tuning and Retrieval-Augmented Generation (RAG), individually and in combination, across selected Qwen and Llama models. Selected models will then be evaluated within an agentic workflow that uses external validation and bounded iterative correction. Evaluation will combine reference-based measures with syntactic and logical accuracy, latency and computational-cost measurements, using a separate multi-vendor benchmark for consistent comparison.
The initial findings show that the required dataset, fine-tuning process and evaluation environment are available to carry out the proposed research. They also show that broader training can affect Qwen and Llama differently, supporting the need for controlled comparison. These findings demonstrate the feasibility of the research rather than provide final answers to the research questions. The remaining experiments will identify which techniques improve accuracy and efficiency and will use this evidence to design the final network-configuration pipeline.
Proposed pipeline · RAG, agentic workflows and Batfish validation are next-stage work.
A natural-language requirement for Cisco IOS or Juniper Junos.
An open-weight model, compared with fine-tuning and retrieval-augmented generation.
Planned syntax and logical checks, including Batfish validation.
Planned iterative correction using validation feedback, with computational cost measured.
How can an open-weight LLM-based pipeline be designed to translate natural-language network intents into syntactically and logically accurate Cisco IOS and Juniper Junos configurations while remaining computationally efficient?
How do domain-specific fine-tuning and retrieval-augmented generation, individually and in combination, affect syntactic accuracy, logical accuracy and computational cost?
To what extent does tool-assisted validation and iterative correction improve accuracy, and what additional computational cost does it introduce?
Exact Match (%) · Three-shot prompting · 140 NetConfRAG Cisco cases
| Model | Training scope | Exact Match (%) | Line F1 | BLEU |
|---|---|---|---|---|
| Qwen | Cisco-only | 14.29 | 0.6939 | 49.41 |
| Qwen | Multi-task | 45.71 | 0.7814 | 67.66 |
| Llama | Cisco-only | 2.86 | 0.6934 | 42.93 |
| Llama | Multi-task | 3.57 | 0.6680 | 43.27 |
Source: Thesis 1, Table 7.3, page 69. Similarity to a reference is not proof of operational correctness.
Qwen gained 44 new exact matches with no regressions. Llama gained three and lost two; its Line F1 declined. Zero-shot Exact Match remained 0% for both models.
Dataset composition, formatting and sequence length changed between runs. These preliminary results cannot isolate a single cause. The next stage uses controlled comparisons and operational validation.
The original 103-page Thesis 1 report, including literature review, methodology, initial findings, references and appendices.
If your browser cannot display the preview, use the full-screen or download option above.