<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Mehrdad Mokhtari on Medium]]></title>
        <description><![CDATA[Stories by Mehrdad Mokhtari on Medium]]></description>
        <link>https://medium.com/@mehrdad.mokhtari?source=rss-3099b15986b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*fJI0oJh7APlWNJQzBqMtgw.jpeg</url>
            <title>Stories by Mehrdad Mokhtari on Medium</title>
            <link>https://medium.com/@mehrdad.mokhtari?source=rss-3099b15986b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 10:15:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@mehrdad.mokhtari/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Netmiko vs Paramiko vs NAPALM vs Ansible: Which Tool Does Your Network Need?]]></title>
            <link>https://systemweakness.com/netmiko-vs-paramiko-vs-napalm-vs-ansible-which-tool-does-your-network-need-270d9e948ddc?source=rss-3099b15986b------2</link>
            <guid isPermaLink="false">https://medium.com/p/270d9e948ddc</guid>
            <category><![CDATA[ansible]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[python-programming]]></category>
            <dc:creator><![CDATA[Mehrdad Mokhtari]]></dc:creator>
            <pubDate>Tue, 06 Oct 2026 09:18:31 GMT</pubDate>
            <atom:updated>2026-10-07T10:03:17.982Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*x7VEid9upGR9ytSJfuFt2w.png" /></figure><p>Creating a VLAN on one switch takes a few commands. Creating it on 500 switches means repeating the same commands, checking each result and keeping track of failures. Automation can reduce that work, but it can also spread a mistake across the network much faster.</p><p>Engineers often compare Paramiko, Netmiko, NAPALM and Ansible. Each handles a different part of network automation: SSH connectivity, device CLI interaction, a common interface across vendors, and repeatable workflows.</p><h3>What is network automation?</h3><p>Network automation uses scripts and software to configure devices, collect operational information, back up configurations and check compliance. It makes repeated changes easier to check and review.</p><p>For a VLAN rollout, the workflow checks the current state, applies changes only where needed, verifies the result and records failures. Saving the configuration, where required, is a separate decision. Start with a small test group before expanding the rollout.</p><p>The examples below show the basic syntax and workflow. Install the libraries or Ansible collections used below, use reachable lab addresses, and check your platform’s requirements. These examples have not been tested against your devices.</p><h3>Paramiko: SSH functionality in Python</h3><p>Paramiko is a Python SSH library. It provides authentication, channels, command execution, interactive shells and SFTP. Use it when a custom SSH application or script needs direct control over a session.</p><p>Paramiko does not supply Cisco IOS or Junos prompt handling, configuration-mode transitions or pagination management. Your application must implement those behaviours and decide how to handle authentication failures, timeouts and other exceptions.</p><h3>A basic Paramiko example</h3><p>This example runs a command on a Linux SSH server. Add the server’s independently verified host key to your known_hosts file first. Paramiko rejects unknown host keys by default; the password is requested at runtime.</p><pre>from getpass import getpass<br>import paramiko<br>with paramiko.SSHClient() as client:<br>    client.load_system_host_keys()<br>    client.connect(<br>        hostname=&quot;10.12.2.10&quot;,<br>        username=&quot;admin&quot;,<br>        password=getpass(&quot;SSH password: &quot;),<br>        timeout=10,<br>    )<br>    stdin, stdout, stderr = client.exec_command(&quot;uname -a&quot;, timeout=10)<br>    print(stdout.read().decode())<br>    print(stderr.read().decode())</pre><p>exec_command() requests SSH command execution, which the target must support. Some network devices require invoke_shell() and an interactive CLI instead. A library that handles the device’s CLI can save you from writing prompt and pagination handling yourself. Avoid automatically accepting unknown host keys in production.</p><h3>Netmiko: network CLI interaction</h3><p>Netmiko simplifies CLI interaction with network devices. Its SSH connections use Paramiko, while its device drivers handle tasks such as recognising prompts and preparing terminal sessions. Support and behaviour depend on the selected driver.</p><p>Netmiko can collect show-command output, back up configurations and change interface descriptions. It also works well in custom troubleshooting scripts. It returns command output as text by default; integrations with parsers can return dictionaries or lists when a parsing template matches the command output.</p><h3>Reading interface information</h3><p>Prepopulate your known_hosts file with a verified device key. This example enables strict host-key checking and uses a context manager to close the session.</p><pre>from getpass import getpass<br>from netmiko import ConnectHandler<br>device = {<br>    &quot;device_type&quot;: &quot;cisco_ios&quot;,<br>    &quot;host&quot;: &quot;10.12.2.10&quot;,<br>    &quot;username&quot;: &quot;admin&quot;,<br>    &quot;password&quot;: getpass(&quot;SSH password: &quot;),<br>    &quot;ssh_strict&quot;: True,<br>    &quot;system_host_keys&quot;: True,<br>}<br>with ConnectHandler(**device) as connection:<br>    print(connection.send_command(&quot;show ip interface brief&quot;))</pre><h3>Making a configuration change</h3><p>The next example reuses the device dictionary and opens a new session. It assumes the account has configuration privileges. If enable authentication is required, supply the secret securely and call connection.enable(). Confirm the interface exists before running the commands.</p><pre>commands = [<br>    &quot;interface GigabitEthernet0/1&quot;,<br>    &quot;description Connect_to_Firewall&quot;,<br>    &quot;no shutdown&quot;,<br>]<br>with ConnectHandler(**device) as connection:<br>    print(connection.send_config_set(<br>        commands, error_pattern=r&quot;% (?:Invalid input|Incomplete command|Ambiguous command)&quot;<br>    ))<br>    print(connection.send_command(&quot;show running-config interface GigabitEthernet0/1&quot;))</pre><p>Netmiko handles configuration-mode transitions, but sending commands does not guarantee success, idempotency or persistence after a reboot. The error pattern catches several common IOS errors, not every failure. Check whether the commands took effect. Save the configuration if the changes need to survive a reboot.</p><h3>Paramiko or Netmiko?</h3><p>Paramiko gives you control over SSH sessions. For a Python script that runs commands on a supported network device, Netmiko handles more of the CLI interaction for you. You can build network automation directly with Paramiko, but you will have more device interaction logic to maintain.</p><p>A common CLI automation problem is pagination: the SSH connection succeeds, but the device pauses before returning all the output. With Paramiko, your script must handle the pagination prompt or disable pagination in the session. Netmiko’s device-specific session preparation usually disables pagination for supported platforms, so the script can read the command output without responding to each page.</p><h3>NAPALM: a common interface across vendors</h3><p>NAPALM stands for Network Automation and Programmability Abstraction Layer with Multi-Vendor support. It provides the same method calls across supported network operating systems, so you write less separate code for each vendor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*hVcn79cE2hdpICJFdHt7Qg.png" /></figure><p>For example, a dashboard can request device facts from Cisco, Arista and Juniper devices through the same method. The drivers may use different transports and backend libraries. NAPALM’s IOS and NX-OS SSH drivers use Netmiko; other drivers use libraries for APIs or NETCONF.</p><h3>Retrieving device facts</h3><p>This Cisco IOS example uses strict host-key checking through the driver’s optional arguments. As with the Netmiko example, verify and store the device’s host key before connecting.</p><pre>from getpass import getpass<br>from napalm import get_network_driver<br>driver = get_network_driver(&quot;ios&quot;)<br>with driver(<br>    hostname=&quot;10.12.2.10&quot;,<br>    username=&quot;admin&quot;,<br>    password=getpass(&quot;SSH password: &quot;),<br>    optional_args={&quot;ssh_strict&quot;: True, &quot;system_host_keys&quot;: True},<br>) as device:<br>    facts = device.get_facts()<br>    print(facts[&quot;hostname&quot;])<br>    print(facts[&quot;os_version&quot;])<br>    print(facts[&quot;interface_list&quot;])</pre><p>get_facts() returns a dictionary containing hostname, vendor, model, operating system version, serial number, uptime, fully qualified domain name and interface_list. The interface list contains names; get_interfaces() provides details such as status, description and speed.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*8OOrfUaQaWWxeXyAfg-aew.png" /></figure><p>NAPALM can also load, compare and commit candidate configurations. Getter support, configuration replacement, merge, rollback and atomicity vary by driver and platform. Check the support matrix and platform caveats before designing a deployment workflow. A common API does not make configuration syntax identical across vendors.</p><h3>NAPALM or Netmiko?</h3><p>NAPALM lets your application use the same method calls across supported platforms. Netmiko gives you direct CLI access when you need a command that a NAPALM getter does not cover. An application can use both.</p><h3>Ansible: repeatable automation workflows</h3><p>Ansible is an automation engine for network devices, servers and other systems. You describe tasks in YAML playbooks and execute them against an inventory. Collections supply platform modules, and connection plugins provide access through interfaces such as SSH CLI, HTTP APIs and NETCONF.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*c_HKHGfMADvNpl6xRuuiAg.png" /></figure><p>Inventory and variables separate device settings from tasks. Roles let you reuse groups of tasks, and Jinja2 templates generate configuration from your data.</p><p>Ansible Vault encrypts stored secrets. Playbooks can be version-controlled in Git and run through CI/CD pipelines; your team still needs to set up version control and pipeline execution.</p><h3>Creating a VLAN with a resource module</h3><p>This playbook uses cisco.ios.ios_vlans to ensure VLAN 50 is named VOICE. The merged state applies the supplied VLAN configuration without replacing the entire VLAN configuration.</p><pre>---<br>- name: Configure voice VLAN<br>  hosts: switches<br>  gather_facts: false<br>  connection: ansible.netcommon.network_cli<br>  vars:<br>    ansible_network_os: cisco.ios.ios<br>  tasks:<br>    - name: Ensure VLAN 50 is named VOICE<br>      cisco.ios.ios_vlans:<br>        config:<br>          - vlan_id: 50<br>            name: VOICE</pre><p>The playbook changes the running configuration. Verify the VLAN and decide when to save. For CLI configuration that a resource module does not cover, use ios_config. Specify parents for the configuration hierarchy and use full commands when matching against the running configuration. Idempotency depends on the module, its inputs and device behaviour; YAML alone does not guarantee it.</p><h3>How the tools fit together</h3><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*PyyWZhpEnF3A4BLUQW2M-Q.png" /></figure><p>The architecture illustration is conceptual, rather than a fixed dependency chain. Ansible does not always use Netmiko or NAPALM. Its connection plugins and modules determine how it accesses devices. NAPALM also uses different backends for different drivers.</p><p>For a mixed-vendor inventory dashboard, NAPALM may provide the common facts you need, with Netmiko handling unsupported CLI queries. For a standard configuration rollout maintained by several engineers, Ansible provides an inventory and reusable playbooks. For a custom SSH application, Paramiko offers more direct session control.</p><h3>Choosing a starting point</h3><ul><li>Paramiko: custom SSH clients, channels and session behaviour.</li><li>Netmiko: Python scripts that collect output or configure supported network CLIs.</li><li>NAPALM: applications that need common getters and supported configuration operations across vendors.</li><li>Ansible: repeatable workflows using inventories, roles and platform modules.</li></ul><p>Don’t choose a tool just because you have many devices. A small team may find Ansible playbooks easier to maintain together; a large network may need a custom Python application. Check that the tool supports your devices and returns the data you need. Think about how failures will be handled and who will maintain the code.</p><h3>Before expanding beyond the lab</h3><p>Before a rollout, record the current state and test the change on devices that match your network. Decide what should stop the rollout and how you will recover from a failure. Set timeouts and record errors so you can identify devices that need attention. Avoid opening more sessions than your devices and management systems can handle. Store credentials outside the source code and verify host keys before connecting.</p><p>An SSH connection can succeed even when the device rejects a configuration command. Check what changed before moving to the next device, including whether the network still behaves as expected. If the change needs to survive a reboot, check that it has been saved.</p><h3>My recommendation</h3><p>Pick one task you already understand, such as collecting interface information or checking an NTP setting. Use the comparisons above to choose a tool, then try it on a few lab devices. Check the output and see what happens when a device cannot be reached.</p><p>You can write your first Ansible playbook before mastering Python. Learn Python as you need custom data processing or integrations. Expand your automation once you understand both its results and its failures.</p><h3>Official documentation</h3><ul><li><a href="https://docs.paramiko.org/en/stable/api/client.html">Paramiko SSH client and host-key policies</a></li><li><a href="https://github.com/ktbyers/netmiko">Netmiko examples and supported platforms</a></li><li><a href="https://napalm.readthedocs.io/en/latest/base.html">NAPALM getters and configuration methods</a></li><li><a href="https://napalm.readthedocs.io/en/latest/support/index.html">NAPALM driver support and caveats</a></li><li><a href="https://docs.ansible.com/projects/ansible/latest/collections/cisco/ios/ios_vlans_module.html">Ansible Cisco IOS VLAN resource module</a></li><li><a href="https://docs.ansible.com/projects/ansible/latest/collections/cisco/ios/ios_config_module.html">Ansible Cisco IOS configuration module</a></li></ul><blockquote>If you found it useful, don’t forget to <strong>👏 clap, 🔄 share, and 💬 follow</strong> for more Network Security, Linux &amp; Windows, and DevOps &amp; Cloud insights.</blockquote><p>📲 Follow me on social media:<br>🔗 LinkedIn: <a href="https://www.linkedin.com/in/bornaly/">https://www.linkedin.com/</a><a href="https://www.linkedin.com/in/mmokhta">in/mmokhta</a><br>✍️ Medium: <a href="https://medium.com/@mehrdad.mokhtari1370">https://medium.com/@mehrdad.mokhtari1370</a><br>♛ GitHub: <a href="https://github.com/MEMOKHTARI">https://github.com/MEMOKHTARI</a><br>⛪ GitLab: <a href="https://gitlab.com/Mehrdad261">[github.com/MEMOKHTARI</a>]</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=270d9e948ddc" width="1" height="1" alt=""><hr><p><a href="https://systemweakness.com/netmiko-vs-paramiko-vs-napalm-vs-ansible-which-tool-does-your-network-need-270d9e948ddc">Netmiko vs Paramiko vs NAPALM vs Ansible: Which Tool Does Your Network Need?</a> was originally published in <a href="https://systemweakness.com/">System Weakness</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Cisco SD-WAN Runs in Production: Cloud OnRamp, Application Experience and Operating at Scale…]]></title>
            <link>https://systemweakness.com/how-cisco-sd-wan-runs-in-production-cloud-onramp-application-experience-and-operating-at-scale-0e6f50cd02c1?source=rss-3099b15986b------2</link>
            <guid isPermaLink="false">https://medium.com/p/0e6f50cd02c1</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Mehrdad Mokhtari]]></dc:creator>
            <pubDate>Fri, 02 Oct 2026 08:52:52 GMT</pubDate>
            <atom:updated>2026-10-13T11:15:46.115Z</atom:updated>
            <content:encoded><![CDATA[<h3>How Cisco SD-WAN Runs in Production: Cloud OnRamp, Application Experience and Operating at Scale: Part 7</h3><h4>Steering SaaS and cloud traffic, recovering quality from imperfect transports, seeing what the fabric is doing, and designing controllers and regions that survive failure the final part of the series.</h4><p>Part 6 ended with an uncomfortable idea: once a branch uses DIA, security no longer lives in a building called the data centre. It lives in the path selected for each flow. A green template or an up SIG tunnel proves only that a component exists, counters and session state prove that a packet met it.</p><p>The same idea applies to everything this final part covers. Application performance, cloud connectivity and operational visibility are also properties of the path a flow takes, not of the features configured on a device. A fabric can be correctly built, correctly secured and still deliver a poor Teams call, a slow ERP screen or an outage nobody can explain at 08:40 on a Monday.</p><p>Return to the retail organisation from Part 6. The 200 stores now use DIA for Microsoft 365 and guest traffic, private applications ride the overlay, and internet traffic goes through a SIG. Over the past year, the business has moved its ERP into AWS, put file services in Azure, and started running store briefings over Teams. Three new complaints arrive in the same week:</p><ul><li>Stores on one broadband provider report choppy Teams calls in the morning.</li><li>Head office says the ERP is “slower since the migration”, even though AWS latency looks fine.</li><li>Operations cannot tell whether a quiet vManage dashboard means the fabric is healthy or simply that nobody enabled the right telemetry.</li></ul><p>Each complaint maps to a part of this article: where the traffic actually goes now, how the fabric squeezes quality out of imperfect transports, how engineers know what is happening, and how the design holds up when controllers, regions or providers fail. The final section ties all seven parts into a single working model.</p><blockquote><em>Naming note: from the 20.12/17.12 generation Cisco renamed vManage, vSmart and vBond to SD-WAN Manager, SD-WAN Controller and SD-WAN Validator. This article uses the new names, with the old ones where CLI output or older documentation still uses them.</em></blockquote><h3>Where the traffic actually goes now: Cloud OnRamp for SaaS</h3><p>Application-Aware Routing, as Part 5 described it, depends on BFD probes running inside IPsec tunnels between WAN edges. That works well for overlay traffic. It tells you nothing about a DIA path to Microsoft 365, because there is no SD-WAN tunnel at the other end. The DIA tracker from Part 5 proves that an internet path exists. It does not prove that the path from ISP B to Microsoft’s nearest front door is performing well this morning.</p><p>Cloud OnRamp for SaaS fills that gap. The edge sends HTTP probes towards each enabled SaaS application over every candidate exit, measures loss and latency, and computes a quality score (vQoE, on a 0–10 scale) per application per path. The best-scoring exit is then used for that application’s traffic.</p><p>The candidate exits depend on the site’s role:</p><ul><li><strong>DIA site: </strong>the edge can use its own internet interfaces directly.</li><li><strong>Client site and gateway site</strong>: a branch without a usable local exit can reach the SaaS application through the overlay to a gateway site, such as a regional hub with better internet connectivity, which breaks out on its behalf.</li><li><strong>SIG tunnels: </strong>on newer releases (the 20.5/17.5 generation onwards on IOS XE SD-WAN), SIG tunnels can also be used as exits, so SaaS path selection and cloud inspection are not mutually exclusive.</li></ul><p>For Microsoft 365, Cisco follows Microsoft’s endpoint categories. You can steer only the <em>Optimise</em> category, which includes Exchange Online, SharePoint, and Teams media and is the latency-sensitive traffic Microsoft recommends sending direct, or include <em>Allow</em> and default endpoints as well. Newer releases can also use Microsoft’s telemetry on path quality. Check the release notes for your Manager version before relying on that feature, as it is version-dependent.</p><h4>The configuration workflow</h4><p>Cloud OnRamp for SaaS is driven from SD-WAN Manager rather than built by hand on each device:</p><ol><li>Enable Cloud OnRamp for SaaS under <strong>Administration › Settings</strong>.</li><li>Under <strong>Configuration › Cloud OnRamp for SaaS</strong>, select the applications and the service VPN they live in.</li><li>Define the DIA sites and the interfaces they may use, then the client and gateway sites where breakout happens elsewhere.</li><li>Confirm the prerequisites from Part 5 are already in place: NAT on the DIA interface, working DNS, and a route or data-policy path for the application traffic.</li></ol><p>That last step is where Part 6’s lesson comes back in. Cloud OnRamp’s path decision interacts with your centralised data policy, and the exact interaction depends on the release and the policy model in use. A broad sequence that matches the same SaaS application earlier, whether to force it into SIG or to NAT it out directly, can override the steering you think you configured. As in Part 6, the policy sequence decides what happens, regardless of what the dashboard says.</p><p>Steering also hides a security decision. If Microsoft 365 leaves directly through local DIA because that path scored best, it does not go through the SIG. Some organisations accept that for the <em>Optimise</em> category, because it is a defined set of Microsoft endpoints. Others require SIG-based exits. Either choice is valid, finding out after an audit that the tool made the choice for you is not.</p><h4>Verification</h4><p>On IOS XE SD-WAN, the CLI still uses the feature’s original internal name, <em>CloudExpress</em>:</p><pre>show sdwan cloudexpress applications<br>show sdwan cloudexpress local-exits<br>show sdwan cloudexpress gateway-exits</pre><p>The applications view shows each enabled application, the exit currently selected (local or gateway) and the interface or gateway system IP in use. The local-exit and gateway-exit views show the measured loss and latency per candidate exit. Comparing them answers the important question: did the edge pick the path you would have picked, and why? On legacy vEdge platforms, the equivalent commands drop the sdwan keyword.</p><p>In Manager, <strong>Monitor › Cloud OnRamp for SaaS</strong> plots vQoE over time per site and application. This view will explain the morning Teams complaint later in this article.</p><h4>When the score is good, and users still complain</h4><p>A common failure involves DNS rather than the WAN. The branch breaks out locally, but its DNS queries are still forwarded to resolvers in the data centre. Microsoft and most large SaaS providers use DNS to send a user to a nearby front door, so the answer the user receives is near the data centre, not the store. The probe may score well, but the connection still travels far to an edge chosen for a different location.</p><p>The fix is a DNS design decision: resolve SaaS domains locally, or through a resolver that is geographically close to the breakout point. To verify, compare the address a branch client resolves with the address the same name resolves to from a resolver in the branch’s region. The second failure is simpler. The vQoE score measures the path to the SaaS service. It does not measure the application’s own health. If the score is good while Teams is bad for everyone everywhere, check the provider’s status page before changing your fabric.</p><h3>Extending the fabric into AWS and Azure: Cloud OnRamp for Multicloud</h3><p>The ERP migration followed a familiar path. The cloud team built a VPC in AWS, the network team terminated a pair of IPsec VPNs from the data-centre firewalls on a Virtual Private Gateway, and branches reached the ERP by hairpinning through the data centre. It worked. It also added a data-centre round trip to every ERP transaction from every store, and it put cloud reachability behind the same backhaul that DIA was meant to remove.</p><p>Cloud OnRamp for Multicloud takes a different approach. It treats the cloud as another set of sites in the fabric, not an external network on the far side of a VPN.</p><h4>How it works in AWS</h4><p>SD-WAN Manager uses the AWS account credentials or IAM role you provide to build a transit VPC containing a pair of Catalyst 8000V cloud gateways. These are ordinary WAN edges. They form control connections and BFD sessions, and they appear in OMP like any branch. The cloud gateways connect to an AWS Transit Gateway. Depending on the release, that attachment is either IPsec VPN with BGP or TGW Connect (GRE with BGP), and BGP exchanges routes between the fabric and the TGW.</p><p>The workload VPCs, which Cisco calls host VPCs, are attached to the Transit Gateway. Their tags map to SD-WAN service VPNs. This mapping keeps the segmentation from Part 3 intact in the cloud: a host VPC tagged for the ERP segment is reachable from VPN 10 and from nowhere else. Behind the scenes, this mapping becomes separate TGW route tables, one per segment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cuE6JmFqxjTA-b6ofbC0aA.png" /></figure><p>The store now reaches the ERP over the overlay, directly to the cloud gateways in the region. AAR and the SLA classes from Part 5 apply to that path like any other, because it is just another set of tunnels.</p><h4>How it works in Azure</h4><p>In Azure, the design centres on Azure Virtual WAN. The Catalyst 8000V instances are deployed as a network virtual appliance inside the Virtual WAN hub. They peer with the hub’s router over BGP, and VNets connected to the hub are mapped to SD-WAN segments through tags. Google Cloud has an equivalent integration based on Network Connectivity Centre. Where the internet is not an acceptable middle mile, Cloud OnRamp for Multicloud Interconnect can build private connections through partners such as Megaport and Equinix instead.</p><h4>Design decisions that matter more than the wizard</h4><ul><li><strong>Overlapping address space.</strong> A Transit Gateway cannot route between two attached VPCs that use the same CIDR. Cloud teams that grew organically often have several 10.0.0.0/16 VPCs. Resolve this before onboarding, not during it.</li><li><strong>Cloud gateway sizing.</strong> The C8000V instance type and throughput licence set the ceiling for every branch flow into that region. Size from measured branch-to-cloud traffic and leave headroom for failover, when one gateway carries the load of both.</li><li><strong>Cost.</strong> TGW attachments and data processing, inter-region traffic and internet egress are all billed. A design that sends Azure-to-AWS traffic through two cloud gateways and a branch is technically valid and financially surprising.</li><li><strong>Ownership.</strong> Manager automates the TGW and its attachments, but the host VPC subnets also need route tables that send branch prefixes towards the TGW. Whether your release’s automation manages that or the cloud team’s Terraform does, someone must own it explicitly.</li></ul><h4>Verification and the one-way failure</h4><p>Two weeks after go-live, a store can open the ERP login page but times out on reports generated from a new reporting subnet. The control plane looks healthy: the cloud gateways have control connections, BFD sessions are up, and OMP shows the ERP prefixes.</p><pre>show sdwan control connections<br>show sdwan bfd sessions<br>show sdwan omp routes vpn 10<br>show bgp vpnv4 unicast vrf 10 summary</pre><p>On the cloud gateway, the first two commands confirm it&#39;s a healthy member of the fabric. The OMP view confirms that branches have learned the cloud prefixes. The BGP summary confirms the gateway’s session with the TGW is up and shows how many prefixes it receives.</p><p>The fault was in the cloud. The new reporting subnet had its own VPC route table, created by a cloud engineer’s Terraform module, with no route for store prefixes towards the TGW attachment. Traffic arrived from the store, but the replies followed the subnet’s default route towards an internet gateway and were lost. The fix was one route entry. The prevention was a tagging and route-table standard that the cloud team’s pipeline enforces for every new subnet. This is Part 6’s coverage lesson again, in a different team’s repository: <em>building the attachment is not the same as covering every subnet behind it</em>.</p><h3>Application experience beyond AAR</h3><p>Part 5 dealt with brownouts by moving traffic. When the broadband tunnel’s loss exceeds the voice SLA class, AAR shifts voice to a compliant path. That only works if a better path exists and the fabric detects the problem quickly enough.</p><p>Both assumptions fail more often than designs admit. A store with a broadband line and a 4G backup can see 1.5% loss on both during a busy evening. AAR has nowhere better to go. Detection speed is also a design choice: classic AAR calculates statistics over the BFD app-route poll interval and multiplier. With the default ten-minute poll interval, AAR responds over minutes, not seconds. Enhanced Application-Aware Routing, available from the 17.10/20.10 generation on supported platforms, uses inline data-plane measurements to react much faster. However, it still moves traffic only when somewhere better exists.</p><p>When no better path exists, the remaining tools change how traffic behaves on its current path.</p><h4>QoS: decide whose packets wait</h4><p>Engineers new to SD-WAN sometimes assume the overlay makes QoS unnecessary. The opposite is true. IPsec copies the inner DSCP marking to the outer header by default, but an internet provider ignores your markings. The only queue you control is the egress queue on your own WAN interface. If that interface transmits at line rate into a 100 Mbps service delivered on a gigabit port, congestion happens in the provider’s equipment, where your voice packets have no priority.</p><p>The fix is to shape slightly below the actual service rate, so your router becomes the bottleneck and queues intelligently behind that shaper. In SD-WAN Manager, this is built as a localised policy: class maps assign traffic to forwarding classes, a QoS map assigns queues and bandwidth, and the WAN interface template sets the shaping rate. On an IOS XE SD-WAN edge, the result is a hierarchical policy along these lines:</p><pre>policy-map WAN-SHAPE<br> class class-default<br>  shape average 95000000<br>  service-policy WAN-QUEUING<br>!<br>policy-map WAN-QUEUING<br> class VOICE<br>  priority level 1<br>  police rate percent 15<br> class BUSINESS-CRITICAL<br>  bandwidth remaining ratio 40<br> class class-default<br>  bandwidth remaining ratio 20<br>  random-detect<br>!<br>interface GigabitEthernet0/0/1<br> service-policy output WAN-SHAPE</pre><p>The parent policy shapes the 100 Mbps circuit to 95 Mbps, leaving headroom for IPsec and transport overhead. The child policy gives voice a strict-priority queue, policed so that a misconfigured or malicious marking cannot starve every other class. Remaining bandwidth is shared by ratio, and RED on the default class drops bulk TCP early rather than allowing tail drops to affect everything. Manager generates its own class names, which differ from these. Treat this as a reading aid for what Manager pushes, not as a paste-ready template.</p><p>Hub sites need one more control. A data-centre edge on a 1 Gbps circuit can easily overrun a 20 Mbps store link. Per-tunnel QoS, configured with hub and spoke roles, shapes traffic towards each spoke at the hub according to the bandwidth the spoke advertises. Adaptive QoS, where supported, adjusts the spoke’s shaper when the real throughput of a broadband or cellular link changes over the day. Both are release- and platform-dependent, confirm support before designing around them.</p><pre>show policy-map interface GigabitEthernet0/0/1 output<br>show sdwan app-route statistics</pre><p>The first command shows per-class offered rate, queue depth and drops. Drops in the default class during busy periods are expected. Drops in the voice class mean either that the policer is too small for the number of concurrent calls, or that something else is marked EF. The second command lets you compare loss and latency on the tunnel with what the queues are doing. Drops on the tunnel but not in your queues point to the provider, while drops in your queues point to your policy.</p><h4>Forward Error Correction: recover packets instead of re-routing them</h4><p>QoS decides which packets wait. It cannot recover a packet that the provider lost. For voice and video, a retransmission is useless because the packet arrives after it should have been played. Forward Error Correction (FEC) handles this by sending extra information in advance.</p><p>The sending edge groups overlay packets into blocks of four and sends one parity packet for each block. If the receiving edge loses any single packet in that block, it reconstructs it from the other three and the parity packet. The cost is roughly 25% additional bandwidth for protected traffic. FEC has two limits worth stating plainly. It recovers one loss per block, so burst loss that removes two packets from the same block is not recovered. It also operates only inside the overlay, DIA traffic gets no FEC.</p><p>FEC is applied as a loss-correction action in a centralised data policy and has two modes:</p><ul><li><strong>FEC Adaptive: </strong>enabled only when measured loss on the tunnel exceeds a configured threshold. This is the sensible default for most voice designs because it costs nothing on a clean path.</li><li><strong>FEC Always: </strong>protects every packet regardless of measured loss. Useful on a link known to be chronically poor, such as a rural DSL line, where waiting for the threshold means the first seconds of every problem are lost.</li></ul><h4>Packet duplication: send it twice</h4><p>Packet duplication goes further. The edge sends each selected packet over two different tunnels, normally on different transports, and the receiving edge keeps the first copy and discards the duplicate. A call survives complete loss on one path, not just a single dropped packet, which makes it the right tool for traffic that cannot tolerate gaps at all.</p><p>The cost is doubled bandwidth for that traffic, on both paths. That is reasonable for a dozen voice calls across two broadband providers. It becomes expensive when it pushes video conferencing over a metered 4G link, and pointless when both tunnels share the same last mile, because both copies are lost in the same fibre cut.</p><pre>show sdwan tunnel statistics fec<br>show sdwan tunnel statistics pkt-dup</pre><p>These show, per tunnel, how many FEC and duplicate packets were sent and received, and how many lost packets were reconstructed or covered by a duplicate. A rising reconstruction count is evidence that FEC is helping. A very high count on one tunnel is evidence that the circuit has a problem the provider should be fixing, FEC is masking it, not solving it. Raise the ticket anyway.</p><h4>TCP optimisation: the long fat pipe</h4><p>The ERP complaint from head office turned out to be a different problem. Reports were exported as large files from AWS in Ireland to an office in Singapore. Round-trip time was about 180 ms and loss was low. Throughput was still poor, because a single TCP flow on a high-latency path is limited by window size and by how its congestion control reacts to occasional loss. The path was healthy, one TCP connection simply could not use it well.</p><p>On supported IOS XE SD-WAN platforms, TCP optimisation is provided by the AppQoE service. The edge acts as a TCP proxy, terminating the client’s connection locally and running an optimised connection across the WAN. Cisco’s implementation uses BBR congestion control, which is much less sensitive to random loss and latency than traditional loss-based algorithms. It is enabled through an AppQoE feature template and a <em>TCP optimisation</em> action in the centralised data policy for the selected traffic. Deploying it at both ends of the path, the branch and the cloud or data-centre gateway, gives the most predictable results.</p><pre>show sdwan appqoe tcpopt status<br>show sdwan appqoe flow vpn-id 10 server-port 443</pre><p>The first command confirms the AppQoE service is running and TCP optimisation is enabled. The second lists the flows in VPN 10 that are being optimised for a given server port. If the flows you expect are not listed, check the data-policy match before investigating TCP behaviour.</p><blockquote><em>TCP optimisation does nothing for UDP, so voice, video and QUIC are untouched. A growing share of SaaS traffic now uses QUIC. Before buying platform memory and licences for TCP optimisation, use SAIE to check how much of the traffic you want to improve is actually TCP.</em></blockquote><h4>Matching the tool to the problem</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GFsLdQLULCFh0ah2Kx39yg.png" /></figure><h3>Day 2: knowing what the fabric is actually doing</h3><p>Every mechanism in this series produces evidence: control connections, OMP routes, BFD statistics, policy counters, firewall sessions, vQoE scores, queue drops. Day-2 operations depend on knowing which evidence answers which question, and on collecting it before the incident, not during it.</p><h4>Alarms, events and what Manager shows</h4><p>SD-WAN Manager receives events from devices, such as a BFD session going down, a control connection flapping or an interface changing state, and correlates them into alarms. That distinction matters during an outage. One store losing its broadband circuit creates many BFD events, one for every tunnel on that TLOC, but should produce a small number of meaningful alarms such as a TLOC or site going down. Teams that forward raw events to an on-call channel quickly learn to ignore it. Forward alarms with severity, and keep events for investigation.</p><p>The monitoring dashboards cover device, tunnel and application health. They show where a problem is well. They are less good at explaining why, which is where the CLI and flow-level telemetry come in.</p><h4>SAIE: application visibility has a storage bill</h4><p>SD-WAN Application Intelligence Engine (SAIE), called DPI in older releases, identifies applications using NBAR2 on IOS XE SD-WAN and reports per-application usage to Manager. It is enabled in a localised policy with <em>application visibility</em>, and flow-level records are enabled with <em>flow visibility</em> (cflowd).</p><p>The common mistake is to enable everything, everywhere, at the most detailed level. Statistics volume grows with sites, flows and retention, and Manager’s statistics database becomes the most expensive part of the deployment. A workable pattern is:</p><ul><li>Enable application visibility broadly, because per-application usage is useful at every site.</li><li>Enable flow visibility selectively, or export cflowd/IPFIX records to an external collector designed for long retention.</li><li>Tune the SAIE aggregation and statistics retention settings in Manager to match the questions you actually need answered.</li><li>Keep the NBAR2 protocol pack current, an outdated pack misclassifies newer SaaS traffic, and data policy then matches the wrong application.</li></ul><pre>show sdwan app-fwd cflowd flows<br>show ip nbar protocol-pack active</pre><p>The first command shows the flows the edge is currently exporting, with VPN, addresses, ports and application where known. The second shows which protocol pack is loaded, which answers the question “why is this application classified as generic SSL?” more often than people expect.</p><p>Above Manager, the analytics service (vAnalytics, now packaged as Cisco SD-WAN Analytics) provides longer-term trends, baselines and comparisons between carriers and regions. It is licence-dependent, and its packaging has changed several times, so check what your subscription includes. Its value is in answering “is ISP B worse than ISP A in the north-west every weekday morning?”, which is very hard to answer from per-device views.</p><p>Recent Manager releases also include Network-Wide Path Insight (NWPI). It traces selected traffic for a site and VPN across the fabric and shows the path taken, the policy applied and where drops occurred. It is the closest thing the platform offers to “follow the packet” without a laptop at each end.</p><h4>A triage ladder for the CLI</h4><p>When the GUI points to a site but cannot explain the fault, work upwards through the layers this series has built, in the same order every time:</p><pre>! 1. Trust and control plane (Parts 1-3)<br>show sdwan control connections<br>show sdwan control local-properties<br>show sdwan omp peers</pre><pre>! 2. Routes and data plane (Parts 3-5)<br>show sdwan omp routes vpn 10<br>show sdwan bfd sessions<br>show sdwan app-route statistics</pre><pre>! 3. Policy decisions (Parts 4-6)<br>show sdwan policy from-vsmart<br>show sdwan policy data-policy-filter</pre><pre>! 4. Application and platform<br>show sdwan app-fwd cflowd flows<br>show sdwan cloudexpress applications<br>show platform hardware qfp active datapath utilization</pre><p>Each step either clears a layer or finds the fault. A missing control connection explains missing routes. Missing routes explain missing BFD sessions. Healthy BFD with poor app-route statistics explains an AAR decision. Correct statistics with unexpected data-policy counters explain a flow that took the wrong path. Starting at the top of this list prevents the most expensive troubleshooting mistake: investigating a QoS policy on a site whose control connections have been down since Friday.</p><h4>Back to Monday morning</h4><p>The choppy Teams calls followed that ladder. Control connections and BFD were healthy at every affected store. App-route statistics showed elevated loss on ISP B tunnels in one region between 08:00 and 10:00, but Teams media used DIA, so AAR was never involved. Cloud OnRamp’s vQoE view would have shown the same degradation on the ISP B local exit, except the affected stores were not included. Cloud OnRamp for SaaS had been enabled for a 40-store pilot and never extended.</p><p>The fix was to add the remaining stores as DIA sites with both internet interfaces as candidate exits. The edge then moved Teams to ISP A during the morning degradation, and the analytics data gave the carrier manager the evidence to escalate. The lesson repeats Part 6: <strong>a feature configured for a subset of sites protects only that subset</strong>. Audit coverage, don&#39;t assume it.</p><h4>Changing a fabric without breaking it</h4><p>Centralised policy is powerful because the Controllers distribute it fabric-wide. That is also why a bad change is dangerous: a mistake in a sequence that matches every site reaches every site at once. Operational practice that holds up in production includes:</p><ul><li><strong>Preview every change.</strong> Manager shows the configuration difference before a template or policy push. Read it, particularly for policy, where a reordered sequence changes behaviour without changing any individual line.</li><li><strong>Canary through site lists.</strong> Introduce new behaviour in a sequence that matches a small pilot site list, measure it, then widen the list. The policy structure from Part 4 makes this easy if you designed site lists with it in mind.</li><li><strong>Know the rollback timer.</strong> Device templates have a rollback timer. If a pushed configuration breaks the device’s control connections, the edge reverts to its previous configuration when the timer expires. This protects against locking out a remote site, but not against a change that keeps the control plane up while breaking user traffic.</li><li><strong>Treat the fabric as code where you can.</strong> Manager has a REST API, and the CiscoDevNet sdwan Terraform provider and Ansible collections can manage templates and policy objects. Version-controlled, peer-reviewed policy changes make Part 6&#39;s “old DIA exception at sequence 10” far less likely to survive unnoticed.</li></ul><p>Finally, extend the failure-drill matrix from Part 6 beyond security: fail a Controller, fail a Manager cluster node, introduce loss and latency on a lab transport with Linux tc netem to watch AAR, FEC and duplication react, and shut down one cloud gateway in a maintenance window. A design that has never been failed deliberately is still an assumption.</p><h3>Designing for scale and failure</h3><p>A 200-store fabric with two controllers in one data centre works well until the retailer acquires a European chain and an APAC distributor. Six months later there are 1,200 sites on three continents. The design questions change from “does it work?” to “what fails together, and how far does the damage spread?”</p><h4>What happens when each controller disappears</h4><p>The three controller roles fail differently, and the design should reflect that:</p><ul><li><strong>SD-WAN Validator (vBond)</strong> is needed for onboarding and re-authentication. Deploy several, publish them behind one DNS name that resolves to all their addresses, and an edge will try each in turn. Losing every Validator does not stop forwarding, it stops edges from joining or rejoining the fabric.</li><li><strong>SD-WAN Controller (vSmart)</strong> reflects OMP routes and distributes policy and data-plane keys. Edges normally hold sessions to two Controllers, and Controllers peer with each other so each has the same view. If an edge loses all its Controllers, OMP graceful restart keeps it forwarding with the last known routes and keys. The default graceful-restart timer is 43,200 seconds, or 12 hours, but keys still rekey on their own interval. Plan for a Controller outage to be shorter than both, and check the timers you have actually configured.</li><li><strong>SD-WAN Manager (vManage)</strong> is the management plane. Losing it does not affect forwarding, but it removes visibility and the ability to change anything. Production deployments run a three-node cluster, or larger at scale, so that the application, statistics, configuration database and messaging services survive a node failure. A second cluster in another location, configured for disaster recovery, covers the loss of the whole site.</li></ul><h4>Controller affinity: putting edges on the right controllers</h4><p>With Controllers in London, Frankfurt and Singapore, it makes little sense for a store in Manchester to depend on Singapore while London is healthy. Controller groups and affinity express that preference. Each Controller is assigned a group ID, and each edge is given an ordered list of groups it should prefer:</p><pre>! SD-WAN Controller in London and Frankfurt<br>system<br> controller-group-id 1<br>!<br>! SD-WAN Controller in Singapore<br>system<br> controller-group-id 2<br>!<br>! UK store edge<br>system<br> controller-group-list 1 2<br> max-omp-sessions 2<br>!<br>sdwan<br> interface GigabitEthernet0/0/1<br>  tunnel-interface<br>   max-control-connections 2<br> interface Cellular0/2/0<br>  tunnel-interface<br>   max-control-connections 1<br>   exclude-controller-group-list 2</pre><p>The UK edge prefers group 1 and falls back to group 2 only if group 1 is unavailable. The broadband TLOC keeps two control connections for resilience. The cellular TLOC keeps one, and never to Singapore, which limits control traffic on a metered link. Manager generates this from the system and VPN 0 interface templates. Affinity is a preference, not a partition. Unless a group is explicitly excluded, an edge will fail over to it, so the Controllers in every group must hold identical policy. The manager handles that when it activates centralised policy.</p><p>The common mistake is placing both Controllers of a group in the same data centre, on the same hypervisor cluster or behind the same internet circuit. The configuration shows two Controllers, but the design has one failure domain.</p><h4>The tunnel arithmetic that forces hierarchy</h4><p>Part 4 showed how control policy shapes topology. Scale is where that stops being optional. In a full mesh, each edge builds a tunnel to every TLOC it can reach on every remote site. With 200 sites and two TLOCs each, every edge holds 199 × 4 = 796 tunnels and BFD sessions. With 1,200 sites, the number rises to almost 4,800, far beyond what a branch router should carry and well beyond what is useful. A store in Leeds has no reason to hold a tunnel to a warehouse in Osaka.</p><p>Before multi-region fabric, the answer was control policy: hub-and-spoke or regional meshes built from TLOC filtering and route manipulation. That works, but inter-region traffic then depends on carefully maintained policy, and every region still shares one flat control plane.</p><h4>Multi-Region Fabric</h4><p>Multi-Region Fabric (MRF), introduced in the 17.7/20.7 generation, turns that hierarchy into architecture. Each site belongs to an access region. Within a region, edges mesh, or follow whatever topology the regional policy defines. Border routers sit in both the access region and a core region, and connect the regions to each other. Traffic from Leeds to Osaka goes from the Leeds edge to a European border router, across the core to an APAC border router, and then to Osaka. Border routers re-originate routes between regions, so an edge holds detail for its own region and reaches everything else through its border routers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n6UosmU-m3DeD0TtXeakew.png" /></figure><p>At device level, the region membership and role are system properties. On border routers, each tunnel interface is assigned to either the core or the access region:</p><pre>! Access-region edge<br>system<br> region 1<br>!<br>! Border router<br>system<br> region 1<br> role border-router<br>!<br>sdwan<br> interface GigabitEthernet0/0/2<br>  tunnel-interface<br>   region core</pre><p>This shows the shape of the configuration, Manager generates it from the system and VPN 0 templates. Later releases added secondary regions, which allow direct paths that bypass the core where needed, and subregions. Check the configuration guide for your release before designing around them.</p><p>MRF brings several practical benefits. Tunnel counts stay regional. The core can use a different transport, such as a premium backbone or a cloud provider’s network, from the one used in access regions. Each region can have its own Controllers, policy and failure domain. The trade-offs are just as specific. Border routers become critical aggregation points, so they need pairs, independent power and diverse transports. Policies written for a flat fabric, especially AAR or data policies that assume a direct tunnel between any two sites, need reviewing, because the path between regions now goes through border routers.</p><pre>show sdwan control connections<br>show sdwan omp routes vpn 10 detail<br>show sdwan bfd sessions</pre><p>On an MRF-capable release, the control-connection and OMP detail output show region information. Use it to confirm that an inter-region prefix is learned from a border router and not directly, and that an access edge holds BFD sessions only inside its own region and to its border routers. An access edge with sessions to remote-region sites indicates that region assignment or TLOC filtering hasn&#39;t been applied as intended.</p><h3>Seven parts, one mental model</h3><p>Looking back across the series, every part answered the same question at a different layer: <strong>why did this packet take this path, and how do you prove it?</strong> Put together, the layers stack like this:</p><pre>  7  Operate     Cloud OnRamp, QoS/FEC/TCP opt, telemetry, HA, regions<br>  6  Protect     ZBFW, UTD, DNS security, SIG, fail-open vs fail-closed<br>  5  Steer       AAR, SLA classes, DIA, service chaining, brownouts<br>  4  Decide      Control policy (topology) vs data policy (per flow)<br>  3  Carry       IPsec, BFD, TLOCs, VPN segmentation<br> 1-3 Trust       Certificates, Validator, Controllers, Manager, OMP<br>                 ───────────────────────────────────────────────<br>                 Troubleshoot from the bottom up. Design from the top down.</pre><p>The foundation is trust and control. Nothing forms until certificates are valid and the edge has authenticated to the Validator, Controllers and Manager. OMP then distributes routes, TLOCs and keys through the Controllers. That is why every troubleshooting ladder in this series starts with control connections.</p><p>Above it is the data plane. IPsec tunnels between TLOCs carry segmented VPNs, and BFD inside them continuously measures each path. Policy then makes two different kinds of decisions: control policy shapes what each edge learns, and therefore the topology, while data policy decides what each flow does. Most confusing behaviour in production comes from mixing those two up.</p><p>Steering uses those measurements. AAR moves traffic between compliant paths, DIA and service chaining decide where traffic leaves the overlay, and brownouts show that “up” and “usable” are different states. Security follows the path, because once traffic breaks out locally, the enforcement point has to move with it. This part added the operational layer: steering into SaaS and cloud, improving quality when no better path exists, seeing what the fabric is doing, and designing it to fail in contained, predictable ways.</p><p>Two habits connect all seven parts. The first is to <strong>follow the packet, not the dashboard</strong>. A green template, an up tunnel or a configured feature describes intent. Counters, sessions and route tables describe what happened. The second is to <strong>treat coverage as something to verify</strong>. The bypassed inspection in Part 6 and the missing Cloud OnRamp sites here were the same mistake: a correct feature applied to fewer sites, VPNs or subnets than everyone believed.</p><p>If you design from the top of that stack, what each traffic class needs, where it should go, what protects it and how you will know, and troubleshoot from the bottom, you will rarely be surprised by a fabric for long. SD-WAN does not remove network engineering. It makes the engineer’s decisions explicit, central and fast. That is a good thing, as long as every decision is one you made deliberately.</p><p>That brings the series to a close. Thank you to everyone who followed it from the first control connection to the last border router. If something in your own deployment behaved in a way none of these parts explained, describe it in the responses. Those cases are usually where the next lesson is.</p><p><strong>Part 1–1:</strong> <a href="https://www.linkedin.com/pulse/1-introduction-sd-wan-mehrdad-mokhtari-0nrvf/?trackingId=okHYbxY%2BQx6GCWhtDbawug%3D%3D">https://www.linkedin.com/pulse/1-introduction-sd-wan-mehrdad-mokhtari-0nrvf/?trackingId=okHYbxY%2BQx6GCWhtDbawug%3D%3D</a></p><p><strong>Part 1–2: </strong><a href="https://www.linkedin.com/pulse/12-what-sd-wan-mehrdad-mokhtari-ro2qe/?trackingId=okHYbxY%2BQx6GCWhtDbawug%3D%3D">https://www.linkedin.com/pulse/12-what-sd-wan-mehrdad-mokhtari-ro2qe/?trackingId=okHYbxY%2BQx6GCWhtDbawug%3D%3D</a></p><p><strong>Part1–3:</strong> <a href="https://medium.com/stackademic/how-cisco-sd-wan-works-control-plane-data-plane-and-orchestration-explained-part-1-3-9128c0020a22">https://www.linkedin.com/pulse/13-how-cisco-sd-wan-works-mehrdad-mokhtari-h4ufe/ trackingId=okHYbxY%2BQx6GCWhtDbawug%3D%3D</a></p><p><strong>Part 2:</strong> <a href="https://www.linkedin.com/pulse/14-cisco-sd-wan-main-principles-mehrdad-mokhtari-bfrde/?trackingId=okHYbxY%2BQx6GCWhtDbawug%3D%3D">https://www.linkedin.com/pulse/14-cisco-sd-wan-main-principles-mehrdad-mokhtari-bfrde/?trackingId=okHYbxY%2BQx6GCWhtDbawug%3D%3D</a></p><p><strong>Part 3: </strong><a href="https://medium.com/system-weakness/how-cisco-sd-wan-actually-secures-the-network-tunnels-certificates-and-segmentation-part-3-91fc4d7b3d54">https://medium.com/system-weakness/how-cisco-sd-wan-actually-secures-the-network-tunnels-certificates-and-segmentation-part-3-91fc4d7b3d54</a></p><p><strong>Part 4: </strong><a href="https://medium.com/stackademic/how-cisco-sd-wan-policy-actually-works-control-policy-data-policy-and-why-engineers-mix-them-up-2c4ab7d7bc3b">How Cisco SD-WAN Policy Actually Works: Control Policy, Data Policy, and Why Engineers Mix Them Up — Part 4</a></p><p><strong>Part 5: </strong><a href="https://medium.com/system-weakness/how-cisco-sd-wan-traffic-actually-flows-dia-service-chaining-and-what-breaks-when-a-transport-d0dd102a3443">https://medium.com/system-weakness/how-cisco-sd-wan-traffic-actually-flows-dia-service-chaining-and-what-breaks-when-a-transport-d0dd102a3443</a></p><p><strong>Part 6:</strong> <a href="https://medium.com/@mehrdad.mokhtari/how-cisco-sd-wan-secures-direct-internet-access-zbfw-utd-and-secure-internet-gateways-part-6-1db4cfcf6d47">https://medium.com/@mehrdad.mokhtari/how-cisco-sd-wan-secures-direct-internet-access-zbfw-utd-and-secure-internet-gateways-part-6-1db4cfcf6d47</a></p><p>If you found it useful, don’t forget to 👏 clap, 🔄 share, and 💬 follow for more Network Security, Linux &amp; Windows, and DevOps &amp; Cloud insights.</p><p>📲 Follow me on social media:</p><p>🔗 LinkedIn: <a href="https://www.linkedin.com/in/mmokhta">https://www.linkedin.com/in/mmokhta</a></p><p>✍️ Medium: <a href="https://medium.com/@mehrdad.mokhtari1370">https://medium.com/@mehrdad.mokhtari1370</a></p><p>♛ GitHub: <a href="https://github.com/MEMOKHTARI">https://github.com/MEMOKHTARI</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0e6f50cd02c1" width="1" height="1" alt=""><hr><p><a href="https://systemweakness.com/how-cisco-sd-wan-runs-in-production-cloud-onramp-application-experience-and-operating-at-scale-0e6f50cd02c1">How Cisco SD-WAN Runs in Production: Cloud OnRamp, Application Experience and Operating at Scale…</a> was originally published in <a href="https://systemweakness.com/">System Weakness</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[NFS on RHEL 10: From a Manual Build to a Verified Ansible Playbook — has a journey (by hand →…]]></title>
            <link>https://blog.stackademic.com/nfs-on-rhel-10-from-a-manual-build-to-a-verified-ansible-playbook-has-a-journey-by-hand-0851459627bf?source=rss-3099b15986b------2</link>
            <guid isPermaLink="false">https://medium.com/p/0851459627bf</guid>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[Mehrdad Mokhtari]]></dc:creator>
            <pubDate>Tue, 15 Sep 2026 14:57:21 GMT</pubDate>
            <atom:updated>2026-09-17T00:45:34.368Z</atom:updated>
            <content:encoded><![CDATA[<h3><em>NFS on RHEL 10: From a Manual Build to a Verified Ansible Playbook</em> — has a journey (by hand → automated) and front-loads the keywords</h3><p><em>Build an NFS server and client from scratch, make the mount persistent, verify the complete read/write path, and troubleshoot the failures engineers see most often.</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NNazYWItEGLu4UlCy-wqKQ.png" /></figure><p>A new RHEL 10 application server needs a shared directory hosted by another Linux machine. The network path is open, both systems can resolve one another, and neither has been configured for NFS.</p><p>That is the deployment built here. The server will export/shared; the client will mount it at /mnt/shared. The verification covers more than the mount itself, because a successful mount does not prove that the application can write to the directory.</p><p>The example uses this addressing:</p><pre>RHEL 10 NFS server:  nfs-server  192.168.0.2  /shared<br>                                      |<br>                                      | NFSv4 over TCP 2049<br>                                      v<br>RHEL 10 NFS client:  nfs-client  192.168.0.20  /mnt/shared</pre><p>192.168.0.0/24 is a private example network. Substitute addresses and network ranges appropriate to the target environment. In production, permit only the clients that genuinely need the export.</p><h3>Before touching NFS</h3><p>Start on both machines by confirming the operating-system release and basic name resolution:</p><pre>cat /etc/redhat-release<br>getent hosts nfs-server nfs-client</pre><p>The first command should identify Red Hat Enterprise Linux release 10. getent checks the same name-service path that applications use, including /etc/hosts and DNS. If using hostnames rather than addresses in /etc/exports or /etc/fstab, make sure resolution is reliable before continuing.</p><p>Confirm also that the client can route to the server:</p><pre>ping -c 3 192.168.0.2<br>ip route get 192.168.0.2</pre><p>Ping may be filtered in some networks, so a failed ping is evidence rather than a verdict. The route lookup is still useful because it shows the interface, source address and next hop the client will use.</p><h3>Configure the RHEL 10 NFS server</h3><h4>Install the NFS utilities</h4><p>On nfs-server, install the package that supplies the server daemons and administration tools:</p><pre>sudo dnf install nfs-utils</pre><p>dnf resolves and installs the package and its dependencies from the enabled RHEL repositories. The same package is used on the client later, but the client does not need to run the NFS server service.</p><h4>Create the directory and set useful permissions</h4><p>Create the directory that will be exported:</p><pre>sudo mkdir /shared<br>sudo groupadd --system nfs-share-writers<br>sudo chgrp nfs-share-writers /shared<br>sudo chmod 2770 /shared</pre><p>This is deliberately not chmod 777. The directory is writable by its owner and the nfs-share-writers group, inaccessible to other users, and the leading 2 sets the set-group-ID bit. Files and directories created beneath /shared will inherit the directory&#39;s group, which is usually more manageable for a shared working area.</p><p>Add the intended users on both systems to a group with the same numeric GID. NFS with the common sec=sys security flavour sends numeric user and group IDs; matching names alone are not enough.</p><p>Check the chosen GID on the server:</p><pre>getent group nfs-share-writers</pre><p>Then create or adjust the matching group on the client, using the same numeric value. For example, if the server reports GID 995:</p><pre>sudo groupadd --system --gid 5000 nfs-share-writers<br>sudo usermod -aG nfs-share-writers &lt;application-user&gt;</pre><p>Do not copy 995 blindly. Use the GID from the server, and note that a logged-in user normally needs a new login session before supplementary-group changes take effect.</p><h4>Define the export</h4><p>Edit /etc/exports:</p><pre>sudo vi /etc/exports</pre><p>Add this line:</p><pre>/shared 192.168.0.0/24(rw,sync,root_squash)</pre><p>The syntax matters. There must be whitespace between the exported path and the client specification, but no whitespace between 192.168.0.0/24 and its opening parenthesis. A stray space there changes the meaning of the rule and can expose the directory more broadly than intended.</p><p>The options do the following:</p><ul><li>rw permits read and write NFS requests. It does not bypass the directory&#39;s normal Linux ownership, mode bits, ACLs or SELinux policy.</li><li>ro is the read-only alternative and is also the default when no access mode is specified.</li><li>sync makes the server reply after earlier changes have been committed to stable storage. It favours predictable data integrity over the performance advantage of asynchronous replies.</li><li>root_squash maps requests from client UID/GID 0 to the anonymous identity. A root user on a client therefore does not automatically become root on the server-side filesystem.</li><li>no_root_squash disables that protection. It is occasionally required by tightly controlled workloads, but it lets client root act as server root within the export. Treat it as an exception requiring a documented reason, not a convenient fix for permissions.</li></ul><p>root_squash is the safer production default. It limits the damage a compromised or misconfigured client can do. It does not replace correct ownership, consistent UID/GID allocation, or stronger authentication where the trust boundary requires it.</p><p>The example grants access to the whole /24 because it makes the lab easy to follow. A narrower rule is better when only one client needs access:</p><pre>/shared 192.168.0.20(rw,sync,root_squash)</pre><p>Avoid wildcards and broad networks unless they reflect a genuine requirement.</p><h4>Start the service and apply the export</h4><p>Enable the service at boot and start it now:</p><pre>sudo systemctl enable --now nfs-server<br>sudo systemctl status nfs-server</pre><p>enable creates the boot-time dependencies; --now starts the unit in the current session. The status should show the service as active. Use q to leave the status view.</p><p>Now read /etc/exports, rebuild the active export table and display what changed:</p><pre>sudo exportfs -rav</pre><p>The three switches are worth understanding:</p><ul><li>-r re-exports directories, synchronising the active table with /etc/exports and files under /etc/exports.d.</li><li>-a applies the action to all entries.</li><li>-v prints verbose output showing what was exported.</li></ul><p>To inspect the active export table without reloading it, use:</p><pre>sudo exportfs -v</pre><p>That distinction is useful during troubleshooting: exportfs -rav changes or refreshes active state; exportfs -v only reports it. /etc/exports is the configuration source, not a command. Use cat /etc/exports to inspect the file and exportfs -v to see what the kernel is currently exporting. If they disagree, reload the exports and look for parsing errors.</p><h4>Open the firewall for the protocol in use</h4><p>For the NFSv4-focused deployment in this article, permit the predefined NFS service and reload firewalld:</p><pre>sudo firewall-cmd --permanent --add-service=nfs<br>sudo firewall-cmd --reload<br>sudo firewall-cmd --list-services</pre><p>The nfs service permits the NFS endpoint, principally TCP 2049. On a deliberately configured NFSv4-only server, TCP 2049 is the main server-side requirement.</p><p>NFSv3 is different. It relies on supporting RPC services such as rpcbind, mountd, statd and the lock manager. Those services must use known ports, and the corresponding firewall rules must be opened. A typical RHEL configuration includes the nfs, rpc-bind and mountd firewalld services, plus any fixed ports configured for statd and lockd. Do not copy a random list of ports from an old build: decide which NFS versions the server will provide, set fixed NFSv3 RPC ports in /etc/nfs.conf where required, and permit that exact design.</p><h4>Verify the server before moving on</h4><p>Check the service, active exports, protocol versions and listening socket:</p><pre>sudo systemctl status nfs-server<br>sudo exportfs -v<br>cat /proc/fs/nfsd/versions<br>sudo ss -lntp | grep &#39;:2049&#39;<br>sudo firewall-cmd --list-services</pre><p>In /proc/fs/nfsd/versions, a leading + means that version is enabled and - means it is disabled. The socket check should show a TCP listener on port 2049. ss confirms local listening state; it does not prove that the firewall or network permits a remote client to reach the socket.</p><h3>Configure the RHEL 10 NFS client</h3><p>Move to nfs-client and install the same utility package:</p><pre>sudo dnf install nfs-utils</pre><p>If the server offers NFSv3, or both v3 and v4, query its mount service for the advertised exports:</p><pre>showmount -e 192.168.0.2</pre><p>The output should list /shared and the permitted client network. This test needs care on a strictly NFSv4-only server: showmount queries the older mount protocol and may fail even while NFSv4 itself works correctly. In that case, test the known NFSv4 export directly rather than opening NFSv3 services merely to make showmount succeed.</p><p>Create the local mount point and mount the export:</p><pre>sudo mkdir -p /mnt/shared<br>sudo mount -t nfs 192.168.0.2:/shared /mnt/shared</pre><p>-t nfs selects the NFS filesystem helper. Unless a version is specified with -o vers=..., the RHEL client negotiates the newest version supported by both sides. For a deployment standardised on NFSv4.2, make that choice explicit:</p><pre>sudo mount -t nfs -o vers=4.2 192.168.0.2:/shared /mnt/shared</pre><p>Now inspect the result from several useful angles:</p><pre>findmnt /mnt/shared<br>df -hT /mnt/shared<br>mount | grep &#39; /mnt/shared &#39;</pre><p>findmnt is the clearest targeted check: it reports the source, target, filesystem type and effective mount options. df -hT shows the filesystem type and space reported by the server. The mount output is more verbose, but it exposes negotiated options such as the NFS version and transport.</p><h3>Prove the complete read/write path</h3><p>Do not declare success because mount returned no error. Test as the account that will actually use the share:</p><pre>sudo -u &lt;application-user&gt; touch /mnt/shared/testfile<br>ls -ln /mnt/shared/testfile</pre><p>ls -ln displays numeric IDs, which is particularly useful for NFS. On nfs-server, verify the same object:</p><pre>ls -ln /shared/testfile</pre><p>The file should appear at both paths with the same UID and GID. If root can mount the export but the application account cannot create the file, the NFS transport is working; the remaining problem is normally identity, ownership, permissions, an ACL, or SELinux policy.</p><p>The test file can be removed once both views agree:</p><pre>sudo -u &lt;application-user&gt; rm /mnt/shared/testfile</pre><h3>Make the mount persistent</h3><p>Add the following entry to /etc/fstab on the client:</p><pre>192.168.0.2:/shared /mnt/shared nfs defaults,_netdev 0 0</pre><p>defaults supplies the normal mount defaults. _netdev tells the boot system that this filesystem depends on networking, so it should not be treated like a local disk that is available immediately. The final two zeroes disable dump processing and filesystem checks for this network mount.</p><p>If the design requires a fixed NFS version, preserve that decision in the entry:</p><pre>192.168.0.2:/shared /mnt/shared nfs defaults,_netdev,vers=4.2 0 0</pre><p>Test the entry before rebooting:</p><pre>sudo umount /mnt/shared<br>sudo mount -a<br>findmnt /mnt/shared</pre><p>mount -a Processes the eligible entries in /etc/fstab. Running it now catches typographical errors, invalid options and reachability problems while the session is still available. Finding out during the next unattended reboot is a much less pleasant test method.</p><p>For systems that must boot even when the NFS server is temporarily unavailable, assess systemd automount options such as x-systemd.automount and nofail against the application&#39;s requirements. They change failure behaviour, so they should be a deliberate operational choice rather than copied into every NFS entry.</p><h3>Troubleshooting the failures that matter</h3><p>The quickest route to a fix is to work in layers: route, port, NFS service, export policy, mount, then file permissions. Repeatedly editing /etc/exports will not repair a missing route.</p><h4>mount.nfs: Connection refused</h4><p>On the server, check whether the service is active and listening:</p><pre>sudo systemctl status nfs-server<br>sudo ss -lntp | grep &#39;:2049&#39;<br>sudo firewall-cmd --list-services</pre><p>On the client, test the exact destination:</p><pre>ping -c 3 192.168.0.2<br>ip route get 192.168.0.2<br>nc -zv 192.168.0.2 2049</pre><p>nc -zv performs a zero-I/O TCP connection test and reports verbosely. A successful connection proves that the client can reach a listener on TCP 2049; it does not validate the export path or authorisation. A refusal normally means the address is reachable but nothing is accepting the connection, while a timeout points more towards filtering, routing or a silent network failure.</p><p>Confirm that the IP or hostname in the mount command is really the NFS server. If using a hostname, compare its resolution on both machines:</p><pre>getent ahosts nfs-server</pre><h4>mount.nfs: access denied by server</h4><p>This means the client reached NFS, but the server rejected the requested export or client identity. On the server:</p><pre>cat /etc/exports<br>sudo exportfs -v</pre><p>Check that /shared is the exported path and that the client&#39;s actual source address falls inside the permitted host or network. A multihomed client may leave through a different interface than expected; ip route get 192.168.0.2 shows the selected source address.</p><p>After correcting the file, apply it and inspect the result:</p><pre>sudo exportfs -rav<br>sudo exportfs -v</pre><p>Also check for the subtle syntax error discussed earlier: 192.168.0.0/24(rw) is not equivalent to 192.168.0.0/24 (rw).</p><h4>The mount works, but writing fails</h4><p>Start with the effective mount and export modes:</p><pre>findmnt -no SOURCE,FSTYPE,OPTIONS /mnt/shared<br>sudo exportfs -v</pre><p>Then inspect the server-side path using numeric identities:</p><pre>namei -l /shared<br>ls -ldn /shared<br>getfacl -p /shared</pre><p>An rw export permits write requests to reach the server. The server still evaluates the directory owner, group, mode bits and ACLs against the request&#39;s UID and GIDs. Compare the application&#39;s identity on both systems:</p><pre>id &lt;application-user&gt;<br>getent group nfs-share-writers</pre><p>If a root write fails while an authorised non-root user succeeds, root_squash is probably doing exactly what it was designed to do. Do not switch to no_root_squash as a reflex. Give the application account correct access instead.</p><h4>SELinux is enforcing</h4><p>Check the mode on both systems:</p><pre>getenforce</pre><p>Leave SELinux enforcing. A normal NFS client mount is labelled with the nfs_t type, and normal filesystem access often needs no manual policy change. Restrictions usually appear when a confined service, rather than an interactive user, accesses the mounted share. For example, Apache may require its workload-specific httpd_use_nfs boolean.</p><p>Find evidence before changing policy:</p><pre>sudo ausearch -m AVC -ts recent<br>sudo journalctl -t setroubleshoot --since &#39;15 minutes ago&#39;<br>getsebool -a | grep nfs</pre><p>If an AVC denial identifies a relevant boolean, inspect its description and enable only the setting the workload requires. For an Apache workload, that might be:</p><pre>sudo setsebool -P httpd_use_nfs on</pre><p>The -P makes the change persistent and can take a little time while the policy is rebuilt. Different confined services have different booleans and rules, so do not apply the Apache example to an unrelated daemon. If no suitable boolean exists, build a narrowly scoped policy from the recorded denial or involve the system&#39;s SELinux policy owner. Disabling SELinux removes a security control and hides the real cause.</p><h4>showmount Fails, but an NFSv4 mount works</h4><p>This is not contradictory. showmount Depends on the NFSv3-era mount service; a server intentionally configured for NFSv4 only does not need to expose that service. Verify NFSv4 with the known export and port:</p><pre>nc -zv 192.168.0.2 2049<br>sudo mount -t nfs -o vers=4.2 192.168.0.2:/shared /mnt/shared<br>findmnt /mnt/shared</pre><p>Use showmount when the server supports the protocol it queries, not as the sole health check for every NFS design.</p><h3>Practical production checks</h3><p>Before handing the service over, review the decisions that tend to become incidents later:</p><ul><li>Restrict each export to the smallest practical set of clients or networks.</li><li>Keep root_squash unless a documented workload requirement justifies the risk of no_root_squash.</li><li>Use group ownership, the setgid bit and ACLs rather than opening the directory with mode 777.</li><li>Standardise the NFS version. NFSv4 simplifies the firewall path; RHEL 10 also supports stronger NFS security designs when the environment requires them.</li><li>Keep numeric UID/GID allocation consistent, or use an appropriate central identity and NFS security design.</li><li>Use stable, unique hostnames and reliable DNS if configuration depends on names. NFSv4 clients should have unique host identities.</li><li>Test persistent mounts withmount -a, and decide how the host should behave if the network or NFS server is unavailable during boot.</li><li>Monitor port reachability, mount availability, latency, capacity and server logs. A mounted filesystem can still become slow or unreachable later.</li><li>Keep SELinux enforcing and make workload-specific policy changes from evidence.</li></ul><p>At this point the path is complete: /shared is exported by nfs-server, /mnt/shared is mounted on nfs-client, the application identity can create a file, and the mount survives a reboot. More importantly, each layer has a test. That is what turns a collection of NFS commands into an operable service.</p><p>And you can see the full Ansible project for it on my GitHub, at the link below:</p><p><a href="https://github.com/MEMOKHTARI/Public_Projects/blob/main/Ansible/NFS-Project">Public_Projects/Ansible/NFS-Project at main · MEMOKHTARI/Public_Projects</a></p><blockquote>If you found it useful, don’t forget to <strong>👏 clap, 🔄 share, and 💬 follow</strong> for more Network Security, Linux &amp; Windows, and DevOps &amp; Cloud insights.</blockquote><p>📲 Follow me on social media:<br>🔗 LinkedIn: <a href="https://www.linkedin.com/in/bornaly/">https://www.linkedin.com/</a><a href="https://www.linkedin.com/in/mmokhta">in/mmokhta</a><br>✍️ Medium: <a href="https://medium.com/@mehrdad.mokhtari1370">https://medium.com/@mehrdad.mokhtari1370</a><br>♛ GitHub: <a href="https://github.com/MEMOKHTARI">https://github.com/MEMOKHTARI</a><br>⛪ GitLab: <a href="https://gitlab.com/Mehrdad261">[github.com/MEMOKHTARI</a>]</p><h3>Before you go</h3><ul><li>Please take a moment to like the post and follow the writer!</li><li>Did you know that over 400,000 developers share what they’re building, learning, and discovering across our platforms every month? Learn how you can contribute <a href="https://plainenglish.io/write-for-us"><strong>here</strong></a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0851459627bf" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/nfs-on-rhel-10-from-a-manual-build-to-a-verified-ansible-playbook-has-a-journey-by-hand-0851459627bf">NFS on RHEL 10: From a Manual Build to a Verified Ansible Playbook — has a journey (by hand →…</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Cisco SD-WAN Secures Direct Internet Access: ZBFW, UTD and Secure Internet Gateways — Part 6]]></title>
            <link>https://systemweakness.com/how-cisco-sd-wan-secures-direct-internet-access-zbfw-utd-and-secure-internet-gateways-part-6-1db4cfcf6d47?source=rss-3099b15986b------2</link>
            <guid isPermaLink="false">https://medium.com/p/1db4cfcf6d47</guid>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[information-security]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <category><![CDATA[network-security]]></category>
            <dc:creator><![CDATA[Mehrdad Mokhtari]]></dc:creator>
            <pubDate>Tue, 08 Sep 2026 07:58:29 GMT</pubDate>
            <atom:updated>2026-10-16T08:14:25.056Z</atom:updated>
            <content:encoded><![CDATA[<p><em>Following internet-bound traffic through stateful inspection, Snort, TLS decryption and SIG failover — and finding the policy paths that quietly bypass them.</em></p><p>Part 5 ended where SD-WAN’s neat abstraction stops helping. A branch packet was NATed out of VPN 0, left the overlay and reached the internet without visiting the central firewall. The path was shorter, and the SaaS application was faster. It also meant that the branch had acquired its own internet edge.</p><p>That change is easy to underestimate. The overlay still encrypts traffic between WAN edges, but it cannot protect a flow after local breakout. NAT hides an inside address and maintains a translation; it does not decide whether a user should reach a newly registered domain, whether a download contains malware, or whether an established TLS session is carrying an exploit.</p><p>Consider a retail estate with 200 sites. Microsoft 365 and guest traffic use DIA, while private applications stay on the overlay. One workstation follows a phishing link. The connection succeeds locally, so the data-centre firewall has no session, no URL event and no malware verdict to investigate. Nothing has failed in routing. The security architecture simply ended at the wrong place.</p><p>Cisco SD-WAN offers three places to put that control back: on the edge, in a cloud-delivered Secure Internet Gateway, or at a regional hub. The hard part isn&#39;t enabling a feature. It is deciding which traffic each control owns, what happens when that control is unavailable, and how an engineer proves that a working session was actually inspected.</p><h3>The branch is now a security boundary</h3><p>DIA moves the internet exit, so the enforcement point has to move with it. A typical branch design builds protection in layers, most of them on the edge itself:</p><ul><li><strong>Zone-Based Firewall (ZBFW)</strong> decides which zones may initiate sessions and maintains state for the return traffic.</li><li><strong>Unified Threat Defense (UTD)</strong> adds intrusion inspection, URL filtering and file reputation to selected flows on supported IOS XE SD-WAN platforms.</li><li><strong>DNS-layer security</strong> stops many destinations before a connection is made.</li><li><strong>A Secure Internet Gateway (SIG)</strong> carries selected internet traffic to a cloud security service such as Cisco Umbrella or Zscaler for inspection there.</li></ul><p>These are not four names for the same thing. A firewall rule can allow HTTPS while URL filtering blocks a high-risk site. DNS security can stop a malicious domain but cannot inspect a connection made directly to an IP address. A healthy SIG tunnel says that a cloud path exists; it does not prove that the data policy sent this particular flow into it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*M7163XGi9yPlOeYB3XqKbw.png" /><figcaption>DIA moves the security boundary to the branch. After local breakout, inspection can go in three places: ZBFW and UTD on the edge, a cloud SIG (Umbrella/Zscaler), or the SD-WAN overlay back to a regional firewall.</figcaption></figure><h3>On-box security starts with state, not signatures</h3><p>The first useful control at the branch is normally ZBFW. It groups service VPNs or interfaces into security zones, then applies a policy to a directional zone pair. Traffic from a user zone to an internet zone may be inspected and allowed; unsolicited traffic in the reverse direction is not admitted merely because the outbound pair exists. Return packets are allowed because the firewall has session state.</p><p>On an IOS XE SD-WAN edge, a VPN-based layout generated by SD-WAN Manager can look like this at device level:</p><pre>policy<br> zone USERS<br>  vpn 10<br> !<br> zone INTERNET<br>  vpn 0<br> !<br> zone-pair ZP-USERS-INTERNET<br>  source-zone USERS<br>  destination-zone INTERNET<br>  zone-policy USERS-OUT<br> !</pre><p>This excerpt establishes the boundaries and attaches the policy; the rules inside USERS-OUT are built in the firewall or unified security policy in Cisco SD-WAN Manager. It is deliberately not a complete paste-ready policy because the generated class maps, application lists and logging statements depend on the release and the chosen policy mode.</p><p>A sensible outbound rule set might permit approved DNS first, inspect sanctioned business applications, pass general web traffic into an advanced inspection profile, and log and drop what remains. The order matters. So does direction: USERS → INTERNET is not the same zone pair as INTERNET → USERS.</p><p>Two details catch experienced routing engineers. First, the self zone protects traffic to and from the router itself and cannot stand in for a transit zone pair. An over-broad self-zone rule can expose management services, while an over-tight one can break control, monitoring or troubleshooting access. Second, interface-based zones are supported on IOS XE SD-WAN from the 20.7.1/17.7.1 generation, and an interface-based policy takes precedence when both interface- and VPN-based policies could apply. A template that looks correct at VPN level may therefore not be the policy actually seeing the packet.</p><h3>Proving that ZBFW is doing work</h3><p>The useful question is not “is a firewall policy attached?” but “did this flow cross the intended zone pair, match the intended rule and create state?” The commands depend on the software train:</p><pre>show sdwan zbfw zonepair-statistics<br>show sdwan zbfw drop-statistics<br>show policy zbfw filter-statistics<br>show policy zbfw sessions</pre><p>show sdwan zbfw zonepair-statistics is available in IOS XE SD-WAN 17.11.1a and later. On earlier supported releases, the show policy zbfw ... views are often the more useful route. An incrementing rule counter and a session showing the expected source, destination and zone pair are stronger evidence than a green template status.</p><p>If the drop statistics show no-zone-pair-present, the packet reached a boundary for which no applicable pair exists. If the session is present but the application still fails, the investigation moves beyond basic state: MTU, TLS handling, UTD verdicts and the return path are now more likely than routing.</p><h3>What UTD actually adds</h3><p>ZBFW answers who may start a conversation. UTD looks further into conversations that the unified security policy selects for advanced inspection. On supported IOS XE SD-WAN devices, Cisco SD-WAN Manager installs a compatible Security Virtual Image and runs the inspection engine as a hosted service. Packets move from the data plane to that service plane, are inspected, and return with a verdict.</p><p>That architecture matters operationally. UTD spends forwarding budget every time it inspects. The router needs supported hardware, storage, memory, a software-compatible security image and the right subscriptions. Catalyst 8200 and 8300 platforms, for example, require at least 8 GB of DRAM and 16 GB of M.2 USB storage for UTD; older ISR and virtual platforms have their own matrices. Check the exact feature entitlement and throughput licence against the platform and current Cisco subscription, rather than assuming it because the security wizard is visible.</p><p>The hosted functions solve different problems:</p><ul><li><strong>Snort IDS/IPS</strong> compares traffic with Talos rules and protocol analysis. IDS mode reports; IPS mode can block. A policy in IDS mode isn&#39;t broken when an attack succeeds; it was never instructed to prevent it.</li><li><strong>URL filtering</strong> makes allow or block decisions using category, reputation and explicit lists. Cloud lookup is lighter on local storage; an on-box database reduces lookup dependence but needs more memory and storage and must remain current.</li><li><strong>Advanced Malware Protection</strong> applies file reputation and malware controls to eligible traffic. Its value depends on the engine being able to see the file content.</li><li><strong>TLS/SSL decryption</strong> makes encrypted sessions inspectable by acting as a transparent proxy. It turns otherwise opaque HTTPS into content that IPS, URL and malware engines can examine.</li></ul><p>Resource profiles allow more Snort instances on platforms that support them. More instances can process packets in parallel and improve throughput, but consume more CPU and memory. The only honest sizing test is on the intended platform with the intended mix of IPS rules, URL lookups, TLS decryption and production-like packet sizes. A headline forwarding figure measured without those services tells you nothing about branch-security capacity.</p><h3>TLS decryption is a PKI project wearing a firewall label</h3><p>When TLS decryption is enabled, the edge terminates the client’s TLS session, validates the remote server, and creates a second encrypted session towards it. Managed clients must trust the enterprise CA the proxy uses. If they do not, users see certificate errors even though routing, DNS and the firewall are all healthy.</p><p>Some traffic should not be decrypted: certificate-pinned applications, categories subject to privacy rules, and protocols or cipher suites the deployed release cannot proxy. Cisco’s TLS policy exposes choices for unsupported versions and ciphers, and a failure mode of open or close. Write those choices as security requirements before rollout. Otherwise, the first certificate-pinned mobile application becomes an emergency bypass rule with no owner and no expiry date.</p><p>A practical deployment starts with a small managed-client group, a documented no-decrypt list, certificate-chain testing and measurements for CPU, memory, latency and throughput. It also tests revocation access and certificate rotation. A TLS policy that works until the enterprise CA expires is only a delayed outage.</p><h3>Checking the engine, not merely the policy</h3><pre>show app-hosting detail appid utd<br>show utd engine standard status<br>show utd engine standard version<br>show utd engine standard utilization cpu<br>show utd engine standard utilization memory<br>show utd engine standard threat-inspection signature update status<br>show utd engine standard logging events<br>show platform hardware qfp active datapath utilization</pre><p>The first two commands establish whether the hosted application and its Snort instances are alive. Version and signature status establish whether the engine is compatible and current. CPU, memory and QFP utilisation show whether inspection is becoming a forwarding constraint. Events explain what the engine decided.</p><p>A common mistake is to see the security policy attached in Manager and stop there. A missing or incompatible Security Virtual Image, an engine that failed to start, stale signatures, or a cloud lookup that cannot leave through VPN 0 can separate policy intent from enforcement. URL filtering and AMP cloud lookups also need working NAT and internet reachability from the hosted service.</p><h3>DNS security is useful, but it is not a SIG tunnel</h3><p>Umbrella DNS integration can redirect client DNS requests to Umbrella and protect them with DNSCrypt. It is a fast, low-overhead way to block known malicious destinations before a TCP or QUIC session exists. A local-domain bypass list keeps internal names on the enterprise resolver.</p><p>In a centralised data policy, the relevant action is explicit:</p><pre>policy<br> data-policy DNS-SECURITY<br>  vpn-list BRANCH-VPNS<br>   sequence 10<br>    match<br>     dns request<br>    action accept<br>     redirect-dns umbrella<br>   default-action accept</pre><p>This excerpt is to add to the existing DIA policy, not a complete breakout policy. Umbrella registration is per VRF, DIA requires NAT, and DNS plus internet reachability must exist through the intended WAN path. Verification begins here:</p><pre>show umbrella config<br>show sdwan umbrella dnscrypt</pre><p>The output should show the intended VRFs registered, and DNSCrypt enabled with a valid certificate. Upstream devices must also handle larger DNSCrypt packets rather than assuming every DNS exchange fits the old 512-byte expectation.</p><p>DNS-layer security still sees only the DNS traffic the edge redirects. A client that uses an uncontrolled public DNS-over-HTTPS service, connects directly to an IP address, or reuses a previously resolved address can sit outside that view. Endpoint policy and firewall controls must prevent those alternate resolution paths if DNS enforcement is meant to be authoritative.</p><h3>Sending breakout to a Secure Internet Gateway</h3><p>When the branch cannot sensibly host the full inspection stack, the edge can build automatic tunnels to a cloud security service. User traffic still breaks out near the branch, but it first travels through an IPsec or GRE tunnel to an Umbrella or Zscaler enforcement point. URL, threat and access policies live in the cloud rather than in every router.</p><p>The Cisco SIG feature template defines the provider, tunnel endpoints, tracker and high-availability relationship. Release details matter. The SIG template and ECMP model arrived in the 20.4.1/17.4.1a generation; automated Zscaler IPsec provisioning requires at least 20.5.1/17.5.1a, and automated Zscaler GRE requires 20.9.1/17.9.1a. Newer deployments may use Cisco’s SSE integration instead. Legacy vEdge behaviour is not identical to IOS XE SD-WAN, particularly around policy fallback, so a cEdge example should not be copied into a mixed fabric without checking the release guide.</p><p>Automatic tunnels also have a mundane dependency that causes disproportionate trouble: the device uses a NAT-outside WAN interface for provider and internet access, and its DNS path must work in the expected transport context. If an upstream firewall blocks UDP 500 or 4500, an IPsec SIG tunnel will not become healthy no matter how correct the central data policy looks.</p><h3>Active and backup are path roles, not decorations</h3><p>A branch normally creates tunnels to two enforcement locations. The active tunnel carries traffic. The backup is established or ready according to the provider and template model, but is selected only when the active path or its tracker fails. Active/active designs can use tunnel weights and ECMP where supported.</p><p>The tracker must test something beyond tunnel encapsulation. A tunnel interface can be up while the provider path cannot pass useful Layer 7 traffic: the same “interface up, service dead” problem that the DIA tracker solved in Part 5. Current Manager releases distinguish up, down and degraded SIG/SSE states when Layer 7 health does not meet the configured SLA.</p><pre>show sdwan secure-internet-gateway tunnels<br>show sdwan secure-internet-gateway umbrella tunnels<br>show sdwan secure-internet-gateway zscaler tunnels<br>show endpoint-tracker</pre><p>The generic tunnel view is available from the 20.9.1/17.9.1a generation. Provider-specific commands are useful on earlier supported integrations and expose provisioning and tunnel state. For an active/backup pair, the expected result is two device-up tunnels with the intended HA roles and an enabled, healthy tracker. A tunnel stuck in provider API or location provisioning is not a routing fault.</p><h3>Route to SIG or policy-steer to SIG?</h3><p>Whole-VPN internet traffic can follow a SIG service default route:</p><pre>ip sdwan route vrf 10 0.0.0.0/0 service sig</pre><p>Selective traffic can be steered by centralised data policy:</p><pre>sequence 100<br> match<br>  destination-data-prefix-list INTERNET<br> action accept<br>  sig<br>  sig-action fallback-to-routing</pre><p>The second example is where failover becomes a security decision. sig-action fallback-to-routing is supported on IOS XE SD-WAN from 17.8.1a with Manager 20.8.1. If every SIG tunnel is down, the packet returns to normal routing. That may mean an overlay default towards a regional firewall, or it may mean uninspected DIA. “Fallback” describes the mechanism, not the security outcome.</p><p>Earlier IOS XE releases and vEdge data-policy steering do not provide the same fallback behaviour. A policy may continue pointing at an unavailable SIG service and black-hole the flow. Route-based SIG redirection can be safer on those releases because the SIG service route is withdrawn when the service is unavailable, allowing another route to win. The routing table still has to be designed so that the winning route is acceptable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9nXzvsorhcmAjAEJvtqGog.png" /><figcaption>Three places to put inspection back: on-box at the branch (keeps enforcing when the cloud is down), a cloud SIG near the destination (one policy for many sites), or a regional hub (heavyweight state, at the cost of backhaul).</figcaption></figure><h3>On-box, SIG or regional hub?</h3><h3>Inspect on the edge when local independence matters</h3><p>On-box ZBFW and selected UTD controls suit sites that must keep enforcing policy when cloud tunnels or a regional hub are unavailable. They also avoid an extra tunnel and can protect traffic between local service VPNs. The cost is distributed compute, security-image lifecycle, signature operations and a different performance ceiling at every router model.</p><h3>Use a SIG when users and applications are already distributed</h3><p>A SIG gives many branches one policy and one logging plane, usually with enforcement close to internet and SaaS destinations. It scales better than maintaining a full threat stack on every small edge. It introduces dependence on provider reachability, tunnel health, DNS, API credentials and the chosen fallback path. Data residency and TLS inspection policy may also decide which enforcement location a site may use.</p><h3>Use a regional hub when inspection needs heavyweight state</h3><p>Regional firewalls make sense when the organisation already operates large inspection clusters, needs specialist features unavailable on the edge, or must centralise decrypted traffic and logs. The trade is the backhaul Part 5 was trying to remove. Capacity, symmetric routing and regional failure domains return as design constraints.</p><h3>Most production designs are hybrid</h3><p>A practical design often runs ZBFW at every branch, redirects DNS everywhere, sends ordinary internet traffic to a SIG, and reserves on-box inspection or regional firewalls for selected applications and locations. That is the useful SD-WAN contribution to SASE: the WAN policy can place a user’s flow on the right security path based on site, VPN and application. It is not the whole SASE story by itself; identity, device posture, ZTNA and cloud policy are separate controls.</p><h3>The bypass that looks like success</h3><p>Return to the retail site. The SIG dashboard is green, browsing works, and the security team sees no Microsoft 365 events. The instinct is to blame provider logging. The actual problem is a policy sequence left over from the DIA rollout:</p><pre>sequence 10<br> match<br>  app-list SAAS<br> action accept<br>  nat use-vpn 0<br>  nat fallback<br>!<br>sequence 20<br> match<br>  destination-data-prefix-list INTERNET<br> action accept<br>  sig<br>  sig-action fallback-to-routing</pre><p>Centralised data policy is the first match. The broad SaaS rule succeeds at sequence 10, performs direct NAT and never reaches sequence 20. The application works precisely because it bypassed inspection.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X0UOSVBmgOjaCz2pVWSurw.png" /><figcaption>The bypass that looks like success: a broad SaaS exception at sequence 10 matches first, NATs the flow out via local DIA with no URL or IPS verdict, so the SIG rule at sequence 20 is never reached.</figcaption></figure><p>The investigation should follow the packet rather than the dashboards:</p><ol><li>Check the applied from-service data policy and its counters with show sdwan policy data-policy-filter &lt;policy-name&gt;. The DIA counter increasing while the SIG counter remains still identifies classification before any tunnel work begins.</li><li>Check show ip nat translations. A direct user translation on the transport interface indicates DIA, not SIG steering.</li><li>Check the ZBFW zone-pair statistics and session table. Confirm which boundary and rule owned the flow.</li><li>Check UTD engine state and events. No event is meaningful until you prove that the flow was selected for the advanced inspection profile.</li><li>Check SIG tunnel roles, tracker state and tunnel counters, then compare the provider log for the same source, destination and time.</li></ol><p>The repair is to narrow or remove the early exception and make the intended security path unambiguous. Moving the SIG sequence above the SaaS exception would send everything matching both lists to SIG; keeping DIA for a genuine exception requires a smaller application or prefix list. After activation, repeat the test and watch the SIG policy and tunnel counters increment.</p><p>A second version of the same bug appears when you add a new service VPN but don&#39;t assign it to the relevant zone or UTD target VPN list. Its traffic follows a valid default route and works, yet never reaches inspection. Template attachment is not coverage. Every new VPN, interface and application exception needs a security-path review.</p><h3>Fail open or fail closed — now with consequences</h3><p>Part 5 asked what should happen when a service disappears. This part supplies the uncomfortable answer: you must choose separately for reachability, firewall state, UTD failure, TLS failure, DNS lookup failure and SIG loss.</p><p>A payment network may fail closed if IPS or the approved SIG path is unavailable. A small sales office may fall back through the overlay to a regional firewall. Guest wireless may use direct DIA with ZBFW and DNS security when SIG is down. Voice signalling may bypass TLS decryption but still require a stateful firewall policy. One global answer is rarely correct.</p><p>You also need to test the decision. Pull the active SIG tunnel and observe the backup. Fail both and verify the actual route, public egress address and security log. Stop the UTD engine in a maintenance window and confirm whether selected traffic passes or drops as designed. Break cloud lookup while leaving the WAN interface up. Expire a lab TLS certificate. These are security tests and availability tests at the same time.</p><p>The final operational record should be a simple matrix for each traffic class: primary path, enforcement point, backup path, inspection retained on backup, expected session impact, owner and verification command. Without it, “fail open” becomes an accidental route and “fail closed” becomes an unexplained outage.</p><h3>Where the security boundary really lives</h3><p>Once a branch uses DIA, security no longer lives at a building called the data centre. It lives in the path selected for each flow. ZBFW provides state at the edge. UTD can add threat inspection where the platform can carry it. DNS security blocks destinations early. SIG tunnels move deeper inspection to the cloud. A regional hub remains valid when centralised controls justify the extra path.</p><p>The engineer’s job is to make those paths deliberate and observable. A green firewall template, running the Snort engine or an up SIG tunnel, proves only that a component exists. Policy counters, session state, tunnel selection and security events together prove that a packet met the control it was supposed to meet.</p><p>Part 7 closes the series with day-two operations: reading fabric health, correlating control-plane and application telemetry, changing policy safely and building the failure drills that keep an SD-WAN design from becoming a collection of untested assumptions.</p><p>If a branch has ever “worked” while quietly bypassing inspection, share what exposed it. Those incidents teach more than a perfect diagram ever will.</p><p>If you found it useful, don’t forget to 👏 clap, 🔄 share, and 💬 follow for more Network Security, Linux &amp; Windows, and DevOps &amp; Cloud insights.</p><p>📲 Follow me on social media:</p><p>🔗 LinkedIn: <a href="https://www.linkedin.com/in/mmokhta">https://www.linkedin.com/in/mmokhta</a></p><p>✍️ Medium: <a href="https://medium.com/@mehrdad.mokhtari1370">https://medium.com/@mehrdad.mokhtari1370</a></p><p>♛ GitHub: <a href="https://github.com/MEMOKHTARI">https://github.com/MEMOKHTARI</a></p><p>⛪ GitLab: [github.com/MEMOKHTARI]</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1db4cfcf6d47" width="1" height="1" alt=""><hr><p><a href="https://systemweakness.com/how-cisco-sd-wan-secures-direct-internet-access-zbfw-utd-and-secure-internet-gateways-part-6-1db4cfcf6d47">How Cisco SD-WAN Secures Direct Internet Access: ZBFW, UTD and Secure Internet Gateways — Part 6</a> was originally published in <a href="https://systemweakness.com/">System Weakness</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Cisco SD-WAN Traffic Actually Flows: DIA, Service Chaining, and What Breaks When a Transport…]]></title>
            <link>https://systemweakness.com/how-cisco-sd-wan-traffic-actually-flows-dia-service-chaining-and-what-breaks-when-a-transport-d0dd102a3443?source=rss-3099b15986b------2</link>
            <guid isPermaLink="false">https://medium.com/p/d0dd102a3443</guid>
            <category><![CDATA[netwrok-security]]></category>
            <category><![CDATA[sd-wan]]></category>
            <category><![CDATA[network-architecture]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Mehrdad Mokhtari]]></dc:creator>
            <pubDate>Thu, 03 Sep 2026 15:21:31 GMT</pubDate>
            <atom:updated>2026-09-04T07:34:13.354Z</atom:updated>
            <content:encoded><![CDATA[<h3>How Cisco SD-WAN Traffic Actually Flows: DIA, Service Chaining, and What Breaks When a Transport Dies — Part 5</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LIVrraXyCDUxAr79feKriw.png" /></figure><p><em>Following a single packet through local internet breakout, a firewall inserted into the overlay, and a mid-session transport failure the point where the policy from Part 4 stops being theory and starts leaving marks on real traffic.</em></p><p>Part 4 ended with a fabric that knew what it was supposed to do. Control policy decided which routes a site could see, data policy decided what happened to a flow once it arrived, and a single centralised policy on vSmart held it all together. The last example in that instalment was almost throwaway: one data policy that pushed guest traffic straight out a local internet circuit with the line nat use-vpn 0.</p><p>That one line hides more than it admits. The moment a packet is NATed onto a local transport, it leaves the tidy world of the overlay, where every tunnel endpoint is a stable system IP and the underlay is invisible, and enters a world where the branch owns its own internet path, its own NAT translations, and its own failure behaviour. The overlay stops protecting it.</p><p>This part follows traffic through three places where that happens:</p><ul><li>a branch breaking out to the internet on its own,</li><li>a firewall forced back into a path that would otherwise go direct,</li><li>and a transport dying with live sessions on it.</li></ul><p>These are the same three threads Part 4 promised, and they share one theme. Routing, policy, and NAT are separate features until a packet has to satisfy all three at once on the same edge router, and then the interactions, not the individual features, generate tickets.</p><h3>Direct Internet Access, Beyond the One-Liner</h3><p>Direct Internet Access exists because backhauling everything is expensive and slow. A branch that sends its Microsoft 365, Zoom, and general web traffic across the overlay to a central data centre, out through a stack of firewalls, and back again pays for that trip twice: once in circuit cost and once in latency to services that are usually closer to the branch than the data centre is. DIA breaks that traffic out locally instead. Guest wireless is the easy case, but the real driver in most enterprises is SaaS.</p><p>There are two ways to build it, and they are not interchangeable.</p><p>The first is the data-policy method from Part 4. It matches a flow, a subnet, a set of ports, and an application once SAIE is in play, and NATs only that flow out the transport VPN. It is the right tool when breakout is selective: these users, this application, everything else stays on the overlay.</p><pre>policy<br> data-policy Guest-DIA<br>  vpn-list VPN-20<br>   sequence 10<br>    match<br>     source-ip 10.20.0.0/16<br>    action accept<br>     nat use-vpn 0<br>     nat fallback<br>   default-action accept<br>apply-policy<br> site-list SPOKES<br>  data-policy Guest-DIA from-service<br></pre><p>The addition here versus Part 4 is NAT fallback. Without it, if the local internet path is unusable, the matched traffic has nowhere to go, the policy has committed it to VPN 0. With it, the edge reverts that flow to the overlay when the DIA path is unhealthy, so guest traffic degrades to backhaul rather than to a black hole.</p><p>The second method is routing-based and lives in the device configuration rather than in centralised policy. NAT is enabled on the transport interface, and a NAT route in the service VPN hands the whole VRF’s default traffic to the global (transport) table to be translated:</p><pre>interface GigabitEthernet1<br> ip nat outside<br>!<br>ip nat route vrf 20 0.0.0.0 0.0.0.0 global</pre><p>vManage’s NAT feature template generates the surrounding plumbing, the overload rule and the system-built ACL that ties inside translations to the outside interface, but those two lines are the mechanism worth recognising. ip nat outside marks the internet-facing interface, and ip nat route vrf 20 … global is what actually leaks VPN 20 into the internet path. This method suits whole-VPN breakout: an entire guest or IoT VRF that should never touch the overlay.</p><p>Which to reach for is a design decision, not a preference. Flow-level, application-selective breakout is a data policy. Blanket, per-VPN breakout is the NAT route. Mixing them on the same VPN without understanding which one wins is a good way to lose an afternoon reading show ip nat translations.</p><h4>The problem: the interface can’t see</h4><p>DIA introduces a failure mode the overlay never had. On the overlay, BFD tells the edge whether a tunnel is alive end to end. With local breakout, there is no tunnel, just a default route pointing at an ISP. The interface can be up, the physical link green, the ISP’s first hop answering, and the internet still unreachable because something has broken upstream. The router has no reason to stop forwarding. Traffic pointed at that path is simply dropped, quietly, while every interface counter looks healthy.</p><p>The fix is an endpoint tracker. It probes a real destination out on the internet at an interval, and only keeps the DIA path in service while that probe succeeds:</p><pre>endpoint-tracker DIA-TRACK<br> tracker-type interface<br> endpoint-ip 208.67.222.222<br> threshold 300<br> interval 30<br> multiplier 3<br>!<br>interface GigabitEthernet1<br> endpoint-tracker DIA-TRACK</pre><p>The tracker probes 208.67.222.222 every 30 seconds and expects a response inside 300 ms; three consecutive failures mark the path down. When it goes down, the edge withdraws the NAT default route, and traffic that was breaking out locally falls back to the overlay, the routing-based equivalent of NAT fallback. Verification is direct:</p><pre>show endpoint-tracker<br>show ip nat translations<br>show ip route vrf 20</pre><p>If the tracker shows Up and translations are incrementing, breakout is genuinely working. If the tracker reads Up but the path is still suspect, the probe target itself may be the problem: the endpoint should be stable, reachable, and not something a captive portal or an ISP will happily answer on the internet’s behalf.</p><p>Common mistake: shipping DIA without a tracker because it works in the lab, where the internet is always up. In production, the day the upstream breaks is the day a branch full of users sits there with a link that says connected, traffic that goes nowhere, and no failover fires because nothing told the router the path was dead.</p><p>Routing can’t paper over a second consequence. A branch doing DIA is a branch with its own door to the internet, no longer behind the central firewall stack. That door has to be guarded locally, an on-box zone-based firewall, a Secure Internet Gateway tunnel to a cloud security service, or both. DIA is a performance decision that is also, unavoidably, a security decision, which is the thread Part 6 picks up. But sometimes the requirement runs the other way: don&#39;t let this traffic out locally; force it through a firewall no matter what the overlay would do on its own. That is service chaining.</p><h3>Service Chaining: Putting a Firewall Back in the Path</h3><p>The hub-and-spoke control policy from Part 4 already bent spoke-to-spoke traffic through a regional hub. Service chaining is a more precise version of the same instinct: instead of steering by topology, it steers a selected set of routes through a named service, a firewall, an IDS, an IPS, that lives at one site, and it does so without hairpinning everything by hand.</p><p>The service is defined on the edge that hosts it. Say the data centre edge sits in front of a firewall whose inside interface is 10.10.10.5 in VPN 10:</p><pre>sdwan<br> service FW vrf 10<br>  ipv4 address 10.10.10.5</pre><p>That edge now advertises a service route into OMP, its own route type, carrying the VPN, a service ID (FW is service ID 1), the originator, and a TLOC. vSmart learns that a firewall exists behind that site. Nothing is steered yet; the fabric simply knows the service is there. The steering is a centralised control policy that sets the service on the routes that must be inspected here, branch traffic destined for data-centre subnets:</p><pre>policy<br> control-policy Inspect-Branch-to-DC<br>  sequence 10<br>   match route<br>    vpn-list VPN-10<br>    prefix-list DC-SUBNETS<br>   action accept<br>    set service FW vpn 10<br>  default-action accept<br>apply-policy<br> site-list SPOKES<br>  control-policy Inspect-Branch-to-DC out</pre><p>The effect is that a spoke resolving a data-centre prefix no longer sees a direct path. It sees a path whose next hop drives the packet to the service node first. The traffic rides the overlay to the hub, is handed to the firewall, comes back to the hub edge, and only then continues to its destination. Confirm the service is actually in OMP before blaming the policy:</p><pre>show sdwan omp services</pre><p>If the firewall’s service route is not listed, the set service action has nothing to resolve to, and the control policy quietly accepts the route without redirecting it the same silent no-op Part 4 warned about, in a new costume.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*i753RVzt6KVgqHw-_D78xw.png" /></figure><p>Cisco edges have probed service nodes for liveness since IOS XE SD-WAN 17.3.1a, and the hosting edge will not advertise the service route until it can resolve the firewall’s address locally. That matters for a design question that has to be answered on purpose: what should happen when the firewall dies? If the service route is withdrawn, traffic can fall back to its normal path, fail open, remain available but uninspected, or the design can keep it closed. Neither is automatically correct. A compliance boundary usually wants fail-closed; an availability-first edge service usually wants fail-open. The point is to decide, because the default is not a security policy.</p><h4>The failure that looks like a firewall bug</h4><p>The subtle killer in service chaining is asymmetry. A stateful firewall must see both directions of a flow. If forward traffic is steered through the firewall at the hub but return traffic finds a different path back to a second hub, a different service node, or a route that bypasses inspection, the firewall sees half a conversation and drops it. The symptom is maddening: connections that establish and then hang, throughput that collapses on large transfers while pings stay clean, problems that appear only for certain site pairs. None of it looks like a steering problem, and engineers burn hours on the firewall before checking the return path.</p><p>The rule that avoids it: both directions of a stateful flow must transit the same service instance. In practice, that means steering the return routes symmetrically and being deliberate about service placement when there is more than one candidate node. A firewall inserted for security becomes an availability and performance dependency the moment it is in the path, which is the honest trade service chaining asks an engineer to accept.</p><h3>When a Transport Dies Mid-Session</h3><p>Everything so far assumed the transports stayed up. They don’t. The scenario that tests all of it: a branch with MPLS and internet, a voice call running over the overlay and a large file transfer in flight, and the MPLS circuit drops, not degrades, mid-session.</p><p>BFD governs what happens next, and it runs inside every overlay tunnel between TLOC colors. Each edge sends BFD hellos across each tunnel and declares the tunnel dead after a number of missed hellos. With default template values that is one hello per second and a multiplier of six, so the tunnel is declared dead after roughly six seconds; both the interval and the multiplier are tunable, and on a large fabric the tuning is a real trade between fast failover and false positives on a naturally lossy link. When BFD declares the MPLS tunnel down, the TLOC and every path over it are withdrawn, OMP reconverges on the surviving internet transport, and traffic re-pins to it.</p><p>Here is the part that surprises people: the voice call and the file transfer usually survive. Both are overlay flows, tunnelled between stable system IPs. The application never sees the transport change. The source and destination it cares about are the overlay endpoints, not the MPLS or internet circuits underneath. Hiding the underlay is the entire reason the overlay exists, and a clean blackout failover is where that design pays off.</p><p>The trouble is that not every failure is a clean blackout, and not every flow is an overlay flow.</p><h4>Blackout versus brownout</h4><p>BFD is a liveness mechanism. It answers whether the tunnel is up, and it answers fast. It does not answer if the tunnel is any good. A circuit that stays up but starts losing 8% of packets and jittering badly is a brownout, and BFD is perfectly content with it, hellos are still getting through. What catches a brownout is Application-Aware Routing, introduced earlier in the series, and it works on a completely different clock.</p><p>AAR builds its loss, latency, and jitter figures by averaging the same BFD hellos over a statistics window. With defaults, that window is long: a ten-minute poll interval, averaged across six intervals, adds up to roughly an hour of smoothing before the numbers settle. That smoothing is deliberate, it stops AAR from yanking traffic around on a single bad second, but it means a brownout can persist for minutes before the SLA math crosses its threshold and AAR moves voice off the degraded transport. Engineers who want faster reaction shorten the poll interval, and the caution is real: shortened too far, without enough hellos left in each bucket, the loss percentage is computed from too small a sample, which produces false SLA violations and traffic that flaps between transports. Faster detection and stable detection pull against each other, and the default errs toward stable.</p><p>So the mental model for a failing transport has two paths. If BFD is down, it is a blackout, and failover is fast and mostly transparent. If BFD is up but the SLA is violated, it is a brownout, handled by AAR on the slow clock. The two produce very different user experiences from what looks, on a status page, like the same amber light.</p><h4>What survives the failover, and what doesn’t</h4><p>Overlay-tunnelled traffic survives, as above. Three things do not, and each traces back to something earlier in this part.</p><p>Locally broken-out DIA sessions break. A NATed flow’s identity is the branch’s public internet address; when that circuit fails, and the flow falls back to the overlay, its path and its translation change, so established sessions reset. New sessions rebuild on the overlay and life goes on, but the in-flight download dies. NAT fallback is failover for reachability, not for sessions.</p><p>Stateful service-chain sessions break if failover lands them on a different firewall. Re-pinning to a surviving transport is fine, re-pinning to a path that reaches a different service node is not, because that node has no state for the existing connection and drops it. This is the asymmetry problem again, arriving through the back door of a transport failure.</p><p>And a rigidly pinned data policy can stop failover happening at all. Part 4 flagged the mistake of using a data policy to hard-set a path for traffic that should follow routing; a transport failure is where that mistake collects its bill. If the policy pins a flow to the MPLS color and offers no alternative, the flow does not gracefully move to the internet transport when MPLS dies, it fails because the policy asserted a path more strongly than the routing table could override. The clean expression of prefer this, fall back naturally belongs in AAR with a backup or in TLOC preference, not in a rigid pin.</p><h4>Troubleshooting a transport failure</h4><p>The methodology is to establish which of the two failures is in play before touching anything, because the fixes diverge. It starts with liveness:</p><pre>show sdwan bfd sessions<br>show sdwan bfd history</pre><p>bfd sessions shows each tunnel’s state, colors, and uptime; bfd history shows the flaps, which is what matters when a session is up now but users report drops. If the relevant session is down, it is a blackout, chase the transport, the TLOC, and physical or ISP faults. If BFD is up, quality comes next:</p><pre>show sdwan app-route stats</pre><p>This is where a brownout shows itself: loss, latency, and jitter per tunnel against the SLA. Healthy BFD with numbers over threshold means AAR should be acting, and if traffic is still on the bad link, the question becomes whether the SLA class, the poll interval, or a competing data policy is holding it there. The control plane should be confirmed intact underneath all of it:</p><pre>show sdwan omp peers<br>show sdwan control connections</pre><p>Common mistake: treating every transport complaint as a blackout and rebooting or re-homing circuits when show sdwan app-route stats would have shown a brownout that no amount of circuit-bouncing fixes. Up is not the same as good, and the two commands that tell them apart take ten seconds to run.</p><h3>Where This Leaves Us</h3><p>The fabric gives traffic a stable overlay; policy gives it intent. This part is where both meet the messy edge of real forwarding, and the recurring lesson is a single sentence: the overlay hides the transport only for traffic that stays inside it. Break out locally with DIA, and the branch owns its own internet path, tracker, and NAT sessions. Force traffic through a service, and the firewall becomes a path dependency with an asymmetry trap and a fail-open-or-closed decision. Lose a transport and overlay flows barely notice, while the things that stepped outside the overlay NATed breakouts, stateful service chains, rigidly pinned flows pay for having done so.</p><p>Every one of those is a deliberate trade, and the engineering skill is knowing the trade is being made. nat use-vpn 0 is not a free performance win, set service FW is not a free security control, and a pinned path is not a free guarantee. Each moves a cost somewhere else, usually into the failure case, which is exactly where nobody is looking until the ticket lands.</p><p>Part 6 follows the security thread this part kept raising and never resolved: once branches break out to the internet on their own, how does the edge stay protected? That is the on-box firewall, the Secure Internet Gateway integration, TLS inspection, and the design question of where enterprise security actually lives once the traffic no longer comes home to be inspected.</p><p>If you have hit one of these in production, a DIA breakout that black-holed, a service chain that dropped half its sessions, or a failover that didn’t , drop it in the comments and I will work it into a later part.</p><p>If you found it useful, don’t forget to 👏 clap, 🔄 share, and 💬 follow for more Network Security, Linux &amp; Windows, and DevOps &amp; Cloud insights.</p><p>📲 Follow me on social media:</p><p>🔗 LinkedIn: <a href="https://www.linkedin.com/in/mmokhta">https://www.linkedin.com/in/mmokhta</a></p><p>✍️ Medium: <a href="https://medium.com/@mehrdad.mokhtari1370">https://medium.com/@mehrdad.mokhtari1370</a></p><p>♛ GitHub: <a href="https://github.com/MEMOKHTARI">https://github.com/MEMOKHTARI</a></p><p>⛪ GitLab: [github.com/MEMOKHTARI]</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d0dd102a3443" width="1" height="1" alt=""><hr><p><a href="https://systemweakness.com/how-cisco-sd-wan-traffic-actually-flows-dia-service-chaining-and-what-breaks-when-a-transport-d0dd102a3443">How Cisco SD-WAN Traffic Actually Flows: DIA, Service Chaining, and What Breaks When a Transport…</a> was originally published in <a href="https://systemweakness.com/">System Weakness</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Cisco SD-WAN Policy Actually Works: Control Policy, Data Policy, and Why Engineers Mix Them Up…]]></title>
            <link>https://blog.stackademic.com/how-cisco-sd-wan-policy-actually-works-control-policy-data-policy-and-why-engineers-mix-them-up-2c4ab7d7bc3b?source=rss-3099b15986b------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c4ab7d7bc3b</guid>
            <category><![CDATA[netwrok-security]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Mehrdad Mokhtari]]></dc:creator>
            <pubDate>Mon, 17 Aug 2026 11:05:55 GMT</pubDate>
            <atom:updated>2026-09-17T00:45:38.840Z</atom:updated>
            <content:encoded><![CDATA[<h3>How Cisco SD-WAN Policy Actually Works: Control Policy, Data Policy, and Why Engineers Mix Them Up — Part 4</h3><p>A practical walkthrough of where each policy type runs, what it can and can’t match on, the order packets are evaluated in, and the design mistakes that make a deployment behave unexpectedly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3RqnLfWwHJD17vrlwn--Kg.jpeg" /></figure><p>By the end of Part 3, the fabric was, in a sense, finished. Every device had proven its identity with a certificate, tunnels came up on their own without anyone matching phase 1 and phase 2 parameters, and traffic sat inside VPNs that stayed isolated across every site. Left alone, that fabric will happily give full any-to-any reachability inside each VPN. vSmart reflects every OMP route it learns to every edge allowed to see it, and packets follow whatever best path OMP calculated.</p><p>That default is almost never what an enterprise actually wants.</p><p>Real networks want spokes to reach each other only through a regional hub. They want guest traffic broken out to the internet locally instead of hauled back to a data centre. They want a specific application forced onto a specific transport, a firewall inserted into a path, or a partner VPN kept away from anything sensitive. None of that comes from the underlay. It comes from policy, and policy in Cisco SD-WAN is where many otherwise solid deployments start behaving strangely, usually because the engineer reached for the wrong kind.</p><p>Two policies can look almost identical in the vManage UI and produce completely different results because one never touches a packet and the other never touches a route.</p><p>A note on naming: Cisco has rebranded the platform to Catalyst SD-WAN. vManage is now SD-WAN Manager, vSmart is SD-WAN Controller, and vBond is SD-WAN Validator. The mechanics are identical. This series keeps the vManage, vSmart, and vBond names for continuity, and the CLI on IOS-XE still uses sdwan throughout.</p><h4>Two Different Questions</h4><p>Every policy in this system answers one of two questions, and keeping them separate is the whole game.</p><ul><li>Control policy answers which routes and TLOCs a site is even allowed to know about. It shapes the topology and the routing table itself.</li><li>Data policy answers, given the routes a site already has, what should happen to a particular packet. It steers, drops, marks, or breaks traffic out based on the actual flow.</li></ul><p>Control policy operates on the control plane, on OMP routes and TLOCs. It never inspects a single user packet. Data policy operates on the forwarding plane, on real traffic matched by a 6-tuple of source and destination IP, ports, protocol, and DSCP. It has no concept of OMP routes.</p><p>That difference in what each one can match on is the root of nearly every design mistake in this area. If you find yourself wanting to steer all traffic to 10.50.0.0/16 out the MPLS circuit, that is a routing decision and belongs in control policy. If you want to steer all traffic from the voice subnet, on UDP, to the internet, that is a flow decision and belongs in data policy. Ask the wrong question, and you will write a policy that matches perfectly and does nothing.</p><h4>Where Policy Lives</h4><p>A second axis trips people up, independent of control versus data: centralised versus localised.</p><p>Centralised policy is authored in vManage and activated on vSmart. Both control policy and data policy are centralised. Localised policy is a different animal: the per-device configuration pushed into a device template and executed entirely on the edge router: interface ACLs, route policy for the local BGP or OSPF edge, QoS class maps, shapers, and policers. Nothing about localised policy involves vSmart.</p><p>What confuses people is where the work physically happens. A centralised data policy is configured on vManage and activated on vSmart, but vSmart never forwards user traffic, so it packages the policy and pushes it down to the relevant edge routers over OMP, and the edges enforce it. Control policy is the opposite: it is authored centrally and executed on vSmart, because only the route reflector needs to manipulate OMP advertisements. The edge never sees the control policy itself, only the filtered set of routes that survived it.</p><ul><li><strong>Control policy</strong>: authored in vManage, executed on vSmart, operates on OMP routes and TLOCs, bound to a site-list with direction in or out.</li><li><strong>Data policy</strong>: authored in vManage, activated on vSmart but executed on the edge router, operates on the packet 6-tuple, bound to a site-list plus a VPN-list with a direction.</li><li><strong>Localised policy</strong>: authored per device, executed on the edge, handling local routes, ACLs, and QoS, with no controller involvement.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tBrm-rP9rFs5oNiT3w_EyA.png" /></figure><h4>Control Policy: Reshaping What Sites Can See</h4><p>The cleanest way to see what control policy does is to build a hub-and-spoke topology on a fabric that defaults to full mesh.</p><p>Say an organisation has one regional hub and forty branch spokes, all in VPN 10. Out of the box, vSmart advertises every spoke’s routes and TLOCs to every other spoke, so any branch can build a direct tunnel to any other branch. Many enterprises do not want that. They want inter-branch traffic to transit the hub firewall, and they do not want forty sites each holding thirty-nine tunnels.</p><p>Control policy fixes this by changing what the spokes are told. On vSmart, you write a policy that, for spoke sites, replaces the next-hop TLOC on inter-branch routes with the hub’s TLOC:</p><pre>policy<br> control-policy Spoke-to-Hub<br>  sequence 10<br>   match route<br>    vpn-list VPN-10<br>    prefix-list ALL-BRANCH<br>   action accept<br>    set tloc-list HUB-TLOCS<br>  default-action accept<br> lists<br>  site-list SPOKES<br>   site-id 100-140<br>  site-list HUB<br>   site-id 10<br>apply-policy<br> site-list SPOKES<br>  control-policy Spoke-to-Hub out</pre><p>Two things do all the work. The set tloc-list HUB-TLOCS action rewrites the next hop so spokes learn every branch prefix as if it lived behind the hub. And the direction, control-policy … out, applies the policy to routes as vSmart advertises them out to the spokes, not as it receives them.</p><p>That direction keyword is the most misunderstood part of control policy. The in direction acts on OMP updates arriving at vSmart from the edges, out acts on updates leaving vSmart towards the edges. Most topology and path-preference work is done out, because you are shaping each site’s view of the network. Reverse it, and the policy is valid, activates without error, and quietly does nothing useful.</p><p>Verification happens on the spoke, not on vSmart, and engineers often forget to look there. On a cEdge spoke:</p><pre>show sdwan omp routes vpn 10</pre><p>If the policy is working, every remote branch prefix shows the hub’s system IP and TLOC as the next hop, not the originating branch. If you still see direct branch TLOCs, the policy either didn&#39;t match, was applied in instead of out, or wasn&#39;t bound to the right site-list. Confirm the raw TLOC view with:</p><pre>show sdwan omp tlocs</pre><p>There is a safety property worth naming: because vSmart never advertises the direct spoke-to-spoke paths, those paths do not exist in the spoke’s routing table at all. This isn&#39;t traffic being permitted and then blocked; the route was filtered in the control plane, so there is nothing to block. It is the same structural-enforcement idea from Part 3, now applied to reachability instead of segmentation.</p><h4>Data Policy: Acting on the Traffic Itself</h4><p>Control policy can’t help when the decision depends on something OMP does not carry, such as a port, a DSCP value, or an application. That is where data policy comes in, and the classic example is direct internet access for a specific class of traffic.</p><p>Take the same branches, now with guest wireless in VPN 20 and a local biz-internet circuit at each site. The business does not want guest traffic tunnelled back to the data centre and hairpinned out a central firewall; it wants it dropped straight onto the local internet link. Routing alone cannot express guest subnet, internet-bound, break out locally, but a data policy can:</p><pre>policy<br> data-policy Guest-DIA<br>  vpn-list VPN-20<br>   sequence 10<br>    match<br>     source-ip 10.20.0.0/16<br>    action accept<br>     nat use-vpn 0<br>  default-action accept<br>apply-policy<br> site-list SPOKES<br>  data-policy Guest-DIA from-service</pre><p><strong><em>“nat use-vpn 0”</em></strong> tells the edge to NAT the flow and send it out the transport VPN directly rather than into an overlay tunnel. The from-service direction evaluates the policy on traffic arriving from the service, or LAN, side of the router; the other options are from-tunnel for traffic coming off the overlay and all for both.</p><p>Because this is a data policy, verification is completely different from a control policy. The policy actually rides down to the edge, so you check that the edge received it and that packets are hitting it:</p><pre>show sdwan policy from-vsmart</pre><p>That shows the centralised data and app-route policy the edge pulled from vSmart. If your policy name is not there, the edge never got it, usually a site-list binding problem, and no amount of staring at vManage will tell you that. Once it is present, the counters show whether traffic is matching:</p><pre>show sdwan policy access-list-counters</pre><p>Common mistake: engineers build a clean data policy, activate it, and test from a device that is not in the bound site-list, or test traffic that enters from-tunnel when the policy only matches from-service. The policy is fine; the scope is wrong. Confirm with show sdwan policy from-vsmart on the specific edge you are testing from before doubting the policy logic.</p><h4>When Data Policy and App-Aware Routing Both Apply</h4><p>Earlier in the series, we covered the application-performance side of SD-WAN: SLA-driven path selection that moves voice off a transport once loss or latency crosses a threshold. Application-Aware Routing is technically its own flavour of centralised data policy, an app-route-policy, and on a real edge it can be active at the same time as a plain data policy. When both match the same flow, evaluation order matters, and it is a genuine source of &quot;why is this traffic on the wrong link&quot; tickets.</p><p>On the edge, traffic is processed in this sequence:</p><ol><li>Local ingress ACL (localised policy on the LAN interface)</li><li>Application-Aware Routing policy (SLA-based path selection)</li><li>Centralised data policy</li></ol><p>AAR gets first crack at path selection based on SLA, and what happens next depends on tunnel state, which has changed across releases. On modern code (20.6.1 and later), if the AAR preferred colors meet SLA and overlap with the data policy’s colors, the router uses the common colors. If the AAR colors fail SLA but the data policy colors are healthy, it uses the data policy colors. If no transport meets SLA at all, the data policy colors win outright. In older code, the data policy colors took precedence more aggressively.</p><p>The practical lesson: if an AAR policy steers voice by SLA and a separate data policy pins the same subnet to a transport, the result is often the intersection of the two rather than whichever you wrote last. Two correct policies can combine into a path you did not intend. When a flow lands somewhere unexpected, check both, and check the code version, because the tie-break logic differs across releases.</p><h4>The Mistake That Generates the Most Tickets</h4><p>Step back from any single feature and the recurring failure is the same: using the wrong policy type for the intent.</p><p>An engineer wants all traffic to a particular data-centre subnet to prefer MPLS. They reach for a data policy matching that destination and setting a preferred color. It works until MPLS goes down and they troubleshoot why traffic isn&#39;t failing over, because a data policy that hard-sets a path can override the routing table’s ability to reroute. The cleaner expression of prefer this path but fall back naturally usually belongs in control policy, via TLOC preference, or in AAR with a backup, not in a rigid data-policy path pin.</p><p>The mirror-image mistake is trying to do flow-level steering with control policy. Someone wants to send a specific application out the internet circuit and writes a control policy, but control policy cannot match an application or a port. It only sees prefixes and TLOCs. The policy commits cleanly and changes nothing, because the application was never visible to it.</p><p>A quick gut-check before writing anything:</p><ul><li>Prefixes, TLOCs, which sites learn which routes, topology → control policy.</li><li>Applications, ports, DSCP, source or destination flows, DIA, service chaining → data policy.</li><li>SLA (loss, latency, jitter) driving path choice → app-aware routing.</li><li>Local ACLs, edge routing, QoS queues and shaping → localised policy.</li></ul><p>Most ‘SD-WAN is behaving unexpectedly’ incidents go away the moment you classify the intent correctly and confirm you used the matching tool.</p><h4>One Active Policy, and Why Edits Feel Heavy</h4><p>A last operational reality that surprises people: only one centralised policy can be active on vSmart at a time. That single policy is a container. It holds all your control-policy and data-policy definitions and the apply-policy blocks that bind each one to site-lists. You do not activate policies individually; you activate the whole bundle.</p><p>So changing one small thing means editing and re-pushing the active centralised policy, and vManage recomputes and redistributes the affected pieces. On a large fabric, this is not free, which is why mature teams treat the centralised policy as a change-controlled artefact rather than something they poke at live during business hours. It also means a careless edit to one sequence can ripple further than expected, because everything shares one document.</p><p>Localised policy has no such constraint; it lives in device templates and is scoped per device, which is one more reason to push edge-specific behaviour, ACLs and QoS, down into localised policy rather than bloating the centralised policy with things that do not need to be central.</p><h4>Where This Leaves Us</h4><p>The fabric from Part 3 gives you identity, encryption, and segmentation for free. Policy is what turns that neutral fabric into a network that matches a business’s intent, and the whole discipline comes down to knowing which of two questions you are answering. Control policy shapes the routes a site can see and runs on vSmart. Data policy acts on real packets and runs on the edge. Everything else, direction keywords, site-list scoping, the AAR interaction, the single-active-policy rule—hangs off that one distinction.</p><p>Part 5 moves from how policy is structured to how traffic actually flows once policy, routing, and NAT interact: direct internet access designs, service chaining a firewall into the path, and the failure modes that show up when a branch loses a transport mid-session. That is where the theory in this instalment starts leaving marks on real traffic.</p><p>If there is a specific policy scenario you have hit in production, a hub-and-spoke gone wrong, a DIA breakout, or an AAR-versus-data-policy conflict, drop it in the comments, and I will work it into a later part.</p><p>If you found it useful, don’t forget to <strong>👏 clap, 🔄 share, and 💬 follow</strong> for more Network Security, Linux &amp; Windows, and DevOps &amp; Cloud insights.</p><p>📲 Follow me on social media:<br>🔗 LinkedIn: <a href="https://www.linkedin.com/in/bornaly/">https://www.linkedin.com/</a><a href="https://www.linkedin.com/in/mmokhta">in/mmokhta</a><br>✍️ Medium: <a href="https://medium.com/@mehrdad.mokhtari1370">https://medium.com/@mehrdad.mokhtari1370</a><br>♛ GitHub: <a href="https://github.com/MEMOKHTARI">https://github.com/MEMOKHTARI</a><br>⛪ GitLab: <a href="https://gitlab.com/Mehrdad261">[github.com/MEMOKHTARI</a>]</p><h3>Before you go</h3><ul><li>Please take a moment to like the post and follow the writer!</li><li>Did you know that over 400,000 developers share what they’re building, learning, and discovering across our platforms every month? Learn how you can contribute <a href="https://plainenglish.io/write-for-us"><strong>here</strong></a>.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c4ab7d7bc3b" width="1" height="1" alt=""><hr><p><a href="https://blog.stackademic.com/how-cisco-sd-wan-policy-actually-works-control-policy-data-policy-and-why-engineers-mix-them-up-2c4ab7d7bc3b">How Cisco SD-WAN Policy Actually Works: Control Policy, Data Policy, and Why Engineers Mix Them Up…</a> was originally published in <a href="https://blog.stackademic.com/">Stackademic</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Why Your SD-WAN Fabric Won’t Come Up: A Root CA Setup Guide for Windows Server 2019]]></title>
            <link>https://systemweakness.com/why-your-sd-wan-fabric-wont-come-up-a-root-ca-setup-guide-for-windows-server-2019-fef62ed3e83d?source=rss-3099b15986b------2</link>
            <guid isPermaLink="false">https://medium.com/p/fef62ed3e83d</guid>
            <category><![CDATA[network-security]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[security]]></category>
            <dc:creator><![CDATA[Mehrdad Mokhtari]]></dc:creator>
            <pubDate>Mon, 10 Aug 2026 08:36:14 GMT</pubDate>
            <atom:updated>2026-08-11T11:17:22.509Z</atom:updated>
            <content:encoded><![CDATA[<p>A practical guide to building an internal certificate authority and using it to establish trust between Cisco SD-WAN control components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ti2amPRPNn6l80YklWl3TA.png" /></figure><p>It was 7 PM on a Friday, and the SD-WAN rollout was still stuck. The routing policies looked correct, and every WAN link was up, but the controllers would not establish control connections. After working through the logs, I found the problem: the certificate chain was not trusted. It is an easy part of the deployment to overlook. The network can be reachable and correctly configured, but the fabric will not come up until every controller can validate the certificates presented by the others. This guide follows the Windows Server 2019 Root CA setup I used to establish that trust.</p><p>The first step is installing Active Directory Certificate Services. Log in to the Windows server, go to Server Manager, select Add Roles and Features, and then click Install. In the Select role service, choose the first option and then press Next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/777/1*Su1Nfl5Pqc-QFiGtyn2Zlw.png" /></figure><p>On the role services screen, I selected two options:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/783/1*CjD4Mfm23iS931nSexUjhw.png" /></figure><p>☑ <strong>Certification Authority</strong>: is the actual CA service. It lets the Windows Server issue, sign, revoke, and manage certificates.<br>☑<strong> Certification Authority Web Enrollment</strong>: installs the classic IIS-based browser interface. After installation and configuration, you normally access it using:</p><pre>http://&lt;CA-server&gt;/certsrv</pre><p>Click Next, confirm all the settings, and press the Install button. The roles have been installed, but your CA has <strong>not yet been configured</strong>. This post-deployment configuration is mandatory before CertSvc can operate properly. So continue into <strong>Configure Active Directory Certificate Services.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/774/1*3GxpcSkjyvVcvQ_Bh0pTzQ.png" /></figure><p>Certification Authority Web Enrollment depends on the CA being configured first. I’ll configure the CA now and set up Web Enrollment afterwards. Select “Certificate Authority” and click Next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/747/1*_3xxtL578CJLDMchZtd7pw.png" /></figure><p>The next window offers two types of setup:</p><ul><li><strong>Enterprise CA:</strong> use for domain-joined environments needing AD integration, templates, and auto-enrollment.</li><li><strong>Standalone CA:</strong> use for independent/offline Root CAs or manual certificate issuance without AD dependency.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*Vce0NSU32FViLSgAq0dIKg.png" /></figure><p>Next come the cryptography settings.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*C_ylJrT6xD3sM-Ge501fzg.png" /></figure><p>In the next window, enter a name for the certificate and set its validity period, then press Configure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*0bWDYkVIaQBjqtFTfYam9A.png" /></figure><p>With the CA configured, the next step is installing the Root CA certificate into vManage and the other controllers. Open a browser and browse to the Root CA server’s IP address.</p><pre>http://&lt;server-IP&gt;/certsrv</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cqJZ2WxNl8ma-wFq2WF3CQ.png" /><figcaption>http://&lt;server-IP&gt;/certsrv</figcaption></figure><p>Once the Root CA is configured, <strong>download its Base64-encoded CA certificate, which will serve as the trust anchor for validating certificates issued within the SD-WAN environment.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/953/1*1YNWA9Q9vgv-1cnAa3RbVA.png" /></figure><p>With Active Directory Certificate Services configured and the public Root CA certificate exported, it’s time to move to SD-WAN Manager and configure trust for the control components.</p><p>Before working with the controller certificates, make sure the licence has been assigned to vManage. Log in and open <strong>Administration </strong>&gt; <strong>Settings</strong>. Check that the organisation name (It must be identical in both places), vBond address, and the other system values match the settings entered through the CLI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mdaVr4hpEC3lSipl7OCaRg.png" /></figure><p>Controller Certificate Authentication determines which certificate authority is trusted for the control-component certificates. vManage offers four options for this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9NUjDuZZtkRer1s3Lhh0Tg.png" /></figure><ul><li><strong>Cisco (Recommended) </strong>— Manager submits each CSR to Cisco PKI and installs the returned certificate automatically. The control components must be associated with the correct Smart Account and Virtual Account, and Manager must be able to reach the required Cisco services. This option also automates renewal.</li><li><strong>DigiCert </strong>— This option remains available for older deployments that already use the DigiCert certificate chain. It is generally retained for compatibility rather than selected for a new fabric.</li><li><strong>Manual</strong> — Manager generates a CSR for each component, but the remaining steps are manual. Download the request, have it signed through the Cisco PKI process, and upload the returned certificate. This is useful when Manager cannot reach the Cisco services directly, but renewal must also be handled manually.</li><li><strong>Enterprise Root Certificate</strong> — This option uses our organisation’s certificate authority. Import the public Root CA certificate into Manager, generate a CSR for each control component, have the enterprise CA sign each request, and then install the returned certificates.</li></ul><p>For this deployment, I select Enterprise Root Certificate and then click Proceed. Select File, upload the public Root CA certificate exported from the Windows CA, and click Save and Import. Do not upload the CA private key.</p><p>Next, open Configuration &gt; Certificates and select Controllers. On newer releases, this tab is called Control Components. Find the required component, open the three-dot menu, and select Generate CSR. Copy the request or download it as a file, then keep it for the signing step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E0iIeqwu7pY83Q77bOJsGg.png" /></figure><p>The uploaded Root CA certificate is the trust anchor; SD-WAN Manager does not use it to sign certificates. Manager creates the CSR, the Windows CA signs the request, and the signed certificate is then installed through Manager.</p><p>On the certificate enrolment page, submit an advanced certificate request. In the next window, select Advanced Certificate Request,</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/841/1*ZZduMAHgXpJJ2NmhC20XKw.png" /></figure><p>then choose the option to submit a Base64-encoded CMC or PKCS #10 request. Open the CSR downloaded from SD-WAN Manager in a text editor. Copy everything from BEGIN CERTIFICATE REQUEST through END CERTIFICATE REQUEST and paste it into the Saved Request field. This process applies to SD-WAN Manager, the Controller, and the Validator.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1017/1*1gPNPmQk2Jfc6BPKDPrJkg.png" /></figure><p>Return to the Windows CA server, open the Certification Authority, and issue all the pending requests.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*zsPfmdwJDNTJ33MKGRlIIg.png" /></figure><p>After issuing the certificate, return to the certificate enrolment page and click View the status of a pending certificate request, then download and save it to your system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qKcFFMrjW_kKkoMmn2E5HA.png" /></figure><h4>Install the signed certificate</h4><p>Return to SD-WAN Manager and open Configuration &gt; Certificates. Depending on the software release, this page may display Controllers or Control Components.</p><p>Select Install Certificate, upload the file issued by the Windows CA, and confirm the installation. Manager uses the certificate details to match it to the component that created the CSR.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PK7pVQCjpsI0ugXbDXvd7A.png" /></figure><p>Repeat the same process for each remaining control component:</p><p>1. Generate the CSR in SD-WAN Manager.<br>2. Submit the CSR to the Windows CA.<br>3. Download the signed certificate in Base 64 format.<br>4. Install the signed certificate through SD-WAN Manager.</p><p>Generate a separate CSR for every component. Each request is tied to the private key held by that component, so one signed certificate cannot be reused across the fabric.</p><h4>Verifying the certificate status</h4><p>After installing the signed certificates, return to the certificate page and check every control component. Each certificate should show as installed and valid, with the expected start and expiry dates.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VFt6cKj0vp8qevf-06erAQ.png" /></figure><p>Compare the organisation name in the certificates with the value configured in the SD-WAN system settings. Even a difference in capitalisation or spacing can stop a control connection while basic IP reachability continues to work.</p><p>Check that every component trusts the same CA chain. When an issuing CA is involved, include its intermediate certificate with the public Root CA certificate. The CA private key must remain on the CA and should never be exported for this procedure.</p><p>Check the control connections from the CLI or the monitoring page. Manager, Controller, and Validator should now authenticate and connect. If one component remains down, check the certificate dates, organisation name, system clock, DNS, and network reachability before requesting a replacement certificate.</p><p>Operational considerations</p><p>The simple Root CA design shown here is intended for a lab or controlled test. In production, keep the Root CA protected — often offline — and use an issuing CA for routine certificate requests. This limits exposure of the Root CA private key.</p><p>Before handing the deployment over, record the certificate expiry dates and assign responsibility for renewal. Test the renewal procedure while the environment is still accessible. If these certificates expire, an otherwise healthy fabric can lose its control connections.</p><p>That Friday night outage came down to a single missing link in the chain of trust. With SD-WAN Manager trusting the enterprise CA and every control component holding its own signed certificate, the fabric can finally authenticate itself — and stay up.</p><p>If you found it useful, don’t forget to <strong>👏 clap, 🔄 share, and 💬 follow</strong> for more Network Security, Linux &amp; Windows, and DevOps &amp; Cloud insights.</p><p>📲 Follow me on social media:<br>🔗 LinkedIn: <a href="https://www.linkedin.com/in/bornaly/">https://www.linkedin.com/</a><a href="https://www.linkedin.com/in/mmokhta">in/mmokhta</a><br>✍️ Medium: <a href="https://medium.com/@mehrdad.mokhtari1370">https://medium.com/@mehrdad.mokhtari1370</a><br>♛ GitHub: <a href="https://github.com/MEMOKHTARI">https://github.com/MEMOKHTARI</a><br>⛪ GitLab: <a href="https://gitlab.com/Mehrdad261">https://github.com/MEMOKHTARI</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fef62ed3e83d" width="1" height="1" alt=""><hr><p><a href="https://systemweakness.com/why-your-sd-wan-fabric-wont-come-up-a-root-ca-setup-guide-for-windows-server-2019-fef62ed3e83d">Why Your SD-WAN Fabric Won’t Come Up: A Root CA Setup Guide for Windows Server 2019</a> was originally published in <a href="https://systemweakness.com/">System Weakness</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How Cisco SD-WAN Actually Secures the Network: Tunnels, Certificates, and Segmentation — Part 3]]></title>
            <link>https://systemweakness.com/how-cisco-sd-wan-actually-secures-the-network-tunnels-certificates-and-segmentation-part-3-91fc4d7b3d54?source=rss-3099b15986b------2</link>
            <guid isPermaLink="false">https://medium.com/p/91fc4d7b3d54</guid>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[network-security]]></category>
            <category><![CDATA[sd-wan]]></category>
            <category><![CDATA[cisco]]></category>
            <category><![CDATA[cybersecurity]]></category>
            <dc:creator><![CDATA[Mehrdad Mokhtari]]></dc:creator>
            <pubDate>Mon, 03 Aug 2026 13:48:25 GMT</pubDate>
            <atom:updated>2026-08-04T14:57:37.799Z</atom:updated>
            <content:encoded><![CDATA[<p>Engineers running into Cisco SD-WAN security for the first time tend to relax once they see it’s a coherent system instead of a pile of buzzwords. Terms like “encrypted overlay” and “zero trust” show up in nearly every vendor deck, but the underlying mechanics rarely get walked through in detail. How a router actually earns its place on the fabric, and what happens the moment two sites start exchanging traffic.</p><p>This part covers that underlying model. Earlier parts in the series addressed why SD-WAN exists, how the control, data, and management planes are structured, and how application performance is protected. This instalment opens up the security layer beneath all of it.</p><h4>Nothing Talks Until It Proves Who It Is</h4><p>Before any user data crosses the fabric, every device has to prove its identity, and it does that with certificates instead of credentials. That process starts back at manufacturing, long before a router ever reaches a customer site.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dMRSSp7S__GA67Bqwwn5cw.png" /></figure><p>Each vEdge or cEdge device carries a unique identity certificate that ties it back to a broader public key infrastructure. Hardware vEdge platforms typically get a chassis-based certificate installed at the factory and tied to the device’s serial number. cEdge routers running IOS-XE SD-WAN, along with the virtual and cloud-hosted editions, get enrolled afterwards instead, either through Cisco’s own PKI or through an enterprise root certificate authority for organisations that need their own trust anchor. In that second case, vManage generates a certificate signing request, an administrator gets it signed externally, and the signed certificate is loaded back onto the controllers. Before any router is allowed onto the fabric, its serial number and chassis identifier also need to show up on vManage’s authorised device list, pulled either from Cisco’s Plug and Play portal or uploaded by hand. Hence, the certificate on its own isn’t enough without a matching record on the controller side. When a router boots for the first time and reaches out to vBond, the orchestrator handling initial authentication and NAT traversal, it presents that certificate for validation against the trusted root. This check goes both ways: the router also confirms it’s actually talking to a legitimate vSmart controller and vManage instance, not something sitting in between pretending to be one, since vSmart and vManage carry certificates signed by that same trust chain.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ncV8SHcmH1GCoZFrK1oLzQ.png" /></figure><p>In practice, that means a stolen router can’t just be plugged into any internet connection and join the fabric. Without a certificate signed by a trusted certificate authority, and without a matching serial number record on vManage, the device never establishes control-plane sessions. It never gets OMP routes, TLOC information, or encryption keys. Compare that to a traditional branch router, where having the device in hand plus a configuration file was often all it took to get onto the network.</p><h4>Tunnel Formation: IPsec Without Manual Negotiation</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UqgJyMqWPQwVScQq5o2c1A.png" /></figure><p>Anyone who’s manually configured site-to-site IPsec knows the drill: matching phase 1 and phase 2 parameters across two devices, then troubleshooting tunnels that refuse to come up because a lifetime value or proposal set doesn’t match on one side. SD-WAN gets rid of nearly all of that manual negotiation, but it’s still worth knowing what’s happening under the hood instead of treating it as a black box.</p><p>Once a vEdge or cEdge router establishes an OMP session with vSmart, it starts receiving TLOC routes that describe how to reach other sites. OMP, the overlay management protocol, acts as the control-plane routing protocol for the fabric, and if you already know BGP, the concept will feel familiar. It runs as a permanent session between each edge router and vSmart, advertises routes with attributes used for best-path selection, and lets vSmart work as a route reflector that can apply centralised policy over what gets advertised to whom. OMP carries three categories of routes:</p><ul><li><strong><em>OMP routes</em></strong>, representing service-side prefixes reachable from a given site.</li><li><strong><em>TLOC routes</em></strong>, describing the transport connections themselves, and</li><li><strong><em>Service routes</em></strong>, advertising the location of network services such as firewalls or load balancers reachable through the fabric.</li></ul><p>A TLOC, short for transport locator, is defined by three things: the system IP, a fixed per-device identifier that stays the same no matter which physical interface or address is in use; the color, a tag describing what kind of circuit is underneath, such as MPLS, biz-internet, public-internet, LTE, or one of several private colors; and the encapsulation type, either IPsec or GRE. Color ends up doing a lot more work than a simple label. It shapes path preference in policy, determines whether a TLOC counts as running over a public or private underlay, and even controls whether tunnels can form between different colors in the first place. Colors marked restricted will only build tunnels to a matching color at other sites, while colors marked no-restrict can form tunnels across mismatched colors, which is exactly what you need in hybrid designs mixing MPLS with broadband or cellular circuits at different sites.</p><p>Alongside TLOC routes, vSmart also hands out the encryption keys needed to bring up each tunnel. These get rotated periodically instead of being negotiated per tunnel through a Diffie-Hellman exchange the way classic IKE works. Deployments with FIPS compliance requirements can opt into pairwise-key or IKE-based negotiation instead, but the default model relies on this centralised key distribution.</p><h4>Segmentation: Keeping Traffic Structurally Isolated</h4><p>Security in this model isn’t just about encrypting traffic between sites, it also governs whether traffic can reach a given destination at all. That’s handled through VPNs in the SD-WAN-specific sense of the word, which have nothing to do with consumer VPN services.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5NbnpUILefVf_7RXzjkdLg.png" /></figure><p>In Cisco SD-WAN, a VPN corresponds to a VRF, a separate routing and forwarding table.</p><ul><li><strong><em>VPN 0</em></strong> is reserved for transport-side connectivity across the WAN interfaces themselves</li><li><strong><em>VPN 512</em></strong> is reserved for out-of-band management.</li><li><strong><em>VPN 1–511 and 513–65K</em></strong> are available for business traffic, organised according to whatever numbering scheme an organisation chooses.</li></ul><p>This structure lets a single physical fabric carry fully isolated traffic for different parts of an organisation. Finance systems can sit in one VPN, guest wireless in another, and operational technology or SCADA equipment in a third, with no reachability between them unless it’s explicitly permitted through policy. This isolation happens at the routing-table level, not through access lists bolted on afterwards, and it’s also enforced at the control-plane level: centralised policies on vSmart can restrict which TLOC and VPN routes get advertised to which sites in the first place, so unauthorised paths often never show up in the routing table at all instead of being blocked downstream.</p><p>Segmentation also travels with the traffic across the entire fabric. A device in VPN 10 at one site stays in VPN 10 at every other site automatically, with no need to manually configure matching access lists at each location the way you would with legacy hardware.</p><h4>Why This Matters in Practice</h4><p>You won’t see any of this in day-to-day operation, and it rarely shows up in a product demo either. But if you’ve ever managed encryption keys across dozens of routers by hand, or spent an afternoon tracing why a segmentation policy worked at one site and not another, this is exactly where the difference comes from.</p><p>Strip it back, and the model comes down to three things: identity gets verified before any device joins the fabric, encryption gets applied automatically without touching every device by hand, and segmentation gets enforced structurally instead of through lists someone has to maintain manually. Put side by side with a traditional branch router deployment, it’s a genuinely different security posture.</p><p>Part 4 gets into how policy is actually built and evaluated in this system, the distinction between control policies and data policies, and why mixing the two up is one of the most common reasons an SD-WAN deployment starts behaving unexpectedly.</p><p>If there’s a specific piece of SD-WAN security you’d like covered in more depth, let me know in the comments.</p><p>If you found it useful, don’t forget to <strong>👏 clap, 🔄 share, and 💬 follow</strong> for more Network Security, Linux &amp; Windows, and DevOps &amp; Cloud insights.</p><p>📲 Follow me on social media:<br>🔗 LinkedIn: <a href="https://www.linkedin.com/in/bornaly/">https://www.linkedin.com/</a><a href="https://www.linkedin.com/in/mmokhta">in/mmokhta</a><br>✍️ Medium: <a href="https://medium.com/@mehrdad.mokhtari1370">https://medium.com/@mehrdad.mokhtari1370</a><br>♛ GitHub: <a href="https://github.com/MEMOKHTARI">https://github.com/MEMOKHTARI</a><br>⛪ GitLab: <a href="https://gitlab.com/Mehrdad261">https://github.com/MEMOKHTARI</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=91fc4d7b3d54" width="1" height="1" alt=""><hr><p><a href="https://systemweakness.com/how-cisco-sd-wan-actually-secures-the-network-tunnels-certificates-and-segmentation-part-3-91fc4d7b3d54">How Cisco SD-WAN Actually Secures the Network: Tunnels, Certificates, and Segmentation — Part 3</a> was originally published in <a href="https://systemweakness.com/">System Weakness</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Route Reflectors: The One BGP Command Everyone Runs, and Almost Nobody Fully Understands]]></title>
            <link>https://systemweakness.com/route-reflectors-the-one-bgp-command-everyone-runs-and-almost-nobody-fully-understands-28299f1234b1?source=rss-3099b15986b------2</link>
            <guid isPermaLink="false">https://medium.com/p/28299f1234b1</guid>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <dc:creator><![CDATA[Mehrdad Mokhtari]]></dc:creator>
            <pubDate>Mon, 27 Jul 2026 10:21:34 GMT</pubDate>
            <atom:updated>2026-07-28T11:42:58.292Z</atom:updated>
            <content:encoded><![CDATA[<p>Every iBGP deployment eventually hits the same wall: full-mesh iBGP requires every router to peer with every other router in the AS, and that count grows as N(N-1)/2. At ten routers, that’s manageable. At a hundred, it’s a configuration and control-plane nightmare that no one actually runs in production. Route reflection is the fix, and it’s one of the most deployed BGP features in existence, but it’s also one of the most casually misunderstood. Most people know the one command that turns a peer into a client. Fewer can explain why ORIGINATOR_ID and CLUSTER_LIST exist, or what actually breaks when a cluster boundary is drawn in the wrong place.</p><h4>The Problem Route Reflectors Solve</h4><p>In a normal iBGP mesh, split-horizon rules exist to prevent loops: a route learned from one iBGP peer is never re-advertised to another iBGP peer. That rule is safe and simple, but it only works if every router is directly peered with every other router, because otherwise a route learned by one router has no path to reach a router it isn’t peered with. Full mesh guarantees that path exists. It just doesn’t scale, because every new router means a new session on every existing router, and every session is its own TCP state machine, its own keepalive traffic, and its own policy surface to maintain.</p><p>A route reflector breaks that rule on purpose, in a controlled way. One or more routers are designated reflectors. Other routers become their clients. Clients only need to peer with the reflector, not with each other, and the reflector takes on the job of re-advertising routes between clients that would otherwise never see each other’s updates. The number of sessions drops from a full mesh to a hub-and-spoke, and the AS can scale to hundreds of routers without every router carrying hundreds of sessions.</p><h4>How Reflection Actually Works</h4><p>The reflection logic follows three simple rules, and almost every operational surprise with route reflectors traces back to one of them.</p><p>A route learned from a client is reflected to all other clients and to all non-client iBGP peers. The reflector takes whatever a client sends it and forwards it to everyone else that needs to see it — that’s the whole point of the feature.</p><p>A route learned from a non-client iBGP peer is reflected only to clients, never to other non-client peers. Non-clients are assumed to already be part of a full mesh with each other, so re-advertising between them would be redundant at best and a loop risk at worst.</p><p>A route learned from an eBGP peer is advertised to everyone, clients and non-clients alike, exactly as it would be without reflection. Reflection only changes iBGP-to-iBGP behaviour; it never touches how eBGP-learned routes propagate inside the AS.</p><p>Notice what’s conspicuously absent here: NEXT_HOP is not rewritten by the reflector. Reflection relays the update at the control-plane level; it doesn’t re-originate the route the way a full iBGP speaker would. A client receiving a reflected route still needs to resolve the original NEXT_HOP through the IGP, exactly as covered in the NEXT_HOP recursion discussion in earlier parts of this series. Adding a route reflector to a topology does not remove the requirement for next-hop-self or IGP reachability to the original advertiser; it only removes the full-mesh session requirement.</p><h4>ORIGINATOR_ID and CLUSTER_LIST: Preventing Loops</h4><p>Removing the split-horizon guarantee creates a real loop risk, so route reflection adds two attributes specifically to close it.</p><p>ORIGINATOR_ID records the router ID of the router that originally injected the route into the AS. A reflector adds this attribute when it reflects a route to iBGP peers. If a router ever receives a route back with its own router ID in ORIGINATOR_ID, it discards it immediately, no matter what else looks valid about the path. It’s a last-resort loop breaker, and it works even if there’s an accidental cycle somewhere in the topology.</p><p>CLUSTER_LIST is the reflector-side equivalent of AS_PATH. Every reflector belongs to a cluster, identified by a cluster ID, and when a reflector reflects a route, it prepends its own cluster ID to the CLUSTER_LIST. If a reflector ever sees its own cluster ID already present in a route’s CLUSTER_LIST, it drops the route. That’s what stops loops in hierarchical designs — without it, a route could circulate endlessly between a cluster and a higher-tier reflector.</p><p>Both attributes are visible in show bgp ipv4 unicast &lt;prefix&gt; output on IOS-XE, and they’re the first place to look when a prefix is unexpectedly missing from a router that should have received it. If a route disappears between one client and another, check whether ORIGINATOR_ID or CLUSTER_LIST is causing a legitimate drop before assuming it’s a policy or filtering problem.</p><h4>Designing for Redundancy: Clusters and Hierarchies</h4><p>A single route reflector is a single point of failure for every client attached to it, which makes redundant reflectors the default production design, not an optional extra. The standard pattern is two reflectors per cluster, both configured with the same cluster ID, each client peering with both. Because they share a cluster ID, they’re treated as one logical reflector for loop-prevention purposes, and a route reflected by one won’t be rejected by the other on cluster-ID grounds.</p><p>It’s worth being deliberate about what the two reflectors in a cluster are to each other. They should also peer with one another directly, as normal iBGP (non-client) peers, so that a route received by only one of them still reaches the other and gets reflected to every client. Skip that peering, and you end up with a scenario where half your clients see a path the other half doesn’t, purely because of which reflector happened to receive it first.</p><p>Larger networks extend this into a hierarchy: edge clusters of reflectors and clients, feeding into a core tier of reflectors that treat the edge reflectors as their clients. CLUSTER_LIST is what makes this safe. Each tier adds its own cluster ID to the list, so a route climbing up through the hierarchy and back down cannot loop back into a cluster it already passed through. Cluster boundaries should map to something physical and stable, a point of presence or a data centre, because that’s what keeps the hierarchy predictable when you’re trying to track down a propagation delay under time pressure.</p><h4>Configuration Walkthrough</h4><p>On IOS-XE, turning a normal iBGP peer into a reflector client is a single line added to the reflector’s configuration. Nothing changes on the client side; from the client’s perspective, it’s just an ordinary iBGP session.</p><pre>router bgp 65001<br>bgp cluster-id 1.1.1.1<br>neighbour 10.0.0.2 remote-as 65001<br>neighbor 10.0.0.2 update-source Loopback0<br>neighbor 10.0.0.2 route-reflector-client<br>neighbour 10.0.0.3 remote-as 65001<br>neighbor 10.0.0.3 update-source Loopback0<br>neighbor 10.0.0.3 route-reflector-client</pre><p>The explicit cluster ID is optional if a reflector has only one client-facing cluster; IOS-XE defaults it to the reflector’s router ID. It’s worth setting it explicitly anyway, since that’s what you’ll need to match across a redundant pair — relying on a default that happens to match the router ID tends to cause problems later, usually the first time someone renumbers a loopback.</p><p>To verify a client relationship and see the attributes in action:</p><pre>show bgp ipv4 unicast neighbors 10.0.0.2<br>! Confirms Route-Reflector Client under the neighbor capabilities section<br>show bgp ipv4 unicast &lt;prefix&gt;<br>! Look for Originator and Cluster list fields on reflected paths</pre><h4>Common Pitfalls</h4><p>A route reflector is not automatically part of the best-path decision for its own forwarding. If a reflector doesn’t need the route itself, it will still faithfully reflect it, but that doesn’t mean the reflector installs it in its own FIB unless the reflector is also a legitimate transit point. Don’t assume a reflector’s own routing table tells you what its clients are seeing.</p><p>Route reflection only changes how a route propagates; it has no effect on policy. LOCAL_PREF, MED, and AS_PATH still apply exactly as described in the best-path algorithm; the reflector doesn’t strip or ignore them. If a client is picking an unexpected best path, look at the standard 11-step algorithm first. The reflector is very rarely the actual cause of a bad path selection, even though it’s often the first thing people suspect because it’s the newest piece in the topology.</p><p>Mixing client and non-client roles for the same peer on different address families is legal but easy to lose track of. A peer can be a route-reflector-client for IPv4 unicast and a regular non-client peer for VPNv4, and if that’s not intentional and documented, it produces exactly the kind of asymmetric visibility that takes hours to diagnose.</p><h4>Mini Lab: Build and Break a Route Reflector Topology</h4><p>This lab is designed to run in any topology tool that supports Cisco IOS-XE or IOS images (GNS3, EVE-NG, or CML all work). Five routers, thirty minutes, and you’ll have watched reflection, ORIGINATOR_ID, and CLUSTER_LIST do their jobs firsthand, then broken each of them on purpose.</p><p>Topology: R1 and R2 are iBGP clients. RR1 and RR2 are a redundant reflector pair sharing cluster ID 1.1.1.1, each peered with both R1 and R2 and with each other. R5 is an eBGP edge router peering with RR1 only, injecting an external prefix into the AS. All routers are in AS 65001 except R5, which is in AS 65010. Use loopback addresses for all iBGP sessions and a simple underlay IGP (OSPF area 0 is fine) to make every loopback reachable before touching BGP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Gj36A078JE5KIlIJOOUUXA.png" /><figcaption>Figure: the redundant route-reflector topology used throughout this lab — R1 and R2 are clients of both RR1 and RR2, RR1 and RR2 share cluster-id 1.1.1.1 and peer with each other, and R5 injects an eBGP-learned prefix via RR1.</figcaption></figure><p>Addressing used below:</p><pre>R1 = 10.0.0.1, <br><br>R2 = 10.0.0.2, <br><br>RR1 = 10.0.0.3, <br><br>RR2 = 10.0.0.4, <br><br>R5 = 10.0.0.5,   all /32 loopbacks reachable via OSPF.</pre><p>Step 1: Configure the clients. On R1 and R2, bring up iBGP sessions to both RR1 and RR2 using next-hop-independent loopback peering. Nothing reflector-specific happens here; both sides just look like ordinary iBGP.</p><pre>router bgp 65001<br>bgp router-id 10.0.0.1<br>neighbor 10.0.0.3 remote-as 65001<br>neighbor 10.0.0.3 update-source Loopback0<br>neighbor 10.0.0.4 remote-as 65001<br>neighbor 10.0.0.4 update-source Loopback0<br>network 192.168.1.0 mask 255.255.255.0<br><br>! Repeat the same block on R2 with its own router-id and a network statement<br>  for 192.168.2.0/24</pre><p>Step 2: Configure the reflectors. On RR1, peer with both clients as a route-reflector-client, and peer with RR2 as a normal non-client iBGP session so the pair stays synchronised.</p><pre>router bgp 65001<br>bgp router-id 10.0.0.3<br>bgp cluster-id 1.1.1.1<br>neighbor 10.0.0.1 remote-as 65001<br>neighbor 10.0.0.1 update-source Loopback0<br>neighbor 10.0.0.1 route-reflector-client<br>neighbor 10.0.0.2 remote-as 65001<br>neighbor 10.0.0.2 update-source Loopback0<br>neighbor 10.0.0.2 route-reflector-client<br>neighbor 10.0.0.4 remote-as 65001<br>neighbor 10.0.0.4 update-source Loopback0<br><br>! Mirror this on RR2 with router-id 10.0.0.4, the same cluster-id 1.1.1.1, <br>and RR1 (10.0.0.3) as its non-client peer</pre><p>Step 3: Verify baseline propagation. From R2, confirm you can see R1’s prefix even though R1 and R2 have no direct session with each other.</p><pre>show bgp ipv4 unicast 192.168.1.0<br><br>! Expect two paths, one via RR1 and one via RR2, <br>  each showing Originator: 10.0.0.1 and a Cluster list of 1.1.1.1</pre><p>R1 and R2 never spoke to each other directly, yet R2 has R1’s route anyway, complete with an ORIGINATOR_ID pointing back to R1’s router ID and a CLUSTER_LIST recording that it passed through cluster 1.1.1.1 — which is where reflection stops being theoretical.</p><p>Step 4: Add the external route. Bring up R5 as an eBGP peer of RR1 only, in AS 65010, advertising 203.0.113.0/24. Then confirm the prefix reaches R1 and R2, which have no direct or indirect awareness of R5 beyond what RR1 tells them.</p><pre>neighbor 10.0.0.5 remote-as 65010<br><br>! configured on RR1, no route-reflector-client keyword since <br>R5 is an eBGP peer, not an iBGP client<br><br>show bgp ipv4 unicast 203.0.113.0 (on R1 or R2)<br><br>! Expect the path to show up with no Originator or Cluster list <br>  attributes at all, because this route came in over eBGP and was<br>  advertised normally, not reflected</pre><p>Step 5: Break the redundancy peering on purpose. Shut down the iBGP session between RR1 and RR2 (shutdown under neighbour 10.0.0.4 on RR1, or the equivalent on RR2), then withdraw R1’s network statement and re-add it so it’s re-advertised as a fresh update.</p><pre>neighbor 10.0.0.4 shutdown<br><br>! run on RR1, then on R1: no network 192.168.1.0 mask 255.255.255.0, <br>  followed by network 192.168.1.0 mask 255.255.255.0 again</pre><p>Now check R2 again. If R1 happened to be sending its update to RR1 and RR1 can no longer relay to RR2, R2 will still have the route, because it also peers with RR1 directly. The interesting failure only shows up in larger topologies where a client peers with just one reflector; in this lab, disable R1’s session to RR2 as well (leaving only R1 to RR1) to see the real effect: with the RR1-RR2 link down, RR2 has nothing to reflect to R2 for this prefix, and show bgp ipv4 unicast 192.168.1.0 on R2 will show only one path, or none, depending on which sessions you disabled.</p><p>It’s the same redundancy gap from earlier in the article, just visible now instead of theoretical — two reflectors without a session between them don’t reliably back each other up, even if each one has plenty of clients of its own.</p><p>Step 6: Restore the RR1-RR2 session and R1’s second peering, then deliberately misconfigure RR2 with a different cluster ID.</p><pre>no neighbor 10.0.0.4 shutdown<br><br>! on RR1; then on RR2, change bgp cluster-id 1.1.1.1 to bgp <br>cluster-id 2.2.2.2<br><br>clear ip bgp * soft<br><br>! run on all five routers to force full re-advertisement under the <br>  new cluster configuration</pre><p>With RR1 in cluster 1.1.1.1 and RR2 in cluster 2.2.2.2, they are no longer treated as one logical reflector. Watch show bgp ipv4 unicast 192.168.1.0 on R2 closely: you should now see paths from both reflectors carrying different CLUSTER_LIST values, and if your topology has any additional cross-links, this is exactly the misconfiguration that produces duplicate or inconsistent paths across clients. Set the cluster ID back to 1.1.1.1 on RR2 before moving on, and confirm the CLUSTER_LIST values converge back to matching entries.</p><p>Once you’ve run through all six steps, none of this is abstract anymore: you’ve watched a route reach a router it never peered with, watched ORIGINATOR_ID and CLUSTER_LIST get attached in real time, and broken redundancy on purpose to see exactly what fails. Most people running BGP in production never see any of that — they just trust that one config line to do its job.</p><p>If you found it useful, don’t forget to <strong>👏 clap, 🔄 share, and 💬 follow</strong> for more Network Security, Linux &amp; Windows, and DevOps &amp; Cloud insights.</p><p>📲 Follow me on social media:<br>🔗 LinkedIn: <a href="https://www.linkedin.com/in/bornaly/">https://www.linkedin.com/</a><a href="https://www.linkedin.com/in/mmokhta">in/mmokhta</a><br>✍️ Medium: <a href="https://medium.com/@mehrdad.mokhtari1370">https://medium.com/@mehrdad.mokhtari1370</a><br>♛ GitHub: <a href="https://github.com/MMOKHTA">https://github.com/MMOKHTA</a><br>⛪ GitLab: <a href="https://github.com/MMOKHTA">https://github.com/MMOKHTA</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28299f1234b1" width="1" height="1" alt=""><hr><p><a href="https://systemweakness.com/route-reflectors-the-one-bgp-command-everyone-runs-and-almost-nobody-fully-understands-28299f1234b1">Route Reflectors: The One BGP Command Everyone Runs, and Almost Nobody Fully Understands</a> was originally published in <a href="https://systemweakness.com/">System Weakness</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[BGP Under the Hood: Path Selection and Policy Architecture — Part 2]]></title>
            <link>https://systemweakness.com/bgp-under-the-hood-path-selection-and-policy-architecture-part-2-196c942d7888?source=rss-3099b15986b------2</link>
            <guid isPermaLink="false">https://medium.com/p/196c942d7888</guid>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[netwrok-security]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[data-center]]></category>
            <dc:creator><![CDATA[Mehrdad Mokhtari]]></dc:creator>
            <pubDate>Mon, 06 Jul 2026 09:36:02 GMT</pubDate>
            <atom:updated>2026-07-09T11:23:03.095Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>The session is up. The neighbour is Established. And the traffic is still going the wrong way.</blockquote><h3>When the Session Is Up, and the Real Work Begins</h3><p>In Part 1, I walked through everything BGP needs just to exist — the TCP dependency, the RIB pipeline, the FSM states that engineers misread, and the eBGP/iBGP split that silently breaks routing tables in ways that look like something else entirely. If you haven’t read that, I’d start there, because this part assumes you already understand why a session gets stuck in Active and what a NOTIFICATION error code is telling you.</p><p>Once the session reaches Established, a different class of problem takes over. The neighbour is up. Routes are arriving. And yet the traffic still goes the wrong way, or the route you wanted installed isn’t in the forwarding table, or a policy change you made last week had no visible effect. That’s where BGP path selection and policy architecture live. This is the part most engineers learn by accident, one production incident at a time. I’m going to try to give you a structured version of that experience.</p><h3>The 11-Step Best-Path Algorithm: Order Is Everything</h3><p>BGP is not a distance-vector protocol that picks the shortest hop count. It is a policy engine that runs a deterministic, ordered decision process against every prefix with multiple candidate paths. The algorithm exits as soon as a winner is found. That detail alone explains a category of silent failures.</p><p>Here are the 11 steps, in the order BGP actually evaluates them.</p><p><strong>Step 1: Next-hop reachability.</strong> Before anything else, BGP discards any path whose NEXT HOP is not resolvable in the routing table. This is not a tie-breaker. It is a hard gate. A path that fails here doesn’t participate in the rest of the selection process at all. I covered this in Part 1, but it’s worth repeating in this context: if you add a more specific prefix that covers your NEXT HOP address to a black-hole route, all paths learned through that peer silently disappear from the FIB even though the BGP session stays up and the routes remain in Loc-RIB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*hYNwn5ktnGVxVULfp-RLAw.png" /></figure><p><strong>Step 2: Weight.</strong> Weight is a Cisco-proprietary attribute. It is local to the router, it is never sent to any peer, and it has no standardised equivalent. Higher weight wins. This makes it the most operator-controlled attribute in the entire algorithm, but only on the specific router where you configure it. Engineers who try to use Weight for network-wide traffic engineering quickly discover this limitation. It is the right tool when you want a single router to prefer one upstream over another, and nothing else.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x8YPuzThH1cY6P8A0T5-3Q.jpeg" /></figure><p><strong>Step 3: LOCAL PREF.</strong> This is the first attribute that crosses iBGP sessions. It is carried inside an AS and used to influence which exit point every router in the AS prefers for a given prefix. Higher LOCAL PREF wins. The default is 100. Raise it on the path you prefer. Lower it on the path you want as a backup. This is the standard knob for influencing outbound traffic from your AS. If someone on your team says “we need to prefer the primary upstream for all traffic”, the answer starts with LOCAL PREF.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q5KlDmmgb2SF0y_Mw2XNqQ.jpeg" /></figure><p><strong>Step 4: Locally originated routes.</strong> A route originated by the local router via a network statement or aggregate wins over a route learned from any peer, when all other attributes are equal to this point. You will rarely use this as an intentional tie-breaker, but it helps explain why a locally injected prefix sometimes wins selection in a way that’s not immediately obvious.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aaTQoVJWaay2CunQmZhFWA.jpeg" /></figure><p><strong>Step 5: AS PATH length.</strong> Shorter AS PATH wins. This is the most commonly abused attribute in BGP. AS PATH prepending is the standard mechanism for making your AS look less attractive to inbound traffic — you repeat your own AS number multiple times to inflate the path length as seen by other ASes. It works when your peers respect AS PATH length. It does not work on peers that have configured a policy to ignore AS PATH during path selection, and several large providers do exactly that for their own internal reasons.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qObGyVPJIU27OmOm3EUJNQ.jpeg" /></figure><p><strong>Step 6: Origin code.</strong> IGP (i) wins over EGP (e), which wins over Incomplete (?). In practice, you will almost never make a routing decision based on origin code deliberately. What you will do is notice that a prefix originated via a network statement (IGP) is beating a redistributed prefix (Incomplete) at this step and wonder why. Now you know.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MI9WiXwIhPOb6Sv-h26H-g.jpeg" /></figure><p><strong>Step 7: MED.</strong> Multi-Exit Discriminator is a suggestion from one AS to an adjacent AS about how to enter the advertising AS. Lower MED is preferred. I want to spend more time here because MED has more non-obvious behaviour than almost any other BGP attribute.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*vdFIMv_UsFfHjViSWXOFvQ.png" /></figure><h3>MED: The Attribute That Does the Opposite of What You Expect</h3><p>MED is optional and non-transitive. That means it is stripped before BGP forwards a prefix to any third AS. If you’re advertising a prefix to an upstream transit provider and you want to influence which of their ingress points they use to reach you, MED is the tool. But only if that provider compares MEDs across their own paths back to you, and only if they haven’t stripped it at their border.</p><p>Here is the non-obvious behaviour that causes real incidents: by default, most implementations only compare MED between paths that came from the same neighbouring AS. If you receive prefix 10.0.0.0/8 from AS 64500 via two different paths, BGP will compare their MEDs. If you receive the same prefix from AS 64500 through one path and from AS 64501 through another, BGP will not compare MEDs between those two paths at all. It will skip MED and move to the next tie-breaker.</p><p>The configuration option that changes this is “bgp always-compare-med”. It forces MED comparison regardless of the advertising AS. Enabling it can change path selection for hundreds of prefixes immediately. Do not enable it without understanding which paths it will affect first.</p><p>The second MED trap: if a peer sends you a prefix without a MED attribute, some implementations treat the missing MED as 0 (most preferred), and others treat it as infinity (least preferred). The “bgp bestpath med missing-as-worst” command on IOS-XE changes this behaviour. If you have peers that inconsistently set MED, this one setting can flip which path you install.</p><p><strong>Step 8: eBGP over iBGP.</strong> A path learned from an external peer is preferred over the same prefix learned from an internal peer. This is almost never a deliberate tie-breaker. It shows up in troubleshooting when you’re trying to understand why a route reflector client is installing a specific path.</p><p><strong>Step 9: IGP metric to the BGP NEXT HOP.</strong> Among paths that are still tied after all of the above, BGP prefers the one whose NEXT HOP is reachable via the lowest IGP metric. This is the step where your interior routing costs influence BGP decisions. It’s also a subtle reason why hot-potato routing happens: a router close to an exit point will have a low IGP metric to that exit’s NEXT HOP, and will prefer that path even if a different exit would be better end-to-end for the traffic.</p><p><strong>Step 10: Router ID.</strong> If everything else is equal, BGP prefers the path from the peer with the lowest BGP Router ID. This is the true last resort before the final step. If you’re ever debugging an unexpected path selection and nothing in steps 1 through 9 explains it, look at Router IDs.</p><p><strong>Step 11: Peer IP address.</strong> For paths from the same peer (multiple sessions), the lowest peer IP address wins. You will rarely hit this step in practice. But it is there.</p><p>You will rarely engineer deliberately around steps 8 through 11. What they do explain is why a tie at steps 1 through 7 can produce a path selection outcome that looks completely arbitrary until you remember the algorithm has to finish somehow. If a path selection result ever seems random after you’ve checked LOCAL PREF, AS PATH, MED, and origin code, the answer is usually sitting at step 9, 10, or 11.</p><h3>Why the Algorithm Exits Early Matters More Than the Steps Themselves</h3><p>The most important operational insight from this list is that BGP stops as soon as a winner is clear. If one path has a higher LOCAL PREF, step 3 decides the winner and steps 4 through 11 never run. This is why your MED configuration has no effect: if LOCAL PREF is different between the candidate paths, MED is never evaluated. Engineers who configure MED without checking that LOCAL PREF is equal on all competing paths spend a long time wondering why nothing changed.</p><p>The same logic applies to AS PATH prepending. If the peer you’re sending the prepended path to has a LOCAL PREF policy that already prefers another exit, your prepending has no effect at all. The algorithm exits at step 3, and your carefully crafted AS PATH is invisible.</p><h3>Route Maps and Prefix Lists: Policy Architecture</h3><p>BGP policy in IOS-XE is applied through route-maps, and route-maps match prefixes through prefix-lists, AS PATH access-lists, or community values. These are three separate matching tools, and conflating them is the most common source of policy mistakes I have seen.</p><p>A prefix-list matches on IP prefix and prefix length. It is the right tool when you want to permit or deny specific prefixes. The “le” keyword means “less than or equal to prefix length”. A statement like permit 192.168.0.0/16 le 24 allows any prefix within that block with a length between /16 and /24, and an implicit deny drops everything else. Forgetting the final deny is harmless when the prefix-list is used as a match condition inside a route-map, but it is a meaningful difference when applied directly as a distribute-list.</p><p>An AS PATH access-list matches on the AS PATH attribute using a regular expression. This is the right tool for a policy that needs to operate on the origin or transit AS of a route, not just its prefix.</p><pre>ip as-path access-list 10 permit ^64500$</pre><p>The caret and dollar anchor match AS paths that consist of exactly AS 64500, meaning prefixes originated by that AS with no transit. This is commonly used to filter out transit routes from a customer peer while accepting their own prefixes.</p><p>Route-maps tie these together and allow you to set attributes when a match occurs.</p><pre>route-map INBOUND-POLICY permit 10<br>match ip address prefix-list FILTER-IN<br>set local-preference 200<br><br>route-map INBOUND-POLICY permit 20<br>match as-path 10<br>set local-preference 150</pre><p>A route-map is evaluated top-down. The first matching clause wins. An implicit deny sits at the end of every route-map. If a prefix doesn’t match any permit clause, it is dropped. I have debugged more than one incident where a route-map was added with a new clause at the bottom, but an earlier clause was catching the prefix first because the match conditions overlapped. Always check route-map sequence numbers before adding new clauses.</p><h3>Communities: The Policy Signal That Scales</h3><p>BGP communities are 32-bit values attached to a prefix as an attribute. They are transitive, meaning they can cross AS boundaries if you explicitly send them. They are the mechanism that makes scalable policy possible.</p><p>The idea is simple: instead of maintaining a long prefix-list that you update every time a peer adds a new prefix, you tag prefixes with a community at the point of entry and write your policy against the community value. When the community tag changes, you update one place.</p><pre>route-map SET-COMMUNITY permit 10<br> match ip address prefix-list CUSTOMER-PREFIXES<br> set community 65001:100</pre><p>The format 65001:100 is the standard two-part community notation where the first number is typically the AS number and the second is a locally significant value. Large communities (RFC 8092) extend this to a three-part format for ASes that require greater granularity, but the principle remains the same.</p><p>Well-known communities are standardised. NO-EXPORT (65535:65281) tells a receiving router not to advertise the prefix to eBGP peers. NO-ADVERTISE (65535:65282) tells it not to advertise the prefix to any peer. These are useful for prefixes you want to share internally but never leak externally. I have used NO-EXPORT in situations where a failover prefix needed to be visible within our AS but should never reach the public routing table.</p><h3>The Silent Failure: Policy That Runs but Does Nothing</h3><p>Here is a scenario I have been called in to look at more than once. An engineer applies an inbound route-map to a peer to set LOCAL PREF on a specific set of prefixes. They verify the route-map syntax, check the prefix-list, and confirm the match logic looks right. The BGP session comes back up. And then “show bgp ipv4 unicast” shows the same paths with the same LOCAL PREF as before the change.</p><p>The most common cause is that the route-map was applied to the peer, but the session was not reset. BGP doesn’t retroactively re-evaluate already-received routes when you add or modify a policy. You need to trigger a route refresh or do a soft reset.</p><pre>clear ip bgp &lt;peer-ip&gt; soft in</pre><p>This sends a ROUTE-REFRESH message to the peer (if the peer supports the capability, which almost all modern implementations do), asking them to re-send their full table. Your router then applies the updated inbound policy to all received routes. Without this step, your route-map is applied only to new updates, not to the full table already sitting in Adj-RIB-In.</p><p>The second common cause: the route-map sequence has an implicit deny at the end, and the prefix you’re trying to match isn’t matching any permit clause. Instead of setting LOCAL PREF on the target prefix, the route-map is dropping it entirely. The engineer sees the prefix disappear from “show bgp” and concludes the policy failed silently, when actually it failed loudly by withdrawing the route. Use “debug ip bgp &lt;peer-ip&gt; updates in” during a soft reset on a test peer to see exactly which prefixes are being permitted and which are being dropped by each clause.</p><h3>Outbound Traffic Engineering in Practice</h3><p>Let me walk through a real scenario. You have two upstream providers, Provider A and Provider B. Provider A is your primary transit. Provider B is your backup. You want all traffic to leave via Provider A unless it goes down.</p><p>The cleanest implementation is LOCAL PREF on inbound routes. You apply an inbound route map to the Provider A session that sets LOCAL PREF to 200. You apply an inbound route map to the Provider B session that sets LOCAL PREF to 100. Every prefix arrives via both providers, but the paths from Provider A win at step 3 of the best-path algorithm. When Provider A goes down, the paths from Provider B remain in the Loc-RIB with LOCAL PREF 100 and immediately become the best paths without any operator intervention.</p><p>This is preferable to using prefix-list filtering as the primary mechanism because it preserves the backup paths in the Loc-RIB. If you were to filter inbound routes from Provider B entirely, a Provider A failure would leave you with no installed paths until you removed the filter.</p><p>For inbound traffic engineering (getting traffic into your AS the way you want it), the primary mechanisms are AS PATH prepending and MED. Prepend your AS on advertisements to Provider B to make that path look longer. Peers who respect AS PATH length will prefer Provider A’s path. Set a higher MED on advertisements to Provider B to achieve the same effect as peers within the same adjacent AS.</p><p>The hard lesson here is that neither mechanism is guaranteed to work. Provider A might strip MED at its border. An upstream might have LOCAL PREF policies that override your prepending. Inbound traffic engineering is always advisory, not authoritative. If precise control over inbound paths matters, BGP communities offered by your upstream provider are usually more reliable — most transit providers publish community values that let customers set LOCAL PREF directly on the provider’s routers.</p><h3>Verification Sequence for Path Selection Issues</h3><p>When a path selection outcome doesn’t match your expectations, I use this sequence to quickly identify the cause.</p><p>First, check what the router is selecting and why:</p><pre>show bgp ipv4 unicast &lt;prefix&gt;</pre><p>The output shows all candidate paths and marks the best path with a greater-than sign. Read the attributes for each path: LOCAL PREF, AS PATH length, MED, and the peer address. Walk the 11 steps manually for the two paths you’re comparing. The algorithm tells you exactly which step made the decision.</p><p>Second, if the inbound policy should have modified the attributes but didn’t, check whether the session was soft-reset after the policy was applied:</p><pre>show bgp neighbors &lt;peer-ip&gt; | include policy<br>clear ip bgp &lt;peer-ip&gt; soft in</pre><p>Third, if a prefix is in “show bgp” but not in “show ip route”, go back to NEXT HOP resolution:</p><pre>show bgp ipv4 unicast &lt;prefix&gt;<br>show ip route &lt;next-hop-address&gt;</pre><p>Fourth, if the outbound policy isn’t advertising a route you expect a peer to receive:</p><pre>show bgp neighbors &lt;peer-ip&gt; advertised-routes</pre><p>If the prefix is absent here, either an outbound route-map filters it or the route doesn’t meet the advertisement criteria (it must be the best path in the Loc-RIB). Check “show bgp ipv4 unicast” first to confirm the route is the best path, then check the outbound route-map.</p><h3>What’s Coming in Part 3</h3><p>Part 2 covered the decision engine: how BGP picks its best path, the attributes that drive it, and the policy mechanisms that let operators shape it. Part 3 will move into BGP scalability and advanced features: confederation and route reflector design at scale, BGP PIC (Prefix Independent Convergence) and how it changes failover behaviour, graceful restart and its interaction with the control plane, and the operational patterns that separate networks that handle full-table churn cleanly from those that don’t. If you’ve ever watched a maintenance window turn into a 20-minute BGP reconvergence event you didn’t anticipate, Part 3 is built for that.</p><p>This is Part 2 of a 4-part BGP deep-dive series.</p><p>If you found it useful, don’t forget to <strong>👏 clap, 🔄 share, and 💬 follow</strong> for more Network Security, Linux &amp; Windows, and DevOps &amp; Cloud insights.</p><p>📲 Follow me on social media:<br>🔗 LinkedIn: <a href="https://www.linkedin.com/in/bornaly/">https://www.linkedin.com/</a><a href="https://www.linkedin.com/in/mmokhta">in/</a><a href="https://www.linkedin.com/in/mehrdad-mokhtari/">mehrdad-mokhtari</a><br>✍️ Medium: <a href="https://medium.com/@mehrdad.mokhtari1370">https://medium.com/@mehrdad.mokhtari</a><br>♛ Github: <a href="https://github.com/MEMOKHTARI">https://github.com/MEMOKHTARI</a><br>⛪ GitLab: <a href="https://github.com/MMOKHTA">https://github.com/MMOKHTA</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=196c942d7888" width="1" height="1" alt=""><hr><p><a href="https://systemweakness.com/bgp-under-the-hood-path-selection-and-policy-architecture-part-2-196c942d7888">BGP Under the Hood: Path Selection and Policy Architecture — Part 2</a> was originally published in <a href="https://systemweakness.com/">System Weakness</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>