<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Tom Xiao on Medium]]></title>
        <description><![CDATA[Stories by Tom Xiao on Medium]]></description>
        <link>https://medium.com/@tom.xiao_3937?source=rss-2597b5c2304b------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*eho2SAzdwhkLe7T1V0-0lA.png</url>
            <title>Stories by Tom Xiao on Medium</title>
            <link>https://medium.com/@tom.xiao_3937?source=rss-2597b5c2304b------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 01:42:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tom.xiao_3937/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Data on EKS Platform Series 2 — Setup IDE for DEP platform (WIP)]]></title>
            <link>https://medium.com/@tom.xiao_3937/data-on-eks-platform-series-2-setup-ide-for-dep-platform-wip-a6fc0fc52402?source=rss-2597b5c2304b------2</link>
            <guid isPermaLink="false">https://medium.com/p/a6fc0fc52402</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[k8s]]></category>
            <dc:creator><![CDATA[Tom Xiao]]></dc:creator>
            <pubDate>Mon, 01 Jan 2024 15:55:49 GMT</pubDate>
            <atom:updated>2024-01-31T05:23:57.012Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data on EKS Platform Series 2 — Setup IDE for DEP platform</h3><p>Following our <a href="https://medium.com/@tom.xiao_3937/data-on-eks-platform-series-1-a-comprehensive-guide-to-permission-management-in-big-data-62fecd35519d">last blog</a>, we will setup an IDE which provides ad-hoc querying and data exploration.</p><p>In our IDE, data developers can use Hive or Trino SQL to develop, test, and request deployment of jobs. Business analysts can query data themselves and download data once approved. Every query goes through Ranger and Solr for data authorization and auditing.</p><p>The IDE provides a collaborative environment for different user personas with built-in governance. Data engineers have the tools to develop pipelines and ETL jobs. Analysts have self-service access to data for exploratory analysis. Queries are logged and access controlled to meet security policies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*6Hwsw3qIYU_bepyF.png" /><figcaption>Demo</figcaption></figure><h3>TL;DR: Quick Deployment with CloudFormation</h3><p>For those eager to get started with deploying a big data platform on EKS for demo, you can immediately kick off the process using my provided CloudFormation templates. Simply add the template to your AWS CloudFormation console, and the deployment of the entire stack, from the EKS cluster to the integrated Apache Ranger, will be automated for you.</p><p>Here’s how to do it:</p><ol><li>Go to AWS CloudFormation: Log into your AWS Management Console and navigate to the CloudFormation service.</li><li>Create New Stack: Select ‘Create Stack’ &gt; ‘With new resources (standard)’.</li><li>Specify Template: Download the template file from GitHub. Choose ‘Upload a template file’ and upload the provided CloudFormation template file. (Or paste the following url in the template url section: <a href="https://dep-public.s3.ap-northeast-1.amazonaws.com/series-2/dep.template"><strong>https://dep-public.s3.ap-northeast-1.amazonaws.com/series-2/dep.template</strong></a>)</li><li>Configure Stack Options: Set your stack name and other parameters required by the template.</li><li>Review and Launch: Review your configuration details, acknowledge any required capabilities, and click ‘Create stack’.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bqq_gnoZ-L_nayNuG7oreQ.gif" /></figure><p>After a few minutes, you’ll have a fully functional big data platform ready for your analytical workloads.</p><h3>How To</h3><p>After the template is successfully created, go to AWS web console and make sure the two default schemas are created as requested in S3 bucket (in my case, ‘<em>dep</em>’ and ‘<em>ppe_dep</em>’ are created in bucket ‘<em>dep-837282828706-ap-northeast-1</em>’).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UczPaZoyP70J4xMf0Uxbww.gif" /><figcaption>Default Databases Creation</figcaption></figure><p>In your command line tools, type the following to make sure your pods are running normally. And then forward the web application to your localhost.</p><pre>aws eks update-kubeconfig --name dep-eks-cluster-ap-northeast-1 --region ap-northeast-1<br><br>## Check pod status.<br>kubectl get pods --namespace hadoop<br>kubectl get pods --namespace platform<br>kubectl get pods --namespace trino<br><br>## Port Forwarding<br>kubectl port-forward service/dep-web 3000:8080 --namespace platform</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pUZoLgWXy-7xQjwL_iCXAQ.gif" /></figure><p>Now the DEP web application is available in port 3000. Login with your predefined username (parameter: <em>LdapUser</em>) and password (parameter: <em>LdapUserPassword</em>).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UwJn7UkxJjy4F-L6ne0fIQ.png" /><figcaption>Login Page</figcaption></figure><p>Upon logging in, you will find several applications on the main page. At this moment, only the IDE is available for demo.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7xjDQk5MgR5vn5mDpoQMGg.png" /><figcaption>Main Page</figcaption></figure><p>Clicking on ‘IDE’ and you will be redirect to the IDE application where you will find 3 databases on the left-hand side (just ignore the default one). The other two schemas are created by the time we create this template.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LYYlx4MaY0KaiVHujcNUQQ.png" /><figcaption>IDE — Database</figcaption></figure><p>On the bottom of this page, you will see your query histories.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OVC0wQZv3MZyLsqAZTPC2Q.png" /><figcaption>IDE — Query History</figcaption></figure><p>You are also able to query data, execute DDLs and save your SQLs into a remote file locating on AWS S3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PgIr30QU5UDjUFeQDDA0_A.png" /><figcaption>IDE — Run your query &amp; save as files.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xxQfkOv_-ruiuDYmMNfbZg.png" /><figcaption>IDE — Examine your personal file on IDE.</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Z2ihhNGzyyU3g8Klf19A7Q.png" /><figcaption>IDE — Examine your personal file on AWS S3.</figcaption></figure><h3>Explanation of Template Parameters</h3><p>Template parameters are the same as those in <a href="https://medium.com/@tom.xiao_3937/data-on-eks-platform-series-1-a-comprehensive-guide-to-permission-management-in-big-data-62fecd35519d">Series 1</a>. You may refer to the previous blog for explanations.</p><h3>Template Resources</h3><p>Apart from the resources created in the first template, we add a few more Kubernetes service to achieve our goal: to setup IDE platform.</p><h4>Apollo Config Center (Including Apollo Admin / Apollo Config)</h4><p>This will implement apollo admin and apollo config service in namespace <em>platform</em> to our EKS cluster.</p><blockquote>Apollo is a reliable configuration management system. It can centrally manage the configurations of different applications and different clusters. It is suitable for microservice configuration management scenarios.</blockquote><p>It helps us to manage platform-level configurations after we deploy the complete DEP.</p><p>Images are available in the following registry:</p><ul><li>public.ecr.aws/y8d7x2g6/dep-public/apollo-adminservice</li><li>public.ecr.aws/y8d7x2g6/dep-public/apollo-configservice</li><li>public.ecr.aws/y8d7x2g6/dep-public/apollo-portal</li></ul><pre>## dep-public/series-2/dep-eks.template<br><br>Resources:<br>  ApolloProdService:<br>    Type: AWSQS::Kubernetes::Helm<br>    DependsOn:<br>      - EbsCsiAddon<br>      - EksCluster<br>      - ClusterNamespace<br>      - PlatformNodeGroup<br>    Properties:<br>      ClusterID: !Ref EksCluster<br>      Repository: https://tomxiaoyz.github.io/big-data-platform-public/<br>      Chart: dep/apollo-service<br>      Namespace: platform<br>      Name: apollo-service<br>      Version: 1.0.0<br>      ValueYaml: !Sub |<br>        configdb:<br>          host: ${DbEndpoint}<br>          port: ${DbPort}<br>          dbName: apollo_config_prod<br>          userName: ${DbUserName}<br>          password: ${DbUserPassword}<br>          service:<br>            enabled: true<br>            type: ExternalName<br>        affinity:<br>          nodeAffinity:<br>            requiredDuringSchedulingIgnoredDuringExecution:<br>              nodeSelectorTerms:<br>                - matchExpressions:<br>                    - key: nodegroup<br>                      operator: In<br>                      values:<br>                        - platform<br>        env: prod<br><br>  ApolloPortal:<br>    Type: AWSQS::Kubernetes::Helm<br>    DependsOn:<br>      - EbsCsiAddon<br>      - EksCluster<br>      - ClusterNamespace<br>      - PlatformNodeGroup<br>      - OpenldapService<br>      - ApolloProdService<br>    Properties:<br>      ClusterID: !Ref EksCluster<br>      Repository: https://tomxiaoyz.github.io/big-data-platform-public/<br>      Chart: dep/apollo-portal<br>      Namespace: platform<br>      Name: apollo-portal<br>      Version: 1.0.0<br>      ValueYaml: !Sub |<br>        config:<br>          envs: prod<br>          metaServers:<br>            prod: http://apollo-service-prod-apollo-configservice:8080<br>          profiles: github,ldap<br>          files:<br>            application-ldap.yml: |<br>              spring:<br>                ldap:<br>                  base: &quot;dc=${LdapDomainName},dc=com&quot;<br>                  username: &quot;cn=${LdapAdmin},dc=${LdapDomainName},dc=com&quot;<br>                  password: &quot;${LdapAdminPassword}&quot;<br>                  searchFilter: &quot;(uid={0})&quot;<br>                  urls:<br>                  - &quot;ldap://openldap.platform.svc.cluster.local:389&quot;<br>                main:<br>                  allow-circular-references: true<br>              ldap:<br>                mapping:<br>                  objectClass: &quot;inetOrgPerson&quot;<br>                  loginId: &quot;uid&quot;<br>                  userDisplayName: &quot;cn&quot;<br>                  email: &quot;mail&quot;<br>        portaldb:<br>          host: ${DbEndpoint}<br>          port: ${DbPort}<br>          dbName: apollo_portal<br>          userName: ${DbUserName}<br>          password: ${DbUserPassword}<br>          connectionStringProperties: characterEncoding=utf8&amp;useSSL=false<br>          service:<br>            enabled: true<br>            type: ExternalName<br>        affinity:<br>          nodeAffinity:<br>            requiredDuringSchedulingIgnoredDuringExecution:<br>              nodeSelectorTerms:<br>                - matchExpressions:<br>                    - key: nodegroup<br>                      operator: In<br>                      values:<br>                        - platform</pre><h4>Nacos Service</h4><p>This will install Nacos service in namespace <em>platform</em> to our EKS cluster.</p><blockquote>Nacos (official site: <a href="https://nacos.io/">nacos.io</a>) is an easy-to-use platform designed for dynamic service discovery and configuration and service management. It helps you to build cloud native applications and microservices platform easily.</blockquote><p>It helps manage all services deployed to EKS cluster for DEP.</p><p>Image is available in the following registry:</p><ul><li>public.ecr.aws/y8d7x2g6/dep-public/nacos-server</li></ul><pre>## dep-public/series-2/dep-eks.template<br><br>Resources: <br>  NacosService:<br>    Type: AWSQS::Kubernetes::Helm<br>    DependsOn:<br>      - EbsCsiAddon<br>      - EksCluster<br>      - ClusterNamespace<br>      - PlatformNodeGroup<br>    Properties:<br>      ClusterID: !Ref EksCluster<br>      Repository: https://tomxiaoyz.github.io/big-data-platform-public/<br>      Chart: dep/nacos<br>      Namespace: platform<br>      Name: nacos<br>      Version: 1.0.0<br>      ValueYaml: !Sub |<br>        namespace: platform<br>        nacos:<br>          storage:<br>            type: mysql<br>            db:<br>              host: ${DbEndpoint}<br>              name: nacos<br>              port: ${DbPort}<br>              username: ${DbUserName}<br>              password: ${DbUserPassword}<br>              param: characterEncoding=utf8&amp;connectTimeout=1000&amp;socketTimeout=3000&amp;autoReconnect=true&amp;useSSL=false&amp;serverTimezone=UTC<br>        affinity:<br>          nodeAffinity:<br>            requiredDuringSchedulingIgnoredDuringExecution:<br>              nodeSelectorTerms:<br>                - matchExpressions:<br>                    - key: nodegroup<br>                      operator: In<br>                      values:<br>                        - platform</pre><h4>Redis</h4><p>We use Redis to act as a cache backend to store user cache, query cache, etc. It contains one master pod and three replica pods.</p><p>Image is available in the following registry:</p><ul><li>public.ecr.aws/y8d7x2g6/dep-public/redis</li></ul><pre>## dep-public/series-2/dep-eks.template<br><br>Resources:<br>  RedisService:<br>    Type: AWSQS::Kubernetes::Helm<br>    DependsOn:<br>      - EbsCsiAddon<br>      - EksCluster<br>      - ClusterNamespace<br>      - PlatformNodeGroup<br>    Properties:<br>      ClusterID: !Ref EksCluster<br>      Repository: https://tomxiaoyz.github.io/big-data-platform-public/<br>      Chart: redis<br>      Namespace: platform<br>      Name: redis<br>      Version: 1.0.0<br>      ValueYaml: !Sub |<br>        global:<br>          redis:<br>            password: ${RedisPassword}<br>        master:<br>          affinity:<br>            nodeAffinity:<br>              requiredDuringSchedulingIgnoredDuringExecution:<br>                nodeSelectorTerms:<br>                  - matchExpressions:<br>                      - key: nodegroup<br>                        operator: In<br>                        values:<br>                          - platform<br>        replica:<br>          affinity:<br>            nodeAffinity:<br>              requiredDuringSchedulingIgnoredDuringExecution:<br>                nodeSelectorTerms:<br>                  - matchExpressions:<br>                      - key: nodegroup<br>                        operator: In<br>                        values:<br>                          - platform</pre><h4>Prometheus Pushgateway</h4><p>This will install Prometheus pushgateway in namespace platform to our EKS cluster to help gather application-level metrics. And thus we can setup monitors and alerts in the future.</p><blockquote>The Prometheus Pushgateway exists to allow ephemeral and batch jobs to expose their metrics to Prometheus. Since these kinds of jobs may not exist long enough to be scraped, they can instead push their metrics to a Pushgateway. The Pushgateway then exposes these metrics to Prometheus.</blockquote><p>Image is available in the following registry:</p><ul><li>public.ecr.aws/y8d7x2g6/dep-public/prometheus-pushgateway</li></ul><pre>## dep-public/series-2/dep-eks.template<br><br>Resources:<br>  PrometheusPushGatewayService:<br>    Type: AWSQS::Kubernetes::Helm<br>    DependsOn:<br>      - EbsCsiAddon<br>      - EksCluster<br>      - ClusterNamespace<br>      - PlatformNodeGroup<br>    Properties:<br>      ClusterID: !Ref EksCluster<br>      Repository: https://tomxiaoyz.github.io/big-data-platform-public/<br>      Chart: dep/prometheus-pushgateway<br>      Namespace: platform<br>      Name: prometheus-pushgateway<br>      Version: 1.0.0<br>      ValueYaml: |<br>        persistentVolume:<br>          storageClass: ebs-sc<br>        affinity:<br>          nodeAffinity:<br>            requiredDuringSchedulingIgnoredDuringExecution:<br>              nodeSelectorTerms:<br>                - matchExpressions:<br>                    - key: nodegroup<br>                      operator: In<br>                      values:<br>                        - platform</pre><h4>DEP Auth Service</h4><p>This is a Spring Boot-based enterprise solution designed for managing authentication and authorization. Its primary objective is to facilitate secure access control and user management in applications. The service integrates various components to handle user credentials, role-based access permissions, and possibly interactions with external systems for identity verification (as indicated by the presence of LDAP-related classes). Overall, the service serves as a comprehensive backend solution for ensuring secure and efficient user access management in enterprise environments.</p><p>Image is available in the following registry:</p><ul><li>public.ecr.aws/y8d7x2g6/dep-public/auth</li></ul><pre>## dep-public/series-2/dep-eks.template<br><br>Resources:<br>  DepAuthService:<br>    Type: AWSQS::Kubernetes::Helm<br>    DependsOn:<br>      - EbsCsiAddon<br>      - EksCluster<br>      - ClusterNamespace<br>      - PlatformNodeGroup<br>      - ApolloPortal<br>      - ApolloProdService<br>      - NacosService<br>      - RedisService<br>      - PrometheusPushGatewayService<br>    Properties:<br>      ClusterID: !Ref EksCluster<br>      Repository: https://tomxiaoyz.github.io/big-data-platform-public/<br>      Chart: dep/auth<br>      Namespace: platform<br>      Name: dep-auth<br>      Version: 1.0.0<br>      ValueYaml: !Sub |<br>        fullnameOverride: dep-auth<br>        spring:<br>          profiles:<br>            active: prod<br>          application:<br>            name: dep-auth<br>          cloud:<br>            nacos:<br>              discovery:<br>                enabled: true<br>                server_addr: nacos-cs.platform.svc.cluster.local<br>          datasource:<br>            url: jdbc:mysql://${DbEndpoint}:3306/scheduler<br>            username: ${DbUserName}<br>            password: ${DbUserPassword}<br>          redis:<br>            password: ${RedisPassword}<br>            host: redis-master.platform.svc.cluster.local<br>            port: 6379<br>        management:<br>          metrics:<br>            export:<br>              prometheus:<br>                pushgateway:<br>                  baseUrl: http://prometheus-pushgateway.platform.svc.cluster.local:9091<br>        app:<br>          id: dep-auth<br>        apollo:<br>          meta: http://apollo-service-prod-apollo-configservice.platform.svc.cluster.local:8080<br>          bootstrap:<br>            enabled: true<br>            eagerLoad:<br>              enabled: true<br>        ldap:<br>          base_dn: dc=${LdapDomainName},dc=com<br>          user_dn: cn=${LdapAdmin},dc=${LdapDomainName},dc=com<br>          password: ${LdapAdminPassword}<br>          url: ldap://openldap.platform.svc.cluster.local:389<br>          user:<br>            base_dn: ou=People,dc=${LdapDomainName},dc=com<br>          auto:<br>            fresh: true<br>        affinity:<br>          nodeAffinity:<br>            requiredDuringSchedulingIgnoredDuringExecution:<br>              nodeSelectorTerms:<br>                - matchExpressions:<br>                    - key: nodegroup<br>                      operator: In<br>                      values:<br>                        - platform</pre><h4>DEP IDE Service</h4><p>This is a Spring Boot-based application focused on integrating and managing data processing and version control functionalities. It serves as a platform for efficient handling of large data sets, incorporating features for data querying, downloading, and statistical analysis. The inclusion of Git-related modules suggests a strong emphasis on version control, making it suitable for collaborative and iterative development environments. Additionally, the service is designed to interact with various data sources and external APIs, indicating its capability to integrate with different systems and services. Overall, this is a versatile solution for data management and version control, catering to the needs of complex data-driven applications or development environments.</p><p>Image is available in the following registry:</p><ul><li>public.ecr.aws/y8d7x2g6/dep-public/ide</li></ul><pre>## dep-public/series-2/dep-eks.template<br><br>Resources:<br>  DepIdeService:<br>    Type: AWSQS::Kubernetes::Helm<br>    DependsOn:<br>      - EbsCsiAddon<br>      - EksCluster<br>      - ClusterNamespace<br>      - PlatformNodeGroup<br>      - ApolloPortal<br>      - ApolloProdService<br>      - NacosService<br>      - RedisService<br>      - PrometheusPushGatewayService<br>    Properties:<br>      ClusterID: !Ref EksCluster<br>      Repository: https://tomxiaoyz.github.io/big-data-platform-public/<br>      Chart: dep/ide<br>      Namespace: platform<br>      Name: dep-ide<br>      Version: 1.0.0<br>      ValueYaml: !Sub |<br>        fullnameOverride: dep-ide<br>        replicaCount: 1<br>        server:<br>          port: 8080<br>        spring:<br>          profiles:<br>            active: prod<br>          application:<br>            name: dep-ide<br>          cloud:<br>            nacos:<br>              discovery:<br>                enabled: true<br>                server_addr: nacos-cs.platform.svc.cluster.local<br>          datasource:<br>            ide:<br>              jdbcUrl: jdbc:mysql://${DbEndpoint}:3306/scheduler<br>              username: ${DbUserName}<br>              password: ${DbUserPassword}<br>            ranger:<br>              jdbcUrl: jdbc:mysql://${DbEndpoint}:3306/ranger<br>              username: ${DbUserName}<br>              password: ${DbUserPassword}<br>          redis:<br>            password: ${RedisPassword}<br>            host: redis-master.platform.svc.cluster.local<br>            port: 6379<br>        management:<br>          metrics:<br>            export:<br>              prometheus:<br>                pushgateway:<br>                  baseUrl: http://prometheus-pushgateway.platform.svc.cluster.local:9091<br>        s3:<br>          region: ap-northeast-1<br>          bucket: ${S3BucketName}<br>          database: dep<br>          ide:<br>            path: ide_files<br>        hive:<br>          driver: org.apache.hive.jdbc.HiveDriver<br>          url: jdbc:hive2://hadoop-hadoop-hive.hadoop.svc.cluster.local:10000<br>          username: ${LdapUser}<br>          password: ${LdapUserPassword}<br>          ide_url: jdbc:hive2://hadoop-hadoop-hive.hadoop.svc.cluster.local:10000<br>          ide_userName: ${LdapUser}<br>          ldap:<br>            enable: true<br>        trino:<br>          ldap:<br>            enable: false<br>          driver: io.trino.jdbc.TrinoDriver<br>          url: jdbc:trino://trino.trino.svc.cluster.local:8080/hive<br>        webide:<br>          ldap:<br>            url: ldap://openldap.platform.svc.cluster.local:389<br>            base_dn: dc=${LdapDomainName},dc=com<br>            people_base_dn: ou=People,dc=${LdapDomainName},dc=com<br>            user: cn=${LdapAdmin},dc=${LdapDomainName},dc=com<br>            password: ${LdapAdminPassword}<br>          result_size:<br>            hive: 10000<br>            trino: 5000<br>          cache_size: 10<br>        k2:<br>          datadownload:<br>            whitelist:<br>              roleid: 1,24</pre><h4>DEP Web App</h4><p>This is the web application for DEP. In this free version, only the IDE application is provided through this web application.</p><p>Image is available in the following registry:</p><ul><li>public.ecr.aws/y8d7x2g6/dep-public/web</li></ul><pre>## dep-public/series-2/dep-eks.template<br><br>Resources:<br>  DepWebService:<br>    Type: AWSQS::Kubernetes::Helm<br>    DependsOn:<br>      - EbsCsiAddon<br>      - EksCluster<br>      - ClusterNamespace<br>      - PlatformNodeGroup<br>    Properties:<br>      ClusterID: !Ref EksCluster<br>      Repository: https://tomxiaoyz.github.io/big-data-platform-public/<br>      Chart: dep/web<br>      Namespace: platform<br>      Name: dep-web<br>      Version: 1.0.0</pre><h3>Cleaning up resources</h3><ul><li>Go to AWS S3: Log into your AWS Management Console and navigate to the S3 service.</li><li>Delete all the contents in the S3 bucket created in this template.</li><li>Go to AWS CloudFormation: Navigate to the CloudFormation service.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*JSuDIUvc4RvgAgDW.png" /></figure><ul><li>Click ‘Delete’ to delete stack.</li><li>Go to AWS EC2: Navigate to the EC2 service.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*vhwb850GLxSfz8fC.png" /></figure><ul><li>Under ‘Volumes’, select all previously created volumes and delete them all.</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a6fc0fc52402" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data on EKS Platform Series 1— A Comprehensive Guide to Permission Management in Big Data…]]></title>
            <link>https://medium.com/@tom.xiao_3937/data-on-eks-platform-series-1-a-comprehensive-guide-to-permission-management-in-big-data-62fecd35519d?source=rss-2597b5c2304b------2</link>
            <guid isPermaLink="false">https://medium.com/p/62fecd35519d</guid>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[apache-ranger]]></category>
            <category><![CDATA[data-warehouse]]></category>
            <category><![CDATA[k8s]]></category>
            <dc:creator><![CDATA[Tom Xiao]]></dc:creator>
            <pubDate>Wed, 06 Dec 2023 07:58:26 GMT</pubDate>
            <atom:updated>2024-01-28T15:14:33.732Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data on EKS Platform Series 1— A Comprehensive Guide to Permission Management in Big Data Components</h3><h3>1. Introduction</h3><p>In the realm of big data, securing data on cloud platforms is crucial. This blog explores the deployment of Apache Ranger for permission management within the Hadoop ecosystem on Amazon EKS. Through practical case studies, we’ll see how Ranger integrates with Hadoop components like <strong>Hive</strong>, Spark, <strong>Trino</strong>, Yarn, and Hdfs, ensuring secure and efficient data management in a cloud environment. Join us as we navigate these advanced security strategies in the context of Kubernetes and cloud computing.</p><h3>2. About DEP (Data on EKS Platform)</h3><p>The <a href="https://medium.com/@tom.xiao_3937/data-on-eks-platform-c94c051dee0b"><strong>DEP</strong></a> (Data on EKS Platform) is a Kubernetes-based, cloud-native big data platform that revolutionizes the way we handle data in EKS environments. Developed by our company, DEP integrates with familiar components like Hive, Spark, Flink, Trino, HDFS, Kafka, and more, making it a versatile and comprehensive solution for data management and BI platforms.</p><h3>3. Crucial Role of Permission Management in Big Data</h3><p>Effective permission management is crucial for several key reasons:</p><ol><li>Enhanced Security: Proper permission management ensures that sensitive data is only accessible to authorized individuals, thereby safeguarding against unauthorized access and potential security breaches. This is especially important in industries handling large volumes of sensitive or personal data.</li><li>Operational Efficiency: By defining clear user roles and permissions, organizations can streamline workflows and reduce administrative overhead. This system simplifies managing user access, saves time, and minimizes the risk of configuration errors.</li><li>Scalability and Compliance: As businesses grow and evolve, a scalable permission management system helps in smoothly adjusting user roles and access rights. This adaptability is essential for maintaining compliance with various data privacy regulations like GDPR and HIPAA, ensuring that the organization’s data practices are legally sound and up to date.</li><li>Addressing Big Data Challenges: Big data comes with unique challenges like managing large volumes of rapidly evolving data across multiple platforms. Effective permission management helps tackle these challenges by controlling how data is accessed and used, ensuring data integrity and minimizing the risk of data breaches​.</li></ol><h3>4. About Apache Ranger</h3><p>Apache Ranger is a comprehensive framework designed for data governance and security in Hadoop ecosystems. It provides a centralized platform to define, administer, and manage security policies consistently across various Hadoop components. Ranger specializes in fine-grained access control, offering detailed management of user permissions and auditing capabilities.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*z_w6YTIZYlfOsHrW.png" /><figcaption>Image source: <a href="https://kymr.github.io/files/hadoop-summit/security/ranger_architecture.png">https://kymr.github.io/files/hadoop-summit/security/ranger_architecture.png</a></figcaption></figure><h4>4.1 Key Components</h4><p>Ranger’s Architecture and Principles Apache Ranger’s architecture is built to integrate seamlessly with a variety of big data tools like Hadoop, Hive, HBase, and Spark. The key components of Ranger include:</p><ol><li>Ranger Admin Service: This is the central component where all security policies are created and managed. It provides a web-based user interface for policy management and an API for programmatic configuration.</li><li>Ranger Usersync Service: Responsible for syncing user/group information from a directory service like LDAP/AD into Ranger.</li><li>Ranger Plugins: These are installed on each component of the Hadoop ecosystem (like Hive, HBase, etc.). Plugins pull policies from the Ranger Admin Service and enforce them locally.</li><li>Ranger Auditing: Ranger captures access audit logs and stores them for compliance and monitoring purposes. It can integrate with external tools for advanced analytics on these audit logs.</li><li>Ranger KMS (Key Management Service): Provides encryption and key management, extending Hadoop’s HDFS Transparent Data Encryption (TDE).</li></ol><h4>4.2 Permission Model</h4><p>At the core of this model are policies composed of three elements: users, resources, and permissions.</p><ol><li>Users: Ranger allows the grouping of users, where a single user may be a member of multiple groups. It enables the configuration of permissions for individual users or user groups, facilitating granular access control.</li><li>Resources: In Ranger, the term ‘resource’ varies across different big data components. For instance, in HDFS, a resource is identified by its file path, whereas, in Hive, it can be as broad as a Database or as specific as a Table or even a Column.</li><li>Permissions: Ranger is capable of limiting actions such as read, write, and execute on resources. Permissions are typically managed through inclusion and exclusion lists: whitelist (allow), whitelist exclude, blacklist (deny), and blacklist exclude.</li></ol><p>When a Ranger administrator sets up policies for a component and installs the corresponding Ranger plugin, the permissions become active. Now, when a user attempts to access a resource, Ranger initiates a permission validation process as follows:</p><ul><li>The system retrieves all policies associated with the requested resource.</li><li>It then iterates through these policies to determine if the user has permission to access the resource based on the established blacklist and whitelist criteria.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1020/1*BNtN1j0uEhpuUm2_S0_0AQ.png" /></figure><p>From the flowchart, it’s evident that the priority levels for matching policies are as follows:</p><ul><li>Blacklist takes precedence over whitelist.</li><li>Blacklist exclude has a higher priority than blacklist.</li><li>Whitelist exclude has a higher priority than whitelist.</li></ul><h3>5. Practical Case: Deploying Big Data Cluster with Apache Ranger on EKS</h3><p>In this section, we delve into a case study that outlines the deployment of a fully containerized big data environment on Amazon EKS, using AWS CloudFormation for resource provisioning.</p><h4>5.1 Deployment Structure</h4><p>Our EKS-based deployment will encompass the following components:</p><ol><li>S3 Buckets: Leveraged for scalable and durable Hive data storage.</li><li>MySQL Database: Utilized for storing Hive metadata, facilitating efficient metadata retrieval and management.</li><li>EKS Cluster: Comprising three distinct node groups named platform, hadoop, and trino, each tailored for specific operational needs.</li><li>Hadoop Cluster Applications: Including HDFS for distributed storage and YARN for managing cluster resources.</li><li>Trino Cluster Application: To enable distributed SQL query execution for analytics.</li><li>Apache Ranger: Serving as the central security management tool for access policy across the big data components.</li><li>OpenLDAP: Integrated as the LDAP service to provide a centralized user information repository, essential for user authentication and authorization.</li><li>Other Cloud Services Resources: Including a dedicated VPC for network security and isolation.</li></ol><p>By the end of this deployment process, the following benefits will be realized:</p><ul><li>A high-performing, scalable big data platform that can handle complex data workflows with ease.</li><li>Enhanced security through centralized management of authentication and authorization, provided by the integration of OpenLDAP and Apache Ranger.</li><li>Cost-effective infrastructure management and operation, thanks to the containerized nature of services on EKS.</li><li>Compliance with stringent data security and privacy regulations, assured by Apache Ranger’s policy enforcement capabilities.</li></ul><p>Source code is available at:</p><ol><li><a href="https://github.com/TomXiaoYZ/dep-public">TomXiaoYZ/dep-public (github.com)</a></li><li><a href="https://github.com/TomXiaoYZ/big-data-platform-public">TomXiaoYZ/big-data-platform-public (github.com)</a></li></ol><h4>Prerequisites</h4><ol><li>Setup an free AWS account for testing purpose. You may refer to <a href="https://aws.amazon.com/free/"><strong>this page</strong></a> for access your free tiers.</li><li>Configure proper permissions for your account.</li><li>Follow this <a href="https://www.stacksimplify.com/aws-eks/eks-cluster/install-aws-eksctl-kubectl-cli/"><strong>guidance</strong></a> to install AWS CLI, eksctl CLI, kubectl for AWS EKS.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m17xRFyjAigyQApJkrgt_w.gif" /><figcaption>Prerequisites: Install AWS CLI, eksctl CLI, kubectl for AWS EKS</figcaption></figure><h4>TL;DR: Quick Deployment with CloudFormation</h4><p>For those eager to get started with deploying a big data environment on EKS with Apache Ranger, you can immediately kick off the process using my provided CloudFormation templates. Simply add the template to your AWS CloudFormation console, and the deployment of the entire stack, from the EKS cluster to the integrated Apache Ranger, will be automated for you.</p><p>Here’s how to do it:</p><ol><li>Go to AWS CloudFormation: Log into your AWS Management Console and navigate to the CloudFormation service.</li><li>Create New Stack: Select ‘Create Stack’ &gt; ‘With new resources (standard)’.</li><li>Specify Template: Download the template file from GitHub. Choose ‘Upload a template file’ and upload the provided CloudFormation template file. (Or paste this url directly: <a href="https://dep-public.s3.ap-northeast-1.amazonaws.com/series-1/dep.template">https://dep-public.s3.ap-northeast-1.amazonaws.com/series-1/dep.template</a>)</li><li>Configure Stack Options: Set your stack name and other parameters required by the template.</li><li>Review and Launch: Review your configuration details, acknowledge any required capabilities, and click ‘Create stack’.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RbKhN47UXgL1stN5CTM6Qg.gif" /><figcaption>TL;DR: Quick Deployment with CloudFormation</figcaption></figure><p>After a few minutes, you’ll have a fully functional big data environment with robust security management ready for your analytical workloads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gCvX1hLYK2u8yKn5WzV0fA.png" /><figcaption>Resources Creation Successfully</figcaption></figure><p>Now it&#39;s time to see how this template works.</p><p>In aws web console, find the name of your EKS cluster. In this case, it is ‘dep-demo-eks-cluster-ap-northeast-1’ for me.</p><pre>aws eks update-kubeconfig --name dep-eks-cluster-ap-northeast-1 --region ap-northeast-1<br><br>## Check pod status.<br>kubectl get pods --namespace hadoop<br>kubectl get pods --namespace platform<br>kubectl get pods --namespace trino</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TAgf5_RylpM7tk0M-sPeqQ.gif" /><figcaption>Check Pod Status</figcaption></figure><p>Let’s see how ranger admin works in this template.</p><pre>kubectl port-forward service/ranger-admin 6080:6080 --namespace platform</pre><p>After ranger admin is successfully forwarded to port 6080 of localhost, go to ‘localhost:6080’ in your browser. Login with username ‘admin’ and ranger password entered above.</p><p>By default, we already created two ranger policies: <em>hive</em> and <em>trino</em> and grant all access to the ldap user created above (<em>depadmin </em>in my case).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Dhsw4XToMJ4-LbqzQfBc0Q.png" /><figcaption>Hive Policy</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PCdQ5PEsXdSBDuK5OI0-wQ.png" /><figcaption>Trino Policy</figcaption></figure><p>Also, ldap user sync service is setup and will automatically sync all users from the ldap service created in this template.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qoBIbHpJ8mbRDPird-d_pw.png" /><figcaption>Ranger Usersync Service</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*a54--4FVaHuBj4hG8XdSuw.gif" /></figure><h4>5.2 Explanation of template parameters</h4><pre>## dep-public/series-1/dep.template<br><br>---<br>AWSTemplateFormatVersion: &#39;2010-09-09&#39;<br>Transform: AWS::Serverless-2016-10-31<br>Description: &#39;Deploy Dep (owner: tom.xiao@hotmail.com)&#39;<br>Metadata:<br>  LICENSE: Apache License Version 2.0<br>  AWS::CloudFormation::Interface:<br>    ParameterGroups:<br>      - Label:<br>          default: DEP Basic Configuration<br>        Parameters:<br>          - EnvironmentName<br>          - CliUsername<br><br>      - Label:<br>          default: VPC Network Configuration<br>        Parameters:<br>          - VpcCIDR<br>          - PrivateSubnet1CIDR<br>          - PrivateSubnet2CIDR<br>          - PublicSubnet1CIDR<br>          - PublicSubnet2CIDR<br>          - RemoteAccessCIDR<br>          - AccessCIDR<br><br>      - Label:<br>          default: S3 Configuration<br>        Parameters:<br>          - S3BucketName<br><br>      - Label:<br>          default: MySQL Configuration<br>        Parameters:<br>          - DBAllocatedStorage<br>          - DBAutoMinorVersionUpgrade<br>          - DBBackupRetentionPeriod<br>          - DBInstanceClass<br>          - DBIops<br>          - DBMasterUsername<br>          - DBMasterUserPassword<br>          - DBMultiAZ<br>          - DBStorageType<br><br>      - Label:<br>          default: Ldap Configuration<br>        Parameters:<br>          - LdapDomainName<br>          - LdapAdmin<br>          - LdapAdminPassword<br>          - LdapUser<br>          - LdapUserPassword<br><br>      - Label:<br>          default: Redis Configuration<br>        Parameters:<br>          - RedisPassword<br><br>      - Label:<br>          default: Others<br>        Parameters:<br>          - RangerPassword<br><br>    ParameterLabels:<br>      # DEP Basic Configuration<br>      EnvironmentName:<br>        default: Set your Dep application name.<br>      # Cli username<br>      CliUsername:<br>        default: The aws username to grant eks cli access to.<br><br>      # VPC Network Configuration<br>      VpcCIDR:<br>        default: VPC CIDR.<br>      PrivateSubnet1CIDR:<br>        default: Private subnet 1 CIDR.<br>      PrivateSubnet2CIDR:<br>        default: Private subnet 2 CIDR.<br>      PublicSubnet1CIDR:<br>        default: Public subnet 1 CIDR.<br>      PublicSubnet2CIDR:<br>        default: Public subnet 2 CIDR.<br>      RemoteAccessCIDR:<br>        default: Allowed WebServer external access CIDR.<br>      AccessCIDR:<br>        default: Permitted IP range.<br><br>      # S3 Configuration<br>      S3BucketName:<br>        default: S3 bucket name for Hive data storage.<br><br>      # MySQL Configuration<br>      DBAllocatedStorage:<br>        default: The size of the database in gigabytes (GiB).<br>      DBAutoMinorVersionUpgrade:<br>        default: Select true/false to setup Auto Minor Version upgrade.<br>      DBBackupRetentionPeriod:<br>        default: The number of days for which automatic DB snapshots are retained.<br>      DBInstanceClass:<br>        default: The name of the compute and memory capacity class of the Amazon mysql DB instance.<br>      DBIops:<br>        default: DB Iops. Used only when io1 specified for the StorageType property.<br>      DBMasterUsername:<br>        default: DB admin username.<br>      DBMasterUserPassword:<br>        default: DB admin password.<br>      DBMultiAZ:<br>        default: Specifies if the database instance is a multiple Availability Zone deployment.<br>      DBStorageType:<br>        default: The storage type associated with this database instance.<br><br>      # Ldap Configuration<br>      LdapDomainName:<br>        default: Ldap domain name.<br>      LdapAdmin:<br>        default: Ldap admin username.<br>      LdapAdminPassword:<br>        default: Ldap admin password.<br>      LdapUser:<br>        default: Ldap user to grant admin access to.<br>      LdapUserPassword:<br>        default: Password for Ldap user to grant admin access to.<br><br>      # Redis Configuration<br>      RedisPassword:<br>        default: Redis root password.<br><br>      RangerPassword:<br>        default: Admin password for ranger.<br><br>Parameters:<br>  # DEP Basic Configuration<br>  EnvironmentName:<br>    Description: Set your dep application name.<br>    Default: dep<br>    Type: String<br><br>  CliUsername:<br>    Description: The aws username to grant eks cli access to.<br>    Type: String<br><br>  # VPC Network Configuration<br>  VpcCIDR:<br>    AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/([0-9]|[1-2][0-9]|3[0-2]))$<br>    ConstraintDescription: Must be a valid IP range in x.x.x.x/x notation<br>    Default: 10.99.0.0/16<br>    Description: CIDR Block for the VPC.<br>    Type: String<br>  PrivateSubnet1CIDR:<br>    AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/([0-9]|[1-2][0-9]|3[0-2]))$<br>    ConstraintDescription: Must be a valid IP range in x.x.x.x/x notation<br>    Default: 10.99.128.0/20<br>    Description: CIDR block for private subnet 1 located in Availability Zone 1.<br>    Type: String<br>  PrivateSubnet2CIDR:<br>    AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/([0-9]|[1-2][0-9]|3[0-2]))$<br>    ConstraintDescription: Must be a valid IP range in x.x.x.x/x notation<br>    Default: 10.99.144.0/20<br>    Description: CIDR block for private subnet 2 located in Availability Zone 2.<br>    Type: String<br>  PublicSubnet1CIDR:<br>    AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/([0-9]|[1-2][0-9]|3[0-2]))$<br>    ConstraintDescription: Must be a valid IP range in x.x.x.x/x notation<br>    Default: 10.99.0.0/20<br>    Description: CIDR Block for the public DMZ subnet 1 located in Availability Zone 1.<br>    Type: String<br>  PublicSubnet2CIDR:<br>    AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/([0-9]|[1-2][0-9]|3[0-2]))$<br>    ConstraintDescription: Must be a valid IP range in x.x.x.x/x notation<br>    Default: 10.99.16.0/20<br>    Description: CIDR Block for the public DMZ subnet 2 located in Availability Zone 2.<br>    Type: String<br>  RemoteAccessCIDR:<br>    AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/([0-9]|[1-2][0-9]|3[0-2]))$<br>    ConstraintDescription: CIDR block parameter must be in the form x.x.x.x/x<br>    Default: 0.0.0.0/0<br>    Description: Allowed CIDR block for external SSH access to the WebServers.<br>    Type: String<br>  AccessCIDR:<br>    AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/([0-9]|[1-2][0-9]|3[0-2]))$<br>    ConstraintDescription: Must be a valid IP range in x.x.x.x/x notation<br>    Default: 0.0.0.0/0<br>    Description: &#39;The CIDR IP range that is permitted to access odoo instances.<br>      Note: a value of 0.0.0.0/0 will allow access from ANY ip address.&#39;<br>    Type: String<br><br>  # S3 Configuration<br>  S3BucketName:<br>    Default: &#39;&#39;<br>    Description: S3 bucket name for Hive data storage. By default &#39;dep-{AccountId}-{Region}&#39;<br>    Type: String<br><br>  # MySQL Configuration<br>  DBAllocatedStorage:<br>    ConstraintDescription: must be between 5 and 4096 GiB. If Iops specified, AllocatedStorage<br>      must be at least 100 GiB and with minimum Iops value of 1000<br>    Default: &#39;50&#39;<br>    Description: The size of the database in gigabytes (GiB)<br>    MaxValue: &#39;4096&#39;<br>    MinValue: &#39;5&#39;<br>    Type: Number<br>  DBAutoMinorVersionUpgrade:<br>    AllowedValues:<br>      - &#39;true&#39;<br>      - &#39;false&#39;<br>    Default: &#39;false&#39;<br>    Description: Select true/false to setup Auto Minor Version upgrade<br>    Type: String<br>  DBBackupRetentionPeriod:<br>    Default: &#39;7&#39;<br>    Description: The number of days for which automatic DB snapshots are retained.<br>    Type: String<br>  DBInstanceClass:<br>    AllowedValues:<br>      - db.m6g.large<br>      - db.m6g.xlarge<br>      - db.m6g.2xlarge<br>      - db.m6g.4xlarge<br>      - db.m6g.8xlarge<br>      - db.m6g.12xlarge<br>      - db.m6g.16xlarge<br>      - db.m5.large<br>      - db.m5.xlarge<br>      - db.m5.2xlarge<br>      - db.m5.8xlarge<br>    ConstraintDescription: Must select a valid database instance type.<br>    Default: db.m5.large<br>    Description: The name of the compute and memory capacity class of the Amazon mysql DB instance.<br>    Type: String<br>  DBIops:<br>    AllowedValues:<br>      - &#39;1000&#39;<br>      - &#39;2000&#39;<br>      - &#39;3000&#39;<br>      - &#39;4000&#39;<br>      - &#39;5000&#39;<br>      - &#39;6000&#39;<br>      - &#39;7000&#39;<br>      - &#39;8000&#39;<br>      - &#39;9000&#39;<br>      - &#39;10000&#39;<br>    ConstraintDescription: &#39;1000 Iops min and increased in 1K increments. &#39;<br>    Default: &#39;1000&#39;<br>    Description: DB Iops. Used only when io1 specified for the StorageType property<br>    Type: Number<br>  DBMasterUsername:<br>    Default: depadmin<br>    Type: String<br>  DBMasterUserPassword:<br>    Default: XwgAST0%QsgTx[<br>    Type: String<br>  DBMultiAZ:<br>    AllowedValues:<br>      - &#39;true&#39;<br>      - &#39;false&#39;<br>    Default: &#39;false&#39;<br>    Description: Specifies if the database instance is a multiple Availability Zone<br>      deployment.<br>    Type: String<br>  DBStorageType:<br>    AllowedValues:<br>      - standard<br>      - gp2<br>      - io1<br>    Default: standard<br>    Description: The storage type associated with this database instance<br>    Type: String<br><br>  LdapDomainName:<br>    Description: Ldap domain name.<br>    Type: String<br>    Default: &#39;dep&#39;<br>  LdapAdmin:<br>    Description: Ldap admin username.<br>    Type: String<br>    Default: &#39;admin&#39;<br>  LdapAdminPassword:<br>    Description: Ldap admin password.<br>    Type: String<br>    Default: &#39;admin123456&#39;<br>  LdapUser:<br>    Description: Ldap user to grant admin access to.<br>    Type: String<br>    Default: &#39;depadmin&#39;<br>  LdapUserPassword:<br>    Description: Password for Ldap user to grant admin access to.<br>    Type: String<br>    Default: &#39;dep123456&#39;<br><br>  RedisPassword:<br>    Description: Redis root password.<br>    Type: String<br>    Default: &#39;depredis123456&#39;<br><br>  RangerPassword:<br>    Description: Admin password for ranger.<br>    Type: String<br>    Default: &#39;dqwW988Ca012n4!&#39;</pre><p>DEP Basic Configuration</p><ul><li>EnvironmentName: This parameter allows you to set a unique name for your DEP application, helping identify resources related to your deployment.</li><li>CliUsername : The aws cli username to grant EKS access to. This allows you to execute eksctl &amp; kubectl commands in your command line tools.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q0EbVAGkeHGMCbvOmg18Yg.png" /><figcaption>Example of CliUsername</figcaption></figure><p>VPC Network Configuration</p><ul><li>VpcCIDR: Defines the IP range for your VPC, creating an isolated network space for your resources.</li><li>PrivateSubnet1CIDR &amp; PrivateSubnet2CIDR: These specify the IP ranges for private subnets where your EKS and database resources will reside, ensuring secure internal access.</li><li>PublicSubnet1CIDR &amp; PublicSubnet2CIDR: These are for public subnets, typically used for resources that need to be accessed over the internet, like load balancers.</li><li>RemoteAccessCIDR: This controls which external IP ranges are allowed SSH access to the servers, enhancing security by restricting access.</li><li>AccessCIDR: Specifies the range of IP addresses that are permitted to access the instances, a critical setting for controlling access to your application.</li></ul><p>S3 Configuration</p><ul><li>S3BucketName : S3 bucket name for Hive data storage. By default ‘dep-{AccountId}-{Region}’.</li></ul><p>MySQL Configuration</p><ul><li>DBAllocatedStorage: Determines the initial storage capacity allocated for your database.</li><li>DBAutoMinorVersionUpgrade: Allows you to enable automatic updates to newer minor versions for your MySQL database.</li><li>DBBackupRetentionPeriod: Sets how many days your automated database backups are retained.</li><li>DBInstanceClass: Selects the class of your database instance, balancing compute and memory capacity according to your workload.</li><li>DBIops: Specifies the provisioned IOPS for the database, enhancing performance for I/O-intensive applications.</li><li>DBMasterUsername &amp; DBMasterUserPassword: These parameters are for setting the administrative credentials for your MySQL database.</li><li>DBMultiAZ: Enables deployment across multiple availability zones for high availability.</li><li>DBStorageType: Lets you choose the type of storage used by the database instance.</li></ul><p>Ldap Configuration</p><ul><li>LdapDomainName: Sets the domain name for the LDAP service.</li><li>LdapAdmin &amp; LdapAdminPassword: These are the credentials for the LDAP administrator, crucial for setting up and managing the LDAP service.</li><li>LdapUser &amp; LdapUserPassword : These are the credentials for the default LDAP users with admin access.</li></ul><p>Redis Configuration</p><ul><li>RedisPassword : Root password for Redis service.</li></ul><p>Ranger Configuration</p><ul><li>RangerPassword : Admin password for Ranger admin service.</li></ul><h4>5.3 Create VPC</h4><p>This will automate the creation of a network architecture that adheres to AWS best practices for high availability and security.</p><pre>## dep-public/series-1/dep.template<br><br>Resources:<br>  DepVpcStack:<br>    Type: AWS::CloudFormation::Stack<br>    Properties:<br>      TemplateURL: &#39;https://dep-public.s3.ap-northeast-1.amazonaws.com/series-1/dep-vpc.template&#39;<br>      Parameters:<br>        EnvironmentName: !Ref EnvironmentName<br>        PrivateSubnet1CIDR: !Ref PrivateSubnet1CIDR<br>        PrivateSubnet2CIDR: !Ref PrivateSubnet2CIDR<br>        PublicSubnet1CIDR: !Ref PublicSubnet1CIDR<br>        PublicSubnet2CIDR: !Ref PublicSubnet2CIDR<br>        VpcCIDR: !Ref VpcCIDR</pre><pre>## dep-public/series-1/dep-vpc.template<br><br>AWSTemplateFormatVersion: 2010-09-09<br>Description:  This template deploys a VPC, with a pair of public and private subnets spread<br>  across two Availability Zones. It deploys an internet gateway, with a default<br>  route on the public subnets. It deploys a pair of NAT gateways (one in each AZ),<br>  and default routes for them in the private subnets.<br><br>Parameters:<br>  EnvironmentName:<br>    Description: An environment name that is prefixed to resource names<br>    Type: String<br><br>  VpcCIDR:<br>    Description: Please enter the IP range (CIDR notation) for this VPC<br>    Type: String<br>    Default: 10.192.0.0/16<br><br>  PublicSubnet1CIDR:<br>    Description: Please enter the IP range (CIDR notation) for the public subnet in the first Availability Zone<br>    Type: String<br>    Default: 10.192.10.0/24<br><br>  PublicSubnet2CIDR:<br>    Description: Please enter the IP range (CIDR notation) for the public subnet in the second Availability Zone<br>    Type: String<br>    Default: 10.192.11.0/24<br><br>  PrivateSubnet1CIDR:<br>    Description: Please enter the IP range (CIDR notation) for the private subnet in the first Availability Zone<br>    Type: String<br>    Default: 10.192.20.0/24<br><br>  PrivateSubnet2CIDR:<br>    Description: Please enter the IP range (CIDR notation) for the private subnet in the second Availability Zone<br>    Type: String<br>    Default: 10.192.21.0/24<br><br>Resources:<br>  VPC:<br>    Type: AWS::EC2::VPC<br>    Properties:<br>      CidrBlock: !Ref VpcCIDR<br>      EnableDnsSupport: true<br>      EnableDnsHostnames: true<br>      Tags:<br>        - Key: Name<br>          Value: !Join<br>            - &#39;-&#39;<br>            - - !Ref EnvironmentName<br>              - vpc<br><br>  InternetGateway:<br>    Type: AWS::EC2::InternetGateway<br>    Properties:<br>      Tags:<br>        - Key: Name<br>          Value: !Join<br>            - &#39;-&#39;<br>            - - !Ref EnvironmentName<br>              - igw<br><br>  InternetGatewayAttachment:<br>    Type: AWS::EC2::VPCGatewayAttachment<br>    Properties:<br>      InternetGatewayId: !Ref InternetGateway<br>      VpcId: !Ref VPC<br><br>  PublicSubnet1:<br>    Type: AWS::EC2::Subnet<br>    Properties:<br>      VpcId: !Ref VPC<br>      AvailabilityZone: !Select [ 0, !GetAZs &#39;&#39; ]<br>      CidrBlock: !Ref PublicSubnet1CIDR<br>      MapPublicIpOnLaunch: true<br>      Tags:<br>        - Key: Name<br>          Value: !Join<br>            - &#39;-&#39;<br>            - - !Ref EnvironmentName<br>              - public-subnet-1<br>        - Key: kubernetes.io/role/elb<br>          Value: 1<br><br>  PublicSubnet2:<br>    Type: AWS::EC2::Subnet<br>    Properties:<br>      VpcId: !Ref VPC<br>      AvailabilityZone: !Select [ 1, !GetAZs  &#39;&#39; ]<br>      CidrBlock: !Ref PublicSubnet2CIDR<br>      MapPublicIpOnLaunch: true<br>      Tags:<br>        - Key: Name<br>          Value: !Join<br>            - &#39;-&#39;<br>            - - !Ref EnvironmentName<br>              - public-subnet-2<br>        - Key: kubernetes.io/role/elb<br>          Value: 1<br><br>  PrivateSubnet1:<br>    Type: AWS::EC2::Subnet<br>    Properties:<br>      VpcId: !Ref VPC<br>      AvailabilityZone: !Select [ 0, !GetAZs  &#39;&#39; ]<br>      CidrBlock: !Ref PrivateSubnet1CIDR<br>      MapPublicIpOnLaunch: false<br>      Tags:<br>        - Key: Name<br>          Value: !Join<br>            - &#39;-&#39;<br>            - - !Ref EnvironmentName<br>              - private-subnet-1<br>        - Key: kubernetes.io/role/internal-elb<br>          Value: 1<br><br>  PrivateSubnet2:<br>    Type: AWS::EC2::Subnet<br>    Properties:<br>      VpcId: !Ref VPC<br>      AvailabilityZone: !Select [ 1, !GetAZs  &#39;&#39; ]<br>      CidrBlock: !Ref PrivateSubnet2CIDR<br>      MapPublicIpOnLaunch: false<br>      Tags:<br>        - Key: Name<br>          Value: !Join<br>            - &#39;-&#39;<br>            - - !Ref EnvironmentName<br>              - private-subnet-2<br>        - Key: kubernetes.io/role/internal-elb<br>          Value: 1<br><br>  NatGateway1EIP:<br>    Type: AWS::EC2::EIP<br>    DependsOn: InternetGatewayAttachment<br>    Properties:<br>      Domain: vpc<br><br>  NatGateway2EIP:<br>    Type: AWS::EC2::EIP<br>    DependsOn: InternetGatewayAttachment<br>    Properties:<br>      Domain: vpc<br><br>  NatGateway1:<br>    Type: AWS::EC2::NatGateway<br>    Properties:<br>      AllocationId: !GetAtt NatGateway1EIP.AllocationId<br>      SubnetId: !Ref PublicSubnet1<br><br>  NatGateway2:<br>    Type: AWS::EC2::NatGateway<br>    Properties:<br>      AllocationId: !GetAtt NatGateway2EIP.AllocationId<br>      SubnetId: !Ref PublicSubnet2<br><br>  PublicRouteTable:<br>    Type: AWS::EC2::RouteTable<br>    Properties:<br>      VpcId: !Ref VPC<br>      Tags:<br>        - Key: Name<br>          Value: !Join<br>            - &#39;-&#39;<br>            - - !Ref EnvironmentName<br>              - public-route-table<br><br>  DefaultPublicRoute:<br>    Type: AWS::EC2::Route<br>    DependsOn: InternetGatewayAttachment<br>    Properties:<br>      RouteTableId: !Ref PublicRouteTable<br>      DestinationCidrBlock: 0.0.0.0/0<br>      GatewayId: !Ref InternetGateway<br><br>  PublicSubnet1RouteTableAssociation:<br>    Type: AWS::EC2::SubnetRouteTableAssociation<br>    Properties:<br>      RouteTableId: !Ref PublicRouteTable<br>      SubnetId: !Ref PublicSubnet1<br><br>  PublicSubnet2RouteTableAssociation:<br>    Type: AWS::EC2::SubnetRouteTableAssociation<br>    Properties:<br>      RouteTableId: !Ref PublicRouteTable<br>      SubnetId: !Ref PublicSubnet2<br><br>  PrivateRouteTable1:<br>    Type: AWS::EC2::RouteTable<br>    Properties:<br>      VpcId: !Ref VPC<br>      Tags:<br>        - Key: Name<br>          Value: !Join<br>            - &#39;-&#39;<br>            - - !Ref EnvironmentName<br>              - private-route-table-1<br><br>  DefaultPrivateRoute1:<br>    Type: AWS::EC2::Route<br>    Properties:<br>      RouteTableId: !Ref PrivateRouteTable1<br>      DestinationCidrBlock: 0.0.0.0/0<br>      NatGatewayId: !Ref NatGateway1<br><br>  PrivateSubnet1RouteTableAssociation:<br>    Type: AWS::EC2::SubnetRouteTableAssociation<br>    Properties:<br>      RouteTableId: !Ref PrivateRouteTable1<br>      SubnetId: !Ref PrivateSubnet1<br><br>  PrivateRouteTable2:<br>    Type: AWS::EC2::RouteTable<br>    Properties:<br>      VpcId: !Ref VPC<br>      Tags:<br>        - Key: Name<br>          Value: !Join<br>            - &#39;-&#39;<br>            - - !Ref EnvironmentName<br>              - private-route-table-2<br><br>  DefaultPrivateRoute2:<br>    Type: AWS::EC2::Route<br>    Properties:<br>      RouteTableId: !Ref PrivateRouteTable2<br>      DestinationCidrBlock: 0.0.0.0/0<br>      NatGatewayId: !Ref NatGateway2<br><br>  PrivateSubnet2RouteTableAssociation:<br>    Type: AWS::EC2::SubnetRouteTableAssociation<br>    Properties:<br>      RouteTableId: !Ref PrivateRouteTable2<br>      SubnetId: !Ref PrivateSubnet2<br><br>Outputs:<br>  VPC:<br>    Description: A reference to the created VPC.<br>    Value: !Ref VPC<br><br>  PublicSubnets:<br>    Description: A list of the public subnets.<br>    Value: !Join [ &quot;,&quot;, [ !Ref PublicSubnet1, !Ref PublicSubnet2 ]]<br><br>  PrivateSubnets:<br>    Description: A list of the private subnets.<br>    Value: !Join [ &quot;,&quot;, [ !Ref PrivateSubnet1, !Ref PrivateSubnet2 ]]<br><br>  PublicSubnet1:<br>    Description: A reference to the public subnet in the 1st Availability Zone.<br>    Value: !Ref PublicSubnet1<br><br>  PublicSubnet2:<br>    Description: A reference to the public subnet in the 2nd Availability Zone.<br>    Value: !Ref PublicSubnet2<br><br>  PrivateSubnet1:<br>    Description: A reference to the private subnet in the 1st Availability Zone.<br>    Value: !Ref PrivateSubnet1<br><br>  PrivateSubnet2:<br>    Description: A reference to the private subnet in the 2nd Availability Zone.<br>    Value: !Ref PrivateSubnet2<br><br>  CidrBlock:<br>    Description: The primary IPv4 CIDR block for the VPC. For example, 10.0.0.0/16.<br>    Value: !GetAtt VPC.CidrBlock</pre><h4>5.4 Create security groups</h4><p>This template sets up the necessary network security parameters for a MySQL database and an EKS cluster in AWS, ensuring that the appropriate ports are open and traffic is allowed from and to the specified IP ranges. It’s a crucial step in setting up a secure and functional environment for applications that require database and Kubernetes cluster services.</p><pre>## dep-public/series-1/dep.template<br><br>Resources:<br>  DepSecurityGroupStack:<br>    Type: AWS::CloudFormation::Stack<br>    DependsOn:<br>      - DepVpcStack<br>    Properties:<br>      TemplateURL: &#39;https://dep-public.s3.ap-northeast-1.amazonaws.com/series-1/dep-securitygroup.template&#39;<br>      Parameters:<br>        EnvironmentName: !Ref EnvironmentName<br>        AccessCIDR: !GetAtt DepVpcStack.Outputs.CidrBlock<br>        VPCID: !GetAtt DepVpcStack.Outputs.VPC<br>        VPCCIDR: !GetAtt DepVpcStack.Outputs.CidrBlock</pre><pre>## dep-public/series-1/dep-securitygroup.template<br><br>AWSTemplateFormatVersion: 2010-09-09<br>Description: Dep Security Groups template<br>Parameters:<br>  EnvironmentName:<br>    Description: An environment name that is prefixed to resource names<br>    Type: String<br>  AccessCIDR:<br>    AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/([0-9]|[1-2][0-9]|3[0-2]))$<br>    ConstraintDescription: Must be a valid IP range in x.x.x.x/x notation<br>    Description: &#39;The CIDR IP range that is permitted to access. Note: a value of<br>      0.0.0.0/0 will allow access from ANY ip address&#39;<br>    Type: String<br>  VPCID:<br>    Description: VPC ID of your existing Virtual Private Cloud (VPC) where you want<br>      to depoy RDS.<br>    Type: AWS::EC2::VPC::Id<br>  VPCCIDR:<br>    AllowedPattern: ^(([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])\.){3}([0-9]|[1-9][0-9]|1[0-9]{2}|2[0-4][0-9]|25[0-5])(\/([0-9]|[1-2][0-9]|3[0-2]))$<br>    ConstraintDescription: Must be a valid IP range in x.x.x.x/x notation<br>    Description: The CIDR block for VPC<br>    Type: String<br><br>Resources:<br>  MySQLRDSSecurityGroup:<br>    Type: AWS::EC2::SecurityGroup<br>    Properties:<br>      GroupName: !Sub ${EnvironmentName}-mysql-sg<br>      GroupDescription: Allow access to MySQL Port<br>      VpcId: !Ref VPCID<br>      SecurityGroupIngress:<br>        - IpProtocol: tcp<br>          FromPort: 3306<br>          ToPort: 3306<br>          CidrIp: !Ref VPCCIDR<br>      SecurityGroupEgress:<br>        - IpProtocol: tcp<br>          FromPort: 80<br>          ToPort: 80<br>          CidrIp: 0.0.0.0/0<br>        - IpProtocol: tcp<br>          FromPort: 443<br>          ToPort: 443<br>          CidrIp: 0.0.0.0/0<br><br>  EKSSecurityGroup:<br>    Type: AWS::EC2::SecurityGroup<br>    Properties:<br>      GroupName: !Sub ${EnvironmentName}-eks-sg<br>      GroupDescription: EKS security group<br>      VpcId: !Ref VPCID<br>      SecurityGroupIngress:<br>        - IpProtocol: tcp<br>          FromPort: 0<br>          ToPort: 65535<br>          CidrIp: !Ref AccessCIDR<br>      SecurityGroupEgress:<br>        - IpProtocol: tcp<br>          FromPort: 0<br>          ToPort: 65535<br>          CidrIp: 0.0.0.0/0<br>      Tags:<br>        - Key: !Sub kubernetes.io/cluster/${EnvironmentName}-eks-cluster<br>          Value: shared<br><br>Outputs:<br>  MySQLRDSSecurityGroup:<br>    Description: MySQL Security Group<br>    Value: !Ref MySQLRDSSecurityGroup<br>  EKSSecurityGroup:<br>    Description: EKS Security Group<br>    Value: !Ref EKSSecurityGroup</pre><h4>5.5 Create S3 Bucket</h4><p>This template automates the setup of an S3 bucket for the Data Engineering Platform (DEP) with policies that help manage the storage costs effectively.</p><blockquote>*Note:</blockquote><blockquote>1. Before you clean up cloudformation resources, you should manually delete contents in previously created S3 bucket to avoid unsuccessful deletion. <br>2. Use this demo only for demo.</blockquote><pre>## dep-public/series-1/dep.template<br><br>Conditions:<br>  CustomS3BucketName: !Equals [&#39;&#39;, !Ref S3BucketName]<br><br>Resources:<br>  DepS3Stack:<br>    Type: AWS::CloudFormation::Stack<br>    Properties:<br>      TemplateURL: &#39;https://dep-public.s3.ap-northeast-1.amazonaws.com/series-1/dep-s3.template&#39;<br>      Parameters:<br>        BucketName: !If [CustomS3BucketName, !Ref S3BucketName, !Join [&#39;-&#39;, [!Ref EnvironmentName, !Ref AWS::AccountId, !Ref AWS::Region]]]</pre><pre>## dep-public/series-1/dep-s3.template<br><br>AWSTemplateFormatVersion: 2010-09-09<br>Description: &#39;Create Dep S3 Bucket&#39;<br>Parameters:<br>  # S3 Configuration<br>  BucketName:<br>    Type: String<br>Resources:<br>  DepS3:<br>    Type: &#39;AWS::S3::Bucket&#39;<br>    #    DeletionPolicy: Retain<br>    Properties:<br>      BucketName: !Ref BucketName<br>      LifecycleConfiguration:<br>        Rules:<br>          - Id: IntelligentTieringTransition<br>            Status: Enabled<br>            Transitions:<br>              - TransitionInDays: 7<br>                StorageClass: INTELLIGENT_TIERING<br>          - Id: NoncurrentVersionExpirationInDays<br>            Status: Enabled<br>            NoncurrentVersionExpirationInDays: 1<br>Outputs:<br>  BucketName:<br>    Description: S3 Bucket Name<br>    Value: !Ref DepS3</pre><h4>5.6 Create IAM roles and assign policies</h4><p>This template facilitates the creation of specialized IAM roles that are integral to managing and operating a DEP environment on AWS, ensuring that each service and node group within EKS has the appropriate permissions for operation and access to necessary resources.</p><ul><li>The DepIamRole is suited for administrative tasks that require broad permissions across multiple AWS services, including EKS management, Lambda functions, API Gateway, CloudFormation, and S3.</li><li>The DepEksWorkerIamRole is tailored for EKS worker nodes, granting permissions necessary for EKS operations, access to the ECR, and S3 bucket interaction.</li></ul><pre>## dep-public/series-1/dep.template<br><br>Resources:<br>  DepIamStack:<br>    Type: AWS::CloudFormation::Stack<br>    DependsOn:<br>      - DepS3Stack<br>    Properties:<br>      TemplateURL: &#39;https://dep-public.s3.ap-northeast-1.amazonaws.com/series-1/dep-iam.template&#39;<br>      Parameters:<br>        EnvironmentName: !Ref EnvironmentName<br>        S3BucketName: !GetAtt DepS3Stack.Outputs.BucketName</pre><pre>## dep-public/series-1/dep-iam.template<br><br>AWSTemplateFormatVersion: 2010-09-09<br>Description: Dep IAM Role<br>Parameters:<br>  EnvironmentName:<br>    Description: An environment name that is prefixed to resource names<br>    Type: String<br>  S3BucketName:<br>    Description: S3 bucket name to grant access to.<br>    Type: String<br><br>Resources:<br>  DepIamRole:<br>    Type: AWS::IAM::Role<br>    Properties:<br>      Description: &quot;IAM role created for managing dep resources&quot;<br>      AssumeRolePolicyDocument:<br>        Version: 2012-10-17<br>        Statement:<br>          - Effect: Allow<br>            Principal:<br>              Service:<br>                - eks.amazonaws.com<br>                - resources.cloudformation.amazonaws.com<br>                - cloudformation.amazonaws.com<br>                - lambda.amazonaws.com<br>                - apigateway.amazonaws.com<br>                - events.amazonaws.com<br>            Action:<br>              - &quot;sts:AssumeRole&quot;<br>      Path: &quot;/&quot;<br>      RoleName: !Join<br>        - &quot;-&quot;<br>        - - !Ref EnvironmentName<br>          - &quot;iam-role&quot;<br>      ManagedPolicyArns:<br>        - arn:aws:iam::aws:policy/AmazonEKSClusterPolicy<br>        - arn:aws:iam::aws:policy/AmazonEKSServicePolicy<br>        - arn:aws:iam::aws:policy/SecretsManagerReadWrite<br>        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole<br>        - arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy<br>      Policies:<br>        - PolicyName: DEPS3AccessPolicy<br>          PolicyDocument:<br>            Version: 2012-10-17<br>            Statement:<br>              - Action:<br>                  - s3:*<br>                Effect: Allow<br>                Resource:<br>                  - !Join<br>                    - &quot;&quot;<br>                    - - &quot;arn:aws:s3:::&quot;<br>                      - !Ref S3BucketName<br>                  - !Join<br>                    - &quot;&quot;<br>                    - - &quot;arn:aws:s3:::&quot;<br>                      - !Ref S3BucketName<br>                      - &quot;/*&quot;<br>        - PolicyName: ResourceTypePolicy<br>          PolicyDocument:<br>            Version: &quot;2012-10-17&quot;<br>            Statement:<br>              - Effect: Allow<br>                Action:<br>                  - &quot;sts:GetCallerIdentity&quot;<br>                  - &quot;eks:CreateCluster&quot;<br>                  - &quot;eks:DeleteCluster&quot;<br>                  - &quot;eks:DescribeCluster&quot;<br>                  - &quot;eks:ListTagsForResource&quot;<br>                  - &quot;eks:UpdateClusterVersion&quot;<br>                  - &quot;eks:UpdateClusterConfig&quot;<br>                  - &quot;eks:TagResource&quot;<br>                  - &quot;eks:UntagResource&quot;<br>                  - &quot;iam:PassRole&quot;<br>                  - &quot;sts:AssumeRole&quot;<br>                  - &quot;lambda:UpdateFunctionConfiguration&quot;<br>                  - &quot;lambda:DeleteFunction&quot;<br>                  - &quot;lambda:GetFunction&quot;<br>                  - &quot;lambda:InvokeFunction&quot;<br>                  - &quot;lambda:CreateFunction&quot;<br>                  - &quot;lambda:UpdateFunctionCode&quot;<br>                  - &quot;ec2:DescribeVpcs&quot;<br>                  - &quot;ec2:DescribeSubnets&quot;<br>                  - &quot;ec2:DescribeSecurityGroups&quot;<br>                  - &quot;kms:CreateGrant&quot;<br>                  - &quot;kms:DescribeKey&quot;<br>                  - &quot;logs:CreateLogGroup&quot;<br>                  - &quot;logs:CreateLogStream&quot;<br>                  - &quot;logs:DescribeLogGroups&quot;<br>                  - &quot;logs:DescribeLogStreams&quot;<br>                  - &quot;logs:PutLogEvents&quot;<br>                  - &quot;cloudwatch:ListMetrics&quot;<br>                  - &quot;cloudwatch:PutMetricData&quot;<br>                Resource: &quot;*&quot;<br>        - PolicyName: EbsCsi<br>          PolicyDocument:<br>            Version: 2012-10-17<br>            Statement:<br>              - Action:<br>                  - &quot;kms:Decrypt&quot;<br>                  - &quot;kms:GenerateDataKeyWithoutPlaintext&quot;<br>                  - &quot;kms:CreateGrant&quot;<br>                Effect: Allow<br>                Resource: &quot;*&quot;<br>        - PolicyName: EksAutoScaling<br>          PolicyDocument:<br>            Version: 2012-10-17<br>            Statement:<br>              - Action:<br>                  - &quot;autoscaling:DescribeAutoScalingGroups&quot;<br>                  - &quot;autoscaling:DescribeAutoScalingInstances&quot;<br>                  - &quot;autoscaling:DescribeLaunchConfigurations&quot;<br>                  - &quot;autoscaling:DescribeTags&quot;<br>                  - &quot;autoscaling:SetDesiredCapacity&quot;<br>                  - &quot;autoscaling:TerminateInstanceInAutoScalingGroup&quot;<br>                  - &quot;ec2:DescribeLaunchTemplateVersions&quot;<br>                Effect: Allow<br>                Resource: &quot;*&quot;<br><br>  DepEksWorkerIamRole:<br>    Type: AWS::IAM::Role<br>    Properties:<br>      Description: &quot;IAM role created for managing eks node groups&quot;<br>      AssumeRolePolicyDocument:<br>        Version: 2012-10-17<br>        Statement:<br>          - Effect: Allow<br>            Principal:<br>              Service:<br>                - eks.amazonaws.com<br>                - resources.cloudformation.amazonaws.com<br>                - cloudformation.amazonaws.com<br>                - lambda.amazonaws.com<br>                - apigateway.amazonaws.com<br>                - events.amazonaws.com<br>                - ec2.amazonaws.com<br>            Action:<br>              - &quot;sts:AssumeRole&quot;<br>      RoleName: !Join<br>        - &quot;-&quot;<br>        - - !Ref EnvironmentName<br>          - &quot;worker-role&quot;<br>      ManagedPolicyArns:<br>        - arn:aws:iam::aws:policy/AmazonEKSWorkerNodePolicy<br>        - arn:aws:iam::aws:policy/AmazonEKS_CNI_Policy<br>        - arn:aws:iam::aws:policy/AmazonEC2ContainerRegistryReadOnly<br>        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole<br>        - arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy<br>        - arn:aws:iam::aws:policy/SecretsManagerReadWrite<br>      Policies:<br>        - PolicyName: DEPS3AccessPolicy<br>          PolicyDocument:<br>            Version: 2012-10-17<br>            Statement:<br>              - Action:<br>                  - s3:*<br>                Effect: Allow<br>                Resource:<br>                  - !Join<br>                    - &quot;&quot;<br>                    - - &quot;arn:aws:s3:::&quot;<br>                      - !Ref S3BucketName<br>                  - !Join<br>                    - &quot;&quot;<br>                    - - &quot;arn:aws:s3:::&quot;<br>                      - !Ref S3BucketName<br>                      - &quot;/*&quot;<br>        - PolicyName: ResourceTypePolicy<br>          PolicyDocument:<br>            Version: 2012-10-17<br>            Statement:<br>              - Action:<br>                  - &quot;eks:CreateCluster&quot;<br>                  - &quot;eks:DeleteCluster&quot;<br>                  - &quot;iam:PassRole&quot;<br>                  - &quot;sts:AssumeRole&quot;<br>                  - &quot;lambda:InvokeFunction&quot;<br>                  - &quot;lambda:CreateFunction&quot;<br>                Effect: Allow<br>                Resource: !Sub arn:aws:eks:${AWS::Region}:${AWS::AccountId}:cluster/${EnvironmentName}-eks-cluster/*<br>        - PolicyName: EbsCsi<br>          PolicyDocument:<br>            Version: 2012-10-17<br>            Statement:<br>              - Action:<br>                  - &quot;kms:Decrypt&quot;<br>                  - &quot;kms:GenerateDataKeyWithoutPlaintext&quot;<br>                  - &quot;kms:CreateGrant&quot;<br>                Effect: Allow<br>                Resource: &quot;*&quot;<br>        - PolicyName: AutoScaler<br>          PolicyDocument:<br>            Version: 2012-10-17<br>            Statement:<br>              - Action:<br>                  - &quot;autoscaling:DescribeAutoScalingGroups&quot;<br>                  - &quot;autoscaling:DescribeAutoScalingInstances&quot;<br>                  - &quot;autoscaling:DescribeLaunchConfigurations&quot;<br>                  - &quot;autoscaling:DescribeTags&quot;<br>                  - &quot;autoscaling:SetDesiredCapacity&quot;<br>                  - &quot;autoscaling:TerminateInstanceInAutoScalingGroup&quot;<br>                  - &quot;eks:DescribeNodegroup&quot;<br>                Effect: Allow<br>                Resource: &quot;*&quot;<br><br>  DepLambdaIamRole:<br>    Type: AWS::IAM::Role<br>    Properties:<br>      Description: &quot;IAM role created for lambda execution&quot;<br>      RoleName: !Join<br>        - &quot;-&quot;<br>        - - !Ref EnvironmentName<br>          - &quot;lambda-role&quot;<br>      AssumeRolePolicyDocument:<br>        Version: &#39;2012-10-17&#39;<br>        Statement:<br>          - Effect: Allow<br>            Principal:<br>              Service:<br>                - eks.amazonaws.com<br>                - lambda.amazonaws.com<br>            Action: sts:AssumeRole<br>      Path: &quot;/&quot;<br>      ManagedPolicyArns:<br>        - !Sub &#39;arn:${AWS::Partition}:iam::aws:policy/AmazonEKSClusterPolicy&#39;<br>        - !Sub &#39;arn:${AWS::Partition}:iam::aws:policy/AmazonEKSServicePolicy&#39;<br>        - arn:aws:iam::aws:policy/service-role/AWSLambdaENIManagementAccess<br>        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole<br><br>Outputs:<br>  DepIamRole:<br>    Description: IAM role created for managing dep resources<br>    Value: !GetAtt DepIamRole.Arn<br>  DepEksWorkerIamRole:<br>    Description: IAM role created for managing eks node groups<br>    Value: !GetAtt DepEksWorkerIamRole.Arn<br>  DepLambdaIamRole:<br>    Description: IAM role created for lambda execution<br>    Value: !GetAtt DepLambdaIamRole.Arn</pre><h4>5.7 Create ec2 ssh key</h4><pre>## dep-public/series-1/dep.template<br><br>Resources:  <br>  DepEc2SshKeyPair:<br>    Type: AWS::EC2::KeyPair<br>    Properties:<br>      KeyName: !Sub ${EnvironmentName}-eks-worker-key-pair</pre><h4>5.8 Enable third-party extension for EKS to manage Kubernetes resources</h4><p>This template extends the native functionality of CloudFormation, allowing it to interact with and manage EKS clusters and Kubernetes resources more effectively.</p><p>Reference: <a href="https://aws.amazon.com/cn/quickstart/architecture/cloudformation-resource-providers/">AWS CloudFormation Resource Types for Kubernetes (amazon.com)</a></p><pre>## dep-public/series-1/dep.template<br><br>Resources:<br>  # Enable third-party cloudformation extension to manage eks cluster<br>  DepCloudformationExtensionsStack:<br>    Type: AWS::CloudFormation::Stack<br>    Properties:<br>      TemplateURL: &#39;https://dep-public.s3.ap-northeast-1.amazonaws.com/series-1/dep-extensions.template&#39;</pre><pre>## dep-public/series-1/dep-extensions.template<br><br>AWSTemplateFormatVersion: &#39;2010-09-09&#39;<br>Description: Extensions<br>Resources:<br>  # AWSQS::EKS::Cluster<br>  AWSQSEKSClusterExecutionRole:<br>    Type: AWS::IAM::Role<br>    Properties:<br>      RoleName: !Join<br>        - &#39;-&#39;<br>        - - &#39;AWSQSExecutionIAMRole&#39;<br>          - !Select [2, !Split [ &#39;/&#39;, !Ref AWS::StackId ]]<br>      MaxSessionDuration: 8400<br>      AssumeRolePolicyDocument:<br>        Version: &#39;2012-10-17&#39;<br>        Statement:<br>        - Effect: Allow<br>          Principal:<br>            Service:<br>            - resources.cloudformation.amazonaws.com<br>            - cloudformation.amazonaws.com<br>            - lambda.amazonaws.com<br>          Action: sts:AssumeRole<br>      Path: &#39;/&#39;<br>      Policies:<br>      - PolicyName: !Join<br>          - &#39;-&#39;<br>          - - &#39;AWSQSExecutionIAMRole&#39;<br>            - &#39;policy&#39;<br>            - !Select [2, !Split [ &#39;/&#39;, !Ref AWS::StackId ]]<br>        PolicyDocument:<br>          Version: &#39;2012-10-17&#39;<br>          Statement:<br>          - Effect: Allow<br>            Action:<br>            - sts:GetCallerIdentity<br>            - eks:CreateCluster<br>            - eks:DeleteCluster<br>            - eks:DescribeCluster<br>            - eks:ListTagsForResource<br>            - eks:UpdateClusterVersion<br>            - eks:UpdateClusterConfig<br>            - eks:TagResource<br>            - eks:UntagResource<br>            - lambda:UpdateFunctionConfiguration<br>            - lambda:DeleteFunction<br>            - lambda:GetFunction<br>            - lambda:InvokeFunction<br>            - lambda:CreateFunction<br>            - lambda:UpdateFunctionCode<br>            - ec2:DescribeVpcs<br>            - ec2:DescribeSubnets<br>            - ec2:DescribeSecurityGroups<br>            - kms:CreateGrant<br>            - kms:DescribeKey<br>            Resource:<br>            - &#39;*&#39;<br>          - Effect: Allow<br>            Action:<br>            - iam:PassRole<br>            Resource:<br>            - !Sub arn:aws:iam::${AWS::AccountId}:role/*<br>            Condition:<br>              StringEquals:<br>                iam:PassedToService:<br>                - eks.amazonaws.com<br>                - lambda.amazonaws.com<br>              StringLike:<br>                iam:AssociatedResourceArn:<br>                - !Sub arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:*<br>                - !Sub arn:aws:eks:${AWS::Region}:${AWS::AccountId}:cluster/*<br>      Tags:<br>      - Key: Name<br>        Value: !Join<br>          - &#39;-&#39;<br>          - - &#39;AWSQSExecutionIAMRole&#39;<br>            - !Select [2, !Split [ &#39;/&#39;, !Ref AWS::StackId ]]<br><br>  AWSQSEKSClusterLogDeliveryRole:<br>    Type: AWS::IAM::Role<br>    Properties:<br>      RoleName: !Join<br>        - &#39;-&#39;<br>        - - &#39;AWSQSLogDeliveryIAMRole&#39;<br>          - !Select [2, !Split [ &#39;/&#39;, !Ref AWS::StackId ]]<br>      MaxSessionDuration: 8400<br>      AssumeRolePolicyDocument:<br>        Version: &#39;2012-10-17&#39;<br>        Statement:<br>        - Effect: Allow<br>          Principal:<br>            Service:<br>            - resources.cloudformation.amazonaws.com<br>            - cloudformation.amazonaws.com<br>          Action: sts:AssumeRole<br>      Path: &#39;/&#39;<br>      Policies:<br>      - PolicyName: !Join<br>          - &#39;-&#39;<br>          - - &#39;AWSQSLogDeliveryIAMRole&#39;<br>            - &#39;policy&#39;<br>            - !Select [2, !Split [ &#39;/&#39;, !Ref AWS::StackId ]]<br>        PolicyDocument:<br>          Version: &#39;2012-10-17&#39;<br>          Statement:<br>          - Effect: Allow<br>            Action:<br>            - logs:CreateLogGroup<br>            - logs:CreateLogStream<br>            - logs:DescribeLogGroups<br>            - logs:DescribeLogStreams<br>            - logs:PutLogEvents<br>            - cloudwatch:ListMetrics<br>            - cloudwatch:PutMetricData<br>            Resource:<br>            - &#39;*&#39;<br>      Tags:<br>      - Key: Name<br>        Value: !Join<br>          - &#39;-&#39;<br>          - - &#39;AWSQSLogDeliveryIAMRole&#39;<br>            - !Select [2, !Split [ &#39;/&#39;, !Ref AWS::StackId ]]<br><br>  AWSQSEKSClusterResourceVersion:<br>    Type: AWS::CloudFormation::ResourceVersion<br>    Properties:<br>      TypeName: AWSQS::EKS::Cluster<br>      ExecutionRoleArn: !GetAtt AWSQSEKSClusterExecutionRole.Arn<br>      LoggingConfig:<br>        LogGroupName: !Join<br>          - &#39;-&#39;<br>          - - &#39;/aws/cloudformation/registry/AWSQSEKSClusterLogGroup&#39;<br>            - !Select [2, !Split [ &#39;/&#39;, !Ref AWS::StackId ]]<br>        LogRoleArn: !GetAtt AWSQSEKSClusterLogDeliveryRole.Arn<br>      SchemaHandlerPackage: &#39;https://dep-public.s3.ap-northeast-1.amazonaws.com/extensions/awsqs-eks-cluster.zip&#39;<br><br>  AWSQSEKubernetesHelmVersion:<br>    Type: AWS::CloudFormation::ResourceVersion<br>    Properties:<br>      TypeName: AWSQS::Kubernetes::Helm<br>      ExecutionRoleArn: !GetAtt AWSQSEKSClusterExecutionRole.Arn<br>      LoggingConfig:<br>        LogGroupName: !Join<br>          - &#39;-&#39;<br>          - - &#39;/aws/cloudformation/registry/AWSQSEKSClusterLogGroup&#39;<br>            - !Select [2, !Split [ &#39;/&#39;, !Ref AWS::StackId ]]<br>        LogRoleArn: !GetAtt AWSQSEKSClusterLogDeliveryRole.Arn<br>      SchemaHandlerPackage: &#39;https://dep-public.s3.ap-northeast-1.amazonaws.com/extensions/awsqs-kubernetes-helm.zip&#39;<br><br>  AWSQSEKubernetesResourceVersion:<br>    Type: AWS::CloudFormation::ResourceVersion<br>    Properties:<br>      TypeName: AWSQS::Kubernetes::Resource<br>      ExecutionRoleArn: !GetAtt AWSQSEKSClusterExecutionRole.Arn<br>      LoggingConfig:<br>        LogGroupName: !Join<br>          - &#39;-&#39;<br>          - - &#39;/aws/cloudformation/registry/AWSQSEKSClusterLogGroup&#39;<br>            - !Select [2, !Split [ &#39;/&#39;, !Ref AWS::StackId ]]<br>        LogRoleArn: !GetAtt AWSQSEKSClusterLogDeliveryRole.Arn<br>      SchemaHandlerPackage: &#39;https://dep-public.s3.ap-northeast-1.amazonaws.com/extensions/awsqs-kubernetes-apply.zip&#39;<br><br>  AWSQSEKubernetesResourceGetVersion:<br>    Type: AWS::CloudFormation::ResourceVersion<br>    Properties:<br>      TypeName: AWSQS::Kubernetes::Get<br>      ExecutionRoleArn: !GetAtt AWSQSEKSClusterExecutionRole.Arn<br>      LoggingConfig:<br>        LogGroupName: !Join<br>          - &#39;-&#39;<br>          - - &#39;/aws/cloudformation/registry/AWSQSEKSClusterLogGroup&#39;<br>            - !Select [2, !Split [ &#39;/&#39;, !Ref AWS::StackId ]]<br>        LogRoleArn: !GetAtt AWSQSEKSClusterLogDeliveryRole.Arn<br>      SchemaHandlerPackage: &#39;https://dep-public.s3.ap-northeast-1.amazonaws.com/extensions/awsqs-kubernetes-get.zip&#39;<br><br>  AWSQSEKSClusterResourceDefaultVersion:<br>    Type: AWS::CloudFormation::ResourceDefaultVersion<br>    Properties:<br>      TypeVersionArn: !Ref AWSQSEKSClusterResourceVersion<br><br>  AWSQSEKubernetesHelmDefaultVersion:<br>    Type: AWS::CloudFormation::ResourceDefaultVersion<br>    Properties:<br>      TypeVersionArn: !Ref AWSQSEKubernetesHelmVersion<br><br>  AWSQSEKubernetesResourceDefaultVersion:<br>    Type: AWS::CloudFormation::ResourceDefaultVersion<br>    Properties:<br>      TypeVersionArn: !Ref AWSQSEKubernetesResourceVersion<br><br>  AWSQSEKubernetesResourceGetDefaultVersion:<br>    Type: AWS::CloudFormation::ResourceDefaultVersion<br>    Properties:<br>      TypeVersionArn: !Ref AWSQSEKubernetesResourceGetVersion</pre><h4>5.9 Create RDS MySQL instance</h4><p>This template is designed to create an Amazon RDS (Relational Database Service) instance with a MySQL database, along with the necessary networking and security configurations.</p><pre>## dep-public/series-1/dep.template<br><br>Resources:<br>  DepDBInstanceStack:<br>    Type: AWS::CloudFormation::Stack<br>    DependsOn:<br>      - DepVpcStack<br>      - DepSecurityGroupStack<br>    Properties:<br>      TemplateURL: &#39;https://dep-public.s3.ap-northeast-1.amazonaws.com/series-1/dep-mysql.template&#39;<br>      Parameters:<br>        DBInstanceIdentifier: !Join<br>          - &#39;-&#39;<br>          - - !Ref EnvironmentName<br>            - mysql<br>        DBAllocatedStorage: !Ref DBAllocatedStorage<br>        DBAutoMinorVersionUpgrade: !Ref DBAutoMinorVersionUpgrade<br>        DBBackupRetentionPeriod: !Ref DBBackupRetentionPeriod<br>        DBInstanceClass: !Ref DBInstanceClass<br>        DBIops: !Ref DBIops<br>        DBMasterUsername: !Ref DBMasterUsername<br>        DBMasterUserPassword: !Ref DBMasterUserPassword<br>        DBMultiAZ: !Ref DBMultiAZ<br>        DBStorageType: !Ref DBStorageType<br>        CustomDBSecurityGroup: !GetAtt DepSecurityGroupStack.Outputs.MySQLRDSSecurityGroup<br>        Subnet1ID: !GetAtt DepVpcStack.Outputs.PublicSubnet1<br>        Subnet2ID: !GetAtt DepVpcStack.Outputs.PublicSubnet2<br>        VPCID: !GetAtt DepVpcStack.Outputs.VPC</pre><pre>## dep-public/series-1/dep-mysql.template<br><br>---<br>AWSTemplateFormatVersion: &#39;2010-09-09&#39;<br>Description: Aurora cluster template<br>Parameters:<br>  DBInstanceIdentifier:<br>    Description: A name for the DB instance. If you specify a name, AWS CloudFormation converts it to lowercase.<br>      If you don&#39;t specify a name, AWS CloudFormation generates a unique physical ID and uses that ID for the DB instance.<br>    Type: String<br>    MinLength: &#39;1&#39;<br>    MaxLength: &#39;63&#39;<br>  DBAllocatedStorage:<br>    ConstraintDescription: must be between 5 and 4096 GiB. If Iops specified, AllocatedStorage<br>      must be at least 100 GiB and with minimum Iops value of 1000.<br>    Default: &#39;50&#39;<br>    Description: The size of the database in gigabytes (GiB).<br>    MaxValue: &#39;4096&#39;<br>    MinValue: &#39;5&#39;<br>    Type: Number<br>  DBAutoMinorVersionUpgrade:<br>    AllowedValues:<br>      - &#39;true&#39;<br>      - &#39;false&#39;<br>    Default: &#39;false&#39;<br>    Description: Select true/false to setup Auto Minor Version upgrade.<br>    Type: String<br>  DBBackupRetentionPeriod:<br>    Default: &#39;7&#39;<br>    Description: The number of days for which automatic DB snapshots are retained.<br>    Type: String<br>  DBInstanceClass:<br>    AllowedValues:<br>      - db.m6g.large<br>      - db.m6g.xlarge<br>      - db.m6g.2xlarge<br>      - db.m6g.4xlarge<br>      - db.m6g.8xlarge<br>      - db.m6g.12xlarge<br>      - db.m6g.16xlarge<br>      - db.m5.large<br>      - db.m5.xlarge<br>      - db.m5.2xlarge<br>      - db.m5.8xlarge<br>    ConstraintDescription: Must select a valid database instance type.<br>    Default: db.m5.large<br>    Description: The name of the compute and memory capacity class of the Amazon mysql DB instance.<br>    Type: String<br>  DBIops:<br>    AllowedValues:<br>      - &#39;1000&#39;<br>      - &#39;2000&#39;<br>      - &#39;3000&#39;<br>      - &#39;4000&#39;<br>      - &#39;5000&#39;<br>      - &#39;6000&#39;<br>      - &#39;7000&#39;<br>      - &#39;8000&#39;<br>      - &#39;9000&#39;<br>      - &#39;10000&#39;<br>    ConstraintDescription: &#39;1000 Iops min and increased in 1K increments. &#39;<br>    Default: &#39;1000&#39;<br>    Description: DB Iops. Used only when io1 specified for the StorageType property.<br>    Type: Number<br>  DBMasterUsername:<br>    Type: String<br>  DBMasterUserPassword:<br>    Type: String<br>  DBMultiAZ:<br>    AllowedValues:<br>      - &#39;true&#39;<br>      - &#39;false&#39;<br>    Default: &#39;false&#39;<br>    Description: Specifies if the database instance is a multiple Availability Zone deployment.<br>    Type: String<br>  DBStorageType:<br>    AllowedValues:<br>      - standard<br>      - gp2<br>      - io1<br>    Default: standard<br>    Description: The storage type associated with this database instance.<br>    Type: String<br>  CustomDBSecurityGroup:<br>    Description: MySQL Security Group.<br>    Type: AWS::EC2::SecurityGroup::Id<br>  Subnet1ID:<br>    Description: The ID of the private subnet in Availability Zone 1.<br>    Type: &#39;AWS::EC2::Subnet::Id&#39;<br>  Subnet2ID:<br>    Description: The ID of the private subnet in Availability Zone 2.<br>    Type: &#39;AWS::EC2::Subnet::Id&#39;<br>  VPCID:<br>    Description: ID of the VPC you are deploying into (e.g., vpc-0343606e).<br>    Type: &#39;AWS::EC2::VPC::Id&#39;<br>    Default: &#39;&#39;<br>Conditions:<br>  IOPSStorageType:<br>    !Equals<br>    - !Ref DBStorageType<br>    - io1<br>Resources:<br>  DBParameterGroup:<br>    Type: AWS::RDS::DBParameterGroup<br>    Properties:<br>      Description: Parameter group of mysql instance<br>      Family: mysql8.0<br>      Parameters:<br>        log_bin_trust_function_creators: 1<br><br>  DBSubnetGroup:<br>    Type: AWS::RDS::DBSubnetGroup<br>    Properties:<br>      DBSubnetGroupDescription: Subnets available for the RDS mysql DB Instance<br>      SubnetIds:<br>        - !Ref Subnet1ID<br>        - !Ref Subnet2ID<br><br>  DBInstance:<br>    Type: AWS::RDS::DBInstance<br>    Properties:<br>      AllocatedStorage: !Ref DBAllocatedStorage<br>      DBInstanceClass: !Ref DBInstanceClass<br>      DBInstanceIdentifier: !Ref DBInstanceIdentifier<br>      Engine: mysql<br>      EngineVersion: &#39;8.0.33&#39;<br>      MasterUsername: !Ref DBMasterUsername<br>      MasterUserPassword: !Ref DBMasterUserPassword<br>      DBParameterGroupName: !GetAtt DBParameterGroup.DBParameterGroupName<br>      DBSubnetGroupName: !Ref DBSubnetGroup<br>      VPCSecurityGroups:<br>        - !Ref CustomDBSecurityGroup<br>      MultiAZ: !Ref DBMultiAZ<br>      StorageType: !Ref DBStorageType<br>      AutoMinorVersionUpgrade: !Ref DBAutoMinorVersionUpgrade<br>      BackupRetentionPeriod: !Ref DBBackupRetentionPeriod<br>      Iops:<br>        !If<br>        - IOPSStorageType<br>        - !Ref DBIops<br>        - !Ref AWS::NoValue<br>      Tags:<br>        - Key: Name<br>          Value: !Sub PostgreSQLDB-${AWS::StackName}<br><br>  DBSecret:<br>    Type: &#39;AWS::SecretsManager::Secret&#39;<br>    Properties:<br>      Name: dep-db-instance-secret<br>      SecretString: !Sub &#39;{&quot;username&quot;:&quot;${DBMasterUsername}&quot;,&quot;password&quot;:&quot;${DBMasterUserPassword}&quot;}&#39;<br><br>Outputs:<br>  RDSEndPoints:<br>    Description: Amazon RDS Endpoint to connect<br>    Value: !Sub ${DBInstance.Endpoint.Address}:${DBInstance.Endpoint.Port}<br>  RDSEndPointAddress:<br>    Description: Amazon RDS Endpoint to connect<br>    Value: !Sub ${DBInstance.Endpoint.Address}<br>  RDSEndPointPort:<br>    Description: Amazon RDS Endpoint to connect<br>    Value: !Sub ${DBInstance.Endpoint.Port}<br>  DBInstanceArn:<br>    Description: The Amazon Resource Name (ARN) for the DB instance.<br>    Value: !GetAtt DBInstance.DBInstanceArn<br>  DBSecretArn:<br>    Description: The ARN of the secret.<br>    Value: !GetAtt DBSecret.Id</pre><h4>5.10 Create EKS cluster and nodegroups</h4><p>This template is designed to create and configure an Amazon EKS (Elastic Kubernetes Service) cluster along with its associated node groups and Kubernetes resources. It provides a comprehensive setup for a Kubernetes environment in AWS, including the EKS cluster, node groups for platform, Hadoop, and Trino, and specific Kubernetes resources like namespaces and storage classes.</p><pre>## dep-public/series-1/dep.template<br><br>Resources:<br>  DepEksStack:<br>    Type: AWS::CloudFormation::Stack<br>    DependsOn:<br>      - DepVpcStack<br>      - DepSecurityGroupStack<br>      - DepIamStack<br>      - DepCloudformationExtensionsStack<br>      - DepDBInstanceStack<br>    Properties:<br>      TemplateURL: &#39;https://dep-public.s3.ap-northeast-1.amazonaws.com/series-1/dep-eks.template&#39;<br>      Parameters:<br>        EnvironmentName: !Ref EnvironmentName<br>        LambdaRoleArn: !GetAtt DepIamStack.Outputs.DepLambdaIamRole<br>        ClusterRoleArn: !GetAtt DepIamStack.Outputs.DepIamRole<br>        WorkerRoleArn: !GetAtt DepIamStack.Outputs.DepEksWorkerIamRole<br>        SecurityGroupIds: !GetAtt DepSecurityGroupStack.Outputs.EKSSecurityGroup<br>        PrivateSubnet1Id: !GetAtt DepVpcStack.Outputs.PrivateSubnet1<br>        PrivateSubnet2Id: !GetAtt DepVpcStack.Outputs.PrivateSubnet2<br>        PublicSubnet1Id: !GetAtt DepVpcStack.Outputs.PublicSubnet1<br>        PublicSubnet2Id: !GetAtt DepVpcStack.Outputs.PublicSubnet2<br>        VpcCIDR: !Ref VpcCIDR<br>        AmiType: &#39;AL2_ARM_64&#39;<br>        CapacityType: &#39;SPOT&#39;<br>        InstanceTypes: &#39;m6g.large&#39;<br>        Ec2SshKey: !Ref DepEc2SshKeyPair<br>        DbEndpoint: !GetAtt DepDBInstanceStack.Outputs.RDSEndPointAddress<br>        DbPort: !GetAtt DepDBInstanceStack.Outputs.RDSEndPointPort<br>        DbUserName: !Ref DBMasterUsername<br>        DbUserPassword: !Ref DBMasterUserPassword<br>        S3BucketName: !GetAtt DepS3Stack.Outputs.BucketName<br>        LdapDomainName: !Ref LdapDomainName<br>        LdapAdmin: !Ref LdapAdmin<br>        LdapAdminPassword: !Ref LdapAdminPassword<br>        LdapUser: !Ref LdapUser<br>        LdapUserPassword: !Ref LdapUserPassword<br>        RedisPassword: !Ref RedisPassword<br>        CliUsername: !Ref CliUsername<br>        RangerPassword: !Ref RangerPassword</pre><pre>## dep-public/series-1/dep-eks.template<br><br>AWSTemplateFormatVersion: &#39;2010-09-09&#39;<br>Description: &#39;Create Dep kubernetes resources.&#39;<br>Parameters:<br>  # EKS Cluster Configuration<br>  EnvironmentName:<br>    Type: String<br>  LambdaRoleArn:<br>    Type: String<br>  ClusterRoleArn:<br>    Type: String<br>  WorkerRoleArn:<br>    Type: String<br>  SecurityGroupIds:<br>    Type: String<br>  PrivateSubnet1Id:<br>    Type: String<br>  PrivateSubnet2Id:<br>    Type: String<br>  PublicSubnet1Id:<br>    Type: String<br>  PublicSubnet2Id:<br>    Type: String<br>  VpcCIDR:<br>    Type: String<br><br>  # EKS Nodegroup Configuration<br>  AmiType:<br>    Description: The AMI type for your node group.<br>    Type: String<br>  CapacityType:<br>    Description: The capacity type of your managed node group.<br>    Type: String<br>  DiskSize:<br>    Description: The root device disk size (in GiB) for your node group instances.<br>    Type: Number<br>    Default: 100<br>  ForceUpdateEnabled:<br>    Description: Force the update if the existing node group&#39;s pods are unable to be drained due to a pod disruption budget issue.<br>    Type: String<br>    Default: false<br>  InstanceTypes:<br>    Description: Specify the instance types for a node group.<br>    Type: String<br>  Ec2SshKey:<br>    Description: The Amazon EC2 SSH key name that provides access for SSH communication with the nodes in the managed node group.<br>    Type: String<br><br>  # MySql Configuration<br>  DbEndpoint:<br>    Type: String<br>  DbPort:<br>    Type: String<br>  DbUserName:<br>    Type: String<br>  DbUserPassword:<br>    Type: String<br>  S3BucketName:<br>    Type: String<br><br>  # LDAP Configuration<br>  LdapDomainName:<br>    Description: Ldap domain name.<br>    Type: String<br>  LdapAdmin:<br>    Description: Ldap admin username.<br>    Type: String<br>  LdapAdminPassword:<br>    Description: Ldap admin password.<br>    Type: String<br>  LdapUser:<br>    Description: Ldap user to grant admin access to.<br>    Type: String<br>    Default: &#39;depadmin&#39;<br>  LdapUserPassword:<br>    Description: Password for Ldap user to grant admin access to.<br>    Type: String<br>    Default: &#39;dep123456&#39;<br><br>  # Redis Configuration<br>  RedisPassword:<br>    Description: Redis password.<br>    Type: String<br><br>  CliUsername:<br>    Description: The aws username to grant eks cli access to.<br>    Type: String<br><br>  RangerPassword:<br>    Description: Admin password for ranger.<br>    Type: String<br><br>Resources:<br>  EksCluster:<br>    Type: AWSQS::EKS::Cluster<br>    Properties:<br>      Name: !Join<br>        - &#39;-&#39;<br>        - - !Ref EnvironmentName<br>          - eks-cluster<br>          - !Ref AWS::Region<br>      Version: 1.28<br>      LambdaRoleName: !Ref LambdaRoleArn<br>      RoleArn: !Ref ClusterRoleArn<br>      ResourcesVpcConfig:<br>        SubnetIds:<br>          - !Ref PrivateSubnet1Id<br>          - !Ref PrivateSubnet2Id<br>          - !Ref PublicSubnet1Id<br>          - !Ref PublicSubnet2Id<br>        SecurityGroupIds:<br>          - !Ref SecurityGroupIds<br>        EndpointPrivateAccess: true<br>        EndpointPublicAccess: true<br>      EnabledClusterLoggingTypes: [&quot;audit&quot;]<br>      KubernetesApiAccess:<br>        Roles:<br>          # nodeRole so the extension can update the kubernetes api access<br>          - Arn: !Ref LambdaRoleArn<br>            Username: &quot;Lambda&quot;<br>            Groups: [ &quot;aws-auth-admin&quot; ]<br>          - Arn: !Ref ClusterRoleArn<br>            Username: &quot;Admin&quot;<br>            Groups: [&quot;system:masters&quot;, &quot;kube-system:aws-auth&quot; ]<br>          - Arn: !Ref WorkerRoleArn<br>            Username: &quot;Worker&quot;<br>            Groups: [ &quot;system:masters&quot;, &quot;kube-system:aws-auth&quot;, &quot;system:nodes&quot; ]<br>          ## Temporary use<br>          - Arn: !Sub &quot;arn:${AWS::Partition}:iam::${AWS::AccountId}:user/${CliUsername}&quot;<br>            Username: &quot;CliUser&quot;<br>            Groups: [ &quot;system:masters&quot;, &quot;kube-system:aws-auth&quot; ]<br>      Tags:<br>        - Key: ClusterName<br>          Value: !Join<br>            - &#39;-&#39;<br>            - - !Ref EnvironmentName<br>              - eks-cluster<br><br>  EbsCsiAddon:<br>    Type: AWS::EKS::Addon<br>    DependsOn:<br>      - EksCluster<br>    Properties:<br>      AddonName: aws-ebs-csi-driver<br>      ClusterName: !Ref EksCluster<br><br>  PlatformNodeGroup:<br>    Type: AWS::EKS::Nodegroup<br>    Properties:<br>      AmiType: !Ref AmiType<br>      CapacityType: !Ref CapacityType<br>      ClusterName: !Ref EksCluster<br>      DiskSize: !Ref DiskSize<br>      ForceUpdateEnabled: !Ref ForceUpdateEnabled<br>      InstanceTypes:<br>        - m6g.large<br>        - m6g.xlarge<br>        - m6g.2xlarge<br>        - m6g.4xlarge<br>        - r6g.2xlarge<br>        - c6g.2xlarge<br>        - c6gd.2xlarge<br>        - c6g.large<br>        - c6g.xlarge<br>        - c6gn.xlarge<br>      NodegroupName: platform<br>      NodeRole: !Ref WorkerRoleArn<br>      RemoteAccess:<br>        Ec2SshKey: !Ref Ec2SshKey<br>      ScalingConfig:<br>        DesiredSize: 2<br>        MaxSize: 5<br>        MinSize: 2<br>      Subnets:<br>        - !Ref PublicSubnet1Id<br>        - !Ref PublicSubnet2Id<br>      Labels:<br>        nodegroup: platform<br><br>  HadoopNodeGroup:<br>    Type: AWS::EKS::Nodegroup<br>    Properties:<br>      AmiType: !Ref AmiType<br>      CapacityType: !Ref CapacityType<br>      ClusterName: !Ref EksCluster<br>      DiskSize: !Ref DiskSize<br>      ForceUpdateEnabled: !Ref ForceUpdateEnabled<br>      InstanceTypes:<br>        - m6g.large<br>        - m6g.xlarge<br>        - m6g.2xlarge<br>        - m6g.4xlarge<br>        - r6g.2xlarge<br>        - c6g.2xlarge<br>        - c6gd.2xlarge<br>        - c6g.large<br>        - c6g.xlarge<br>        - c6gn.xlarge<br>      NodegroupName: hadoop<br>      NodeRole: !Ref WorkerRoleArn<br>      RemoteAccess:<br>        Ec2SshKey: !Ref Ec2SshKey<br>      ScalingConfig:<br>        DesiredSize: 2<br>        MaxSize: 5<br>        MinSize: 2<br>      Subnets:<br>        - !Ref PublicSubnet1Id<br>        - !Ref PublicSubnet2Id<br>      Labels:<br>        nodegroup: hadoop<br><br>  TrinoNodeGroup:<br>    Type: AWS::EKS::Nodegroup<br>    Properties:<br>      AmiType: !Ref AmiType<br>      CapacityType: !Ref CapacityType<br>      ClusterName: !Ref EksCluster<br>      DiskSize: !Ref DiskSize<br>      ForceUpdateEnabled: !Ref ForceUpdateEnabled<br>      InstanceTypes:<br>        - m6g.large<br>        - m6g.xlarge<br>        - m6g.2xlarge<br>        - m6g.4xlarge<br>        - r6g.2xlarge<br>        - c6g.2xlarge<br>        - c6gd.2xlarge<br>        - c6g.large<br>        - c6g.xlarge<br>        - c6gn.xlarge<br>      NodegroupName: trino<br>      NodeRole: !Ref WorkerRoleArn<br>      RemoteAccess:<br>        Ec2SshKey: !Ref Ec2SshKey<br>      ScalingConfig:<br>        DesiredSize: 2<br>        MaxSize: 5<br>        MinSize: 2<br>      Subnets:<br>        - !Ref PublicSubnet1Id<br>        - !Ref PublicSubnet2Id<br>      Labels:<br>        nodegroup: trino<br><br>  ClusterNamespace:<br>    Type: AWSQS::Kubernetes::Resource<br>    Properties:<br>      ClusterName: !Ref EksCluster<br>      Namespace: default<br>      Manifest: |<br>        ---<br>        apiVersion: v1<br>        kind: Namespace<br>        metadata:<br>          name: platform<br>        ---<br>        apiVersion: v1<br>        kind: Namespace<br>        metadata:<br>          name: hadoop<br>        ---<br>        apiVersion: v1<br>        kind: Namespace<br>        metadata:<br>          name: trino<br><br>  EbsStorageClass:<br>    Type: AWSQS::Kubernetes::Resource<br>    Properties:<br>      ClusterName: !Ref EksCluster<br>      Namespace: default<br>      Manifest: |<br>        ---<br>        apiVersion: storage.k8s.io/v1<br>        kind: StorageClass<br>        metadata:<br>          name: ebs-sc<br>        provisioner: ebs.csi.aws.com<br>        volumeBindingMode: WaitForFirstConsumer  <br><br>  ClusterAutoscaler:<br>    Type: AWSQS::Kubernetes::Helm<br>    DependsOn:<br>      - EbsCsiAddon<br>      - EksCluster<br>      - ClusterNamespace<br>      - PlatformNodeGroup<br>      - HadoopNodeGroup<br>      - TrinoNodeGroup<br>    Properties:<br>      ClusterID: !Ref EksCluster<br>      Repository: https://tomxiaoyz.github.io/big-data-platform-public/<br>      Chart: dep/cluster-autoscaler<br>      Namespace: kube-system<br>      Name: cluster-autoscaler<br>      Version: 1.0.0<br>      ValueYaml: !Sub |<br>        autoDiscovery:<br>          clusterName: ${EnvironmentName}-eks-cluster-${AWS::Region}<br>        awsRegion: ${AWS::Region}<br><br>Outputs:<br>  Arn:<br>    Description: The ARN of the cluster, such as arn:aws:eks:us-west-2:666666666666:cluster/prod.<br>    Value: !GetAtt EksCluster.Arn<br>  CertificateAuthorityData:<br>    Description: The certificate-authority-data for your cluster.<br>    Value: !GetAtt EksCluster.CertificateAuthorityData<br>  ClusterSecurityGroupId:<br>    Description: The cluster security group that was created by Amazon EKS for the cluster.<br>      Managed node groups use this security group for control plane to data plane communication.<br>    Value: !GetAtt EksCluster.ClusterSecurityGroupId<br><br>  Endpoint:<br>    Description: The endpoint for your Kubernetes API server,<br>      such as https://5E1D0CEXAMPLEA591B746AFC5AB30262.yl4.us-west-2.eks.amazonaws.com.<br>    Value: !GetAtt EksCluster.Endpoint<br>  Name:<br>    Description: EKS cluster name<br>    Value: !Ref EksCluster<br>  OIDCIssuerURL:<br>    Description: The issuer URL for the OIDC identity provider.<br>    Value: !GetAtt EksCluster.OIDCIssuerURL</pre><p>Next, we will start to deploy related applications on our EKS cluster.</p><h4>5.11 Deploy OpenLDAP and phpLDAPadmin services</h4><p>OpenLDAP and phpLDAPadmin images are available in the following registries:</p><ul><li>public.ecr.aws/y8d7x2g6/dep-public/openldap</li></ul><p>To deploy the image using helm in CloudFormation:</p><pre>## dep-public/series-1/dep-eks.template<br><br>Resources:<br>  OpenldapService:<br>    Type: AWSQS::Kubernetes::Helm<br>    DependsOn:<br>      - EbsCsiAddon<br>      - EksCluster<br>      - ClusterNamespace<br>      - PlatformNodeGroup<br>      - EbsStorageClass<br>    Properties:<br>      ClusterID: !Ref EksCluster<br>      Repository: https://tomxiaoyz.github.io/big-data-platform-public/<br>      Chart: dep/openldap<br>      Namespace: platform<br>      Name: openldap<br>      Version: 1.0.0<br>      ValueYaml: !Sub |<br>        global:<br>          ldapDomain: &quot;dc=${LdapDomainName},dc=com&quot;<br>          adminUser: &quot;${LdapAdmin}&quot;<br>          adminPassword: &quot;${LdapAdminPassword}&quot;<br>          configUserEnabled: false<br>          username: &quot;${LdapUser}&quot;<br>          userPassword: &quot;${LdapUserPassword}&quot;<br>        customLdifFiles:<br>          00-root.ldif: |-<br>            # Root creation<br>            dn: dc=${LdapDomainName},dc=com<br>            objectClass: dcObject<br>            objectClass: organization<br>            o: ${LdapDomainName}<br>            dc: ${LdapDomainName}<br>          01-users-group.ldif: |-<br>            dn: ou=People,dc=${LdapDomainName},dc=com<br>            ou: People<br>            objectClass: organizationalUnit<br>            objectClass: top<br>        customAcls: |-<br>          dn: olcDatabase={2}mdb,cn=config<br>          changetype: modify<br>          replace: olcAccess<br>          olcAccess: {0}to *<br>            by dn.exact=gidNumber=0+uidNumber=1001,cn=peercred,cn=external,cn=auth manage<br>            by * break<br>          olcAccess: {1}to attrs=userPassword,shadowLastChange<br>            by self write<br>            by dn=&quot;cn=${LdapAdmin},dc=${LdapDomainName},dc=com&quot; write<br>            by set=&quot;user/employeeType &amp; [ldap_admin]&quot; write<br>            by anonymous auth by * none<br>          olcAccess: {2}to *<br>            by dn=&quot;cn=${LdapAdmin},dc=${LdapDomainName},dc=com&quot; write<br>            by set=&quot;user/employeeType &amp; [ldap_admin]&quot; write<br>            by self read<br>            by * none<br>        persistence:<br>          storageClass: ebs-sc<br>        affinity:<br>          nodeAffinity:<br>            requiredDuringSchedulingIgnoredDuringExecution:<br>              nodeSelectorTerms:<br>                - matchExpressions:<br>                    - key: nodegroup<br>                      operator: In<br>                      values:<br>                        - platform</pre><p>This template will deploy OpenLDAP services in platform namespace, create admin and config users with given username and password and persist ldap users utilizing EBS storage class.</p><p>This template exposes OpenLDAP service on standard port 389 and phpLDAPadmin on port 80.</p><h4>5.12 Deploy ranger-related services</h4><p>This template will setup ranger-admin, ranger-solr and ranger-usersync,</p><p>Ranger images are available in the following registries:</p><ul><li>public.ecr.aws/y8d7x2g6/dep-public/ranger/admin</li><li>public.ecr.aws/y8d7x2g6/dep-public/ranger/solr</li><li>public.ecr.aws/y8d7x2g6/dep-public/ranger/usersync</li></ul><pre>## dep-public/series-1/dep-eks.template<br><br>Resources:<br>  RangerService:<br>    Type: AWSQS::Kubernetes::Helm<br>    DependsOn:<br>      - EbsCsiAddon<br>      - EksCluster<br>      - ClusterNamespace<br>      - PlatformNodeGroup<br>      - OpenldapService<br>    Properties:<br>      ClusterID: !Ref EksCluster<br>      Repository: https://tomxiaoyz.github.io/big-data-platform-public/<br>      Chart: dep/ranger<br>      Namespace: platform<br>      Name: ranger<br>      Version: 1.0.0<br>      ValueYaml: !Sub |<br>        env:<br>          admin:<br>            DB_HOST: ${DbEndpoint}<br>            DB_ROOT_USER: ${DbUserName}<br>            DB_ROOT_PASSWORD: ${DbUserPassword}<br>            DB_NAME: ranger<br>            DB_USER: ${DbUserName}<br>            DB_PASSWORD: ${DbUserPassword}<br>            AUDIT_SOLR_URLS: http://ranger-solr.platform.svc.cluster.local:8983/solr/ranger_audits<br>            POLICYMGR_EXTERNAL_URL: http://ranger-admin.platform.svc.cluster.local:6080<br>            XA_LDAP_URL: ldap://openldap.platform.svc.cluster.local:389<br>            XA_LDAP_BASE_DN: dc=${LdapDomainName},dc=com<br>            XA_LDAP_BIND_DN: cn=${LdapAdmin},dc=${LdapDomainName},dc=com<br>            XA_LDAP_BIND_PASSWORD: ${LdapAdminPassword}<br>            LDAP_USER: ${LdapUser}<br>            LDAP_PASSWORD: ${LdapUserPassword}<br>            RANGER_PASSWORD: ${RangerPassword}<br>          usersync:<br>            POLICY_MGR_URL: http://ranger-admin.platform.svc.cluster.local:6080<br>            SYNC_SOURCE: ldap<br>            SYNC_LDAP_URL: ldap://openldap.platform.svc.cluster.local:389<br>            SYNC_LDAP_BIND_DN: cn=${LdapAdmin},dc=${LdapDomainName},dc=com<br>            SYNC_LDAP_BIND_PASSWORD: ${LdapAdminPassword}<br>            SYNC_LDAP_SEARCH_BASE: dc=${LdapDomainName},dc=com<br>            SYNC_LDAP_USER_SEARCH_BASE: ou=People,dc=${LdapDomainName},dc=com<br>            SYNC_LDAP_USER_NAME_ATTRIBUTE: cn</pre><h4>5.13 Deploy standalone Hive metastore service</h4><p>This template will instance standalone Hive metastore service which act as the metastore service for Trino engine.</p><p>Metadata backend is RDS created in chapter 5.9 and warehouse location is s3://${S3BucketName}/hive/ where bucket name is referenced to the one created in chapter 5.5.</p><p>Image is available in the following registry:</p><ul><li>public.ecr.aws/y8d7x2g6/dep-public/hive-metastore-standalone</li></ul><pre>## dep-public/series-1/dep-eks.template<br><br>Resources:<br>  HiveMetastoreService:<br>    Type: AWSQS::Kubernetes::Helm<br>    DependsOn:<br>      - EbsCsiAddon<br>      - EksCluster<br>      - ClusterNamespace<br>      - HadoopNodeGroup<br>    Properties:<br>      ClusterID: !Ref EksCluster<br>      Repository: https://tomxiaoyz.github.io/big-data-platform-public/<br>      Chart: dep/hive-metastore<br>      Version: 1.0.0<br>      Namespace: hadoop<br>      Name: hive-metastore<br>      ValueYaml: !Sub |<br>        env:<br>          HIVE_DB_JDBC_URL: &quot;jdbc:mysql://${DbEndpoint}:${DbPort}/hive&quot;<br>          HIVE_DB_USER: &quot;${DbUserName}&quot;<br>          HIVE_DB_PASS: &quot;${DbUserPassword}&quot;<br>          HIVE_WAREHOUSE_S3LOCATION: &quot;s3://${S3BucketName}/hive/&quot;</pre><h4>5.14 Deploy Hadoop and Hive services with ranger plugin setup</h4><p>This template will install Hadoop and hive services with ranger plugin setup.</p><p>Images are available in the following registries:</p><ul><li>public.ecr.aws/y8d7x2g6/dep-public/emr</li></ul><p>Within the first image, we install hadoop-hdfs, hadoop-yarn, hiveServer2 as well as hive metastore services.</p><pre>## dep-public/series-1/dep-eks.template<br><br>Resources:<br>  HadoopService:<br>    Type: AWSQS::Kubernetes::Helm<br>    DependsOn:<br>      - EbsCsiAddon<br>      - EksCluster<br>      - ClusterNamespace<br>      - HadoopNodeGroup<br>      - HiveMetastoreService<br>      - OpenldapService<br>    Properties:<br>      ClusterID: !Ref EksCluster<br>      Repository: https://tomxiaoyz.github.io/big-data-platform-public/<br>      Chart: dep/hadoop<br>      Namespace: hadoop<br>      Name: hadoop<br>      Version: 1.0.0<br>      ValueYaml: !Sub |<br>        ldap:<br>          url: ldap://openldap.platform.svc.cluster.local:389<br>          bind_dn: cn=${LdapAdmin},dc=${LdapDomainName},dc=com<br>          bind_password: ${LdapAdminPassword}<br>          base_dn: ou=People,dc=${LdapDomainName},dc=com<br>        hive:<br>          mysql:<br>            endpoint: ${DbEndpoint}<br>            database: hive<br>            port: ${DbPort}<br>            username: ${DbUserName}<br>            password: ${DbUserPassword}<br>          warehouse:<br>            location: s3://${S3BucketName}/hive/<br>        domain_database: ${LdapDomainName}<br><br>  ZookeeperService:<br>    Type: AWSQS::Kubernetes::Helm<br>    DependsOn:<br>      - EbsCsiAddon<br>      - EksCluster<br>      - ClusterNamespace<br>      - HadoopNodeGroup<br>    Properties:<br>      ClusterID: !Ref EksCluster<br>      Repository: https://tomxiaoyz.github.io/big-data-platform-public/<br>      Chart: zookeeper<br>      Namespace: hadoop<br>      Name: zookeeper<br>      Version: 1.0.0<br>      ValueYaml: |<br>        persistence:<br>          storageClass: ebs-sc<br>        affinity:<br>          nodeAffinity:<br>            requiredDuringSchedulingIgnoredDuringExecution:<br>              nodeSelectorTerms:<br>                - matchExpressions:<br>                    - key: nodegroup<br>                      operator: In<br>                      values:<br>                        - hadoop</pre><h4>5.15 Deploy Trino engine with ranger plugin setup</h4><p>This template will install Trino service with ranger plugin setup.</p><p>Also, it creates a catalog configuration refrenced to Hive metastore service created in chapter 5.13. This template will expose jmx metrics to port 9081 so that we can monitor the status of Trino server simultaneously.</p><p>Image is available in the following registry:</p><ul><li>public.ecr.aws/y8d7x2g6/dep-public/trino</li></ul><blockquote>*Note: The Trino service in this demo is not secured by TLS and thus is not recommended for production use.</blockquote><pre>## dep-public/series-1/dep-eks.template<br><br>Resources:<br>  TrinoService:<br>    Type: AWSQS::Kubernetes::Helm<br>    DependsOn:<br>      - EbsCsiAddon<br>      - EksCluster<br>      - ClusterNamespace<br>      - TrinoNodeGroup<br>      - OpenldapService<br>    Properties:<br>      ClusterID: !Ref EksCluster<br>      Repository: https://tomxiaoyz.github.io/big-data-platform-public/<br>      Chart: dep/trino<br>      Namespace: trino<br>      Name: trino<br>      Version: 1.0.0<br>      ValueYaml: !Sub |<br>        server:<br>          workers: 4<br>          exchangeManager:<br>            baseDir: s3://${S3BucketName}/trino/<br>        additionalCatalogs:<br>          hive: |-<br>            hive.metastore-refresh-interval=1m<br>            connector.name=hive<br>            hive.metastore-cache-ttl=60s<br>            hive.metastore=thrift<br>            hive.metastore.uri=thrift://hive-metastore.hadoop.svc.cluster.local:9083<br>            hive.non-managed-table-writes-enabled=true<br>            hive.hdfs.impersonation.enabled=true<br>            hive.recursive-directories=true<br>            hive.allow-drop-table=true<br>            hive.parquet.use-column-names=true<br>            hive.max-partitions-per-writers=3000<br>            parquet.ignore-statistics=true<br>            hive.allow-rename-table=true<br>        additionalConfigProperties:<br>          - retry-policy=TASK<br>          - query.remote-task.max-error-duration=10m<br>          - task-retry-attempts-per-task=3<br>          - retry-initial-delay=10s<br>          - retry-max-delay=1m<br>          - retry-delay-scale-factor=2.0<br>          - fault-tolerant-execution-task-descriptor-storage-max-memory=10GB<br>        additionalExchangeManagerProperties:<br>          - exchange.s3.region=${AWS::Region}<br>        domain_database: ${LdapDomainName}<br>        domain_username: ${LdapUser}</pre><h3>6. Cleaning up resources</h3><ul><li>Go to AWS S3: Log into your AWS Management Console and navigate to the S3 service.</li><li>Delete all the contents in the S3 bucket created in this template.</li><li>Go to AWS CloudFormation: Navigate to the CloudFormation service.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fDBhOBR4e8EIgl3J19RMvA.png" /></figure><ul><li>Click ‘Delete’ to delete stack.</li><li>Go to AWS EC2: Navigate to the EC2 service.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QbSOcdmeODGBk56_I7U7Ag.png" /></figure><ul><li>Under ‘Volumes’, select all previously created volumes and delete them all.</li></ul><h3>7. Conclusion</h3><p>In conclusion, this comprehensive guide on permission management in big data, particularly within the Amazon EKS platform using Apache Ranger, equips users with the essential knowledge and tools for robust data security and management. By implementing the strategies and understanding the components detailed in this guide, users can effectively manage permissions, ensuring data security and compliance in their big data environments.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=62fecd35519d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Recovering Accidentally Transitioned S3 Files from the Glacier Tier]]></title>
            <link>https://medium.com/@tom.xiao_3937/recovering-accidentally-transitioned-s3-files-from-the-glacier-tier-8bb51503cf42?source=rss-2597b5c2304b------2</link>
            <guid isPermaLink="false">https://medium.com/p/8bb51503cf42</guid>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[glacier]]></category>
            <dc:creator><![CDATA[Tom Xiao]]></dc:creator>
            <pubDate>Fri, 06 Oct 2023 02:06:05 GMT</pubDate>
            <atom:updated>2023-10-06T02:06:05.138Z</atom:updated>
            <content:encoded><![CDATA[<p>Are you stuck with S3 files that were accidentally moved to the Glacier storage tier? Do you need an efficient yet simple way to restore them? If yes, then this guide is for you.</p><p>Amazon S3 is a scalable object storage service while Amazon Glacier is a secure, durable, and low-cost storage service for data archiving and long-term backup. Sometimes, files may end up in Glacier storage either intentionally, for cost-saving purposes, or unintentionally due to errors.</p><p>Regardless, the restoration process can be daunting. This guide will help you simplify this process with Python and AWS’s boto3 library.</p><p>Below is the Python script that will get the job done:</p><pre>import multiprocessing as mp<br>import os<br>import sys<br>import shutil<br>import traceback<br>import zipfile<br>import requests<br>import json<br>import socket<br>import boto3<br>import threadpool<br>import time<br>from datetime import timedelta, datetime<br>from pymongo import MongoClient<br>from loguru import logger<br>from botocore.exceptions import ClientError<br>import re<br><br>ACK = &quot;Your Access Key&quot;<br>ACS = &quot;Your Secret Key&quot;<br><br>bucket_name = &quot;Your Bucket Name&quot;<br>s3_remote_dir = &quot;Path to Your S3 Directory&quot;<br><br>s3 = boto3.client(&#39;s3&#39;,  aws_access_key_id=ACK, aws_secret_access_key=ACS)<br><br># local path<br>local_save_path = &#39;./temp/&#39;  # temporarily download files to local path<br><br><br>def _get_all_s3_objects(**base_kwargs):<br>    &quot;&quot;&quot;<br>    Get all objects under s3_remote_dir<br>    &quot;&quot;&quot;<br>    try:<br>        continuation_token = None<br>        while True:<br>            list_kwargs = dict(MaxKeys=1000, **base_kwargs)<br>            if continuation_token:<br>                list_kwargs[&#39;ContinuationToken&#39;] = continuation_token<br>            response = s3.list_objects_v2(**list_kwargs)<br>            yield from response.get(&#39;Contents&#39;, [])<br>            if not response.get(&#39;IsTruncated&#39;):  # At the end of the list?<br>                break<br>            continuation_token = response.get(&#39;NextContinuationToken&#39;)<br>    except:<br>        # send_dingtalk_message(traceback.format_exc())<br>        logger.error(traceback.format_exc())<br><br><br>def head_object(bucket_name, object_name):<br>    s3 = boto3.client(&#39;s3&#39;)<br>    response = None<br>    try:<br>        response = s3.head_object(Bucket=bucket_name, Key=object_name)<br>    except ClientError as e:<br>        logger.error(e)<br>        logger.error(<br>            f&quot;NoSuchBucket, NoSuchKey, or InvalidObjectState error == the object&#39;s, storage class was not GLACIER. {bucket_name} {object_name} &quot;)<br>        return None<br>    return response<br><br><br># resore objects from glacier tier<br>def restore_object(bucket_name, object_name, days, retrieval_type=&#39;Expedited&#39;):<br>    request = {&#39;Days&#39;: days,<br>                &#39;GlacierJobParameters&#39;: {&#39;Tier&#39;: retrieval_type}}<br>    # s3 = boto3.client(&#39;s3&#39;)<br>    try:<br>        s3.restore_object(Bucket=bucket_name, Key=object_name, RestoreRequest=request)<br>    except ClientError as e:<br>        logger.error(e)<br>        logger.error(<br>            f&quot;NoSuchBucket, NoSuchKey, or InvalidObjectState error == the object&#39;s, storage class was not GLACIER. {bucket_name} {object_name} &quot;)<br> <br>        return False<br>    return True<br><br><br>key_content_list = []<br>total_content_size = 0<br><br><br>while True:<br>    doing_count, done_count, need_count = 0, 0, 0<br><br>    s3_objects = _get_all_s3_objects(Bucket=bucket_name, Prefix=s3_remote_dir)<br>    for obj in s3_objects:<br>        key = obj.get(&#39;Key&#39;, None)<br>        file_name = key.split(&quot;/&quot;)[-1]<br>        # In hive, files under glacier tier are then moved to sub-directories called &#39;HIVE_UNION_SUBDIR_*&#39;<br>        # We need to move it to the right place<br>        to_key = re.sub(&quot;\/HIVE_UNION_SUBDIR_[\d]+\/&quot;, &quot;/&quot;, key)<br>        print(key, to_key, file_name)<br>        success = head_object(bucket_name, key)<br>        need_count += 1<br><br>        if success:<br>            if success.get(&#39;Restore&#39;):<br>                print(&#39;Restore {}&#39;.format(success[&#39;Restore&#39;]))<br>                index = success[&#39;Restore&#39;].find(&#39;ongoing-request=\&quot;false\&quot;&#39;)<br>                if -1 == index:<br>                    print(f&quot;{need_count} Under recovering...{key}&quot;)<br>                    doing_count += 1<br>                else:<br>                    print(success[&#39;Restore&#39;][success[&#39;Restore&#39;].find(&#39;expiry-date=&#39;):])<br>                    print(f&quot;{need_count} Recover succeeded...{key}&quot;)<br>                    find_spec_path = file_name.find(&#39;HIVE_UNION_SUBDIR&#39;)<br>                    if -1 != find_spec_path:<br>                        print(&quot;no need to download...{key}&quot;)<br>                    else:<br>                        s3.download_file(bucket_name, key, local_save_path + file_name)<br>                        s3.upload_file(local_save_path + file_name, bucket_name, to_key)<br>                    done_count += 1<br>            else:<br>                print(f&#39;{need_count} neet to recovery... {key}&#39;)<br>                restore_object(bucket_name, key, 10)<br>        print(doing_count, done_count, need_count)<br><br>    if done_count == need_count:<br>        break<br>    time.sleep(15)</pre><h3>Understanding the Code</h3><p>The solution uses the boto3 library to interact with the AWS S3 and Glacier services.</p><p>The _get_all_s3_objects function is created to list all the objects in the specified S3 bucket and directory.</p><p>The head_object function retrieves metadata from an object without returning the object itself. This is useful for checking if a file exists and its status.</p><p>The restore_object function is the heart of the script. It initiates a job to restore a file from Glacier to S3. The Days parameter specifies the lifetime of the temporary copy of the object in the S3 bucket, while GlacierJobParameters sets the speed (tier) of the restoration process.</p><p>The script then loops through the objects in the specified S3 location. For each object, it checks if a restore operation is already in progress or completed using the head_object function. If a restore operation is in progress, it leaves it alone. If the restore operation is complete, it re-uploads the file to the desired location. If the file hasn&#39;t been restored at all, it starts a restore operation.</p><p>The process continues until all files have been restored.</p><h3>In Conclusion</h3><p>The script provides a way to automate the process of restoring files from the Glacier storage tier back to S3. It can be a lifesaver if you’ve got hundreds or even thousands of files to restore.</p><p>Remember to replace the placeholders in the script with your actual AWS credentials, bucket names, and file paths. Happy restoring!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8bb51503cf42" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data on EKS Platform]]></title>
            <link>https://medium.com/@tom.xiao_3937/data-on-eks-platform-c94c051dee0b?source=rss-2597b5c2304b------2</link>
            <guid isPermaLink="false">https://medium.com/p/c94c051dee0b</guid>
            <category><![CDATA[k8s]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[big-data-platform]]></category>
            <category><![CDATA[containers]]></category>
            <dc:creator><![CDATA[Tom Xiao]]></dc:creator>
            <pubDate>Mon, 25 Sep 2023 01:58:56 GMT</pubDate>
            <atom:updated>2023-09-25T01:58:56.263Z</atom:updated>
            <content:encoded><![CDATA[<h3>Overview</h3><p>Since its launch in 2006, the Hadoop ecosystem has gained widespread acclaim. However, as big data components proliferate and usage increases, traditional big data platforms face growing challenges, including inefficient deployment of new components, limited resource sharing between applications, and ineffective operations and maintenance. As architectures remain static, this not only brings more complexity to operations, but also predictably high costs.</p><p>Let me share our company’s experience as an example. Previously, our big data platform relied heavily on AWS cloud native services like EMR and EC2. Due to high costs, complex maintenance, and other issues, over the past six months we have gradually migrated all platform applications to EKS. This has reduced costs by 90%. During this time, data volumes have dropped 50% due to operational changes. By moving to EKS, we can now focus solely on building data applications, while basic ops are handled by our ops team for integrated management, reducing our investment in infrastructure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*-KDxyteqvtNPQWlB" /></figure><p>We believe our experience could benefit any organization undertaking a containerization transition. Therefore, we plan to productize our big data platform and sell it on AWS Marketplace, making adoption easier.</p><p>Our platform leverages Kubernetes and other cloud native technologies, providing a simplified architecture compared to traditional Hadoop. By offering this platform on AWS Marketplace, we aim to help organizations realize the benefits we have seen — 90% cost reduction and simplified ops — when shifting big data to containers on Kubernetes.</p><p>With AWS handling infrastructure provisioning, customers can focus on data applications while our platform provides cluster deployment, autoscaling, monitoring, and more. We believe this turnkey approach will accelerate container adoption for big data use cases.</p><p>Please reach out if you would like to learn more about our journey or get early access when we launch on AWS Marketplace. We welcome any feedback as we work to bring a proven big data container platform to the broader cloud native community.</p><h3>Architecture</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*eKfA9GAF-XxffssW" /></figure><p>Data on EKS Platform (DEP) is a Kubernetes-based cloud-native big data platform developed by our company. Built using open source components, DEP integrates Hive, Spark, Flink, Trino, HDFS, Kafka and more. It provides data development and BI platforms including Superset, Metabase, Jupyter, Zeppelin and custom IDE platform. DEP’s metadata management and data quality monitoring system improve data quality, while data ops and user profiling enable data-driven capabilities. Ops management and permissions ensure security and stability.</p><p>DEP is an all-in-one platform enabling admins, developers, analysts and business users to complete daily tasks. Leveraging Kubernetes auto-scaling, self-healing and simplified ops, DEP provides a managed big data environment. Companies can focus on data while DEP handles infrastructure, monitoring, access controls and more. Compared to legacy Hadoop, DEP reduces costs while empowering employees through faster, easier analytics and insights access.</p><h3>Product Features</h3><p>DEP’s main functions include:</p><ul><li>DEP Core, which contains scheduling center, IDE, metadata management, and data quality monitoring capabilities.</li><li>Nebula, the data requirement module, which helps users easily find, understand, request access to datasets and raise new data requests to data engineers or data analysts.</li><li>Data-driven capabilities including data operations and user profiling to generate insights.</li><li>New features currently in development:</li><li>Model development and deployment to support machine learning workflows. Using models built by data scientists in production applications and at scale on DEP.</li><li>Large language model applications to allow conversational interaction with data and insights using chatbots and virtual assistants.</li></ul><p>DEP aims to provide an integrated environment covering the full data lifecycle. Users can discover datasets, develop models and applications, deploy them at scale, monitor data quality, and enable self-service analytics. The platform leverages Kubernetes to provide flexibility, scalability, and automated management. We welcome customer feedback to prioritize additional capabilities that would maximize the value of DEP.</p><h3>Scheduling Center</h3><p>On DEP, we developed a scheduling center to handle thousands of different types of scheduling tasks across the platform. The scheduling center currently supports the following job types:</p><ul><li><strong>SYNC</strong>: Extract data from business databases (MySQL, PostgreSQL, etc) into the data warehouse ODS layer. Supports full or incremental extracting, customizable data cleansing rules, and partitioning.</li><li><strong>HIVE,SPARK,TRINO</strong>: Supports SQL jobs using different compute engines.</li><li><strong>JAVA</strong>: Supports execution of custom Jar packages.</li><li><strong>SHELL</strong>: Supports execution of other custom jobs.</li><li><strong>EXPORT</strong>: Exports ADS layer data back to business databases for online business queries.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3EnjewWnUNkjweqN" /><figcaption>Overview Page</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*B2FAfzf9g5bOI5DZ" /><figcaption>Status Page</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*llosFOMX5MA2DyaE" /><figcaption>Task Status and Logs</figcaption></figure><p>Additionally, the scheduling center supports automatic lineage determination, DAG diagrams, job logging, and job alerts.</p><p>The scheduling center provides a unified interface to develop, monitor, and manage ETL pipelines, data integration, and processing workloads. By leveraging Kubernetes concepts like controllers, the system is scalable, resilient, and automates repetitive tasks. We plan to continue enhancing the scheduling center with machine learning capabilities for intelligent optimization and automation.</p><h3>IDE</h3><p>On DEP, we developed a custom IDE to provide ad-hoc querying, data development, and job approval workflows.</p><p>In our IDE, data developers can use Hive or Trino SQL to develop, test, and request deployment of jobs. Business analysts can query data themselves and download data once approved. Every query goes through Ranger and Solr for data authorization and auditing.</p><p>The IDE provides a collaborative environment for different user personas with built-in governance. Data engineers have the tools to develop pipelines and ETL jobs. Analysts have self-service access to data for exploratory analysis. Queries are logged and access controlled to meet security policies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*gHX1LbXKI25p2d80" /><figcaption>IDE</figcaption></figure><p>Key benefits include:</p><ul><li>Unified interface for SQL, development, testing before deploying to production</li><li>Collaboration between engineers and analysts</li><li>Data discovery for business users</li><li>Approval workflows for access requests</li><li>Centralized data authorization, auditing, and lineage tracking</li></ul><p>By leveraging the IDE, DEP customers can accelerate analytics development, empower users with self-service, and maintain oversight — all through a single pane of glass. We welcome additional feedback on IDE capabilities that would further enhance the user experience.</p><h3>Metadata Management &amp; Data Quality</h3><p>Metadata management is a crucial process in building a data platform, enabling enterprises to maximize the value of their data assets. Metadata management helps business analysts, architects, data engineers, developers and other stakeholders clearly understand what data an organization has, where it is stored, how to extract, cleanse, maintain it, and guide users.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*b1LqQkdIYV6DnUoZ" /><figcaption>Metadata Management</figcaption></figure><p>A key benefit of metadata management is ensuring data quality. In DEP, we provide additional data quality monitoring capabilities, calculating metrics and alerts at the table, column and custom rule levels. This allows data teams to promptly discover and address anomalies.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*FzlVBCptcXkpoOr_" /><figcaption>Data Quality</figcaption></figure><p>With DEP’s metadata management, companies can:</p><ul><li>Document data sources, schemas, ETL pipelines, dependencies</li><li>Discover data to understand availability across silos</li><li>Track lineage to trust data provenance</li><li>Manage governance policies like security, access</li><li>Monitor data quality with automated alerts</li><li>Guide and train end users on datasets</li></ul><p>Strong metadata management and data quality monitoring provides the foundation for reliable analytics, trusted insights, and data democratization. We welcome additional perspectives on how DEP can further enhance metadata and governance capabilities to maximize value.</p><h3>Request Center</h3><p>We developed Request Center, a data requirements platform, to manage business demands. Business users can submit requests for data applications, dashboards, and data warehouse development. Data developers can promptly follow up and process these requests on the platform.</p><p>The data requirements platform strengthens the connection between business and data teams, reducing communication barriers.</p><p>Key benefits include:</p><ul><li>Central platform for data requests across the organization</li><li>Standardized intake forms and workflows</li><li>Transparency into request status and handling</li><li>Prioritization based on business needs</li><li>Collaboration between business and technical teams</li><li>Audit trail of requirements gathering and delivery</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/0*-JJABZnpUJFP3gTn" /></figure><p><strong>Submit Data Requests</strong></p><p>Request Center allows business users to easily log and track requests, ensuring their needs are captured and worked on. For data teams, it provides a single workflow to manage incoming tasks rather than ad-hoc emails and messages.</p><p>By streamlining data requirements gathering through the Request Center, organizations can align business stakeholders and data teams, accelerate analytics delivery, and use data more strategically. We welcome additional input on how to further enhance and leverage the platform.</p><h3>Other Features</h3><h3>Data Operations &amp; User Profiling</h3><p>Empowering business users to apply data in reasonable and rapid ways to feed online business operations.</p><h3>Permission Center</h3><p>On DEP, we have defined hundreds of granular permissions that can be combined to assign appropriate roles to members.</p><p>For example, data engineers may have access to:</p><ul><li>Create/edit tables, views in Hive</li><li>Develop ETL jobs with Sqoop, Spark, etc</li><li>Schedule and run jobs in production</li></ul><p>Analysts may have permissions like:</p><ul><li>Read access to certain tables and views</li><li>Ability to run ad-hoc queries in the IDE</li><li>Export limited results sets</li></ul><p>Fine-grained authorization allows implementing least privilege access tailored to each user’s responsibilities. Complex permissions can be abstracted into reusable roles mapped to job functions.</p><p>Key benefits include:</p><ul><li>Align access with organizational data security policies</li><li>Prevent unauthorized or unintentional data access</li><li>Simplify user management by bundling permissions into roles</li><li>Gain visibility into who can access what data</li><li>Streamline auditing for compliance</li></ul><p>DEP’s flexible and granular permission model balances security with enabling self-service analytics.</p><h3>Configuration Center</h3><p>On DEP, we provide platform configuration capabilities through a Parameter Center where data administrators can manage table naming conventions, job labels, table tags, and other custom rules. Configurations take effect in the system in real-time.</p><h3>Product Timeline</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/892/0*n_URGGKTWZhKicvC" /></figure><h3>Pricing</h3><h3>Pricing Models</h3><p>We offer annual and monthly subscription plans with one-click deployment through AWS CloudFormation for a ready-to-use solution.</p><h3>Subscription Plans</h3><p>We provide a unified edition for all customers. Monthly fee is $800 and annual fee is $8,000.</p><h3>Value-Added Services</h3><p>We provide additional services like training, technical support, and can customize exclusive solutions based on user needs.</p><p>Key pricing details:</p><ul><li>Predictable subscription model avoids variable cloud costs</li><li>Annual contract offers discounted rate with one yearly payment</li><li>Monthly subscriptions allow flexibility to start small and expand</li><li>Simple per-node pricing, no hidden fees or extra charges</li><li>Add-on services available for custom integrations, training, support</li></ul><p>By leveraging our AWS Marketplace product, customers can deploy DEP quickly with minimal upfront costs. We aim to provide a high-value and scalable platform enabling organizations to realize fast time-to-value. Please reach out for customized pilots, proofs of concept, or pricing discussions.</p><p>For more information about DEP, please feel free to contact <strong>tom.xiao@hotmail.com</strong> if you are interested.</p><p>We will provide updates on the product development progress and usage instructions through additional blogs posts and announcements.</p><p>We welcome you to try out a demo, sign up for early access once launched on AWS Marketplace, or engage in a pilot project to see how DEP can empower your data teams and organization.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c94c051dee0b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setup HTTPs Authentication for Presto / Trino]]></title>
            <link>https://medium.com/@tom.xiao_3937/setup-https-authentication-for-presto-trino-28232cb61bd4?source=rss-2597b5c2304b------2</link>
            <guid isPermaLink="false">https://medium.com/p/28232cb61bd4</guid>
            <category><![CDATA[data-platforms]]></category>
            <category><![CDATA[java]]></category>
            <category><![CDATA[presto]]></category>
            <category><![CDATA[https]]></category>
            <category><![CDATA[trinos]]></category>
            <dc:creator><![CDATA[Tom Xiao]]></dc:creator>
            <pubDate>Sun, 27 Feb 2022 13:13:56 GMT</pubDate>
            <atom:updated>2022-02-27T13:13:56.116Z</atom:updated>
            <content:encoded><![CDATA[<p>Version: Presto 0.261 / Trino 360</p><p>In this article, we are going to enable HTTPS-based front-end password authentication for the Presto CLI or JDBC and ODBC drivers. The Presto CLI sends the username and password to the Presto coordinator, who then validates those credentials using an external LDAP service or user password store file.</p><h4>Step 1: Generate TLS Java Keystore File</h4><p>When using Kerberos and LDAP authentication, the Presto coordinator must be accessed via HTTPS. The Presto coordinator uses Java Keystore files for TLS configuration. These keys are generated using keytool and stored in the Java Keystore file of the Presto coordinator.</p><pre>keytool -genkeypair -alias presto_keystore -keyalg RSA -keystore /etc/presto_keystore.jks</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y1ZxGr5Jev-XQcCiSYXKQQ.png" /></figure><p><strong>Note: Please Make Sure CN is your IP address!</strong></p><h4>Step 2: Generate TLS Java Truststore File</h4><p>The truststore file contains the certificate of a trusted TLS/SSL server or a certificate of a trusted certification authority that identifies the server.</p><p>To ensure secure access to the Presto coordinator over HTTPS, clients can configure a truststore.</p><p>In order for the Presto CLI to trust the Presto coordinator, the coordinator’s certificate must be imported into the CLI’s truststore.</p><pre>keytool -exportcert -keystore /etc/presto_keystore.jks -alias presto_keystore -file /etc/alternatives/jre/lib/security/presto_certificate.cer -storepass {your password}</pre><pre>keytool -import -alias presto_trust -keystore /etc/alternatives/jre/lib/security/cacerts -file /etc/alternatives/jre/lib/security/presto_certificate.cer --storepass changeit -trustcacerts<br># changeit is the default password for Java truststore</pre><p><strong>Step 3: Install a trusted certificate to the client in code</strong></p><p><em>vim ~/InstallCert.java</em></p><pre>/*<br> * Copyright 2006 Sun Microsystems, Inc.  All Rights Reserved.<br> *<br> * Redistribution and use in source and binary forms, with or without<br> * modification, are permitted provided that the following conditions<br> * are met:<br> *<br> *   - Redistributions of source code must retain the above copyright<br> *     notice, this list of conditions and the following disclaimer.<br> *<br> *   - Redistributions in binary form must reproduce the above copyright<br> *     notice, this list of conditions and the following disclaimer in the<br> *     documentation and/or other materials provided with the distribution.<br> *<br> *   - Neither the name of Sun Microsystems nor the names of its<br> *     contributors may be used to endorse or promote products derived<br> *     from this software without specific prior written permission.<br> *<br> * THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS &quot;AS<br> * IS&quot; AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO,<br> * THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR<br> * PURPOSE ARE DISCLAIMED.  IN NO EVENT SHALL THE COPYRIGHT OWNER OR<br> * CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL,<br> * EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO,<br> * PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR<br> * PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF<br> * LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING<br> * NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS<br> * SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.<br> */<br> <br>import java.io.BufferedReader;<br>import java.io.File;<br>import java.io.FileInputStream;<br>import java.io.FileOutputStream;<br>import java.io.InputStream;<br>import java.io.InputStreamReader;<br>import java.io.OutputStream;<br>import java.security.KeyStore;<br>import java.security.MessageDigest;<br>import java.security.cert.CertificateException;<br>import java.security.cert.X509Certificate;<br> <br>import javax.net.ssl.SSLContext;<br>import javax.net.ssl.SSLException;<br>import javax.net.ssl.SSLSocket;<br>import javax.net.ssl.SSLSocketFactory;<br>import javax.net.ssl.TrustManager;<br>import javax.net.ssl.TrustManagerFactory;<br>import javax.net.ssl.X509TrustManager;<br> <br>public class InstallCert {<br>            <br> <br> public static void main(String[] args) throws Exception {<br>            <br>  String host;<br>  int port;<br>  char[] passphrase;<br>  if ((args.length == 1) || (args.length == 2)) {<br>            <br>   String[] c = args[0].split(&quot;:&quot;);<br>   host = c[0];<br>   port = (c.length == 1) ? 443 : Integer.parseInt(c[1]);<br>   String p = (args.length == 1) ? &quot;changeit&quot; : args[1];<br>   passphrase = p.toCharArray();<br>  } else {<br>            <br>   System.out<br>     .println(&quot;Usage: java InstallCert &lt;host&gt;[:port] [passphrase]&quot;);<br>   return;<br>  }<br> <br>  File file = new File(&quot;jssecacerts&quot;);<br>  if (file.isFile() == false) {<br>            <br>   char SEP = File.separatorChar;<br>   File dir = new File(System.getProperty(&quot;java.home&quot;) + SEP + &quot;lib&quot;<br>     + SEP + &quot;security&quot;);<br>   file = new File(dir, &quot;jssecacerts&quot;);<br>   if (file.isFile() == false) {<br>            <br>    file = new File(dir, &quot;cacerts&quot;);<br>   }<br>  }<br>  System.out.println(&quot;Loading KeyStore &quot; + file + &quot;...&quot;);<br>  InputStream in = new FileInputStream(file);<br>  KeyStore ks = KeyStore.getInstance(KeyStore.getDefaultType());<br>  ks.load(in, passphrase);<br>  in.close();<br> <br>  SSLContext context = SSLContext.getInstance(&quot;TLS&quot;);<br>  TrustManagerFactory tmf = TrustManagerFactory<br>    .getInstance(TrustManagerFactory.getDefaultAlgorithm());<br>  tmf.init(ks);<br>  X509TrustManager defaultTrustManager = (X509TrustManager) tmf<br>    .getTrustManagers()[0];<br>  SavingTrustManager tm = new SavingTrustManager(defaultTrustManager);<br>  context.init(null, new TrustManager[] {<br>             tm }, null);<br>  SSLSocketFactory factory = context.getSocketFactory();<br> <br>  System.out<br>    .println(&quot;Opening connection to &quot; + host + &quot;:&quot; + port + &quot;...&quot;);<br>  SSLSocket socket = (SSLSocket) factory.createSocket(host, port);<br>  socket.setSoTimeout(10000);<br>  try {<br>            <br>   System.out.println(&quot;Starting SSL handshake...&quot;);<br>   socket.startHandshake();<br>   socket.close();<br>   System.out.println();<br>   System.out.println(&quot;No errors, certificate is already trusted&quot;);<br>  } catch (SSLException e) {<br>            <br>   System.out.println();<br>   e.printStackTrace(System.out);<br>  }<br> <br>  X509Certificate[] chain = tm.chain;<br>  if (chain == null) {<br>            <br>   System.out.println(&quot;Could not obtain server certificate chain&quot;);<br>   return;<br>  }<br> <br>  BufferedReader reader = new BufferedReader(new InputStreamReader(<br>    System.in));<br> <br>  System.out.println();<br>  System.out.println(&quot;Server sent &quot; + chain.length + &quot; certificate(s):&quot;);<br>  System.out.println();<br>  MessageDigest sha1 = MessageDigest.getInstance(&quot;SHA1&quot;);<br>  MessageDigest md5 = MessageDigest.getInstance(&quot;MD5&quot;);<br>  for (int i = 0; i &lt; chain.length; i++) {<br>            <br>   X509Certificate cert = chain[i];<br>   System.out.println(&quot; &quot; + (i + 1) + &quot; Subject &quot;<br>     + cert.getSubjectDN());<br>   System.out.println(&quot;   Issuer  &quot; + cert.getIssuerDN());<br>   sha1.update(cert.getEncoded());<br>   System.out.println(&quot;   sha1    &quot; + toHexString(sha1.digest()));<br>   md5.update(cert.getEncoded());<br>   System.out.println(&quot;   md5     &quot; + toHexString(md5.digest()));<br>   System.out.println();<br>  }<br> <br>  System.out<br>    .println(&quot;Enter certificate to add to trusted keystore or &#39;q&#39; to quit: [1]&quot;);<br>  String line = reader.readLine().trim();<br>  int k;<br>  try {<br>            <br>   k = (line.length() == 0) ? 0 : Integer.parseInt(line) - 1;<br>  } catch (NumberFormatException e) {<br>            <br>   System.out.println(&quot;KeyStore not changed&quot;);<br>   return;<br>  }<br> <br>  X509Certificate cert = chain[k];<br>  String alias = host + &quot;-&quot; + (k + 1);<br>  ks.setCertificateEntry(alias, cert);<br> <br>  OutputStream out = new FileOutputStream(&quot;jssecacerts&quot;);<br>  ks.store(out, passphrase);<br>  out.close();<br> <br>  System.out.println();<br>  System.out.println(cert);<br>  System.out.println();<br>  System.out<br>    .println(&quot;Added certificate to keystore &#39;jssecacerts&#39; using alias &#39;&quot;<br>      + alias + &quot;&#39;&quot;);<br> }<br> <br> private static final char[] HEXDIGITS = &quot;0123456789abcdef&quot;.toCharArray();<br> <br> private static String toHexString(byte[] bytes) {<br>            <br>  StringBuilder sb = new StringBuilder(bytes.length * 3);<br>  for (int b : bytes) {<br>            <br>   b &amp;= 0xff;<br>   sb.append(HEXDIGITS[b &gt;&gt; 4]);<br>   sb.append(HEXDIGITS[b &amp; 15]);<br>   sb.append(&#39; &#39;);<br>  }<br>  return sb.toString();<br> }<br> <br> private static class SavingTrustManager implements X509TrustManager {<br>            <br> <br>  private final X509TrustManager tm;<br>  private X509Certificate[] chain;<br> <br>  SavingTrustManager(X509TrustManager tm) {<br>            <br>   this.tm = tm;<br>  }<br> <br>  public X509Certificate[] getAcceptedIssuers() {<br>            <br>   throw new UnsupportedOperationException();<br>  }<br> <br>  public void checkClientTrusted(X509Certificate[] chain, String authType)<br>    throws CertificateException {<br>            <br>   throw new UnsupportedOperationException();<br>  }<br> <br>  public void checkServerTrusted(X509Certificate[] chain, String authType)<br>    throws CertificateException {<br>            <br>   this.chain = chain;<br>   tm.checkServerTrusted(chain, authType);<br>  }<br> }<br> <br>}</pre><p>Compile InstallCert.java and run</p><pre>javac InstallCert.java</pre><pre>java InstallCert {coordinator hostname} or<br>java InstallCert {coordinator hostname}:port or<br>java InstallCert {coordinator hostname}:port password</pre><p>Enter 1, enter, and generate a certificate named jssecacerts in the current directory. Copy the certificate to the $JAVA_HOME/jre/lib/security directory</p><pre>mv jssecacerts /etc/alternatives/jre/lib/security/</pre><h4>Step 4. Repeat Step 1~3 on each of your worker node. Remember to replace IP in Step 1 with current node IP.</h4><h4>Step 5. Configure Presto/Trino config.properties.</h4><p>Remain the same on each of your coordinator and worker node.</p><pre>discovery.uri=<a href="http://ip-10-186-17-52.ap-northeast-1.compute.internal:9002/">http://ip-10-186-17-52.ap-northeast-1.compute.internal:9002</a><br>http-server.authentication.type = PASSWORD,CERTIFICATE<br>http-server.https.enabled = true<br>node.internal-address-source = FQDN<br>http-server.http.enabled = true<br>http-server.https.port = 8889<br>http-server.http.port = 9002<br>http-server.https.keystore.path = /etc/presto_keystore.jks<br>http-server.https.keystore.key = {replace with you keystore password}</pre><p>Create an password-authenticator.properties file on the coordinator</p><p>Here you can configure the use of LDAP or password file authentication. We will use password file authentication.</p><p>For more configuration details, please refer to the official website: <a href="https://prestodb.io/docs/current/security/ldap.html">https://prestodb.io/docs/current/security/ldap.html</a></p><pre>password-authenticator.name=file <br>file.password-file=/etc/presto/conf/password.db</pre><h4>Step 6. Create Password File</h4><p>Create an empty password file to get started:</p><pre>touch password.db</pre><p>Add or update password for user ‘test’</p><pre>htpasswd -B -C 10 password.db test</pre><h4>Step 7. Restart Presto/Trino</h4><pre>systemctl restart presto-server<br>systemctl restart trino-server</pre><h3>Congratulation!</h3><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28232cb61bd4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Platform from the ground up-Setup Superset in production (High availability/Async) with…]]></title>
            <link>https://medium.com/@tom.xiao_3937/data-platform-from-the-group-up-setup-superset-in-production-high-availability-async-with-39a2bc9aab4?source=rss-2597b5c2304b------2</link>
            <guid isPermaLink="false">https://medium.com/p/39a2bc9aab4</guid>
            <category><![CDATA[superset]]></category>
            <category><![CDATA[high-availability]]></category>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[bi]]></category>
            <category><![CDATA[data-platforms]]></category>
            <dc:creator><![CDATA[Tom Xiao]]></dc:creator>
            <pubDate>Sun, 27 Feb 2022 12:37:37 GMT</pubDate>
            <atom:updated>2023-09-29T02:30:56.191Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Platform from the ground up-Setup Superset in production (High availability/Async) with Superset, Gunicorn, Gevent, Mysql, Redis, Celery, Flower, Supervisor</h3><p>In this article, we are going to setup a BI tool with <a href="https://superset.apache.org/">Superset</a> in production environment.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fmnBrqsqmFNINCCuaWzmRw.png" /></figure><blockquote>Superset is fast, lightweight, intuitive, and loaded with options that make it easy for users of all skill sets to explore and visualize their data, from simple line charts to highly detailed geospatial charts.</blockquote><h3>Environment Preparation</h3><ul><li>Make sure you have anaconda or miniconda installed on your machine.</li></ul><pre># Download latset version of miniconda<br>wget -c <a href="https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh">https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh</a><br>chmod 777 Miniconda3-latest-Linux-x86_64.sh<br>bash Miniconda3-latest-Linux-x86_64.sh<br>source ~/.bashrc</pre><ul><li>Create a vitual environment named ‘superset’.</li></ul><pre>conda create -n superset python=3.6 <br>conda activate superset</pre><ul><li>Install necessary packages.</li></ul><pre>pip3 install --upgrade setuptools pip </pre><pre># s3werkzeugcache: used for superset S3Cache<br>pip3 install apache-superset redis gevent celery flower gunicorn sqlalchemy==1.3.23 dataclasses mysqlclient pyhive pyhive[presto] psycopg2 s3werkzeugcache<br><br>yum install supervisor openldap-devel python3-devel</pre><h3>Configure Superset</h3><ul><li><strong>(Optional) To find out where is your superset installed.</strong></li></ul><pre># Enter your virtual environment<br>conda activate superset</pre><pre># Find where is superset<br>which superset</pre><pre>- My return: <strong>/opt/miniconda3/envs/superset</strong>/bin/superset</pre><pre># Search for config.py<br>find /opt/miniconda3/envs/superset/ -name config.py</pre><pre>- My return: /opt/miniconda3/envs/superset/lib/python3.6/site-packages/superset/config.py</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yjUmZ2OTbByaKCgcEpzwjA.png" /></figure><ul><li><strong>Create mysql database for superset.</strong></li></ul><pre>create database superset; <br>alter database superset character set latin1; <br>create user &#39;superset&#39;@&#39;%&#39; identified by &#39;superset&#39;; <br>set password for &#39;superset&#39;@&#39;localhost&#39; identified by &#39;superset&#39;; <br>GRANT ALL PRIVILEGES ON superset.* TO &#39;superset&#39;@&#39;%&#39; IDENTIFIED BY &#39;superset&#39; with grant option; <br>GRANT ALL PRIVILEGES ON mysql.* TO &#39;superset&#39;@&#39;%&#39; IDENTIFIED BY &#39;superset&#39; with grant option; <br>GRANT RELOAD ON *.* TO &#39;superset&#39;@&#39;localhost&#39;;</pre><ul><li><strong>Configure Superset config.py</strong></li></ul><pre>vim /opt/miniconda3/envs/superset/lib/python3.6/site-packages/superset/config.py</pre><p><em>a. Default database</em></p><pre>By default, Superset use SQLite as its backend database, which is apprently not suitable for maintaince. Replace SQLALCHEMY_DATABASE_URI with mysql in config.py.</pre><pre><em>SQLALCHEMY_DATABASE_URI = “mysql://superset:superset@localhost/superset?charset=UTF8”</em></pre><p><em>b. Celery configuration</em></p><blockquote>On large analytic databases, it’s common to run queries that execute for minutes or hours. To enable support for long running queries that execute beyond the typical web request’s timeout (30–60 seconds), it is necessary to configure an asynchronous backend for Superset which consists of:</blockquote><blockquote>- one or many Superset workers (which is implemented as a Celery worker), and can be started with the celery worker command, run celery worker --help to view the related options.</blockquote><blockquote>- a celery broker (message queue) for which we recommend using Redis or RabbitMQ</blockquote><blockquote>- a results backend that defines where the worker will persist the query results</blockquote><blockquote>Configuring Celery requires defining a CELERY_CONFIG in your superset_config.py. Both the worker and web server processes should have the same configuration.</blockquote><pre>class CeleryConfig:  # pylint: disable=too-few-public-methods<br>    BROKER_URL = &quot;redis://{username}:{password}<a href="/service/mailto:LKHSDFkjsldfyukjsdlfhls@master.aws-sg-redis-amber-risk-prod.65waxl.apse1.cache.amazonaws.com">@</a>{host}:{port}/{db}&quot;<br>    CELERY_IMPORTS = (&quot;superset.sql_lab&quot;, &quot;superset.tasks&quot;, &quot;superset.tasks.thumbnails&quot;)<br>    CELERY_RESULT_BACKEND = &quot;redis://{username}:{password}<a href="/service/mailto:LKHSDFkjsldfyukjsdlfhls@master.aws-sg-redis-amber-risk-prod.65waxl.apse1.cache.amazonaws.com">@</a>{host}:{port}/{db}&quot;<br>    CELERYD_LOG_LEVEL = &quot;DEBUG&quot;<br>    CELERYD_PREFETCH_MULTIPLIER = 1<br>    CELERY_ACKS_LATE = False<br>    CELERY_ANNOTATIONS = {<br>        &quot;sql_lab.get_sql_results&quot;: {&quot;rate_limit&quot;: &quot;100/s&quot;},<br>        &quot;email_reports.send&quot;: {<br>            &quot;rate_limit&quot;: &quot;1/s&quot;,<br>            &quot;time_limit&quot;: 120,<br>            &quot;soft_time_limit&quot;: 150,<br>            &quot;ignore_result&quot;: True,<br>        },<br>    }<br>    CELERYBEAT_SCHEDULE = {<br>        &quot;email_reports.schedule_hourly&quot;: {<br>            &quot;task&quot;: &quot;email_reports.schedule_hourly&quot;,<br>            &quot;schedule&quot;: crontab(minute=1, hour=&quot;*&quot;),<br>        }<br>    }</pre><blockquote>To setup a result backend, you need to pass an instance of a derivative of from cachelib.base.BaseCache to the RESULTS_BACKEND configuration key in your superset_config.py. You can use Memcached, Redis, S3 (<a href="https://pypi.python.org/pypi/s3werkzeugcache">https://pypi.python.org/pypi/s3werkzeugcache</a>), memory or the file system (in a single server-type setup or for testing), or to write your own caching interface. Your superset_config.py may look something like:</blockquote><pre><em># On S3</em><br>from s3cache.s3cache import S3Cache<br>S3_CACHE_BUCKET = &#39;foobar-superset&#39;<br>S3_CACHE_KEY_PREFIX = &#39;sql_lab_result&#39;<br>RESULTS_BACKEND = S3Cache(S3_CACHE_BUCKET, S3_CACHE_KEY_PREFIX)<br><br><em># On Redis</em><br>from cachelib.redis import RedisCache<br>RESULTS_BACKEND = RedisCache(<br>    host=&#39;localhost&#39;, port=6379, key_prefix=&#39;superset_results&#39;)</pre><p><em>c. Caching Configuration</em></p><blockquote>Superset uses <a href="https://flask-caching.readthedocs.io/">Flask-Caching</a> for caching purpose. For security reasons, there are two separate cache configs for Superset’s own metadata (CACHE_CONFIG) and charting data queried from connected datasources (DATA_CACHE_CONFIG). However, Query results from SQL Lab are stored in another backend called RESULTS_BACKEND, See <a href="https://superset.apache.org/docs/installation/async-queries-celery">Async Queries via Celery</a> for details.</blockquote><blockquote>Configuring caching is as easy as providing CACHE_CONFIG and DATA_CACHE_CONFIG in your superset_config.py that complies with <a href="https://flask-caching.readthedocs.io/en/latest/#configuring-flask-caching">the Flask-Caching specifications</a>.</blockquote><pre>CACHE_CONFIG: CacheConfig = {&quot;CACHE_TYPE&quot;: &quot;redis&quot;, &quot;CACHE_REDIS_URL&quot;: &quot;redis://{username}:{password}<a href="/service/mailto:LKHSDFkjsldfyukjsdlfhls@master.aws-sg-redis-amber-risk-prod.65waxl.apse1.cache.amazonaws.com">@</a>{host}:{port}/{db}&quot;, &quot;CACHE_KEY_PREFIX&quot;: &quot;SUPERSET_VIEW_CACHE_&quot;} </pre><pre>TABLE_NAMES_CACHE_CONFIG: CacheConfig = {&quot;CACHE_TYPE&quot;: &quot;redis&quot;, &quot;CACHE_REDIS_URL&quot;: &quot;redis://{username}:{password}<a href="/service/mailto:LKHSDFkjsldfyukjsdlfhls@master.aws-sg-redis-amber-risk-prod.65waxl.apse1.cache.amazonaws.com">@</a>{host}:{port}/{db}&quot;, &quot;CACHE_KEY_PREFIX&quot;: &quot;SUPERSET_TABLE_NAMES_CACHE_&quot;}</pre><pre>DATA_CACHE_CONFIG: CacheConfig = {&quot;CACHE_TYPE&quot;: &quot;redis&quot;, &quot;CACHE_REDIS_URL&quot;: &quot;redis://{username}:{password}<a href="/service/mailto:LKHSDFkjsldfyukjsdlfhls@master.aws-sg-redis-amber-risk-prod.65waxl.apse1.cache.amazonaws.com">@</a>{host}:{port}/{db}&quot;, &quot;CACHE_KEY_PREFIX&quot;: &quot;SUPERSET_DATA_CACHE_&quot;}</pre><p>d. Flask configuration to enable public access to Superset host and port.</p><pre>vim /opt/miniconda3/envs/superset/lib/python3.6/site-packages/falsk/cli.py</pre><pre>@click.option(&quot;--host&quot;, &quot;-h&quot;, default=&quot;0.0.0.0&quot;, help=&quot;The interface to bind to.&quot;) <br>@click.option(&quot;--port&quot;, &quot;-p&quot;, default=8088, help=&quot;The port to bind to.&quot;)<br># change 8088 to which IP you want to host superset.</pre><ul><li><strong>Initialize Superset</strong></li></ul><pre>superset db upgrade<br>export FLASK_APP=superset<br>flask fab create-admin<br>superset load_examples<br>superset init</pre><p>Now you can run superset with <strong><em>‘superset run’</em></strong> command</p><ul><li>Start Celery and Flower</li></ul><pre>celery worker --app=superset.tasks.celery_app:app --pool=prefork --max-tasks-per-child=128 -O fair -c 4<br>celery flower --app=superset.tasks.celery_app:app --port=7386</pre><p>So far we are able to start Superset, Celery and Flower separately. In the next part we are running these processes with Supervisord.</p><h3>Configure Supervisord</h3><ul><li>Configure gunicorn</li></ul><pre>mkdir /opt/gunicorn<br>vim /opt/gunicorn/gunicorn.py</pre><pre>import multiprocessing</pre><pre>bind = &#39;0.0.0.0:9000&#39; # Bind to your superset port<br>workers = multiprocessing.cpu_count() * 2 + 1<br>threads = 2</pre><pre>timeout = 30<br>backlog = 2048<br>worker_class = &quot;gevent&quot;<br>worker_connections = 2000<br>daemon = False<br>debug = False<br>proc_name = &#39;app&#39;<br>loglevel = &#39;warning&#39;<br>pidfile = &#39;/opt/gunicorn/pid&#39;<br>errorlog = &#39;/opt/gunicorn/superset_error.log&#39;<br>accesslog = &#39;/opt/gunicorn/superset_access.log&#39;</pre><p>Now we are able to start superset with below command:</p><pre>gunicorn -c /opt/gunicorn/gunicorn.py superset:app</pre><ul><li>Configure Supervisord<br>Make sure you have superviosrd installed first.</li></ul><pre># Create log directory for supervisord<br>mkdir -p /opt/supervisor/superset /opt/supervisor/flower /opt/supervisor/celery<br>vim /etc/supervisord.conf</pre><pre>[program:superset]<br>command=/opt/miniconda3/envs/superset/bin/gunicorn -c /opt/gunicorn/gunicorn.py &quot;superset.app:create_app()&quot;<br>directory=/opt/miniconda3/envs/superset/bin/<br>autostart=true<br>startsecs=10<br>autorestart=true<br>startretries=5<br>user=root<br>numprocs=1<br>priority=979<br>exitcodes=0<br>stopsignal=KILL<br>stopwaitsecs=10<br>redirect_stderr=true<br>stdout_logfile_maxbytes=50MB<br>stdout_logfile_backups = 10<br>stdout_logfile=/opt/supervisor/superset/stdout.log<br>stopasgroup=false<br>killasgroup=false</pre><pre>[program:celery]<br>command=/opt/miniconda3/envs/superset/bin/celery worker --app=superset.tasks.celery_app:app --pool=prefork -O fair -c 4<br>directory=/opt/miniconda3/envs/superset/bin/<br>autostart=true<br>startsecs=10<br>autorestart=true<br>startretries=5<br>user=root<br>numprocs=1<br>priority=980<br>exitcodes=0<br>stopsignal=KILL<br>stopwaitsecs=10<br>redirect_stderr=true<br>stdout_logfile_maxbytes=50MB<br>stdout_logfile_backups = 10<br>stdout_logfile=/opt/supervisor/celery/stdout.log<br>stopasgroup=false<br>killasgroup=false</pre><pre>[program:flower]<br>command=/opt/miniconda3/envs/superset/bin/celery flower --app=superset.tasks.celery_app:app --port=7386<br>directory=/opt/miniconda3/envs/superset/bin/<br>autostart=true<br>startsecs=10<br>autorestart=true<br>startretries=5<br>user=root<br>numprocs=1<br>priority=981<br>exitcodes=0<br>stopsignal=KILL<br>stopwaitsecs=10<br>redirect_stderr=true<br>stdout_logfile_maxbytes=50MB<br>stdout_logfile_backups = 10<br>stdout_logfile=/opt/supervisor/flower/stdout.log<br>stopasgroup=false<br>killasgroup=false</pre><ul><li>Now start supervisord</li></ul><pre>service supervisor start</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x_MFmqe-XE1EXbVQBH6r-A.png" /></figure><p>Congratulation!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=39a2bc9aab4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS ElastiCache(Redis) With TLS]]></title>
            <link>https://medium.com/@tom.xiao_3937/aws-elasticache-redis-with-tls-e1243ad5175b?source=rss-2597b5c2304b------2</link>
            <guid isPermaLink="false">https://medium.com/p/e1243ad5175b</guid>
            <category><![CDATA[redis]]></category>
            <category><![CDATA[ssl]]></category>
            <dc:creator><![CDATA[Tom Xiao]]></dc:creator>
            <pubDate>Sun, 06 Feb 2022 08:55:12 GMT</pubDate>
            <atom:updated>2022-02-06T08:55:12.169Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>To help keep your data secure, Amazon ElastiCache and Amazon EC2 provide mechanisms to guard against unauthorized access of your data on the server. By providing in-transit encryption capability, ElastiCache gives you a tool you can use to help protect your data when it is moving from one location to another. For example, you might move data from a primary node to a read replica node within a replication group, or between your replication group and your application.</blockquote><blockquote>In-transit encryption is optional and can only be enabled on Redis replication groups when they are created. You enable in-transit encryption on a replication group by setting the parameter TransitEncryptionEnabled to true (CLI: --transit-encryption-enabled) when you create the replication group. You can do this whether you are creating the replication group using the AWS Management Console, the AWS CLI, or the ElastiCache API. If you enable in-transit encryption, you must also provide a value for CacheSubnetGroup.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c8Z5s34AX_dTAExILhkZ7A.jpeg" /><figcaption>Create ElastiCache Cluster on AWS console with TLS enabled.</figcaption></figure><h3>Connecting to Amazon ElastiCache for Redis nodes enabled with in-transit encryption using redis-cli</h3><ol><li>Download and compile the redis-cli utility. This utility is included in the Redis software distribution.</li><li>At the command prompt of your EC2 instance, type the following commands:</li></ol><p>Amazon Linux 2</p><pre>sudo yum -y install openssl-devel gcc<br>wget <a href="http://download.redis.io/redis-stable.tar.gz">http://download.redis.io/redis-stable.tar.gz</a><br>tar xvzf redis-stable.tar.gz<br>cd redis-stable<br>make distclean<br>make redis-cli BUILD_TLS=yes<br>sudo install -m 755 src/redis-cli /usr/local/bin/</pre><p>Amazon Linux</p><pre>sudo yum install gcc jemalloc-devel openssl-devel tcl tcl-devel clang wget <br>wget <a href="http://download.redis.io/redis-stable.tar.gz">http://download.redis.io/redis-stable.tar.gz</a><br>tar xvzf redis-stable.tar.gz<br>cd redis-stable<br>make redis-cli CC=clang BUILD_TLS=yes<br>sudo install -m 755 src/redis-cli /usr/local/bin/<br>sudo yum install clang <br>CC=clang make<br>sudo make install</pre><p>3. After this, it is recommended that you run the optional make-test command.</p><p>4. Type the following command, substituting the endpoint of your cluster and port for what is shown in this example.</p><pre>redis-cli -h Primary or Configuration Endpoint --tls -p 6379</pre><p>The following example connects to a cluster with encryption and authentication enabled (not recommended)</p><pre>redis-cli -h Primary or Configuration Endpoint --tls -a &#39;your-password&#39; -p 6379</pre><h3>Connecting to Amazon ElastiCache for Redis nodes enabled with in-transit encryption using python packages (take aioredis as an example)</h3><p>aioredis==1.3.0</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Y9PMSfHsG6EQACQPjBI6Tg.jpeg" /></figure><pre>import aioredis<br>pool = aioredis.create_pool(&quot;redis://[host]:6379&quot;, ssl=True)</pre><p>aioredis==2.0.0</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o08KA83jl82Y6V_c2siGzg.jpeg" /></figure><pre>import aioredis<br>redis = aioredis.from_url(https://medium.com/&quot;rediss://[[username]:[password]]@[host]:6379/0&quot;)</pre><p><em>Originally published at </em><a href="https://bottlenose-answer-db6.notion.site/AWS-ElastiCache-Redis-With-TLS-2091071176ee456984290d92a3511be8"><em>https://bottlenose-answer-db6.notion.site</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e1243ad5175b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Network — All You Need to Know (WIP_]]></title>
            <link>https://medium.com/@tom.xiao_3937/aws-network-all-you-need-to-know-wip-bc19beb934bd?source=rss-2597b5c2304b------2</link>
            <guid isPermaLink="false">https://medium.com/p/bc19beb934bd</guid>
            <category><![CDATA[data-platforms]]></category>
            <dc:creator><![CDATA[Tom Xiao]]></dc:creator>
            <pubDate>Mon, 03 Jan 2022 15:05:34 GMT</pubDate>
            <atom:updated>2022-01-03T15:05:34.648Z</atom:updated>
            <content:encoded><![CDATA[<h3>AWS Network — All You Need to Know (WIP)</h3><p>In this article, we are going to build up entire network on AWS to support our cloud services. The architecture is shown below.</p><h3>1. Concepts</h3><h4>1.1 VPC</h4><p>A virtual private cloud — VPC — is a private network space in which you can run your infrastructure. It has an address space (CIDR range) which you choose e.g. 10.0.0.0/16. This determines how many IP addresses you can assign within the VPC. Each server you create inside the VPC will need an IP address so this address space defines the limit of how many resources you can have within the network. The 10.0.0.0/16 address space can use the addresses from 10.0.0.0 to 10.0.255.255, which is 65,536 IP addresses.</p><blockquote>The VPC is the basis of your network on AWS and all new accounts include a default VPC with a subnets in each availability zone.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*b78aqbrje_-gBmN5Fq47DQ.png" /><figcaption>This is a default VPC.</figcaption></figure><h4>1.2 Availability Zone</h4><blockquote>An Availability Zone (AZ) is one or more discrete data centers with redundant power, networking, and connectivity in an AWS Region. AZs give customers the ability to operate production applications and databases that are more highly available, fault tolerant, and scalable than would be possible from a single data center. All AZs in an AWS Region are interconnected with high-bandwidth, low-latency networking, over fully redundant, dedicated metro fiber providing high-throughput, low-latency networking between AZs. All traffic between AZs is encrypted. The network performance is sufficient to accomplish synchronous replication between AZs. AZs make partitioning applications for high availability easy. If an application is partitioned across AZs, companies are better isolated and protected from issues such as power outages, lightning strikes, tornadoes, earthquakes, and more. AZs are physically separated by a meaningful distance, many kilometers, from any other AZ, although all are within 100 km (60 miles) of each other.</blockquote><h4>1.3 Region</h4><blockquote>AWS has the concept of a Region, which is a physical location around the world where we cluster data centers. We call each group of logical data centers an Availability Zone. Each AWS Region consists of multiple, isolated, and physically separate AZs within a geographic area. Unlike other cloud providers, who often define a region as a single data center, the multiple AZ design of every AWS Region offers advantages for customers. Each AZ has independent power, cooling, and physical security and is connected via redundant, ultra-low-latency networks. AWS customers focused on high availability can design their applications to run in multiple AZs to achieve even greater fault-tolerance. AWS infrastructure Regions meet the highest levels of security, compliance, and data protection.</blockquote><blockquote>AWS provides a more extensive global footprint than any other cloud provider, and to support its global footprint and ensure customers are served across the world, AWS opens new Regions rapidly. AWS maintains multiple geographic Regions, including Regions in North America, South America, Europe, China, Asia Pacific, South Africa, and the Middle East.</blockquote><h4>1.4 Subnet</h4><p>A subnet is a section of your VPC, with its own CIDR range and rules on how traffic can flow. Its CIDR range has to be a subset of the VPC’s, for example 10.0.1.0/24 which would allow for IPs from 10.0.1.0 to 10.0.1.255 giving 256 possible IP addresses.</p><p>Subnets are often denominated as ‘public’ or ‘private’ depending on whether traffic can reach them from outside the VPC (the Internet). This visibility is controlled by the traffic routing rules and each subnet can have its own rules.</p><p>A subnet has to be in a specific availability zone within a region so it’s good practice to have a subnet in each zone. If you plan to have public and private subnets then there should be one of each per availability zone.</p><blockquote>CIDR is principally a bitwise, prefix-based standard for the representation of IP addresses and their routing properties. It facilitates routing by allowing blocks of addresses to be grouped into single routing table entries. These groups, commonly called CIDR blocks, share an initial sequence of bits in the binary representation of their IP addresses. IPv4 CIDR blocks are identified using a syntax similar to that of IPv4 addresses: a dotted-decimal address, followed by a slash, then a number from 0 to 32, i.e., a.b.c.d/n. The dotted decimal portion is the IPv4 address. The number following the slash is the prefix length, the number of shared initial bits, counting from the most-significant bit of the address. When emphasizing only the size of a network, the address portion of the notation is usually omitted. Thus, a /20 block is a CIDR block with an unspecified 20-bit prefix.</blockquote><blockquote>An IP address is part of a CIDR block and is said to match the CIDR prefix if the initial n bits of the address and the CIDR prefix are the same. An IPv4 address is 32 bits so an n-bit CIDR prefix leaves 32 − n bits unmatched, meaning that 232−n IPv4 addresses match a given n-bit CIDR prefix. Shorter CIDR prefixes match more addresses, while longer prefixes match fewer. In the case of overlaid CIDR blocks, an address can match multiple CIDR prefixes of different lengths.</blockquote><blockquote>CIDR is also used for <a href="https://en.wikipedia.org/wiki/IPv6_address">IPv6 addresses</a> and the syntax semantic is identical. The prefix length can range from 0 to 128, due to the larger number of bits in the address. However, by convention, a subnet on broadcast MAC layer networks always has 64-bit host identifiers. Larger prefixes are rarely used even on point-to-point links.[<a href="https://en.wikipedia.org/wiki/Wikipedia:Citation_needed">citation needed</a>]</blockquote><h4>1.5 Routing Table</h4><p>A routing table contains rules about how IP packets in the subnets can travel to different IP addresses. There is always a default route table which will only allow traffic to travel locally, within the VPC. If a subnet has no routing table associated with it then it uses the default one. These would be ‘private’ subnets.</p><p>If you want external traffic to be able to get to a subnet then you need to create a routing table with a rule explicitly allowing this. Subnets associated to that routing table would be ‘public’.</p><p>All of the subnets in the default VPCs are associated with a route table which makes them public.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*y80EfZuBdYQ-HZMDEZoNoQ.png" /><figcaption>This is a default routing table.</figcaption></figure><h4>1.6 Internal Gateway</h4><p>The routing table which makes a subnet public needs to reference an Internet gateway to allow the flow of external IP packets into and out of the VPC. You create your Internet gateway and then create a rule which says that packets to 0.0.0.0/0 - all IP addresses - need to go to there.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-m46UwjwHtmP-RNX8-6cPg.png" /><figcaption>This is a default internal gateway.</figcaption></figure><h4>1.7 NAT Gateway</h4><p>If you have an EC2 instance in a private subnet — one which doesn’t allow traffic from the Internet to reach it — then there’s also no way for IP packets to reach the Internet. We need a mechanism for sending those packets out, and then routing the replies correctly. This is called network address translation and is very likely done in your house by your wifi router.</p><p>A NAT gateway is a device which sits in the public subnets, accepts any IP packets bound for the Internet coming from the private subnets, sends those packets on to their destination and then sends the returning packets back to the source.</p><p>It’s not necessary to have NAT gateways if you don’t intend instances in your private subnets to talk outside if your VPC but if you do need to do that e.g. using an external API, SaaS database etc. then you can simply set up an EC2 instance (might be cheaper, depending on your traffic), configured appropriately, or use an AWS managed NAT gateway resource (will be easier to manage because you won’t be doing it).</p><p>Here is what’s happening when a request is send from an VM inside a private subnet to somewhere on the Internet.</p><ul><li>The public subnet contains the NAT gateway</li><li>A request is made from the private subnet to an IP address somewhere on the Internet</li><li>The route table says that it needs to go to the NAT gateway</li><li>The NAT gateway sends it on</li></ul><h4>1.8 Security Group</h4><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bc19beb934bd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Data Platform from the ground up — 1. Cloud Service]]></title>
            <link>https://medium.com/@tom.xiao_3937/data-platform-from-the-ground-up-1-cloud-service-53d2bdb0b584?source=rss-2597b5c2304b------2</link>
            <guid isPermaLink="false">https://medium.com/p/53d2bdb0b584</guid>
            <category><![CDATA[data-platforms]]></category>
            <dc:creator><![CDATA[Tom Xiao]]></dc:creator>
            <pubDate>Fri, 17 Dec 2021 10:07:50 GMT</pubDate>
            <atom:updated>2022-01-03T15:07:20.697Z</atom:updated>
            <content:encoded><![CDATA[<h3>Data Platform from the ground up — 1. Cloud Services Setup (WIP)</h3><p>We are build our data platform on AWS so the very first step is to apply for necessary cloud services. Required resources are listed below:</p><ul><li><a href="https://aws.amazon.com/vpc/">VPC</a>: <em>Amazon Virtual Private Cloud (Amazon VPC) gives you full control over your virtual networking environment, including resource placement, connectivity, and security. Get started by setting up your VPC in the AWS service console. Next, add resources to it such as Amazon Elastic Compute Cloud (EC2) and Amazon Relational Database Service (RDS) instances. Finally, define how your VPCs communicate with each other across accounts, Availability Zones, or AWS Regions. In the example below, network traffic is being shared between two VPCs within each Region.</em></li><li><a href="https://aws.amazon.com/ec2/?ec2-whats-new.sort-by=item.additionalFields.postDateTime&amp;ec2-whats-new.sort-order=desc">EC2</a>: <em>Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides secure, resizable compute capacity in the cloud. It is designed to make web-scale cloud computing easier for developers. Amazon EC2’s simple web service interface allows you to obtain and configure capacity with minimal friction. It provides you with complete control of your computing resources and lets you run on Amazon’s proven computing environment.</em></li><li><a href="https://aws.amazon.com/emr/">EMR</a>: <em>Amazon EMR is a cloud big data platform for running large-scale distributed data processing jobs, interactive SQL queries, and machine learning (ML) applications using open-source analytics frameworks such as </em><a href="https://aws.amazon.com/emr/features/spark/"><em>Apache Spark</em></a><em>, </em><a href="https://aws.amazon.com/emr/features/hive/"><em>Apache Hive</em></a><em>, and </em><a href="https://aws.amazon.com/emr/features/presto/"><em>Presto</em></a><em>.</em></li><li><a href="https://aws.amazon.com/rds/mysql/">RDS for MySQL</a>: <a href="https://aws.amazon.com/rds/mysql/what-is-mysql/"><em>MySQL</em></a><em> is the world’s most popular </em><a href="https://aws.amazon.com/products/databases/open-source-databases/"><em>open source</em></a><em> relational database and Amazon RDS makes it easy to set up, operate, and scale MySQL deployments in </em><a href="https://aws.amazon.com/what-is-cloud-computing/"><em>the cloud</em></a><em>. With Amazon RDS, you can deploy scalable MySQL servers in minutes with cost-efficient and resizable hardware capacity.<br>Amazon RDS for MySQL frees you up to focus on application development by managing time-consuming database administration tasks including backups, software patching, monitoring, scaling and replication.<br>Amazon RDS supports MySQL Community Edition versions 5.6, 5.7, and 8.0 which means that the code, applications, and tools you already use today can be used with Amazon RDS.</em></li><li><a href="https://aws.amazon.com/redis/">Redis</a>: <em>Redis, which stands for Remote Dictionary Server, is a fast, open source, in-memory, key-value data store. The project started when Salvatore Sanfilippo, the original developer of Redis, wanted to improve the scalability of his Italian startup. From there, he developed Redis, which is now used as a database, cache, message broker, and queue.<br>Redis delivers sub-millisecond response times, enabling millions of requests per second for real-time applications in industries like gaming, ad-tech, financial services, healthcare, and IoT. Today, Redis is one of the most popular open source engines today, named the </em><a href="https://insights.stackoverflow.com/survey/2021"><em>“Most Loved” database by Stack Overflow</em></a><em> for five consecutive years. Because of its fast performance, Redis is a popular choice for </em><a href="https://aws.amazon.com/caching/"><em>caching</em></a><em>, session management, gaming, leaderboards, real-time analytics, geospatial, ride-hailing, chat/messaging, media streaming, and pub/sub apps.<br>AWS offers two fully managed services to run Redis. </em><a href="https://aws.amazon.com/memorydb/"><em>Amazon MemoryDB for Redis</em></a><em> is a Redis-compatible, durable, in-memory database service that delivers ultra-fast performance. </em><a href="https://aws.amazon.com/elasticache/"><em>Amazon ElastiCache for Redis</em></a><em> is a fully managed caching service that accelerates data access from primary databases and data stores with microsecond latency. Furthermore, ElastiCache also offers support for Memcached, another popular open source caching engine.</em></li><li><a href="https://aws.amazon.com/msk/">MSK (Kafka)</a>: <em>Securely stream data with a fully managed, highly available Apache Kafka service</em></li><li><a href="https://aws.amazon.com/s3/">S3</a>: An object storage service offering industry-leading scalability, data availability, security, and performance. Customers of all sizes and industries can store and protect any amount of data for virtually any use case, such as data lakes, cloud-native applications, and mobile apps. With cost-effective storage classes and easy-to-use management features, you can optimize costs, organize data, and configure fine-tuned access controls to meet specific business, organizational, and compliance requirements. Similar to <a href="https://cloud.google.com/storage">Google Cloud Storage</a> and <a href="https://cn.aliyun.com/product/oss">Ali Cloud OOS</a>.</li></ul><p><strong>1.1 VPC</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=53d2bdb0b584" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>