<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Raghav Patel on Medium]]></title>
        <description><![CDATA[Stories by Raghav Patel on Medium]]></description>
        <link>https://medium.com/@raghav-patel?source=rss-b5f51c0e64c3------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*OoWYZTR5j6Cs3J0K.jpg</url>
            <title>Stories by Raghav Patel on Medium</title>
            <link>https://medium.com/@raghav-patel?source=rss-b5f51c0e64c3------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Thu, 19 Nov 2026 10:17:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@raghav-patel/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[The Ultimate Guide to PySpark Dependency Management on Dataproc with Conda (2025)]]></title>
            <link>https://raghav-patel.medium.com/stop-your-pyspark-jobs-from-failing-the-data-scientists-guide-to-bulletproof-dependencies-with-00f749e96048?source=rss-b5f51c0e64c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/00f749e96048</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[dataproc]]></category>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[conda]]></category>
            <dc:creator><![CDATA[Raghav Patel]]></dc:creator>
            <pubDate>Mon, 15 Sep 2025 13:56:49 GMT</pubDate>
            <atom:updated>2025-09-15T14:19:13.736Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1016/1*QD8luml8WaN_mMW16QKB9A.png" /></figure><p>If you’re a data scientist using PySpark on Google Cloud Dataproc, you’ve almost certainly faced the dreaded ModuleNotFoundError. You spend hours crafting a perfect script on your local machine, only to have it fail on the cluster due to a missing library or a subtle version conflict. This guide solves that problem for good.</p><p>Here, you’ll find a step-by-step tutorial on mastering <a href="https://spark.apache.org/docs/latest/api/python/index.html#pyspark-overview"><strong>PySpark</strong></a><strong> dependency management</strong>. We’ll show you how to use <a href="https://anaconda.org/anaconda/conda"><strong>Conda</strong></a> to create self-contained, portable environments, ensuring your <a href="https://cloud.google.com/dataproc">Dataproc</a> jobs run reliably and predictably every single time.</p><h3>Why Standard Dataproc Environments Fail for Data Science</h3><p>A shared Google Cloud Dataproc cluster is a minefield for Python dependencies. Different projects often require conflicting library versions:</p><ul><li>One job needs pandas==1.5.3.</li><li>A legacy job requires pandas==1.3.5.</li><li>Your new project needs pandas==2.2.0.</li></ul><p>Installing a single version cluster-wide is not a scalable solution; it will inevitably break other jobs. While initialization actions can help, they are often slow and complex to maintain. This is where managing dependencies on a per-job basis becomes essential.Our Solution: The “Magic Backpack” 🎒</p><h3>The Solution: Isolated Job Environments with Conda</h3><p>The best practice for robust <strong>PySpark dependency management</strong> is to package your environment with your code. We will use <strong>Conda</strong>, a powerful open-source package and environment manager, to create a perfect, isolated “bubble” for each job.</p><p>This Conda environment will contain:</p><ol><li>A specific version of Python.</li><li>The exact versions of all the libraries your job needs (numpy, scikit-learn, pandas, etc.).</li></ol><p>When you submit a job, Dataproc unpacks this environment, and your script runs inside it, completely ignorant of what’s installed on the main cluster. The result is perfect reproducibility and an end to dependency hell.</p><h3>Step-by-Step: Packaging a Conda Environment for Dataproc</h3><p>Let’s walk through the entire workflow, from creating the environment on your Mac to submitting the job.</p><h4>Step 1: Install Miniconda on Your Machine</h4><p>First, you need Conda. We recommend <strong>Miniconda</strong>, a minimal, lightweight installer.</p><ul><li><strong>On macOS:</strong></li></ul><ol><li>Visit the <a href="https://docs.conda.io/en/latest/miniconda.html">Miniconda documentation page</a>.</li><li>Download and run the .pkg installer for your Mac.</li><li>Restart your Terminal. Verify the installation by running conda --version.</li></ol><ul><li><strong>For Windows &amp; Linux:</strong> Instructions are available on the same download page.</li></ul><h4>Step 2: Create and Activate Your Conda Environment</h4><p>Next, define an environment for your project. Be specific with naming and versions. For this example, we’ll create pyspark_numpy_env.</p><ul><li><strong>Create the Environment:</strong> This command creates an environment with Python 3.9, a specific numpy version, and conda-pack—the tool we need to bundle it.</li></ul><pre>conda create -y -n pyspark_numpy_env python=3.9 numpy=1.21.5 conda-pack</pre><ul><li><strong>Activate the Environment:</strong></li></ul><pre>conda activate pyspark_numpy_env</pre><p>Your terminal prompt will now be prefixed with (pyspark_numpy_env).</p><h4>Step 3: Package the Conda Environment with conda-pack</h4><p>With your environment active, use conda-pack to compress it into a single tar.gz file.Bash</p><pre>conda pack -o pyspark_numpy_env.tar.gz</pre><p>This creates the pyspark_numpy_env.tar.gz archive in your current directory. This file is your portable environment.</p><h4>Step 4: Upload Your Script and Environment to GCS</h4><p>Now, place your PySpark script and your packed environment file into a Google Cloud Storage (GCS) bucket where Dataproc can access them.</p><pre># Set your GCS bucket variable<br>export GCS_BUCKET=&quot;gs://your-dataproc-bucket&quot;<br><br># Upload the packaged environment<br>gsutil cp pyspark_numpy_env.tar.gz ${GCS_BUCKET}/envs/<br><br># Upload your PySpark script<br>gsutil cp your_script.py ${GCS_BUCKET}/scripts/</pre><h3>How to Submit a PySpark Job with a Custom Conda Environment</h3><p>This final step uses a gcloud command to submit your job, telling Dataproc where to find and how to use your custom environment.</p><pre># --- Configuration ---<br>export REGION=&quot;us-central1&quot;<br>export CLUSTER_NAME=&quot;your-cluster&quot;<br>export GCS_BUCKET=&quot;gs://your-dataproc-bucket&quot;<br>export SCRIPT_PATH=&quot;${GCS_BUCKET}/scripts/your_script.py&quot;<br>export ENV_PATH=&quot;${GCS_BUCKET}/envs/pyspark_numpy_env.tar.gz&quot;<br><br># --- gcloud Submit Command ---<br>gcloud dataproc jobs submit pyspark ${SCRIPT_PATH} \<br>    --cluster=${CLUSTER_NAME} \<br>    --region=${REGION} \<br>    --archives=${ENV_PATH}#conda_env \<br>    --properties spark.executorEnv.PYSPARK_PYTHON=./conda_env/bin/python,spark.dataproc.driverEnv.PYSPARK_PYTHON=./conda_env/bin/python</pre><h4>Breaking Down the Key Flags:</h4><ul><li>--archives=${ENV_PATH}#conda_env: This is the crucial part. It tells Dataproc to download your .tar.gz file, unpack it on every node, and create a directory named <strong>conda_env</strong> containing its contents. The #conda_env alias is essential.</li><li>--properties=spark.pyspark.python=./conda_env/bin/python: This property overrides Spark&#39;s default Python path. It tells the Spark executors to use the Python executable <em>inside</em> the directory we just unpacked.</li></ul><h3>Conclusion: Reliable Data Science on Dataproc</h3><p>By packaging your Python dependencies with <strong>Conda</strong>, you build more robust and reliable data pipelines on <strong>Google Cloud Dataproc</strong>. This method eliminates one of the most common sources of failure in distributed computing, allowing you to focus on building great data science models, not debugging environments.</p><p><strong>If you found this guide helpful, leave a comment or share it with a colleague!</strong> What other Dataproc challenges are you facing?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=00f749e96048" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Configuring Tetrate (an Enterprise solutions for Istio) Ingress Gateway to Expose Services out-side…]]></title>
            <link>https://raghav-patel.medium.com/configuring-tetrate-an-enterprise-solutions-for-istio-ingress-gateway-to-expose-services-out-side-239e9e961f51?source=rss-b5f51c0e64c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/239e9e961f51</guid>
            <category><![CDATA[istio]]></category>
            <category><![CDATA[elastickubernetesservice]]></category>
            <category><![CDATA[tetrate]]></category>
            <category><![CDATA[ingress]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Raghav Patel]]></dc:creator>
            <pubDate>Wed, 29 Nov 2023 19:16:44 GMT</pubDate>
            <atom:updated>2023-11-29T19:27:38.985Z</atom:updated>
            <content:encoded><![CDATA[<h3>Configuring Tetrate (an Enterprise solutions for Istio) Ingress Gateway to Expose Services out-side Kubernetes cluster</h3><p>Welcome to this guide, where I’ll walk you through the process of setting up the <strong>Tetrate Ingress Gateway</strong> in Kubernetes to make your microservices accessible outside the kubernetes cluster. In this demonstration, We’ll use the AWS manage kubernetes service EKS to run our kubernetes cluster. Also in this post we’ll only focus on install and configure the Ingress gateway and assumed that we’ve already running Kubernetes cluster and installed Tetrate Service Bridge(Enterprice Istio Solution)</p><p>For those unfamiliar with Tetrate, you can refer to the official <a href="https://docs.tetrate.io/">documentation</a>. Tetrate serves as an enterprise solution for Istio, providing powerful features for managing and securing microservices in Kubernetes environments.</p><h3>Prerequisites</h3><p>Before delving into the setup of the Ingress Gateway in Tetrate, it’s essential to familiarize ourselves with key terminologies that will be integral to the configuration process. These terminologies are employed during the Ingress Gateway setup.</p><h4>Organization</h4><p>In the Tetrate Service Bridge (TSB), an organization is the designation given to a corporation that shares a common infrastructure. It serves as the overarching entity managing all individual teams within the corporation.</p><h4>Tenant</h4><p>A tenant represents a group within an organization, such as a team or department. Tenants share organizational resources and possess specific privileges, including read and write access. This structure allows for the effective management of resources among different groups within the organization.</p><h4>Workspace</h4><p>A workspace acts as a meticulously partitioned zone where teams exclusively manage their namespaces. It serves as the centralized area for maintaining all service mesh-related configurations associated with a team’s namespaces. This spans across various Virtual Machines and Kubernetes clusters.</p><h4>Group</h4><p>A group is a logical consolidation of resources within a workspace. It can take the form of a gateway, traffic, or security group, providing a systematic way to organize and manage these resources for streamlined service mesh configuration.</p><p>Now that we have a grasp of the essential terminologies within Tetrate Service Bridge required for setting up the Ingress Gateway, let’s proceed with the assumption that our readers have already established the Organization, Tenant, and Workspace. In this demonstration, our primary focus will be on the creation of Groups. Within these Groups, we will specifically delve into the process of crafting the Ingress Gateway resource.</p><h3>Create the namespace</h3><p>To create the namespace, where we’ll install all the Ingress Resouces, Run the following command:</p><pre>kubectl create ns ingress-gateway</pre><h3>Setup Groups</h3><p>To create the TSB resource <strong>group</strong>, utilize the following command , which creates the TSB group. Replace the placeholder values with your specific information.</p><pre>cat &lt;&lt;EOF &gt; ingress-gateway-group.yaml<br>apiVersion: gateway.tsb.tetrate.io/v2<br>kind: Group<br>metadata:<br>  organization: &lt;org-name&gt; #&lt;--Put your organization name here<br>  tenant: &lt;tenant-name&gt; #&lt;--Put your tenant name here<br>  workspace: &lt;workspace-name&gt; #&lt;--Put your workspace name here<br>  name: ingress-gateway-group #&lt;--Give the name of group<br>spec:<br>  namespaceSelector:<br>    names:<br>      - &quot;*/ingress-gateway&quot; #ingress-gateway is namespace where we&#39;ll deploy ingress-gateway pod.<br>  configMode: BRIDGED<br>EOF</pre><p>After replacing the placeholders with meaningful values, execute the following tctl command to create the group:</p><pre>tctl apply -f ingress-gateway-group.yaml</pre><p>After the group is successfully created, you can verify its existence by logging into the TSB console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*omU0IbEXGtMjEwmdmPciHQ.png" /></figure><h3>Install Tetrate Ingress Gateway</h3><p>Now that we have set up the TSB resource group, the next step is to install the Ingress Gateway resource within that group. The following command generates a YAML file for the Ingress Gateway Resource:</p><pre>cat &lt;&lt;EOF &gt; install-ingress-gateway.yaml<br>apiVersion: install.tetrate.io/v1alpha1<br>kind: IngressGateway<br>metadata:<br>  name: ingress-gateway <br>  namespace: ingress-gateway<br>spec:<br>  kubeSpec:<br>    deployment:<br>      hpaSpec:<br>        maxReplicas: 10<br>        metrics:<br>        - resource:<br>            name: cpu<br>            targetAverageUtilization: 70<br>          type: Resource<br>        minReplicas: 1<br>      replicaCount: 1<br>      resources:<br>        limits:<br>          cpu: 1000m<br>          memory: 2Gi<br>        requests:<br>          cpu: 700m<br>          memory: 128Mi<br>      strategy:<br>        rollingUpdate:<br>          maxUnavailable: 0<br>        type: RollingUpdate<br>    service:<br>      annotations:<br>        service.beta.kubernetes.io/aws-load-balancer-name: &quot;ingress-gateway&quot;<br>      ports:<br>      - name: mtls<br>        port: 15443<br>        targetPort: 15443<br>      - name: http2<br>        port: 80<br>        targetPort: 8080<br>      - name: https<br>        port: 443<br>        targetPort: 8443<br>      type: LoadBalancer<br>EOF</pre><p>Here are some details about the YAML file to help you understand its components:</p><ul><li><strong>spec.kubeSpec.deployment:</strong> This is a Kubernetes specification for the ingress-gateway pod. You can adjust the Horizontal Pod Autoscaler (hpa) settings, such as the maximum and minimum replicas, metrics for scaling, and resource limits like CPU and memory.</li><li><strong>service.annotations:</strong> These are annotations applied to the Kubernetes service for the ingress-gateway pod.</li><li><strong>service.ports:</strong> These are the ports defined for the Kubernetes service, and these ports are also exposed to the load balancers.</li></ul><p>Run below command to create the ingress-gateway resources:</p><pre>kubectl apply -f install-ingress-gateway.yaml</pre><p>Verify the Successful Creation of Ingress-Gateway Resources</p><p>To confirm that the ingress-gateway resources have been successfully created, run the following command:</p><pre>kubectl get all -n ingress-gateway</pre><p>The output should resemble the following:</p><pre>NAME                                   READY   STATUS    RESTARTS   AGE<br>pod/ingress-gateway-685b885bd6-9vmbm   1/1     Running   0          18s<br><br>NAME                      TYPE           CLUSTER-IP      EXTERNAL-IP                                                                PORT(S)              AGE<br>service/ingress-gateway   LoadBalancer   172.20.170.40   aed1c8e12e0fe43b3924f26a95875985-1982569114.ap-south-1.elb.amazonaws.com   15443:30433/TCP,80:31919/TCP,443:31337/TCP   18s<br><br>NAME                              READY   UP-TO-DATE   AVAILABLE   AGE<br>deployment.apps/ingress-gateway   1/1     1            1           18s<br><br>NAME                                         DESIRED   CURRENT   READY   AGE<br>replicaset.apps/ingress-gateway-685b885bd6   1         1         1       18s<br><br>NAME                                                  REFERENCE                    TARGETS         MINPODS   MAXPODS   REPLICAS   AGE<br>horizontalpodautoscaler.autoscaling/ingress-gateway   Deployment/ingress-gateway   &lt;unknown&gt;/70%   1         10        1          18s</pre><p>Now that we have installed the Ingress-Gateway resource in the namespace ingress-gateway, it is ready for configuration to handle routing. Moving forward, we will configure the routing for all traffic coming to the ingress gateway at port 80 with the hostname helloworld.example.com routed to the helloworld service in the namespace sample.</p><h3>Configure Routing of Ingress-gateway</h3><p>To set up the routing for the Ingress-Gateway, use the following YAML configuration file:</p><pre>apiVersion: gateway.tsb.tetrate.io/v2<br>kind: IngressGateway<br>metadata:<br>  organization: &lt;org-name&gt; #&lt;--Put your organization name here<br>  tenant: &lt;tenant-name&gt; #&lt;--Put your tenant name here<br>  workspace: &lt;workspace-name&gt; #&lt;--Put your workspace name here<br>  group: ingress-gateway-group #&lt;--Group Name which we created earlier<br>  name: ingress-gateway #&lt;--Name of IngressGateway <br>spec:<br>  workloadSelector:  #&lt;-- workloadSelector will select the Ingress Gateway pod which we created in the previous step<br>    namespace: ingress-gateway<br>    labels:<br>      app: ingress-gateway #By default, the app: label will be created to the pod of ingress-gateway, check your pod to verify.<br>  http:<br>    - name: helloworld #&lt;-- Just a name of route, it could be any identical name<br>      port: 80 #&lt;--port of Ingress-Gateway where the client will send the traffic<br>      hostname: helloworld.example.com #&lt;--host header in the client request<br>      routing:<br>        rules:<br>          - route:<br>              host: &quot;sample/helloworld.sample.svc.cluster.local&quot; #&lt;-- Full DNS name Kubernetes service following by namespace, where traffic should be routed.<br>              port: 5000 #&lt;-- Port Number of helloworld service</pre><p>To apply this configuration, run the following command:</p><pre>tctl apply -f configure-ingress-gateway.yaml</pre><p>If the command runs successfully, Tetrate accepts the Ingress-Gateway configuration, and it will create all the necessary components in the control-plane. Now, the gateway is ready to accept traffic from outside the Kubernetes cluster.</p><p>To access the helloworld service, create the DNS entry helloworld.example.com pointing to our Ingress-Gateway load balancer URL.</p><h3>Congratulations!!</h3><p>By following the above steps, you can create the Tetrate Ingress Gateway and expose your Kubernetes services outside of the Kubernetes cluster.</p><h3>References:</h3><ol><li><a href="https://docs.tetrate.io/service-bridge/quickstart/ingress_gateway">https://docs.tetrate.io/service-bridge/quickstart/ingress_gateway</a></li><li><a href="https://docs.tetrate.io/service-bridge/refs/tsb/gateway/v2/ingress_gateway">https://docs.tetrate.io/service-bridge/refs/tsb/gateway/v2/ingress_gateway</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=239e9e961f51" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Simplify Range Partition Management in AWS RDS Aurora PostgreSQL with pg_partman and pg_cron]]></title>
            <link>https://raghav-patel.medium.com/simplify-range-partition-management-in-aws-rds-aurora-postgresql-with-pg-partman-and-pg-cron-996219799fcc?source=rss-b5f51c0e64c3------2</link>
            <guid isPermaLink="false">https://medium.com/p/996219799fcc</guid>
            <category><![CDATA[database-optimization]]></category>
            <category><![CDATA[aurora-postgresql]]></category>
            <category><![CDATA[partitioning]]></category>
            <dc:creator><![CDATA[Raghav Patel]]></dc:creator>
            <pubDate>Sat, 17 Jun 2023 20:39:35 GMT</pubDate>
            <atom:updated>2023-06-17T21:14:52.135Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/866/1*TS_0CzOGjwF4xIscgpoIyw.png" /></figure><p>Range partitioning is a powerful technique for managing large datasets in a PostgreSQL database. It allows you to divide your data into multiple smaller and more manageable partitions based on a specified range of values. However, setting up and managing range partitions manually can be complex and time-consuming. In this blog post, I will explore how to simplify range partition management in AWS RDS Aurora PostgreSQL using the pg_partman extension.</p><h3>What is pg_partman?</h3><p>pg_partman is a PostgreSQL extension that provides native partitioning capabilities with additional features for easier partition management. It simplifies the process of creating, maintaining, and managing partitions in a PostgreSQL database.</p><p>Key Capabilities of pg_partman</p><ul><li>Automatic partition creation for time-based partitioning (e.g., monthly partitions).</li><li>Pre-creation of a default partition to capture data outside specified ranges.</li><li>Intelligent retention management for automatically dropping older partitions.</li><li>Dynamic partition maintenance for operations like splitting, merging, and attaching new partitions.</li><li>Monitoring and reporting functions to gather insights into partition sizes and query performance.</li></ul><p>If you want to read more about pg_partman here is the <a href="https://github.com/pgpartman/pg_partman">pg_partman GitHub Repository</a></p><h4>Installing pg_partman in AWS RDS Aurora PostgreSQL</h4><p>To install pg_partman in AWS RDS Aurora PostgreSQL, follow these steps:</p><ol><li>Connect to your RDS Aurora PostgreSQL instance using a client tool.</li><li>It’s recommended to create the separate schema for pg_partman. To create the schema and install pg_partman extension execute following commands:</li></ol><pre>CREATE SCHEMA partman;<br>CREATE EXTENSION pg_partman WITH SCHEMA partman;</pre><h4>Setting up Range Partitioning with pg_partman</h4><p>Setting up range partitioning with pg_partman involves the following steps:</p><p>Choose a column in your table that will be used as the partition key, such as a date or numeric column, and Create the initial parent table manually.</p><p>In below example we’re creating the sales table which partitioned by the <strong>sale_date</strong></p><pre>CREATE TABLE public.sales_data (<br>    id SERIAL,<br>    sale_date DATE,<br>    amount NUMERIC<br>) PARTITION BY RANGE (sale_date);</pre><p>Parent partitioned table sales_data is created, now let’s configure pg_partman to manage the child partition of sales_data with create_parent function.</p><pre>SELECT partman.create_parent(<br>    p_parent_table =&gt; &#39;public.sales_data&#39;,<br>    p_control =&gt; &#39;sale_date&#39;,<br>    p_type =&gt; &#39;native&#39;,<br>    p_interval =&gt; &#39;monthly&#39;,<br>    p_premake =&gt; 6,<br>    p_start_partition =&gt; &#39;2022-04-01&#39;<br>);</pre><p>Let’s break down each parameter of the create_parent function in a simpler language:</p><ul><li>p_parent_table: This is the main table that will be partitioned. It should already exist and include the schema name.</li><li>p_control: This parameter determines the column that will be used for partitioning. It should be a column with either integer or time-based data types.</li><li>p_type: You can choose between two options: &#39;native&#39; or &#39;partman&#39;. &#39;Native&#39; provides better performance and flexibility, while &#39;partman&#39; relies on inheritance.</li><li>p_interval: This specifies the interval or range for each partition. For example, you can use &#39;monthly&#39;, &#39;daily&#39;, or even specify an integer range.</li><li>p_premake: This parameter determines the number of partitions to create in advance to accommodate future data inserts.</li><li>p_start_partition: This optional parameter specifies the start date or value for the first partition to be created. It allows you to define a specific starting point for your partitions based on your data. For example, if you set p_start_partition to &#39;2022-04-01&#39;, the first partition will be created starting from April 1, 2022.</li></ul><p>We have successfully set up pg_partman to manage the partitions of the sales_data table. You can view the configuration in the partman.part_config table. To see all the configuration details, run the following command:</p><pre>SELECT * from partman.part_config;</pre><p>If you want to update the existing pg_partman configuration for a specific table, such as changing the retention period for the child tables of sales_data, you can use the following command:</p><pre>UPDATE partman.part_config <br>SET infinite_time_partitions = true,<br>    retention = &#39;3 months&#39;, <br>    retention_keep_table=true <br>WHERE parent_table = &#39;public.sales_data&#39;;</pre><p>By executing this command, you can modify the retention settings to retain data for a specific duration, such as 3 months, and ensure that the child tables are retained while performing partition maintenance.</p><p>To ensure the regular maintenance of partitions, we need to periodically execute the run_maintenance_proc function provided by pg_partman. To automate this process, we can set up a cron job using the pg_cron extension.</p><h3>Install pg_cron extension in aurora postgres</h3><p>To install the pg_cron extension in AWS Aurora PostgreSQL, follow these steps:</p><ul><li>Modify the cluster parameter group associated with your PostgreSQL DB instance. Add “pg_cron” to the value of the “shared_preload_libraries” parameter. This parameter controls the shared libraries that are loaded when the database starts.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JmcHaiy37q0EkDTVmoPOHw.png" /></figure><ul><li>Restart your PostgreSQL DB instance to apply the changes made to the parameter group. This will ensure that the pg_cron extension is loaded and available for use.</li><li>Run flowing command to install pg_cron extention</li></ul><pre>CREATE EXTENSION pg_cron;</pre><blockquote>Please note that by default, the pg_cron extension saves all its configuration in the PostgreSQL database itself. However, if you want to change the default database where pg_cron stores its data, you can do so by modifying the &quot;cron.database_name&quot; parameter in the cluster parameter group associated with your database instance.</blockquote><h4>Configure pg_cron to run run_maintenance_proc periodically</h4><ul><li>Connect to the PostgreSQL database of your Aurora cluster.</li><li>Run the following command to schedule the execution of the run_maintenance_proc function on a monthly basis:</li></ul><pre>SELECT cron.schedule(&#39;@monthly&#39;, $$CALL partman.run_maintenance_proc()$$);</pre><p>By executing this command, pg_cron will be configured to automatically execute the run_maintenance_proc function every month. This function is responsible for performing maintenance tasks on the partitioned tables managed by pg_partman.</p><p>As mentioned before, pg_cron stores its configuration in the default database “postgres”. However, if you want to change the configuration and run the cron job in a specific database like “sales”, you can do so by executing the following command:</p><pre>UPDATE cron.job SET database = &#39;sales&#39; WHERE jobid = 1;</pre><p>This setup ensures that partition maintenance is regularly performed without manual intervention, helping to optimize the performance and organization of your partitioned data.</p><h3>Congratulations!!! 🥳</h3><p>By utilizing the capabilities of pg_partman and pg_cron extensions in PostgreSQL, you can automate partition management tasks, such as creation, retention, and maintenance, while scheduling their execution at regular intervals. This simplifies data organization, enhances query performance, and improves overall database efficiency.</p><h3>Thank You!</h3><h3>Reference:</h3><ol><li><a href="https://github.com/pgpartman/pg_partman">https://github.com/pgpartman/pg_partman</a></li><li><a href="https://pgxn.org/dist/pg_partman/doc/pg_partman.html">https://pgxn.org/dist/pg_partman/doc/pg_partman.html</a></li><li><a href="https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL_Partitions.html">https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/PostgreSQL_Partitions.html</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=996219799fcc" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>