<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Alexei on Medium]]></title>
        <description><![CDATA[Stories by Alexei on Medium]]></description>
        <link>https://medium.com/@acosetov?source=rss-33deb7e5c5c1------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*c0xd6CCkIsUd-JaYyo35vw.jpeg</url>
            <title>Stories by Alexei on Medium</title>
            <link>https://medium.com/@acosetov?source=rss-33deb7e5c5c1------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 06:07:44 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@acosetov/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to Deploy Paperclip on a VPS: A Step-by-Step Guide with Nginx Authentication]]></title>
            <link>https://medium.com/@acosetov/how-to-deploy-paperclip-on-a-vps-a-step-by-step-guide-with-nginx-authentication-7291a3a26333?source=rss-33deb7e5c5c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/7291a3a26333</guid>
            <dc:creator><![CDATA[Alexei]]></dc:creator>
            <pubDate>Mon, 23 Mar 2026 13:37:53 GMT</pubDate>
            <atom:updated>2026-03-23T13:37:53.557Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V38NkhK_FflP9juyO2FrBg.png" /></figure><p><a href="https://paperclip.ing/">Paperclip</a> is a self-hosted AI agent platform that runs entirely on your own server. In this guide, I’ll walk you through installing it from scratch on a fresh VPS, connecting it to Claude via a Pro subscription, and securing the interface with password protection using Nginx.</p><p>The whole process takes about 15–20 minutes.</p><h3>Prerequisites</h3><ul><li>A VPS or dedicated server running <strong>Ubuntu 24.04</strong></li><li>A <strong>Claude Pro</strong> subscription (or Max / Team / Enterprise)</li><li>Basic familiarity with the terminal</li></ul><h3>Step 1. Create a sudo user</h3><p>Running everything as root is a bad practice. Let&#39;s create a dedicated user instead:</p><pre>adduser myuser<br>usermod -aG sudo myuser</pre><blockquote><em>If you already have a non-root user with sudo privileges, skip this step.</em></blockquote><p>Switch to the new user:</p><pre>su - myuser</pre><h3>Step 2. Install Node.js</h3><p>Paperclip runs on Node.js. We’ll install the latest version from the official Node Source repository:</p><pre>sudo apt update<br>curl -sL https://deb.nodesource.com/setup_25.x -o nodesource_setup.sh<br>sudo bash nodesource_setup.sh<br>sudo apt install nodejs<br>node -v</pre><p>The node -v command at the end will print the installed version make sure it runs without errors.</p><h3>Step 3. Install pnpm</h3><p>Paperclip uses pnpm as its package manager:</p><pre>curl -fsSL https://get.pnpm.io/install.sh | sh -<br>source /home/myuser/.bashrc<br>pnpm -v</pre><h3>Step 4. Install Nginx</h3><p>Nginx will serve as a reverse proxy and provide basic password protection:</p><pre>sudo apt install nginx -y</pre><h3>Step 5. Clone and install Paperclip</h3><pre>git clone https://github.com/paperclipai/paperclip.git<br>cd paperclip<br>pnpm install</pre><h3>Step 6. Verify the first launch</h3><p>Start the application:</p><pre>pnpm dev</pre><p>If everything installed correctly, you’ll see a status screen in the terminal:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eyPFim9ubGhnVpWIArT5fw.png" /></figure><p>Pay attention to these key indicators:</p><ul><li><strong>Auth: ready</strong> — the authentication system is active</li><li><strong>Migrations: already applied</strong> — the database has been initialized</li><li><strong>Server: 3100</strong> — the app is listening on port 3100</li></ul><p>If you see this — everything is working. Stop the server (Ctrl+C) and move on to connecting Claude.</p><h3>Step 7. Install Claude</h3><pre>curl -fsSL https://claude.ai/install.sh | bash<br>echo &#39;export PATH=&quot;$HOME/.local/bin:$PATH&quot;&#39; &gt;&gt; ~/.bashrc &amp;&amp; source ~/.bashrc</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zOXjkVTlBMlphzg6uDr8GQ.png" /></figure><h3>Step 8. Connect Claude to your server</h3><p>Launch Claude:</p><pre>claude</pre><p>In the menu that appears, select option <strong>1. Claude account with subscription</strong> — this works for Pro, Max, Team, and Enterprise plans.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mMtbPH4csLYVsi-QvTbI9w.png" /></figure><p>Claude will generate a unique URL. Open it in a browser on your local machine, log into your account, and grant access. After confirmation, the browser will display a code — copy it and paste it back into the terminal.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bvwDXhWRvvGhshxZX2pCVA.png" /></figure><p>Claude is now connected and ready.</p><h3>Step 9. Run Paperclip in the background with tmux</h3><p>To keep the app running after you close your SSH session, use tmux:</p><pre>tmux new -s paperclip<br>cd ~/paperclip<br>pnpm dev</pre><p>Paperclip is now running in the background. Useful tmux commands:</p><ul><li>Detach from the session without stopping the app: <strong>Ctrl+B</strong>, then <strong>D</strong></li><li>Reattach to the running session: tmux attach -t paperclip</li></ul><h3>Step 10. Configure Nginx with password protection</h3><p>Right now, the Paperclip interface is accessible on port 3100 with no protection. Let&#39;s secure it using Nginx with HTTP Basic Auth.</p><p><strong>Install the password utility and create a user:</strong></p><pre>sudo apt install apache2-utils<br>sudo htpasswd -c /etc/nginx/.htpasswd admin</pre><p>The command will prompt you for a password for the admin user — choose a strong one and save it.</p><p><strong>Clear the default Nginx config:</strong></p><pre>sudo sh -c &#39;&gt; /etc/nginx/sites-available/default&#39;</pre><p><strong>Add the new configuration</strong> to /etc/nginx/sites-available/default:</p><pre>server {<br>    listen 80;<br>    server_name _;<br><br>    auth_basic &quot;Private site&quot;;<br>    auth_basic_user_file /etc/nginx/.htpasswd;<br><br>    location / {<br>        proxy_pass http://127.0.0.1:3100;<br>        proxy_set_header Host $host;<br>        proxy_set_header X-Real-IP $remote_addr;<br>        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;<br>        proxy_set_header X-Forwarded-Proto $scheme;<br>    }<br>}</pre><p>server_name _; means Nginx will accept requests on any IP address or domain. If you have a domain, replace _ with it.</p><p><strong>Restart Nginx:</strong></p><pre>sudo systemctl restart nginx</pre><h3>Open it in your browser</h3><p>Navigate to your server’s IP address. Nginx will greet you with a login prompt:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FQKWAS-kSmEvPJZLIYWUVQ.png" /></figure><p>Enter the username admin and the password you created in the previous step. After logging in, you&#39;ll see the Paperclip onboarding screen:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N78v9KeCBxlqPnAEid6TQQ.png" /></figure><p>Installation complete. Paperclip is running on your server, protected by a password, and connected to Claude.</p><h3>Summary</h3><p>In just a few steps, we:</p><ul><li>Deployed Paperclip on a clean Ubuntu 24.04 VPS</li><li>Connected Claude using a Pro subscription</li><li>Kept the app running persistently in the background with tmux</li><li>Secured the interface with password authentication via Nginx</li></ul><p>Have questions or ran into issues? Drop a comment below.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7291a3a26333" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building UDFs in ClickHouse with Go: A Step-by-Step Guide]]></title>
            <link>https://medium.com/@acosetov/building-udfs-in-clickhouse-with-go-a-step-by-step-guide-813076b167f4?source=rss-33deb7e5c5c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/813076b167f4</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[clickhouse]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[database]]></category>
            <dc:creator><![CDATA[Alexei]]></dc:creator>
            <pubDate>Fri, 06 Jun 2025 10:56:09 GMT</pubDate>
            <atom:updated>2025-06-06T11:07:56.901Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1022/1*_ltnFbgQlb0Z1iCiC8CmNg.png" /></figure><h3>Suggested Introduction</h3><p>As a data engineer diving deeper into ClickHouse, I’ve been captivated by its blazing-fast performance and versatility for analytical workloads. While preparing for my ClickHouse certification, I wanted to challenge myself by exploring its extensibility through User-Defined Functions (UDFs). Why GoLang? Its simplicity, speed, and robust standard library make it an ideal choice for building high-performance UDFs that integrate seamlessly with ClickHouse’s demanding environment. In this article, I’ll guide you through creating an <strong>is_valid_email</strong> UDF in ClickHouse using GoLang, which checks the syntax of email addresses in a string column. I’ll share insights from my journey and practical steps to help you unlock ClickHouse’s potential with the power of Go.</p><h3>Problem Statement</h3><p>To showcase the power of UDFs in ClickHouse with GoLang, we’ll create an <strong>is_valid_email</strong> UDF. This function will take a email column containing email addresses and return a boolean (UInt8 in ClickHouse) indicating whether the email syntax is valid. In this article, we’ll walk through building and integrating this UDF step-by-step using GoLang’s simplicity and robust standard library. Let’s go!!</p><h3>Setting Up the Test Environment</h3><p>Before we dive into creating the <strong>email_validator</strong> UDF, let’s set up a simple ClickHouse table to test our function. We’ll create a users table to store textual data, such as usernames, which we’ll later hash using our Rust-based UDF. Here’s the SQL to create the table:</p><pre>CREATE TABLE users<br>(<br>    id UInt32,<br>    username String,<br>    email String<br>)<br>ENGINE = MergeTree()<br>ORDER BY id;<br><br>INSERT INTO users (id, username, email) VALUES<br>(1, &#39;alice&#39;, &#39;alice@example.com&#39;),<br>(2, &#39;bob&#39;, &#39;bob@example.com&#39;),<br>(3, &#39;charlie&#39;, &#39;charlie@domain&#39;),<br>(4, &#39;alexa&#39;, &#39;alexa@domain..com&#39;);</pre><h3>Implementing a UDF in Go for Email Syntax Validation</h3><p>Let’s implement a User-Defined Function (UDF) in Go to validate the syntax of email addresses.</p><pre># main.go<br>package main<br><br>import (<br> &quot;bufio&quot;<br> &quot;encoding/json&quot;<br> &quot;fmt&quot;<br> &quot;os&quot;<br> &quot;regexp&quot;<br>)<br><br>type Input struct {<br> InputString string `json:&quot;input_string&quot;`<br>}<br><br>type Output struct {<br> IsValid uint8 `json:&quot;is_valid&quot;`<br>}<br><br>func isValidEmail(email string) bool {<br> re := regexp.MustCompile(`^[a-zA-Z0-9._%+-]+@[a-zA-Z0-9-]+(\.[a-zA-Z0-9-]+)*\.[a-zA-Z]{2,}$`)<br> return re.MatchString(email)<br>}<br><br>func main() {<br> scanner := bufio.NewScanner(os.Stdin)<br> for scanner.Scan() {<br>  line := scanner.Text()<br>  if line == &quot;&quot; {<br>   continue<br>  }<br><br>  var input Input<br>  if err := json.Unmarshal([]byte(line), &amp;input); err != nil {<br>   fmt.Fprintf(os.Stderr, &quot;Failed to parse JSON: %v\n&quot;, err)<br>   continue<br>  }<br><br>  isValid := uint8(0)<br>  if isValidEmail(input.InputString) {<br>   isValid = 1<br>  }<br><br>  output := Output{IsValid: isValid}<br>  outputJSON, err := json.Marshal(output)<br>  if err != nil {<br>   fmt.Fprintf(os.Stderr, &quot;Failed to marshal JSON: %v\n&quot;, err)<br>   continue<br>  }<br><br>  fmt.Println(string(outputJSON))<br>  os.Stdout.Sync()<br> }<br><br> if err := scanner.Err(); err != nil {<br>  fmt.Fprintf(os.Stderr, &quot;Error reading input: %v\n&quot;, err)<br> }<br>}</pre><p>Let’s build our Go program and test how it works.</p><pre>go build -o clickhouse_udf_email_validator<br>echo &#39;{&quot;input_string&quot;: &quot;test@example...com&quot;}&#39; | ./clickhouse_udf_email_validator<br>Output: {&quot;is_valid&quot;:0}</pre><p>Great, everything is working. Now, let’s enable ClickHouse to use our function.</p><h3>Configure ClickHouse to enable our function.</h3><p>We need to configure two parameters in the <strong>config.xml</strong> file located at <strong>/etc/clickhouse-server/config.xml</strong>. First, add the following line to specify the configuration file for executable user-defined functions:</p><pre>&lt;user_defined_executable_functions_config&gt;*_function.xml&lt;/user_defined_executable_functions_config&gt;</pre><p>This defines the path to the configuration file for executable user-defined functions.</p><p>Second, add the following line to specify the directory for user script files used by executable user-defined functions:</p><pre>&lt;user_scripts_path&gt;/var/lib/clickhouse/user_files/&lt;/user_scripts_path&gt;</pre><p>This sets the directory where user script files for executable user-defined functions are stored.</p><p>Save the <strong>config.xml</strong> file and restart ClickHouse. We’re almost done — let’s complete the final step by connecting our function.</p><h3>Configuration and Connecting the Go UDF</h3><p>Great, we need to complete two steps:</p><ol><li>Copy our compiled binary file to the directory specified for user script files, which in this case is <strong>/var/lib/clickhouse/user_files/</strong>. Run the following command:</li></ol><pre>cp clickhouse_udf_email_validator /var/lib/clickhouse/user_files/</pre><p>2. Create the <strong>is_valid_email</strong> function with named arguments and the JSONEachRow format using an XML configuration file. Create or edit the file <strong>/etc/clickhouse-server/clickhouse_udf_email_validator_function.xml</strong> with the following content:</p><pre>&lt;functions&gt;<br>    &lt;function&gt;<br>        &lt;type&gt;executable&lt;/type&gt;<br>        &lt;name&gt;is_valid_email&lt;/name&gt;<br>        &lt;return_type&gt;UInt8&lt;/return_type&gt;<br>        &lt;return_name&gt;is_valid&lt;/return_name&gt;<br>        &lt;argument&gt;<br>            &lt;type&gt;String&lt;/type&gt;<br>            &lt;name&gt;input_string&lt;/name&gt;<br>        &lt;/argument&gt;<br>        &lt;format&gt;JSONEachRow&lt;/format&gt;<br>        &lt;command&gt;clickhouse_udf_email_validator&lt;/command&gt;<br>    &lt;/function&gt;<br>&lt;/functions&gt;</pre><p>The function will be named <strong>is_valid_email</strong>. Save the file and restart ClickHouse.</p><h3>Run the Validation Query</h3><p>Now we can test our solution. Open the users table and run the following query:</p><pre>SELECT id, username, email, is_valid_email(email) AS is_valid FROM users;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/799/1*yT8r4vEeE16Ax49vcuYHJA.png" /></figure><p>And like magic, we now have an email validation status — where 1 means the email is valid and 0 means it’s not.</p><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=813076b167f4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Efficiently Exporting Over 1 Million Records from Metabase: A Step-by-Step Guide]]></title>
            <link>https://medium.com/@acosetov/efficiently-exporting-over-1-million-records-from-metabase-a-step-by-step-guide-87a4c507d48c?source=rss-33deb7e5c5c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/87a4c507d48c</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[metabase]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[python]]></category>
            <dc:creator><![CDATA[Alexei]]></dc:creator>
            <pubDate>Tue, 13 Aug 2024 13:12:41 GMT</pubDate>
            <atom:updated>2024-08-13T13:12:41.715Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MtXyrXRkfLCkzPVBKHUYfQ.png" /></figure><p><a href="https://www.metabase.com/">Metabase</a> is a great tool for business intelligence, dashboards, and data visualization. However, the open-source version has some restrictions, such as exporting large datasets. Exporting 100k records is manageable, but occasionally, clients request to export all data from a database, which can exceed 1 million records. In this short post, I will show you how to accomplish this using generated sample data. Let’s get started!</p><ol><li><a href="#2fc1">Prepare demo environment</a></li><li><a href="#74ea">Exporting Over 1 Million Records from Metabase</a></li></ol><h3>Prepare demo environment</h3><p>First of all, we need to prepare our test environment. Don’t worry, it won’t take much time since I have already prepared everything we need. Just follow my steps.</p><p><strong>Step 1</strong>: Clone this Git repository.</p><pre>git clone https://github.com/acosetov/MetabaseTutorials.git</pre><p>This repository contains all the tools and data we need for this tutorial. However, it’s <strong>important that you have Docker and Docker Compose preinstalled</strong>.</p><p><strong>Step 2</strong>: Navigate to the folder and run docker-compose command</p><pre>cd MetabaseTutorials/01-export-1m-records/<br>docker-compose up -d</pre><p>After everything is done, you will have the following:</p><ol><li>PostgreSQL 16</li><li>Metabase connected to PostgreSQL</li><li>A Python script that will randomly generate 1.2 million records</li></ol><p><strong>Step 3</strong>: We need to configure Metabase and connect it to our generated database with the example data.</p><p>Type <em>http://localhost:3000</em> into your browser. You will see the Metabase welcome page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/706/1*fB8D4sFZ_id0xNIUQpbmnQ.png" /></figure><p>Just click “Let’s get started” and and fill out the form.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/762/1*xVl4bVVEW-aUJjKx1rD0nA.png" /></figure><p>On step 4, choose PostgreSQL.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/759/1*jcps0aYD9d20iBfsKSY6AA.png" /></figure><p>Fill out the form with the following details:</p><ul><li><strong>Display name</strong>: ExampleDB (you can provide your own name)</li><li><strong>Host</strong>: postgres</li><li><strong>Port</strong>: 5432</li><li><strong>Database</strong> name: generated_data</li><li><strong>Username</strong>: data_user</li><li><strong>Password</strong>: data_password</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/695/1*OeowqjgTAKLcsdU2DKW24g.png" /></figure><p>Click “Connect database”, then choose your preference in Step 5, and click “Finish”.</p><p><strong>Step 4</strong>: Log in to Metabase using the credentials you entered in the form, and navigate to our ExampleDB using the Metabase UI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0m0IyyupgRrSPRdY5yp35w.png" /></figure><h3>Exporting Over 1 Million Records from Metabase</h3><p>Great! Our demo environment is ready, and you can see the demo data.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*tig4WxOuuC7bQZNVCQ_xYQ.png" /></figure><p>Now let’s check how many records we have. We can do this with the help of Metabase or by using SQL. I will do it with SQL because we will also use SQL for data extraction.</p><p>On the right side of the menu, click on “New” and select “SQL query”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X90YwkiO8TKqaDVeEwXklA.png" /></figure><p>Great! Let’s write an SQL query to check how many records we have in the table.</p><pre>SELECT COUNT(1) FROM users;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PT3Ihogk8HeJf-aWDud2Sg.png" /></figure><p>Great! We have 1.2 million records. Our next step is to extract all this data. It’s very simple we will use the LIMIT and OFFSET commands.</p><p>Let’s first extract the first 1 million records.</p><pre>SELECT * FROM users ORDER BY id LIMIT 1000000 OFFSET 0;</pre><p>Here is our result. Point 1 shows our query, point 2 is the result, and if you click on point 3, you can download the first 1 million records.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OJyt-XvytM1hVBKzPHw5KQ.png" /></figure><p>To extract the next part of the data, just run the following query.</p><pre>SELECT * FROM users ORDER BY id LIMIT 1000000 OFFSET 1000000;</pre><p>Run the query and download the result again.</p><p>That’s all, thank you. As you can see, it’s simple — just increase the OFFSET value, for example, by 1 million, and you will extract data 1 million records at a time. Good luck!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=87a4c507d48c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Discovery: Converting CSV to Parquet with AWS Lambda Trigger]]></title>
            <link>https://medium.com/@acosetov/aws-discovery-converting-csv-to-parquet-with-aws-lambda-trigger-80c6c8802827?source=rss-33deb7e5c5c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/80c6c8802827</guid>
            <category><![CDATA[s3]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[pipeline]]></category>
            <category><![CDATA[aws-lambda]]></category>
            <category><![CDATA[cloud]]></category>
            <dc:creator><![CDATA[Alexei]]></dc:creator>
            <pubDate>Sun, 14 Jul 2024 18:37:47 GMT</pubDate>
            <atom:updated>2024-07-15T06:33:15.988Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*D4x-CrCLCMCOIRYCvCFJVA.png" /></figure><p>Hello. In this episode, we will create a simple pipeline on AWS using a Lambda function to convert a CSV file to Parquet. You may ask why we need to convert CSV to Parquet, and this is a reasonable question. Here are some arguments:</p><ol><li><strong>Columnar Storage</strong>: Parquet is a columnar storage file format, which means it stores data by columns instead of rows. This is particularly efficient for queries that only need to access a subset of columns.</li><li><strong>Compression</strong>: Parquet files are highly compressed, often reducing the file size significantly compared to CSV. This reduces storage costs and improves read performance.</li><li><strong>Efficiency</strong>: Reading and writing Parquet files can be much faster than CSV files, especially for large datasets. This is because Parquet files can be read in chunks and allow for more efficient data scanning.</li></ol><p>Okay let’s go!🏎</p><ol><li><a href="#d6e7">Create S3 buckets</a></li><li><a href="#8f73">Create IAM policy and role</a></li><li><a href="#b84b">Create Lambda function</a></li></ol><h3>Create S3 buckets</h3><p>First, our step will be to create two S3 buckets. One bucket will be for CSV files, and the second will be for converted Parquet files. In a previous post, we configured the AWS CLI. Let’s try to use it to create the buckets.</p><p>Before we begin, let’s review the command to create an S3 bucket:</p><pre>aws s3api create-bucket --bucket **bucket-name** --region **region-name** --create-bucket-configuration LocationConstraint=**region-name**</pre><p><strong>Breakdown of the Command:</strong></p><ul><li><strong>aws s3api create-bucket</strong>: This is the AWS CLI command to create an S3 bucket</li><li><strong>— — bucket bucket-name</strong>: This specifies the name of the bucket you’re creating</li><li><strong>— — region region-name</strong>: This specifies the AWS region where the bucket will be created</li><li><strong>— —create-bucket-configuration LocationConstraint=region-name</strong>: This specifies the location constraint for the bucket, ensuring it is created in the specified region.</li></ul><pre>aws s3api create-bucket --bucket csv-bucket-act24 --region eu-west-3 --create-bucket-configuration LocationConstraint=eu-west-3<br>aws s3api create-bucket --bucket parquet-bucket-act24 --region eu-west-3 --create-bucket-configuration LocationConstraint=eu-west-3</pre><p>Let’s go to our S3 buckets and check what we have. Great! As you can see, we created two buckets using AWS CLI.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VGkeindtJs_wR4cFrPfi-A.png" /></figure><h3>Create an IAM policy and role for your Lambda function</h3><p>Before creating a Lambda function, we need to create a policy and role for our future function. Let’s do this. Write in the AWS Console search bar “IAM” and go to “Policies“.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/990/1*Ikrm-3PSXHVkzcX60a4ZFQ.png" /></figure><p>In the <strong>Policies</strong> section, click on “Create policy”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zSCsZ_XcywQ5vAwsZ1whCQ.png" /></figure><p>On the opened page, choose ‘JSON’ and add the following JSON to the Policy editor and click “Next”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VkzrMrE7QVAvtY_7uP1m9A.png" /></figure><pre>{<br> &quot;Version&quot;: &quot;2012-10-17&quot;,<br> &quot;Statement&quot;: [<br>  {<br>   &quot;Effect&quot;: &quot;Allow&quot;,<br>   &quot;Action&quot;: [<br>    &quot;logs:PutLogEvents&quot;,<br>    &quot;logs:CreateLogGroup&quot;,<br>    &quot;logs:CreateLogStream&quot;<br>   ],<br>   &quot;Resource&quot;: &quot;arn:aws:logs:*:*:*&quot;<br>  },<br>  {<br>   &quot;Effect&quot;: &quot;Allow&quot;,<br>   &quot;Action&quot;: [<br>    &quot;s3:*&quot;<br>   ],<br>   &quot;Resource&quot;: [<br>    &quot;arn:aws:s3:::csv-bucket-act24/*&quot;,<br>    &quot;arn:aws:s3:::csv-bucket-act24&quot;,<br>    &quot;arn:aws:s3:::parquet-bucket-act24/*&quot;,<br>    &quot;arn:aws:s3:::parquet-bucket-act24&quot;<br>   ]<br>  }<br> ]<br>}</pre><p>Add “Policy name”. Example “LambdaS3CwPolicy” and click “Create policy”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZIcEorNyDEEKNWwMTRAqaQ.png" /></figure><p>Let’s check if the policy was created by typing the policy name in the search bar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yz6T78GJkNmlGZ1NxQzXsA.png" /></figure><p>Before creating the “Role”, let me explain what our policy means.</p><pre>{<br>   &quot;Effect&quot;: &quot;Allow&quot;,<br>   &quot;Action&quot;: [<br>    &quot;logs:PutLogEvents&quot;,<br>    &quot;logs:CreateLogGroup&quot;,<br>    &quot;logs:CreateLogStream&quot;<br>   ],<br>   &quot;Resource&quot;: &quot;arn:aws:logs:*:*:*&quot;<br>  }</pre><p><strong>CloudWatch Logs Permissions</strong>:<br> • <em>Actions</em>: logs:PutLogEvents, logs:CreateLogGroup, logs:CreateLogStream<br> • <em>Resource</em>: arn:aws:logs:*:*:*<br> • <em>Effect</em>: Allows the Lambda function to create log groups and log streams, and put log events in any CloudWatch Logs resource.</p><pre>{<br>   &quot;Effect&quot;: &quot;Allow&quot;,<br>   &quot;Action&quot;: [<br>    &quot;s3:*&quot;<br>   ],<br>   &quot;Resource&quot;: [<br>    &quot;arn:aws:s3:::csv-bucket-act24/*&quot;,<br>    &quot;arn:aws:s3:::csv-bucket-act24&quot;,<br>    &quot;arn:aws:s3:::parquet-bucket-act24/*&quot;,<br>    &quot;arn:aws:s3:::parquet-bucket-act24&quot;<br>   ]<br>  }</pre><p><strong>S3 Permissions:</strong><br> • <em>Actions</em>: s3:*<br> • <em>Resources</em>:<br> — arn:aws:s3:::csv-bucket-act24/* (all objects in the csv-bucket-act24 bucket)<br> — arn:aws:s3:::csv-bucket-act24 (the csv-bucket-act24 bucket itself)<br> — arn:aws:s3:::parquet-bucket-act24/* (all objects in the parquet-bucket-act24 bucket)<br> — arn:aws:s3:::parquet-bucket-act24 (the parquet-bucket-act24 bucket itself)<br> • <em>Effect</em>: Allows all S3 actions on the specified buckets and their contents.</p><p>Now let’s proceed to create our role. Move to the role page and click on “Create role”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lOh4EQO2qhKD2TFHg_DP6g.png" /></figure><p>On next page chose “AWS service” and in “Use case” chose “Lambda” and click “Next”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QW7F6LBU6tjTUrYOgYkQ-w.png" /></figure><p>Now we should add our permission policy. Just type in the search our created policy and choose it and click “Next”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DPTCpuy53Qo16fLRVdkSMg.png" /></figure><p>And for our last step, add a role name (LambdaS3CwRole) and click on ‘Create role’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N_6pMzqUW35OT64ICNrzsg.png" /></figure><h3>Create Lambda function</h3><p>Good, we’re almost finished. Our last step is to create a Lambda function. Let’s go to the Lambda section. Just type “Lambda” in the search bar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PlANSbr4DQj0xz0EYGvX6g.png" /></figure><p>On the opened page, click on “Create a function”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bpdtsQ0GStWRVGDQu6gVug.png" /></figure><p>On the ‘Create function’ page, we should add the following details:</p><ol><li>Lambda function name: CSVtoParquetLambda</li><li>Runtime: Python 3.11</li><li>Architecture: x86_64 (default)</li><li>Execution role: Change the default execution role to ‘Use an existing role’, and select the role we created: “LambdaS3WcRole”.</li></ol><p>Then, click on “Create function.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/844/1*CnnWo2t3d5IzcBJEvQUppg.png" /></figure><p>Great! We’ve created our Lambda function. Now let’s add some code and also configure, for example, “Add trigger”. First, let’s add a trigger that will check if there’s a new CSV file in our CSV bucket and notify the Lambda function to run automatically. Click on “Add trigger”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-HZrpy4q2uu-M1ZXvdbBFg.png" /></figure><p>This is our configuration for the trigger:</p><ol><li>Choose “S3&quot;</li><li>Select our S3 bucket, in this case, it’s “csv-bucket-act24&quot;</li><li>In “Suffix”, add “.csv”</li><li>Check the box in the “Recursive invocation” section</li></ol><p>Finally, click “Add”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*_kgSkDC6SPkbd-hDsJlSaA.png" /></figure><p>In the Code section, copy the code below and to save, click on “Deploy”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h-Sk8KkccrE5ilYjK2DpVg.png" /></figure><pre>import boto3<br>import pandas as pd<br>import pyarrow as pa<br>import pyarrow.parquet as pq<br>import io<br><br>s3 = boto3.client(&#39;s3&#39;)<br><br>def lambda_handler(event, context):<br>    # Get the bucket and object key from the event<br>    bucket = event[&#39;Records&#39;][0][&#39;s3&#39;][&#39;bucket&#39;][&#39;name&#39;]<br>    key = event[&#39;Records&#39;][0][&#39;s3&#39;][&#39;object&#39;][&#39;key&#39;]<br>    <br>    # Download the CSV file from S3<br>    csv_obj = s3.get_object(Bucket=bucket, Key=key)<br>    csv_content = csv_obj[&#39;Body&#39;].read().decode(&#39;utf-8&#39;)<br>    <br>    # Read the CSV file into a pandas DataFrame<br>    df = pd.read_csv(io.StringIO(csv_content))<br>    <br>    # Convert the DataFrame to a Parquet file<br>    table = pa.Table.from_pandas(df)<br>    <br>    # Prepare the Parquet file to upload<br>    parquet_buffer = io.BytesIO()<br>    pq.write_table(table, parquet_buffer)<br>    parquet_buffer.seek(0)<br>    <br>    # Define the new S3 object key for the Parquet file<br>    parquet_key = key.replace(&#39;.csv&#39;, &#39;.parquet&#39;)<br>    <br>    # Upload the Parquet file to S3<br>    s3.put_object(Bucket=&#39;parquet-bucket-act24&#39;, Key=parquet_key, Body=parquet_buffer.getvalue())<br>    <br>    return {<br>        &#39;statusCode&#39;: 200,<br>        &#39;body&#39;: f&#39;Successfully converted {key} to {parquet_key}&#39;<br>    }</pre><p>And the last thing we need to add is a <strong>layer</strong> for our Lambda function. Before adding a layer, let’s define what it is. In AWS Lambda, a <strong>layer</strong> is a deployment package that contains libraries, custom runtimes, or other dependencies that you can include in your Lambda function. <strong>Layers</strong> help to manage and share code across multiple Lambda functions, making it easier to keep your codebase DRY (Don’t Repeat Yourself) and streamline the deployment process. In our code, we use the <strong>Pandas</strong> package, and we need to add this package. To add <strong>Pandas</strong>, we will use <strong>AWS SDK for Pandas</strong> (https://aws-sdk-pandas.readthedocs.io/en/stable/)</p><p>Let’s do this. In our Lambda function, find the <strong>Layers</strong> section and click “Add a layer”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oT07Hem9vHiaaf36oskDxw.png" /></figure><p>In “Choose a layer, select “Specify an ARN” and add “arn:aws:lambda:eu-west-3:336392948345:layer:AWSSDKPandas-Python311:16&quot;. It’s important to choose the correct ARN. In my case, it’s for the eu-west-3 region and Python 3.11. In your case, it may be different. To choose the right ARN, use this link: <a href="https://aws-sdk-pandas.readthedocs.io/en/stable/layers.html">https://aws-sdk-pandas.readthedocs.io/en/stable/layers.html</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aWmaKk7Q18hAu02LoL_uVQ.png" /></figure><p>Enter your ARN and click “Add”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/837/1*mNAvHhVewIQjzKEuZVyw0Q.png" /></figure><p>Good, everything is ready. Let’s test our Lambda function. First of all, let’s upload a CSV file to our CSV bucket. I will use AWS CLI and prepare a CSV file generated with the Python library “Faker”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/362/1*pIQuiUDg_7uKQxpA0Qfn2g.png" /></figure><p>Copy our “test.csv” file.</p><pre>aws s3 cp test.csv s3://csv-bucket-act24</pre><p>Now let’s check the result. We should have a Parquet file in our Parquet bucket (“parquet-bucket-act24”).</p><pre>aws s3 ls s3://parquet-bucket-act24</pre><p>Congratulations! You just created your Lambda function.</p><p><strong>P.S.</strong> All materials for this post can be found at the following link: <a href="https://github.com/acosetov/AwsDiscovery/tree/main/CSVtoParquetLambda">AWS Discovery — CSV to Parquet Lambda</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=80c6c8802827" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Discovery: First steps with AWS Cloud. Part 3]]></title>
            <link>https://medium.com/@acosetov/aws-discovery-first-steps-with-aws-cloud-part-3-6e305d72ae88?source=rss-33deb7e5c5c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/6e305d72ae88</guid>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[s3]]></category>
            <dc:creator><![CDATA[Alexei]]></dc:creator>
            <pubDate>Sun, 30 Jun 2024 14:38:54 GMT</pubDate>
            <atom:updated>2024-06-30T14:38:54.026Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/1*qerVDM_7Vf_oY3XJ3VNlqA.png" /></figure><p>Hello friends, in previous posts, we learned how to create a new account(<a href="https://medium.com/@acosetov/aws-discovery-first-steps-with-aws-cloud-part-1-f29b9414518a">post 1</a>) and add some billing controls (<a href="https://medium.com/@acosetov/aws-discovery-first-steps-with-aws-cloud-part-2-0e67a6aabaf6">post 2</a>). In the last post of this series, “First Steps with AWS Cloud,” we covered the following:</p><p>• <a href="#da80">Setup and configure AWS CLI</a><br> • <a href="#403d">Create an S3 bucket using AWS CLI</a></p><h3>Setup and configure AWS CLI</h3><p>The AWS Command Line Interface (CLI) is a unified tool to manage your AWS services. With just one tool to download and configure, you can control multiple AWS services from the command line and automate them through scripts.</p><p>Great. First of all, let’s log in to our administrator account and go to IAM section. We need to create an access key to link our AWS CLI and AWS account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FLVreFw7AYxNuANwuKDI4A.png" /></figure><p>Go to “Users” and select your admin user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jJvZqAhRCjblxDnvywBJmw.png" /></figure><p>Now, go to “Security credentials” and click “Create access key.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nnHj_NFFWEnHVDZSddxSww.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-zJDXX35ECtoQy9lyUPagw.png" /></figure><p>In the next steps, we will configure our access key. Choose “Command Line Interface (CLI),” check the “Confirmation” checkbox, and click “Next.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ANw7iXwYObps69xg4-RlUg.png" /></figure><p>On the next step, we can add a description, but this is optional. We can ignore this step and click “Create access key.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fzVoKEcGrXE8-0292kHsHQ.png" /></figure><p>Good, we are on the finish line. Now just download the CSV file with your details, save it in a safe place, and click “Done.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w0Ch3rlspsgOUPtudxGlyg.png" /></figure><p>Now let’s check what we have in our CSV file. As you can see, we have two parameters: Access key ID and Secret access key. Good, now we can install our AWS CLI and configure it. Let’s go!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lfrlDBVKu8VbyEdAupcJYg.png" /></figure><p>Now, let’s go to the next page: <a href="https://aws.amazon.com/cli/">https://aws.amazon.com/cli/.</a> I will choose the macOS client because that’s what I use. You can choose the client for your operating system.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*_j10aOHS-CpyUS7m-64rMQ.png" /></figure><p>After installation, let’s check if AWS CLI was installed correctly. Just type in your command line: <strong><em>aws — version</em></strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/966/1*YC8bkKUmSpqSUTR7zalD7Q.png" /></figure><p>Good. AWS CLI is installed, and we have the access key. Let’s start configuring it. First, run the command “<strong><em>aws configure”</em></strong>, and enter your <em>access key</em> and <em>secret access key</em> from the file. For the <em>default region name</em>, you can choose what is convenient for you. For example, I’m in Europe and chose “eu-west-3”. You can find all regions under “Available Regions”. I will keep the output format as the default, “json”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2--3Zq_rGUj34sgl2RaUcg.png" /></figure><p>Now let’s test if everything is working. For example, we can get a list of all our S3 buckets “<strong><em>aws s3 ls”</em></strong> (we will get an empty response because we haven’t created any buckets yet).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/294/1*qK3HZ4dvFEBZb_EDIFVpZw.png" /></figure><h3>Create an S3 bucket using AWS CLI</h3><p>We installed AWS CLI. Now let’s create our first S3 bucket using the CLI and check in the console to ensure everything is working. To create an S3 bucket, we need to type the following command: <strong><em>aws s3 mb s3://bucket-name</em></strong>. The bucket name must be globally unique.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/356/1*7DNAp0G1K7m-IjV3YAgIpw.png" /></figure><p>Congratulations! We just created our first bucket. We checked its existence using the command <strong><em>aws s3 ls</em></strong>. Now, let’s also check using the AWS Console. Open S3 in the AWS Console and verify that your bucket exists.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*MsnzcI40L2PFoDzQKPEB-w.png" /></figure><p>Great, we configured our AWS CLI and created our first S3 bucket using the CLI. What’s next? In upcoming posts, we will learn how to create an EC2 instance using both the CLI and AWS Console. We will also cover how to work with S3 buckets using the CLI.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6e305d72ae88" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Discovery: First steps with AWS Cloud. Part 2]]></title>
            <link>https://medium.com/@acosetov/aws-discovery-first-steps-with-aws-cloud-part-2-0e67a6aabaf6?source=rss-33deb7e5c5c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/0e67a6aabaf6</guid>
            <category><![CDATA[sns]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[cloudwatch]]></category>
            <dc:creator><![CDATA[Alexei]]></dc:creator>
            <pubDate>Sun, 23 Jun 2024 10:56:28 GMT</pubDate>
            <atom:updated>2024-06-23T10:56:28.910Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*F__68XtWmXy47ai7.png" /></figure><p>In the previous <a href="https://medium.com/@acosetov/aws-discovery-first-steps-with-aws-cloud-part-1-f29b9414518a">post</a>, we configured our root account with Multi-Factor Authentication (MFA) and created an admin account with added security measures. In this part, we will cover how to monitor and manage our spending on AWS resources. Often, users create AWS resources and forget about them, which can lead to unexpected charges. This guide will help you keep track of your expenses and avoid unnecessary costs.</p><h3>Create a billing alarm using AWS CloudWatch</h3><p>First, let’s go to our root account. Remember that we now need to use MFA to log in.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HBfNvdqgeICBLJmaKZNm4Q.png" /></figure><p>Enter your MFA code and submit it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8GW8NqjSvuZDh_Wyqb3Y5g.png" /></figure><p>Okay, we are logged in to our root account. First, we will enable billing alerts. Just type ‘billing’ in the search bar and open “<a href="https://console.aws.amazon.com/billing/home?#/">Billing and Cost Management</a>”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ey3tZhtf42LpqiBcRJ3GRQ.png" /></figure><p>After that, in the navigation pane, choose “Billing Preferences”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lPMZ9JMGJ2cGM9BFDKGo5Q.png" /></figure><p>As you can see in the “Alert Preferences” section, all alerts are currently not delivered. Let’s change this. All alerts will be received on our root email.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/992/1*USrW-9jZUgwnOAfAW9MJQA.png" /></figure><p>Good. Now that alerts are enabled, let’s create one using CloudWatch. In the search bar, type “cloudwatch” and choose it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nfNihIf9o7XAqWv_2xzd5w.png" /></figure><p>Before creating an alarm, please choose the region “<strong>US East (N. Virginia)</strong>” and after click “Create alarm”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*KLftz7yGDYYArPeFMoEi4w.png" /></figure><p>Click “Create alarm”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6cSPhDVlV4_8YYK_bZXIJg.png" /></figure><p>At this step, we should select the metric.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*E3m_bdeDyDVbQu7nYHCylA.png" /></figure><p>Choose ‘Billing’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*A2JuC4vUF-cbWMoIsuGJyw.png" /></figure><p>Choose ‘Total Estimated Charge’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p5KuYE6vOFiWN5gfuh7lDw.png" /></figure><p>Select “Metrics”. In our case, this is “USD”, with the metric name “EstimatedCharges”. And click to “Select metric”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8d__0GUQ-Q6YwKxTiczMFQ.png" /></figure><p>Now let’s specify the metric and conditions. I chose a period of 6 hours and a threshold value of 10 USD. You can choose your conditions based on your needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/833/1*Qsg0RqPmwIT0CJSIlcEazg.png" /></figure><p>Now we need to configure actions. In this step, we will create an SNS topic to send notifications to your email. The topic name can be anything you want, and for the email endpoint, I will use my root account email. After that, click “Create topic”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/834/1*PODCM6Tfd_uB0dIJPre_aQ.png" /></figure><p>Now choose our topic and go to the next step.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/827/1*Lc37sAFUV_CYyC6lpGON7w.png" /></figure><p>Now add a name for our alarm and click “Next”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/832/1*E5XUQCkLLy5kpIkd4uWIjQ.png" /></figure><p>Now you can review your alarm configuration, and if everything is okay, click “Create alarm”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h0JCKPFGBuULYgBd48TqgQ.png" /></figure><p>We created our first alarm. However, as you can see, there is a warning in “Actions”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bZ_HnLeybDwJr0-nWyOG3A.png" /></figure><p>You should go to your email and confirm that you subscribed to the topic we created for our alarm notifications. Just click ‘Confirm subscription.’ You will be redirected to the confirmation page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/846/1*6JF_5no-RNj2Jw1KUSjXhw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*A18hgYznSYstmOft6ykMVg.png" /></figure><p>After that, you can return to your alarms dashboard. Just refresh the page, and you will see that the warning has disappeared.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1EAlHhxpJqZBxExFZ1GiDw.png" /></figure><p>Congratulations! You have successfully created your first alarm, and more importantly, you now have control over how much you spend.”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0e67a6aabaf6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Discovery: First steps with AWS Cloud. Part 1]]></title>
            <link>https://medium.com/@acosetov/aws-discovery-first-steps-with-aws-cloud-part-1-f29b9414518a?source=rss-33deb7e5c5c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/f29b9414518a</guid>
            <category><![CDATA[tutorial]]></category>
            <category><![CDATA[amazon]]></category>
            <category><![CDATA[mfa]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Alexei]]></dc:creator>
            <pubDate>Tue, 18 Jun 2024 05:32:32 GMT</pubDate>
            <atom:updated>2024-06-18T05:40:07.391Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ebh7rijBCBTA9Eoa5d9Gww.png" /></figure><p>In previous <a href="https://medium.com/@acosetov/aws-discovery-creating-your-first-account-on-aws-cloud-dcf6a3310273">posts</a>, we learned how to create your first AWS account. Our next steps will be to configure:</p><ul><li><a href="#aa65">MFA for the Root account</a></li><li><a href="#f4f6">Create an admin account and add MFA</a></li></ul><h3>MFA for the Root account</h3><p>First, log in to your AWS account using the Root user credentials that you created in the previous post. And go to <a href="https://console.aws.amazon.com/">AWS Management Console</a>. And enter your root email and password.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/1*IzVUGHPBkHTWeOHI3YtOaA.png" /></figure><p>After logging in, you will see your AWS Management Console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RYECB8gBEaJfDjZVB4XkIg.png" /></figure><p>In the search bar, type “IAM” and select “IAM” from the results.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W4LAZRS2HxeuTFcugN9T0A.png" /></figure><p>You will next “IAM” console</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Lojb445Yb4YGe_CnI6_K2A.png" /></figure><p>As you can see, under “Security recommendations,” there is an option to add MFA to our root user. It is crucial to enable MFA for the root user for the following reasons:</p><ol><li><strong>Enhanced Security:</strong> MFA adds an extra layer of protection, making it significantly harder for unauthorized users to access your account even if they have your password.</li><li><strong>Prevent Unauthorized Access:</strong> If your password is compromised, MFA ensures that attackers cannot access your account without the second form of authentication.</li><li><strong>AWS Best Practices:</strong> Enabling MFA is a recommended best practice by AWS to help secure your account and protect your resources.</li></ol><p>Let’s proceed with enabling MFA for the root user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*STvlwtd7d_XbiJkqQG0WBQ.png" /></figure><p>Push “Add MFA”</p><p>Add “Device name” and chose “MFA Device”</p><p>For authentication, I will use “Google Authenticator” on my phone. And click Next.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WGRNlc2f8YWFPvk6h7FzQw.png" /></figure><p>On the next page, you need to click on “Show QR code.” Take your phone and scan this QR code using “Google Authenticator.” Then, enter the code from your virtual app twice. And click “Add MFA”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kejSHKtNqntMnSiT1JYzwg.png" /></figure><p>Congratulations, you have successfully added MFA to your Root account!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZwasX_azwvx_4iCVFWhMLQ.png" /></figure><h3>Create an Admin account and add MFA</h3><p>Now let’s discuss why it is not recommended to use the root account for everyday work.</p><p><em>Unlimited Access:</em></p><p>The root account has unrestricted access to all resources and services in your AWS environment. Any actions performed with the root account, whether intentional or accidental, can have significant impacts, including deletion of critical resources.</p><p><em>Security Risk:</em></p><p>If the root account credentials are compromised, the attacker gains full access to your AWS resources. This can lead to data breaches, loss of data, and unauthorized usage of AWS services, potentially resulting in significant costs and compliance issues.</p><p><em>Best Practice:</em></p><p>AWS best practices emphasize the principle of least privilege. By using IAM (Identity and Access Management) users and roles with only the necessary permissions, you reduce the risk of accidental or malicious changes to your environment.</p><p>Let’s start by creating an Admin user.</p><p>Navigate to the “Users” section in the IAM console and click “Add user.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QpOSDV1bzUJSEkIZ80SYIA.png" /></figure><p>Let’s specify user details.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6_F2bQe1LckDvJU4jNA4xw.png" /></figure><p>At the next step, we need to attach permission options. I will attach policies directly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*umrUB5h0p1CjgbkCpmhHgg.png" /></figure><p>Click “Next”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NewV5UayuNwLcyZ9NWJa8w.png" /></figure><p>Then click ‘Create user’. User created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*u-RP6VcyilNEQ7tVIYXZww.png" /></figure><p>Copy your console sign-in details URL. In my case, it’s <a href="https://381492104235.signin.aws.amazon.com/console.">https://381492104235.signin.aws.amazon.com/console.</a> Now, we can go to the user list and check if our admin user was created.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/745/1*lwr4GvsG4eWGRKYh_nqpYg.png" /></figure><p>Well done! We have now created the admin user. Let’s also add MFA to our admin user. Log out from your root account or simply open a private window in your browser. Then, go to your sign-in URL: <a href="https://381492104235.signin.aws.amazon.com/console,">https://381492104235.signin.aws.amazon.com/console,</a> enter the admin username and password, and click ‘Sign in’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hQJMV3zYZtrqRc5Z3hqQGQ.png" /></figure><p>And here we are, now logged in as the Admin user to your AWS Account. Let’s add MFA to the admin user. The steps are the same as those we followed for the Root account.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2u8rrtPonpQeRj0CJJbzyg.png" /></figure><p>Once again, search for “IAM” in the search bar.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ULKQnAmbJKNqoi8WHlIBtg.png" /></figure><p>As you can see, there are security recommendations to add MFA for your admin account. Let’s do it. Click on “Add MFA”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OeYmiwI5I9d5dvEeojN33w.png" /></figure><p>Now we will follow the same steps as we did for our root user. Select MFA device</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dArR_31_aLaVlItuYh7AVg.png" /></figure><p>Set up device and add MFA</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*kejSHKtNqntMnSiT1JYzwg.png" /></figure><p>And we’re done. We’ve now added MFA for your root and admin accounts. Now you can rest assured knowing that your accounts are protected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tIP6ok3xfLTmGqjOUvjGIA.png" /></figure><p>I’ve divided “First Steps with AWS Cloud” into three parts. In the next post, I will describe how to set up billing alarms, install AWS CLI, and create your first EC2 instance. See you then!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f29b9414518a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[AWS Discovery: Creating Your First Account on AWS Cloud]]></title>
            <link>https://medium.com/@acosetov/aws-discovery-creating-your-first-account-on-aws-cloud-dcf6a3310273?source=rss-33deb7e5c5c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/dcf6a3310273</guid>
            <category><![CDATA[amazon-web-services]]></category>
            <category><![CDATA[beginner]]></category>
            <category><![CDATA[create-account]]></category>
            <category><![CDATA[cloud-computing]]></category>
            <category><![CDATA[amazon]]></category>
            <dc:creator><![CDATA[Alexei]]></dc:creator>
            <pubDate>Sun, 09 Jun 2024 16:04:20 GMT</pubDate>
            <atom:updated>2024-06-09T16:04:20.601Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dnyycwNUZxqUOgeqmEKbaQ.png" /></figure><p>This is the first article in the AWS Discovery series, where we will create our very first account for future work. Let’s go!</p><h3>Let’s get started!</h3><p>To start, create your AWS account by first going to <a href="https://aws.amazon.com/">https://aws.amazon.com/</a> and clicking on “Create an AWS Account.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QjD-mkgVTGVS1UqtoNjq9g.png" /></figure><p>And starting from the next page, we will encounter some registration challenges.</p><p>First, you need to add your email and AWS account name. After that, click on “Verify email address.”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/825/1*BpfnFOFNtT5sqEWxiIDHXQ.png" /></figure><p>Go to your mailbox, and you should receive an email from AWS with a verification code. And click “Verify”</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/807/1*_JSxgCm_3_ZXEct72T42AQ.png" /></figure><p>After the email is verified, we will see another page where we should create a password for our account. Be careful, the password should be secure, but <strong>please don’t forget it</strong>.📝😀</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/788/1*XQrvfSTOswAo-LO2Qv-T_A.png" /></figure><p>On the next page, we have a Contact Information form in two versions:</p><ul><li>Business — for your work, school, or organization</li><li>Personal — for your own projects</li></ul><p>The conditions are the same, but for “Business,” we have an “Organization name” field. Since we are creating an account for our own projects, let’s select “Personal.” Fill in the form with your personal information, and let’s go to the next step.✈</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*EwqLX6r8hXpT91ER2NNiGQ.png" /></figure><p>Okay, let’s go to the next step, security verification. In this step, you need to add your billing information. Amazon will charge you 1💲, but this 1💲 will be returned. This is just for billing verification. Let’s do this.✈</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*IMqptEV7mRndKAJSvOzV9g.png" /></figure><p>In this step, you need to verify your phone number ☎. You can choose to receive the verification code by SMS or Voice Call. In my case, I chose SMS.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/731/1*iSf2woO9cObKUuYY5EU3Iw.png" /></figure><p>When you receive your verification code, fill in the form and continue.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/694/1*P_Xq7z5rwx7k9QXdWZxMnQ.png" /></figure><p>Our last step is to choose your support plan. To start, select “Basic support,” which is free. Let’s go.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/734/1*xwKEElzhF4dIlrp2OVGvyQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/583/1*MJNvGYD-hpKhlGncZyYkTQ.png" /></figure><p>Congratulations! You just created your first AWS account.🥳</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/454/1*Umd31RSjv-Jg5x3TSnoLSg.png" /></figure><p>Now let’s try to log in. Enter the email and password that you used during registration.✅</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aqYuP-55QN7Fv4Mlm5p7BA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/359/1*DcChy1eHwgtYIF2OsyHeeA.png" /></figure><p>Congratulations! You have successfully created your first account and logged in. Great job!🎉</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7-0bKwii6XPKoPRBmViW9A.png" /></figure><p>In the next post, I will describe best practices for securing your AWS account. See you then.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=dcf6a3310273" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Install and Configure Dante Socks5 Proxy on CentOS 7]]></title>
            <link>https://medium.com/@acosetov/install-and-configure-dante-socks5-proxy-on-centos-7-ebdac4f60312?source=rss-33deb7e5c5c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/ebdac4f60312</guid>
            <dc:creator><![CDATA[Alexei]]></dc:creator>
            <pubDate>Thu, 07 Dec 2023 09:57:06 GMT</pubDate>
            <atom:updated>2025-02-28T14:07:43.633Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ExumFG1Bb0frj6neJ4ZdZw.png" /><figcaption>Generated by DALL-E</figcaption></figure><blockquote>Dante is a free and open-source software project that provides a SOCKS server and a SOCKS client, functioning as a proxy server.</blockquote><h4>Step 1: Install the development tools for CentOS</h4><pre>$ yum groupinstall &#39;Development Tools&#39; -y</pre><h4>Step 2: Download and install Dante server</h4><pre>$ wget https://www.inet.no/dante/files/dante-1.4.4.tar.gz<br>$ tar -xvf dante-1.4.4.tar.gz<br>$ cd dante-1.4.4<br>$ ./configure<br>$ make<br>$ make install<br>$ sockd --version </pre><p>Create <strong>stop-start-daemon</strong> to make it easier to control Dante.</p><pre>$ touch /lib/systemd/system/danted.service</pre><pre>[Unit]<br>Description=SOCKS (v4 and v5) proxy daemon (danted)<br>Documentation=man:sockd(8) man:sockd.conf(5)<br>After=network.target<br><br>[Service]<br>Type=forking<br>PIDFile=/var/run/sockd.pid<br>ExecStart=/usr/local/sbin/sockd -f /etc/sockd.conf -D<br><br>[Install]<br>WantedBy=multi-user.target</pre><h4>Step 3: Configure Dante server</h4><p>First, we need to create a log file, a PID file, and a configuration file.</p><pre># Create log file<br>$ touch /var/log/danted.log<br>$ chmod 777 /var/log/danted.log<br><br># Create PID file<br>$ touch /var/run/sockd.pid<br>$ chmod 777 /var/run/sockd.pid<br><br># Create config file<br>$ touch /etc/sockd.conf</pre><p>Now, let’s fill out our configuration file (“sockd.conf”).</p><pre>user.privileged: root<br>user.unprivileged: nobody<br><br>logoutput: /var/log/danted.log<br>debug: 0<br><br>internal.protocol: ipv4<br>internal: eth0 port = 1080<br><br>external.protocol: ipv4<br>external: eth0<br><br>external.rotation: same-same<br>socksmethod: username<br>clientmethod: none<br><br>client pass {<br>        from: 0.0.0.0/0 to: 0.0.0.0/0<br>        log: connect disconnect error ioop<br>        socksmethod: username<br>}<br><br>socks pass {<br>        from: 0.0.0.0/0 to: 0.0.0.0/0<br>        log: connect disconnect error ioop<br>        socksmethod: username<br>}</pre><p>Additionally, let’s enhance security by creating a user with a password, which we will use to access our proxy server.</p><pre>$ useradd --shell /usr/sbin/nologin proxyuser<br>$ passwd proxyuser</pre><p>And for our last step, we should run our Dante server.</p><pre>$ systemctl start danted.service</pre><h4>Step 4: Test our server</h4><p>Let’s conduct a quick test to check our Dante server. We can log in from another server, or we can use our local machine with CURL.</p><pre>$ curl -4 -x socks5://proxyuser:password@proxy_ip:1080 https://ipecho.net/plain; echo<br>#in output we should see our proxy_ip </pre><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ebdac4f60312" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Migrate a MySQL database to PostgreSQL using pgLoader]]></title>
            <link>https://medium.com/@acosetov/migrate-a-mysql-database-to-postgresql-using-pgloader-5d943b9fc51c?source=rss-33deb7e5c5c1------2</link>
            <guid isPermaLink="false">https://medium.com/p/5d943b9fc51c</guid>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[migration]]></category>
            <category><![CDATA[database]]></category>
            <category><![CDATA[postgresql]]></category>
            <dc:creator><![CDATA[Alexei]]></dc:creator>
            <pubDate>Thu, 21 Sep 2023 16:51:30 GMT</pubDate>
            <atom:updated>2023-09-21T16:51:30.596Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1012/1*ha_Fl0F_rajT2yoJtJP0pQ.png" /></figure><p>I have received an interesting task, and I believe it would be beneficial to share it. Perhaps it can help someone.</p><h4><strong>Problem Description</strong></h4><p>I received a dump of a MySQL database, but my current main database is PostgreSQL. I started thinking about how to migrate data from MySQL to PostgreSQL and found a tool called <a href="https://pgloader.io/">PgLoader</a>.</p><h4><strong>What tools we need</strong></h4><ol><li>Mysql</li><li>Postgresql</li><li>PgLoader</li></ol><p>*I will demonstrate the entire process on AlmaLinux 8.</p><h4><strong>Let’s get started</strong></h4><ol><li><strong>Install Mysql 8</strong></li></ol><pre>dnf install mysql-server -y<br>systemctl start mysqld<br>systemctl enable mysqld</pre><p>Now, let’s create a <strong><em>test_db</em></strong> with a table and add some data.</p><pre>mysql -u root -p</pre><pre># create database test_db<br>CREATE DATABASE test_db;<br><br># switch to this database test_db<br>USE test_db;<br><br># create table sample_table<br>CREATE TABLE sample_table (<br>  user_id INT PRIMARY KEY,<br>  first_name VARCHAR(50),<br>  last_name VARCHAR(50),<br>  birthday DATE<br>);<br><br># populate this table with some data<br>INSERT INTO sample_table (user_id, first_name, last_name, birthday) <br>VALUES (1, &#39;Alexei&#39;, &#39;Cosetov&#39;, &#39;1986-05-16&#39;),<br>(2, &#39;Elon&#39;, &#39;Musk&#39;, &#39;1971-06-28&#39;),<br>(3, &#39;Bill&#39;, &#39;Gates&#39;, &#39;1955-10-28&#39;),<br>(4, &#39;Jeff&#39;, &#39;Bezos&#39;, &#39;1964-01-12&#39;),<br>(5, &#39;Mark&#39;, &#39;Zuckerberg&#39;, &#39;1984-05-14&#39;);<br><br># can close the MySQL prompt<br>exit</pre><p><strong>2. Install Postgresql 15</strong></p><p>Let’s install PostgreSQL now. I will use the official PostgreSQL package for CentOS/AlmaLinux.</p><pre># Install the repository RPM:<br>dnf install -y https://download.postgresql.org/pub/repos/yum/reporpms/EL-8-x86_64/pgdg-redhat-repo-latest.noarch.rpm<br><br># Disable the built-in PostgreSQL module:<br>dnf -qy module disable postgresql<br><br># Install PostgreSQL:<br>dnf install -y postgresql15-server<br><br># Optionally initialize the database and enable automatic start:<br>/usr/pgsql-15/bin/postgresql-15-setup initdb<br>systemctl enable postgresql-15<br>systemctl start postgresql-15</pre><p>Now, let’s create a <strong><em>main_db </em></strong>using<strong><em> </em></strong><em>Postgresql.</em></p><pre>su postgres<br>psql</pre><pre># add default password for user postgres<br>ALTER USER postgres PASSWORD &#39;myPassword&#39;;<br># create database main_db<br>CREATE DATABASE main_db;<br>\q</pre><p><strong>3. Install PgLoader</strong></p><p>After installing PostgreSQL from the official repository, you will also find PgLoader available in the official repository. Let’s start.</p><pre>dnf install pgloader -y</pre><p>Installation of PgLoader is complete.</p><h4>Migrate data from MySQL to Postgresql</h4><p>We are ready to go.</p><pre>pgloader mysql://root@localhost/test_db pgsql://postgres:myPassword@localhost/main_db</pre><p>This is the result of running the command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1010/1*xjvZSVolQ8d1rNOySPUI_A.png" /></figure><p>Now let have a look what we have in Postgresql.</p><pre>sudo -u postgres psql</pre><pre># connect to our main_db<br>\c main_db<br><br># select data<br>select * from sample_table;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/420/1*_bC2d8PlKOXTc8bX3KT5xw.png" /></figure><p>We have completed the data migration from MySQL to PostgreSQL.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5d943b9fc51c" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>