<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Shannon Chan on Medium]]></title>
        <description><![CDATA[Stories by Shannon Chan on Medium]]></description>
        <link>https://medium.com/@channeng?source=rss-6c8694088910------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*KPeCdgKmwskNlR7NtenC_Q.jpeg</url>
            <title>Stories by Shannon Chan on Medium</title>
            <link>https://medium.com/@channeng?source=rss-6c8694088910------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 10:35:25 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@channeng/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Introducing Image Search & Price Suggestions]]></title>
            <link>https://medium.com/carousell-insider/introducing-image-search-price-suggestions-ce8e40a0163f?source=rss-6c8694088910------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce8e40a0163f</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[image-search]]></category>
            <category><![CDATA[carousell]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Shannon Chan]]></dc:creator>
            <pubDate>Mon, 25 Feb 2019 01:46:00 GMT</pubDate>
            <atom:updated>2019-03-04T08:44:27.371Z</atom:updated>
            <content:encoded><![CDATA[<p><em>In the past seven years, Carousell has led the way in mobile classifieds and is one of the fastest growing mobile marketplaces in Southeast Asia. Currently, we are in 19 cities across 7 countries. We’re always looking at ways to leverage machine learning to enhance our users’ experience.</em></p><p>Over the course of 2018, we have launched multiple AI-powered features such as <a href="https://medium.com/carousell-insider/deep-image-understanding-at-carousell-121857434837">category and title suggestions</a> as well as <a href="https://medium.com/carousell-insider/introducing-chat-reply-on-carousell-3efdf06a72e1">suggested chat replies</a>.</p><p>Today, we are excited to launch two new features to help our buyers and sellers: Image Search &amp; Price Suggestion. In this post, we will introduce these features and share how we use AI to power them.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/0*weBtSXn72fUX5Iwl" /></figure><p>Shopping on Carousell has always been a visually engaging experience. Just think of all the time we spend scrolling through the feed. We know that when it comes to buying, you really have to see something before you even consider it.</p><p>The inspiration to buy can come at unpredictable times, whether you are out window shopping with friends or browsing social media. In these situations, we often resist the urge for an impulse buy and make a mental note to try searching for it later. However, when we do remember to come back to it we might find it difficult to describe what exactly we saw in words, much less search terms.</p><p>With Image Search, users now have the ability to use pictures to easily find things on Carousell. Instead of typing keywords into the search bar, simply select an image from your camera roll to search for items of similar colour, style and design.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/360/0*JjNj0LwwiSidHtJ0" /></figure><p>As a two-sided marketplace, we serve buyers <strong>and </strong>sellers. While we build features that help buyers find what they want on the platform, we also make it point to solve common problems sellers face.</p><p>What we’ve found is that sellers often struggle to price their products to get the best value. Price too high and you won’t receive any chats and offers. Price too low, and you lose out on potential earnings.</p><p>To figure out an appropriate price, sellers usually search for similar listings to get a sense of market prices before actually listing their product. This can be quite tedious and time-consuming.</p><p>We introduced Price Suggestions to directly address this cumbersome process. Given the listing’s image and title, the feature uses AI and our transaction database to suggest an appropriate price for your listing based on prices of similar products that were successfully sold. We also display a set of similar listings that were recently sold below the suggested prices.</p><p><strong>How it works</strong></p><p>Image Search and Price Suggestions each address one side of the marketplace, be it buyers or sellers. However, both features actually share similar mechanisms under the hood.</p><p>We search for listings of similar attributes, with features extracted using AI-based computer vision and natural language processing. In the case of Price Suggestion, we also compute the price statistics on top of the retrieved results to suggest appropriate price ranges.</p><p><strong>Learning feature embeddings</strong></p><p>Traditionally, extracting features from images requires extensive feature engineering from computer vision algorithms. However, we have already developed a <a href="https://medium.com/carousell-insider/deep-image-understanding-at-carousell-121857434837">neural network with deep convolutional layers</a>. We are able to train our model to map the raw image and the listing title into a shared high<strong>-</strong>dimensional vector space.</p><p>Given an image and a bag-of-ngrams representation of a title, our model learns embeddings for both of them, and maximises their dot-product.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*qMcWyp7sfP0pIKE1" /></figure><p>The learned vector space is where semantically similar titles and visually similar images are located closer to each other. This results in implicit clusters that capture meaningful information about how listings relate to one another.</p><p><strong>Nearest Neighbour Search</strong></p><p>To find similar listings from a given image, we generate feature embeddings from the trained model and perform a nearest neighbour search on a collection of listings also embedded in the same the vector space. Results retrieved are then ranked by closest distance based on the feature embeddings.</p><p>However, performing an exact nearest neighbour search can lead to relatively high latency given the large collection and high dimensionality of the vectors. An additional consideration would be that loading the collection of pre-computed embeddings from disk into memory would take hundreds of seconds.</p><p>Hence, we surveyed several Approximate Nearest Neighbour (ANN) search solutions, and settled on Spotify’s Annoy library. Annoy works by building binary trees through random projections (a hyperplane split between 2 random points) recursively until there are at most K items left in each node.</p><p>In this way, Annoy drastically reduces the search latency and memory footprint while preserving precision. The indices built with Annoy can also be mapped into memory quickly, reducing load time.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/858/0*9M_kSUpF4ItbBX5W" /></figure><p>At time of inference, we generate feature embeddings for a given listing, and use Annoy to perform an approximate nearest neighbour search on top of a pre-computed collection of indexed listings. The results are then surfaced to the user in their search.</p><p><strong>Price Suggestion</strong></p><p>Price Suggestions uses a similar mechanism as Image Search. Using a concatenated set of image and title embeddings, we found that the nearest neighbour search yielded more accurate results than simply using only image search, especially for listings in which brands cannot be reliably inferred from the image.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1002/0*et1T4LPtYH7-8X1E" /></figure><p>We compared the performance of the neural network approach to a baseline full-text search approach through an internal A/B evaluation tool. Staff members were presented a test set of listings, with two sets of results computed — one from the neural network, the other through full-text search.</p><p>We found that results from the neural network approach significantly outperformed the full-text search, and was picked nearly 5 times more often.</p><p>In the process of evaluation, we made several key observations on each approach. Generally, full-text search fell short with generic titles such as “sling bag”, which produced a variety of irrelevant listings. In contrast, the neural network was able to infer the specific type of sling bag from the image, and return results that were more similar to the original sling bag.</p><p>However, the neural network would occasionally fail to take into account certain important text features such as brands, which would heavily influence the price of an item, e.g. “Zara Striped Crop Top”.</p><p>To address the shortcomings of the neural network approach, we applied a final sort on top of the retrieved set of results based on title match. Through computing price percentiles from weighted samples biased towards results of better match in titles, we were able to further improve the accuracy of the suggested price ranges.</p><p>Image Search and Price Suggestion are two of our latest features that we are proud to introduce to our community. We hope that these machine learning powered features can enhance your experience on our marketplace. If you haven’t tried them out, give it a go and you might just surprise yourself!</p><p>Happy buying and selling!</p><p><em>Special thanks to Jethro, Jay, Johnny, Kan, Hoang and many other teammates who have made invaluable contributions to bring these features to live for our users.</em></p><p><em>We are always looking at ways to leverage machine learning to enhance the user experience and are </em><a href="https://careers.carousell.com/?utm_source=articlebanner&amp;utm_medium=medium"><em>currently hiring data scientists and machine learning engineers</em></a><em> to join us in building more features like this.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce8e40a0163f" width="1" height="1" alt=""><hr><p><a href="https://medium.com/carousell-insider/introducing-image-search-price-suggestions-ce8e40a0163f">Introducing Image Search &amp; Price Suggestions</a> was originally published in <a href="https://medium.com/carousell-insider">Carousell Insider</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Celery Scheduler (Part 2): Managing Celery with Supervisor]]></title>
            <link>https://medium.com/@channeng/celery-scheduler-part-2-managing-celery-with-supervisor-2a0c6e7f7a6e?source=rss-6c8694088910------2</link>
            <guid isPermaLink="false">https://medium.com/p/2a0c6e7f7a6e</guid>
            <category><![CDATA[supervisor]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[celery]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Shannon Chan]]></dc:creator>
            <pubDate>Sun, 29 Oct 2017 05:26:11 GMT</pubDate>
            <atom:updated>2018-02-11T02:45:47.033Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*SPKX4Fvnson8bLzJJ5LA-g.png" /></figure><p>Recall in the previous article (<a href="https://medium.com/@channeng/setting-up-a-task-scheduler-application-with-celery-flask-part-1-8652265050dc">Setting up a task scheduler application with Celery &amp; Flask</a>) , we covered:</p><p>1. Setting up Celery with Flask<br>2. Writing and scheduling task in Celery<br>3. Install Redis as the message broker<br>4. Running Celerybeat, Celery worker, and Redis together</p><p>In this article, we will explore how we can use <a href="http://supervisord.org/">Supervisor</a> to manage the 3 running components, Celerybeat, Celery worker, and Redis required for the Celery application to run.</p><p>With Supervisor, we can manage and control the processes running Redis, Celerybeat and celery worker. It has capabilities to ensure these processes run persistently, and monitor them as long-running programs through a consistent interface.</p><p>In other words, you don’t have to worry about having them to run them within a terminal shell or restart these programs when they do happen to crash in any case since Supervisor will manage all these for you with the right configurations.</p><p>Alright let’s get started!</p><p>First, let’s install supervisor.</p><pre># In Ubuntu server <br>sudo apt-get install -y supervisor</pre><pre># Activate virtualenv and install supervisor<br>source ~/.virtualenvs/celery_env/bin/activate<br>pip install supervisor==3.3.3</pre><p>Supervisor works by executing programs through configuration files that it is fed with. These configuration files inform Supervisor which executables to run, the environment variables that come with it, how the output will be handled.</p><p>All programs run under Supervisor must be run in a <a href="http://supervisord.org/subprocess.html#nondaemonizing-of-subprocesses">non-daemonising mode</a> (a.k.a. &#39;foreground mode&#39;, executables which takes control of the terminal and requires you to press Ctrl-C to regain control of your terminal). As you may recall, this was how we ran Redis, Celerybeat and Celery worker in the previous article.</p><p>Now that we get the concepts out of the way, let’s start to create the configs. In the following commands, we’ll create a folder in our project directory to store these configs.</p><pre># Create a folder for supervisord configs<br>cd ~/celery-scheduler<br>mkdir configs<br>cd configs<br>mkdir conf.d</pre><p>In the following file, we’ll create a main config for the supervisor program. The key points that you will need to take note in this file is where the supervisor logs are (/var/log/supervisor), and where the program config files are read from: files = /etc/supervisor/conf.d/*.conf.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/491eb4aea7354d95fdb8c5cb478f9f17/href">https://medium.com/media/491eb4aea7354d95fdb8c5cb478f9f17/href</a></iframe><p><strong>Program configurations</strong></p><p>To tell Supervisor which programs you want to run and how you want them to run, you will have to write program configurations. It’s quite simple, and the configuration options for program definitions are robust.</p><p>The simplest program configuration looks like this:</p><pre>[program:hello_world]<br>command=/bin/hello_word.sh</pre><p>In our case, we have 3 programs that we need to trigger for Celery to run, that is Celerybeat, Celery worker and Redis.</p><p>So we’ll have to create config files for each of these programs and store them in configs/conf.d folder.</p><pre>mkdir conf.d<br>cd conf.d</pre><p>Save the following following program configs in ~/celery-scheduler/configs/conf.d/.</p><ol><li>redis.conf</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac0a566ca8240a57872de31401880a18/href">https://medium.com/media/ac0a566ca8240a57872de31401880a18/href</a></iframe><p>2. celerybeatd.conf</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5567f4c4f1de557727d3b3359f3db741/href">https://medium.com/media/5567f4c4f1de557727d3b3359f3db741/href</a></iframe><p>3. celeryd.conf</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e88ecb2797134513349cc3bc974a45b3/href">https://medium.com/media/e88ecb2797134513349cc3bc974a45b3/href</a></iframe><p>A thing to note about the program configs is that Supervisor requires us to use the <strong>absolute path</strong> to the Python interpreter in a virtualenv. In the above files, I have set them as /path/to/home/.virtualenvs/celery_env/bin.</p><p>You’ll need to replace all /path/to/home in the following program conf files with the absolute path to home. To get absolute path of your home in bash: echo $HOME.</p><p>Recall from supervisor.conf above that our program configs are will be read from: /etc/supervisor/conf.d/*.conf. So we’ll have to copy our the above 3 program config files into /etc/supervisor/.</p><pre># Copy supervisor.conf to /etc/supervisor<br>sudo mkdir /etc/supervisor/<br>cp ~/celery-scheduler/configs/supervisord.conf /etc/supervisor/</pre><pre># Copy program configs to /etc/supervisor/conf.d<br>sudo mkdir /etc/supervisor/conf.d/<br>cp ~/celery-scheduler/configs/conf.d/*.conf /etc/supervisor/conf.d/</pre><p>In our final step, we’ll also have to create the directories which we have configured for Supervisor to store the program logs.</p><pre>sudo mkdir /var/log/supervisor<br>sudo mkdir /var/log/celery<br>sudo mkdir /var/log/redis</pre><p>And that’s it, we’re done!</p><p>Start Supervisor by running the following command: supervisord -c /etc/supervisor/supervisord.conf</p><p>To check that supervisor is successfully running the programs,</p><ol><li>Run ps aux to make sure that program processes are running</li></ol><p>You should see Redis, Celerybeat and Celery worker processes running:</p><pre>USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND<br> root 1 0.0 2.5 56492 12728 ? Ss Oct21 0:05 /usr/bin/python /usr/bin/supervisord — nodaemon<br> root 7 0.0 0.5 31468 2616 ? Sl Oct21 0:30 /home/ubuntu/celery-scheduler/redis-3.2.1/src/redis-server *:6379<br> root 8 0.0 8.3 98060 41404 ? S Oct21 0:00 /home/ubuntu/.virtualenvs/celery_env/bin/python2.7 /home/ubuntu/.virtualenvs/celery_env/bin/celery beat -A ap<br> root 9 0.1 8.4 91652 41900 ? S Oct21 0:42 /home/ubuntu/.virtualenvs/celery_env/bin/python2.7 /home/ubuntu/.virtualenvs/celery_env/bin/celery worker -A<br> root 20 0.0 9.3 99540 46820 ? S Oct21 0:00 /home/ubuntu/.virtualenvs/celery_env/bin/python2.7 /home/ubuntu/.virtualenvs/celery_env/bin/celery worker -A</pre><p>2. Check the program logs</p><pre>tail /var/log/redis/redis.log<br>tail /var/log/celery/beat.log<br>tail /var/log/celery/worker.log<br>tail /var/log/supervisor/supervisord.log</pre><p>You should also see the task print_hello running every minute in your worker.log.</p><pre>tail -f /var/log/celery/worker.log</pre><p>Output:</p><pre> [2017–10–22 03:18:00,050: INFO/MainProcess] Received task: app.tasks.test.print_hello[aa1b7700–1665–4751-ada2–35aba5670d40]<br> [2017–10–22 03:18:00,051: INFO/ForkPoolWorker-1] app.tasks.test.print_hello[aa1b7700–1665–4751-ada2–35aba5670d40]: Hello<br> [2017–10–22 03:18:00,052: INFO/ForkPoolWorker-1] Task app.tasks.test.print_hello[aa1b7700–1665–4751-ada2–35aba5670d40] succeeded in 0.000455291003163s: None</pre><p>If successfully deployed, supervisor logs should display:</p><pre> INFO success: redis entered RUNNING state, process has stayed up for &gt; than 10 seconds (startsecs)<br> INFO success: celerybeat entered RUNNING state, process has stayed up for &gt; than 10 seconds (startsecs)<br> INFO success: celery entered RUNNING state, process has stayed up for &gt; than 10 seconds (startsecs)</pre><p>That’s it! You now have Celery app with persistent processes fully managed by Supervisor.</p><p>In the next article, we’ll see how we can use Docker to automate the deployment of the entire setup on a server.</p><p>Stay tuned!</p><p>P.S. If this article has been helpful for you, give me a clap below to let me know! :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2a0c6e7f7a6e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Celery Scheduler (Part 1): Setting up a Task Scheduler application with Celery & Flask]]></title>
            <link>https://medium.com/@channeng/setting-up-a-task-scheduler-application-with-celery-flask-part-1-8652265050dc?source=rss-6c8694088910------2</link>
            <guid isPermaLink="false">https://medium.com/p/8652265050dc</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[home-automation]]></category>
            <category><![CDATA[business-intelligence]]></category>
            <category><![CDATA[celery]]></category>
            <category><![CDATA[flask]]></category>
            <dc:creator><![CDATA[Shannon Chan]]></dc:creator>
            <pubDate>Sun, 22 Oct 2017 16:41:04 GMT</pubDate>
            <atom:updated>2018-08-25T04:27:28.438Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/512/1*sUA09k79SchtiamEA3tCYQ.png" /></figure><p>As part of the Data team, I’ve found it incredibly useful to have a reliable scheduler application that runs routine tasks. These tasks could be creating reports that power dashboards and visualisation tools, or they can be workflows that feed data to other ingestion pipelines of a production system. That was how I discovered Celery.</p><p>Celery allows you to setup a powerful, distributed and fuss-free application task scheduler. Once you set it up on a server, it can reliably run scheduled tasks at regular defined intervals.</p><p>All you need to do is to define your task method, the schedule to trigger the task, and Celery will handle the rest for you.</p><p>Some interesting uses for your own task scheduler include:</p><ul><li>Home automation (IOT projects)</li><li>Data Workflow management for Business Intelligence (BI)</li><li>Triggering Email campaigns</li><li>Any other routine, periodic tasks</li></ul><p>This series of articles will walk you through how you can set up your own task scheduler application with Celery and Flask, managed by Supervisord and deployed with Docker.</p><p>However, if you are eager to run the finished project, feel free to skip this tutorial and hop over to <a href="https://github.com/channeng/celery-scheduler">Celery Scheduler github project</a> to get started.</p><p>In this article, I will cover the basics of setting up Celery with a web application framework Flask.</p><p>Let’s start by creating a project directory and a new virtual environment to work with!</p><pre># Create virtualenv<br>pip install --upgrade pip<br>pip install --upgrade virtualenv<br>virtualenv -p /usr/bin/python2.7 ~/.virtualenvs/celery_env</pre><pre># Activate virtualenv<br>source ~/.virtualenvs/celery_env/bin/<br>pip install Flask==0.12.2 celery==4.1.0 redis==2.10.6</pre><pre># Create project directory<br>mkdir ~/celery-scheduler<br>cd ~/celery-scheduler</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/595/1*lws1JBFixLLoOhcsDDeTuw.png" /></figure><p>To begin, let’s first set up the Flask framework with Celery.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/943dc560354bc0ca78b226265dea46a9/href">https://medium.com/media/943dc560354bc0ca78b226265dea46a9/href</a></iframe><p>Save this python script as app.py in the main project directory, and in your terminal, run: python ~/celery-scheduler/app.py. Now go to <a href="http://localhost:5000/">http://localhost:5000/</a>. If everything is running fine, you should see “Hello, Flask is up and running!”</p><p>Next, I’m going to isolate the config variables from the application code. We’ll move the configuration variables to a config file.</p><p>In the same folder, save the following script as config.py:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2c93fb796e3faa348235a7393e8f7ad9/href">https://medium.com/media/2c93fb796e3faa348235a7393e8f7ad9/href</a></iframe><p>Now that we have extracted our configuration variables in config.py, we can remove them from app.py. After removing the config variables, app.py should look much simpler, like the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/edcb64ccb92bc3a41179669d8d4c68d1/href">https://medium.com/media/edcb64ccb92bc3a41179669d8d4c68d1/href</a></iframe><p>Notice here we read the config directly into the Flask app. Again, run the command: python ~/celery-scheduler/app.py and go to <a href="http://localhost:5000/">http://localhost:5000/</a> to make sure that everything is running fine.</p><p>At this stage, you should now have 2 files: app.py and config.py within your project directory, ~/celery-scheduler/.</p><p>Now that we have Celery running on Flask, we can set up our first task!</p><p>First, create a new folder app with which we can organise our code.</p><pre>mkdir ~/celery-scheduler/app<br>mv ~/celery-scheduler/app.py ~/celery-scheduler/app/__init__.py</pre><p>Notice, we moved our original file app.py to app/__init__.py, which is really for convenience since it doesn’t change the import path of app.</p><p>Now, we can create a folder to contain all our tasks:</p><pre>mkdir ~/celery-scheduler/app/tasks<br>touch ~/celery-scheduler/app/tasks/__init__.py</pre><p>To create a celery task, define your task method and decorate the python method with (guess what?) @celery.task().</p><p>Let’s write our first task and save the file as test.py within tasks folder:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/35af740855311fd5b84f05f3dfd18940/href">https://medium.com/media/35af740855311fd5b84f05f3dfd18940/href</a></iframe><p>Next, we will have to set up a schedule for celerybeat to trigger the above task. In the main project directory ~/celery-scheduler/, save the following script as celeryconfig.py .</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/deb05ce6d3ba9fc9232b8b6b7d3f28c7/href">https://medium.com/media/deb05ce6d3ba9fc9232b8b6b7d3f28c7/href</a></iframe><p>Celeryconfig contains the configurations for celery to execute the tasks, including import path, task serialization format, and of course, the schedule for which tasks should be triggered.</p><p>In the above, we have set up CELERYBEAT_SCHEDULE to trigger the task test-celery every minute.</p><p>Now that we have a new celeryconfig.py, we’ll need to update app/__init__.py to read from celeryconfig. To update celery with our newly written celeryconfig, we will import the config file and added 2 lines of code:</p><pre>import celeryconfig<br>…<br>celery.config_from_object(celeryconfig)<br>…</pre><p>At this point, app/__init__.py should look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7d763fe8e84c55fa76706deafe544770/href">https://medium.com/media/7d763fe8e84c55fa76706deafe544770/href</a></iframe><p>Great! One last step before we can see task scheduler application running. We will have to set up a message broker for celerybeat to transmit messages to task workers.</p><p>For this, we will use Redis as our message broker.</p><p>Add the script install_redis.sh to the project directory:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f53c1c86fa53079237c5118d23b61925/href">https://medium.com/media/f53c1c86fa53079237c5118d23b61925/href</a></iframe><p>Next, run bash ~/celery-scheduler/install_redis.sh to download and build redis-server.</p><p>With all the pieces set up, let’s spin them up and see our task in action.</p><ol><li>Run Redis: <br>~/celery-scheduler/redis-3.2.1/src/redis-server</li><li>Run celerybeat: <br>~/.virtualenvs/celery_env/bin/celery beat -A app.celery --schedule=/tmp/celerybeat-schedule --loglevel=INFO --pidfile=/tmp/celerybeat.pid</li><li>Run celery worker: <br>~/.virtualenvs/celery_env/bin/celery worker -A app.celery --loglevel=INFO</li></ol><p>On the terminal screen running celerybeat, you should see the scheduler sending the task every minute on the dot:</p><pre>[2017–10–22 23:47:28,861: INFO/MainProcess] beat: Starting…<br>[2017–10–22 23:48:00,019: INFO/MainProcess] Scheduler: Sending due task test-celery (app.tasks.test.print_hello)</pre><p>On the other screen where you trigger celeryd, you should see the task being received and logged by the celery worker:</p><pre>[2017–10–22 23:48:06,890: INFO/MainProcess] Connected to redis://0.0.0.0:6379/0<br>[2017–10–22 23:48:06,898: INFO/MainProcess] mingle: searching for neighbors<br>[2017–10–22 23:48:07,917: INFO/MainProcess] mingle: all alone<br>[2017–10–22 23:48:07,936: INFO/MainProcess] celery@Shannons-MacBook-Air.local ready.<br>[2017–10–22 23:48:08,144: INFO/MainProcess] Received task: app.tasks.test.print_hello[1cf686bf-589d-4bbd-bd3e-a6a707431caa]<br>[2017–10–22 23:48:08,148: INFO/ForkPoolWorker-3] app.tasks.test.print_hello[1cf686bf-589d-4bbd-bd3e-a6a707431caa]: Hello<br>[2017–10–22 23:48:08,160: INFO/ForkPoolWorker-3] Task app.tasks.test.print_hello[1cf686bf-589d-4bbd-bd3e-a6a707431caa] succeeded in 0.0129454089329s: None</pre><p>Great! We’ve got everything up and running now. But do we have to keep each of the programs running in separate screens? Sure, we could use screen or tmux terminal multiplexer to keep them running, but what if any of the programs crash?</p><p>In the part 2 of this series, we will explore how we can use Supervisor to manage redis, celerybeat and celery processes and keep them running in the background.</p><p>Check it out here: <a href="https://medium.com/@channeng/celery-scheduler-part-2-managing-celery-with-supervisor-2a0c6e7f7a6e">Managing Celery with Supervisor</a></p><p>Till then!</p><p>P.S. If this article has been helpful for you, I’d be happy to know so do give me a clap below!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8652265050dc" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>