<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Jakub Drobník on Medium]]></title>
        <description><![CDATA[Stories by Jakub Drobník on Medium]]></description>
        <link>https://medium.com/@drobnik.j?source=rss-f9983ec66fb4------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*T947Pgb7QA_9hxvFtuN4eQ.jpeg</url>
            <title>Stories by Jakub Drobník on Medium</title>
            <link>https://medium.com/@drobnik.j?source=rss-f9983ec66fb4------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 20:01:32 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@drobnik.j/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Updates to key-value store API]]></title>
            <link>https://medium.com/@drobnik.j/updates-to-key-value-store-api-a2c6f6cfc7c3?source=rss-f9983ec66fb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/a2c6f6cfc7c3</guid>
            <category><![CDATA[endpoints]]></category>
            <category><![CDATA[sdk]]></category>
            <category><![CDATA[apify]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Jakub Drobník]]></dc:creator>
            <pubDate>Tue, 09 Mar 2021 12:07:16 GMT</pubDate>
            <atom:updated>2021-03-09T12:13:15.282Z</atom:updated>
            <content:encoded><![CDATA[<p>Over the past few weeks, we have introduced some changes to how the <a href="https://docs.apify.com/api/v2">Apify API</a> <strong>gets</strong> records from <a href="https://docs.apify.com/storage/key-value-store">key-value stores</a> and <strong>uploads</strong> them. These changes improve the API’s stability and usability. While they will mostly happen under the hood, they are relevant to any developer using Apify.</p><figure><img alt="Apify API" src="https://cdn-images-1.medium.com/max/1024/1*g2jdepmtvIDMZqvTaibR8Q.png" /></figure><h4><strong>Main changes:</strong></h4><ul><li>Adjustments to how data are stored</li><li>Unlimited file size when uploading to key-value store</li><li>Key-value store requests served from Apify servers without redirects to S3</li><li>Deprecation of Direct Upload API endpoint</li></ul><h4>Adjustments to how data are stored</h4><p>Until now, every record uploaded using the <a href="https://docs.apify.com/api/v2#/reference/key-value-stores/record/put-record">Put record</a> endpoint was compressed using <a href="https://en.wikipedia.org/wiki/Gzip">gzip</a> before uploading. This has changed. <strong>Records are now stored in the state you upload them. This means it is up to you to decide whether the record is stored compressed or uncompressed.</strong></p><p>You can compress a record and use the <a href="https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding">Content-Encoding request header</a> to let our platform know which compression format it uses. We recommend compressing large key-value records to save storage space and network traffic.</p><p><strong>If you use </strong><a href="https://sdk.apify.com/docs/api/key-value-store#keyvaluestoresetvaluekey-value-options"><strong>Apify SDK</strong></a><strong> or our </strong><a href="https://docs.apify.com/apify-client-js#keyvaluestoreclient-setrecord"><strong>JavaScript API client</strong></a><strong>, your files are compressed automatically by default.</strong> We recommend using the JavaScript API client, which compresses your data before they are sent to our servers and decompresses them when you retrieve them. This keeps your storage costs as low as possible.</p><h4>Unlimited file size when uploading to key-value store</h4><p>To simplify the upload process, we have removed the 9MB-per-file upload limit for key-value stores. You can now upload a file of <strong>any size</strong> when using the <a href="https://docs.apify.com/api/v2#/reference/key-value-stores/record/put-record">Put record</a> API endpoint.</p><p>Previously, unlimited size uploads were only available when using the <a href="https://docs.apify.com/api/v2#/reference/key-value-stores/direct-upload-url">Direct upload</a> endpoint. This means you no longer need to call the <strong>Direct upload</strong> endpoint, get an upload URL, and then send your file to that URL, leaving you more time for important things.</p><h4>Key-value store requests are served from Apify servers without redirects to S3</h4><p>Requests to the <a href="https://docs.apify.com/api/v2#/reference/key-value-stores/record/get-record">Get record</a> endpoint are no longer redirected to AWS S3. It was confusing, and we found that most customers disabled this feature using the disableRedirect parameter.</p><p>From now, all records in key-value stores are served directly from Apify servers, without redirects. Thus, we removed the disableRedirect parameter as it no longer has any effect. While requests using disableRedirect will still work, we recommend <strong>not</strong> using the parameter.</p><h4>Deprecation of Direct Upload API endpoint</h4><p>This endpoint is no longer necessary following the change that allows files of unlimited size to be uploaded using the <a href="https://docs.apify.com/api/v2#/reference/key-value-stores/record/put-record">Put record</a> endpoint. The endpoint is still available and existing integrations will still work; however, we recommend migrating to the <a href="https://docs.apify.com/api/v2#/reference/key-value-stores/record/put-record">Put record</a> endpoint.</p><p>If you know someone who will find these updates useful, please share them. We are continuously working on updates and improvements, so keep an eye out for our newsletter every month and for new blog posts on how to use the Apify platform.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a2c6f6cfc7c3" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to debug an infinite loop in Node.js production code]]></title>
            <link>https://medium.com/@drobnik.j/how-to-debug-an-infinite-loop-in-node-js-production-code-9ec0e1442da0?source=rss-f9983ec66fb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ec0e1442da0</guid>
            <category><![CDATA[debugging]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[nodejs]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Jakub Drobník]]></dc:creator>
            <pubDate>Thu, 27 Aug 2020 16:00:49 GMT</pubDate>
            <atom:updated>2020-08-28T09:25:59.454Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Print the stack trace of an infinite loop in a Node.js application running in a Docker container using <a href="https://www.gnu.org/software/gdb/">GNU debugger</a> (GDB) and Kubernetes’ <strong>livenessProbe</strong>.</blockquote><p>Debugging infinite loops in <a href="https://nodejs.org/">Node.js</a> code locally is easy — just launch the debugger, break the execution, see where your code is stuck, fix and you’re done. However, in production systems, this becomes much more complicated.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sqYjgTpG7gcPVLd_3jgOAA.png" /><figcaption>Infinite loop in container CPU usage graph</figcaption></figure><p>Node.js doesn’t have any out-of-the-box tool to break a running program and print its stack trace. So, when your production code suddenly peaks at 100% CPU, it’s tricky to find out where exactly it got stuck. Neither -prof nor -cpu-prof (native Node.js profiling tools provided with the V8 debugger) helped since the infinite loop in application code was caused non-deterministically.</p><p>At <a href="https://apify.com/">Apify</a>, we had this type of problem in a production application running inside a stateless <a href="https://kubernetes.io/">Kubernetes</a> (K8s) container. The application is a simple <a href="https://expressjs.com/">express.js</a> based web server. This article describes the solution that worked for us. Hopefully, it can also help you.</p><blockquote><strong>TL;DR</strong> — We used a script based on this <a href="https://gist.github.com/mraleph/6453431#file-infinite-loop-trick-log-L4">GitHub</a> gist, which attaches the <a href="https://www.gnu.org/software/gdb/">GNU debugger</a> (GDB) to Node.js processes to print the leaking code’s stack trace. We had to run the script with K8s’ <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/"><strong>livenessProbe</strong></a> check to get the stack trace and save it to a <a href="https://kubernetesbyexample.com/pv/">persistent volume</a>.</blockquote><h4>Using the GDB debugger in the app container</h4><p>As a Node.js developer with a basic knowledge of <a href="https://v8.dev/">V8</a> and the underlying C++ code, you probably haven’t used GDB for debugging your Node.js applications. You probably have no use for it most of the time but in this specific case, GDB proved to be extremely useful.</p><p>GDB allows you to attach the debugger to a running Node.js process and set up a breakpoint in C++ where the infinite loop occurs. This place in V8 is called the <a href="https://thlorenz.com/v8-dox/build/v8-3.25.30/html/d4/d71/classv8_1_1internal_1_1_stack_guard.html"><strong>stack guard</strong></a> and we got the idea to use it from this <a href="https://gist.github.com/mraleph/6453431#file-infinite-loop-trick-log-L4">GitHub gist</a> (it includes an explanation of the whole script if you need to know more).</p><p>With some basic knowledge of GDB and V8’s stack guard, you can reproduce the steps that cause the infinite loop and print the stack trace of your app’s code where it occurs. The code below attaches a breakpoint to the stack guard and prints the stack trace.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5fbf88803c9a47e6fb80d5517ca0bd1c/href">https://medium.com/media/5fbf88803c9a47e6fb80d5517ca0bd1c/href</a></iframe><p>You can easily test it by running a simple Docker container with GDB installed. First, run an infinite loop, then run the GDB command.</p><p>Below are the steps for testing it in your local terminal using Docker.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/07128eb028d9a27e578ea07c35ea9756/href">https://medium.com/media/07128eb028d9a27e578ea07c35ea9756/href</a></iframe><p>After running these commands, your terminal should display <strong>myLoop</strong> function’s stack trace.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2P031EdG4BkcFmalQaczPw.png" /><figcaption>The printed stack trace in terminal</figcaption></figure><h4>Update K8s deployment to use the GDB script</h4><p>Now you know how to get the infinite loop’s stack trace, you can use it in the production container. First, add GDB to your Docker container. In this case, update the Dockerfile using the commands used in the test.</p><pre>apt-get update<br>apt-get install gdb</pre><p>Below is the Dockerfile for this scenario.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3fd868812a7300743c05c291e6f21652/href">https://medium.com/media/3fd868812a7300743c05c291e6f21652/href</a></iframe><p>Now you have GDB installed in your Docker container, you need to ensure that the GDB command will be attached in case of an infinite loop. As mentioned above, our loop was caused non-deterministically, so we used the liveness probe command to find it.</p><p>In our case, we had a basic HTTP liveness probe check set up. It checks the <strong>/health-check</strong> path every 5 seconds, allowing 3 failed attempts.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0253ca805ad8852cf55ad660e73e6e55/href">https://medium.com/media/0253ca805ad8852cf55ad660e73e6e55/href</a></iframe><p>If this probe fails a 4th time, the K8s scheduler pronounces the container as <strong>dead</strong> and replaces it in the pool. This place in the container’s runtime where the container is pronounced as dead is the place where the GDB command will need to run.</p><p>You want to preserve the loop-causing behavior; however, if the health check fails, the GDB script should run and save the infinite loop’s stack trace into a specific file. The <strong>bash</strong> script below does exactly that.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/04e4a2f97a1c666f218e268d3d1074bb/href">https://medium.com/media/04e4a2f97a1c666f218e268d3d1074bb/href</a></iframe><p>This saves the script as <strong>liveness_probe.sh</strong> into your app’s root directory. You can see that the bash script does exactly the same as the HTTP liveness probe. However, if the health check fails 4 times, it runs the GDB command and prints the stack trace.</p><p>To use this script in our app, we needed to edit the liveness probe in the K8s deployment specification as shown below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dc85b279eb9cbadd12688e9e2f81a404/href">https://medium.com/media/dc85b279eb9cbadd12688e9e2f81a404/href</a></iframe><p>This ensures our health check script runs every 40 seconds, which is enough time to run HTTP probe 4 times every 5 seconds. But be careful: since we’re using a debugger here, we need to allow processes using <a href="https://man7.org/linux/man-pages/man2/ptrace.2.html">process trace</a> with the SYS_PTRACE flag.</p><p>We can do this using <strong>securityContext </strong>in K8s deployment.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/016543dd099931ab5392f636765f0eb3/href">https://medium.com/media/016543dd099931ab5392f636765f0eb3/href</a></iframe><h4>Saving the stack trace file to a persistent volume</h4><p>Once you are able to track and print the loop into a specific file, you need to ensure that the file will not be deleted after the restart. The application runs as stateless, so after the container restarts, you lose all the data in memory and storage.</p><p>To attach a persistent volume to your K8s pod, you can follow <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-persistent-volume-storage/">these steps</a>. The attachable volume is a little different on each K8s-managed cluster. Our app uses the <a href="https://aws.amazon.com/eks/">AWS Elastic Kubernetes Service</a> (EKS), which is easily compatible with the <a href="https://aws.amazon.com/efs/">Elastic File System</a> (EFS).</p><p>You can do a very basic setup of EFS by running the command below.</p><pre><em>aws efs create-file-system</em></pre><p>From the output, you will need the <strong>FileSystemId</strong> property for further use. To attach EFS as a persistent volume to your EKS cluster, launch the <a href="https://docs.aws.amazon.com/eks/latest/userguide/efs-csi.html">Amazon EFS CSI Driver</a>. After installing it, let your application know about it by creating a <strong>StorageClass</strong> K8s resource.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bc0144f87160855a5c76c6c72d00591d/href">https://medium.com/media/bc0144f87160855a5c76c6c72d00591d/href</a></iframe><p>Next, create a <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/">persistent volume</a> and <a href="https://kubernetes.io/docs/concepts/storage/volumes/#persistentvolumeclaim">persistent volume claim</a>.<br>Note: Use <strong>FileSystemId</strong> as <strong>volumeHandle</strong>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/51e9bf1a9b8227282a6bcd7afe9f89bb/href">https://medium.com/media/51e9bf1a9b8227282a6bcd7afe9f89bb/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7a256fe3fb9740e46d8ee2b95613d956/href">https://medium.com/media/7a256fe3fb9740e46d8ee2b95613d956/href</a></iframe><p>Finally, mount the persistent volume claim to the deployment.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f3eb9accc91d814d9bf7025630a0beef/href">https://medium.com/media/f3eb9accc91d814d9bf7025630a0beef/href</a></iframe><p>When the persistent volume is set up, use SSH to connect it to one of the app’s containers. The files containing stack traces will be in the <strong>debugger</strong> folder.</p><h4>Conclusion</h4><p>To summarize, our app had a non-deterministic infinite loop, which occurred only on production. We identified it by attaching the GNU debugger to the app’s Node.js processes, which allowed us to print the leaking code’s stack trace. We then ran Kubernetes’ <strong>livenessProbe</strong> check to get the stack trace and save it to a persistent volume.</p><p>In our case, the infinite loop was caused by a third-party package.</p><p>We hope you will find this article useful if you encounter an infinite loop in your Node.js application.</p><p>Additionally, we added <a href="https://kubernetes-csi.github.io/docs/sidecar-containers.html">a sidecar container</a> into k8s cluster to sync stack trace files directly to AWS S3 bucket. If you are interested in how we did it, let us know in the comments, and we will describe it in a future blog post.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ec0e1442da0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The new version of Apify integration for Zapier is out!]]></title>
            <link>https://medium.com/@drobnik.j/the-new-version-of-apify-integration-for-zapier-is-out-bbd366430bef?source=rss-f9983ec66fb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbd366430bef</guid>
            <category><![CDATA[zapier]]></category>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[integration]]></category>
            <category><![CDATA[apify]]></category>
            <dc:creator><![CDATA[Jakub Drobník]]></dc:creator>
            <pubDate>Wed, 26 Jun 2019 16:08:58 GMT</pubDate>
            <atom:updated>2019-06-26T16:08:58.611Z</atom:updated>
            <content:encoded><![CDATA[<p>We are proud to announce that the new version of <a href="https://zapier.com/apps/apify/integrations">Apify integration for Zapier</a> has been released. <a href="https://zapier.com/">Zapier</a> is a tool that lets you connect 1,500+ apps and automatically move data between them so you can focus on your most important work. The new integration for Zapier now lets you connect <a href="https://apify.com/actors">Apify Actors</a> — cloud programs that can perform arbitrary web scraping, data processing or web automation tasks. You can connect dozens of existing actors from <a href="https://apify.com/store">Apify Store</a>, develop your own or buy new ones from our approved freelance developers on <a href="https://apify.com/marketplace">Apify Marketplace</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NrdvFuvV10soXdgydQ38Eg.png" /><figcaption><a href="https://zapier.com/apps/apify/integrations">Apify integration for Zapier</a></figcaption></figure><p>For example, with the new integration, you can automatically extract prices from the websites of your competitors, get a notification using your communication tool of choice when they’re selling products cheaper than you, or automatically reduce your own prices by updating your external database or by calling an API.</p><blockquote><strong>In other words, Apify lets you extend the Zapier ecosystem of 1,500+ apps with an unlimited number of new custom-built actions and triggers.</strong></blockquote><p>If you were already working with the previous version of Apify integration for Zapier, you can migrate to the new version using <a href="https://kb.apify.com/articles/2981117-migrate-your-zapier-integration-from-legacy-crawlers-to-actor-tasks">the migration guide</a>. You can jump to setting up integration right away with the <a href="https://kb.apify.com/articles/3034235-getting-started-with-apify-integration-for-zapier">Getting started with Zapier &lt;&gt; Apify tutorial</a>.</p><h3>How it works</h3><p>Zapier lets you connect multiple apps in a workflow called a <strong><em>Zap</em></strong> and then automatically move data between the apps. In Zaps, the apps can be used in three different ways: as <strong>Triggers, Actions</strong> and <strong>Searches</strong>.</p><h4>Triggers</h4><p>When an app is used to initiate a Zap, it’s called a trigger. For example, the trigger can be a <a href="https://zapier.com/apps/email/integrations">received inbound email</a> or a <a href="https://zapier.com/apps/google-sheets/integrations">new row added to a Google Sheets</a> document. The Apify app provides the following triggers:</p><ul><li><strong>Finished Actor Run</strong> — Triggers a workflow whenever an Apify actor finishes.</li><li><strong>Finished Task Run</strong> — Triggers a workflow whenever an Apify task finishes. Note that tasks are saved configurations of actors.</li></ul><p>These triggers are useful for attaching custom actions that should occur when Apify actors or tasks finish, regardless if they were invoked using a scheduler or API. They can be used to process the results, send notifications, etc.</p><h4>Actions</h4><p>When the app performs an activity in the Zap workflow, it’s called an action. For example, an action can <a href="https://zapier.com/apps/slack/integrations">send a notification to Slack</a> or <a href="https://zapier.com/apps/dropbox/integrations">upload a document to Dropbox</a>. The Apify app for Zapier provides the following actions:</p><ul><li><strong>Run Actor </strong>— Invokes an Apify actor with a specified input configuration and options.</li><li><strong>Run Task </strong>— Invokes an Apify task, optionally overriding input configuration and options.</li><li><strong>Set Key-Value Store Record</strong> — Writes a record into the Apify’s <a href="https://apify.com/docs/storage#key-value-store">Key-value store</a> data storage.</li></ul><p>A great thing about running an actor from Zapier is that all input configuration fields are pre-filled with the actor default values, which lets you start quickly. The new version of the Apify app for Zapier also enables running actors or tasks synchronously. This is great if you just want to run a simple action (e.g. extract data from a single web page) and then use the data in the next step of your Zap.</p><h4>Searches</h4><p>If the app provides some data that can be used in the Zap workflow, it’s called a search. The Apify app provides the following searches:</p><ul><li><strong>Find Last Actor Run </strong>—<strong> </strong>Get results from the last run of a specific Apify actor.</li><li><strong>Find Last Task Run</strong> — Get results from the last run of a specific Apify task.</li><li><strong>Get Key-Value Store Record</strong> — Reads a record from the Apify’s K<a href="https://apify.com/docs/storage#key-value-store">ey-value store</a> data storage.</li><li><strong>Fetch Dataset Items</strong> — Reads one or more items from the Apify’s <a href="https://apify.com/docs/storage#dataset">Dataset</a> storage.</li></ul><p>You will find searches valuable if you need to store dynamic values across multiple Zaps or between a Zap and an actor run.</p><h4>Usage</h4><p>With all the Apify’s triggers, actions and searches for Zapier, you can easily set up workflows that leverage data from the web or automate workflows on the web. For example, you could schedule an actor to run every day at 10 am to scrape the lunch menu of your favorite restaurant and then send it to your company email or Slack. Or you can monitor your competitor’s prices by setting up a workflow which scrapes prices from an online store and saves them to <a href="https://zapier.com/apps/google-sheets/integrations">Google Sheets</a> or <a href="https://zapier.com/apps/airtable/integrations">Airtable</a>. The options are literally unlimited.</p><p>To get started, have a look at this <a href="https://kb.apify.com/articles/3034235-getting-started-with-apify-integration-for-zapier">step-by-step guide</a> to how you can create your first Zapier workflow with Apify. Here are several popular Zap templates that you can use:</p><figure><a href="https://zapier.com/apps/apify/integrations/slack/53690/get-slack-notifications-for-finished-apify-tasks"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ij6Wn-6pMtse9GBlgucIyQ.png" /></a></figure><figure><a href="https://zapier.com/apps/apify/integrations/google-sheets/53687/save-datasets-from-finished-apify-tasks-to-google-sheets-document"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WH4UQ3tvurA3bwzpHBBOmQ.png" /></a></figure><figure><a href="https://zapier.com/apps/apify/integrations/google-drive/53692/upload-dataset-from-finished-apify-actor-run-to-google-drive"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6liFFoZ02VtGIbW-HyykNg.png" /></a></figure><figure><a href="https://zapier.com/apps/apify/integrations/dropbox/53695/add-datasets-from-finished-apify-tasks-to-dropbox-folder"><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8hL9qcY7NPTWu3TskeZt8A.png" /></a></figure><p>And that’s it! If you have any questions, comments or any inspiring use cases that you’d like to share with us (such as <a href="http://adrienrahier.com/blog/how-i-used-push-by-zapier-to-publish-wordpress-articles-in-one-click/">How to publish Wordpress article in one click</a> from Adrien Rahier), we’d love to hear from you! Just fire off an email to <a href="/service/mailto:hello@apify.com">hello@apify.com</a></p><p>Happy integration of Zapier with Apify!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbd366430bef" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Get data from Google Places not provided by official API]]></title>
            <link>https://medium.com/@drobnik.j/get-data-from-google-places-not-provided-by-official-api-2d930918deb6?source=rss-f9983ec66fb4------2</link>
            <guid isPermaLink="false">https://medium.com/p/2d930918deb6</guid>
            <category><![CDATA[google-maps]]></category>
            <category><![CDATA[api]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[google-places]]></category>
            <category><![CDATA[crawler]]></category>
            <dc:creator><![CDATA[Jakub Drobník]]></dc:creator>
            <pubDate>Thu, 17 Jan 2019 09:32:59 GMT</pubDate>
            <atom:updated>2019-01-17T09:32:59.032Z</atom:updated>
            <content:encoded><![CDATA[<p>Every now and then someone wants to get data from Google Places using an Apify crawler. When starting to cooperate on such a project in the past, we always suggested that the user could instead try the <a href="https://developers.google.com/places/web-service/search">Google Places API</a>. But most users replied that they need information that the API does not provide. So we decided to come up with a generic solution.</p><p>The result is an actor that crawls Google Maps. The actor fills out a search query in <a href="https://www.google.com/maps">Google Maps search</a> and goes through each item in the result. The output item for a single place contains:</p><ul><li>Basic information</li><li>Popular times histogram</li><li>Photos</li><li>Reviews</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wvjfQZegZUknOkhPiTEtYQ.png" /></figure><p>The benefits of using an actor are that you can get all user reviews, all place photos and the popular times histogram. If you’re using the <a href="https://developers.google.com/places/web-service/intro">Google Places API</a>, you can only get 5 user reviews, 10 place photos and the popular times histogram is not included in the data.</p><h3>How to use the actor</h3><p>The best thing about Apify public actors is that you can use them for free! Start off on the <a href="https://www.apify.com/drobnikj/crawler-google-places">actor library page</a>, where you can click on the <em>Try actor </em>button (if you’re not an Apify user, <a href="https://my.apify.com/sign-up">create an account</a> first). You will be redirected to the actor run console page, where you can specify which places you want to get data about.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DyRI3CsR9XAPZZk_Uq23-w.png" /></figure><p>The actor provides the following configuration options. <strong>Search </strong>is used to specify the search string with which you want to search places, e.g. <em>pubs</em> in our example. In the <strong>Proxy configuration, </strong>you can set up any proxy IPs you want to use for the crawler. As this option is for advanced Apify users (read more about Apify Proxy <a href="https://www.apify.com/docs/proxy">here</a>), let’s just choose <em>Apify proxy (automatic)</em> for our example.</p><p>If you want to search places in a specific location, you can use <strong>Viewport point latitude, Viewport point longitude, Viewport zoom level. </strong>For example, we want to search for pubs in Berlin. We have to find the latitude and longitude for Berlin (you can use this handy <a href="https://www.latlong.net/convert-address-to-lat-long.html">lat and long converter</a> for that). The converter gives us 52.520008 for latitude and 13.404954 for longitude, so let’s enter these values into the configuration. <strong>Viewport zoom level</strong> is used to set up the zoom level of the viewport. This is a number from 1 to 20, where 20 means that map viewport is maximum zoomed in. In our example, we’ll use 14, which give us a viewport for the whole Berlin city center when combined with the selected latitude and longitude. You can check <a href="https://www.google.com/maps/search/pubs/@52.509532,13.4139396,14z">selected viewport on Google Maps</a>.</p><p>The last configuration is <strong>Max crawled places,</strong> which is used to limit the results count. There are hundreds of pubs in Berlin, so we’ll limit our results to get up to 50 pubs.</p><p>Then run the actor using the <em>Run</em> button under the configuration. After a few minutes, you should see the <a href="https://api.apify.com/v2/datasets/WHyHta3rRNMQMiCLs/items?format=json">results in a Apify dataset</a>, which you can find in the tabs menu on the actor run page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aSPCjgbBjymzi98kgdGFxg.png" /></figure><p>The best format for the actor is JSON because there are a few nested arrays in each result and these can make table formats (HTML, CSV, Excel) unreadable.</p><p>And that’s it! We’d love to hear feedback from you. If you’ve found any additional data you want to get from Google Places or any issues, you can submit an issue on the <a href="https://github.com/drobnikj/crawler-google-places/issues">actor GitHub repo</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2d930918deb6" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>