<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by ilker cam on Medium]]></title>
        <description><![CDATA[Stories by ilker cam on Medium]]></description>
        <link>https://medium.com/@cam.ilker?source=rss-131cf0ebbd42------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*hvv2kbX_rZ7rW-eKPJc6Cg.jpeg</url>
            <title>Stories by ilker cam on Medium</title>
            <link>https://medium.com/@cam.ilker?source=rss-131cf0ebbd42------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 05:53:43 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@cam.ilker/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[iOS AppDelegate lifecycle — done right]]></title>
            <link>https://medium.com/@cam.ilker/ios-appdelegate-lifecycle-done-right-fb5f50acf350?source=rss-131cf0ebbd42------2</link>
            <guid isPermaLink="false">https://medium.com/p/fb5f50acf350</guid>
            <category><![CDATA[amplitude]]></category>
            <category><![CDATA[swift]]></category>
            <category><![CDATA[xcode]]></category>
            <category><![CDATA[ios]]></category>
            <dc:creator><![CDATA[ilker cam]]></dc:creator>
            <pubDate>Fri, 17 Sep 2021 15:10:07 GMT</pubDate>
            <atom:updated>2021-09-17T15:18:35.683Z</atom:updated>
            <content:encoded><![CDATA[<h3>iOS Silent Notification Problem — How they mess up your app&#39;s lifecycle and Amplitude session events</h3><p>After so many years in iOS development, I realized that I have been bootstrapping new projects wrong from the beginning. As many of the samples you can find online (including 3&#39;rd party SDK integration samples), I too was following this convention to launch my apps;</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ffec2715badd7798fbcf503a3a0edd40/href">https://medium.com/media/ffec2715badd7798fbcf503a3a0edd40/href</a></iframe><p>What can go wrong with this? you can find similar samples online where you ditch storyboards and go hero with layout constraints, right? For me, a lot of things did go wrong. Especially if you’re putting content-available in your notifications. AKA: Silent notifications for background fetching.</p><p>Apple’s developer documentation defines silent notifications as;</p><blockquote>To deliver a background notification, the system wakes your app in the background. On iOS it then calls your app delegate’s <a href="https://developer.apple.com/documentation/uikit/uiapplicationdelegate/1623013-application">application(_:didReceiveRemoteNotification:fetchCompletionHandler:)</a> method. On watchOS, it calls your extension delegate’s <a href="https://developer.apple.com/documentation/watchkit/wkextensiondelegate/3152235-didreceiveremotenotification">didReceiveRemoteNotification(_:fetchCompletionHandler:)</a> method. Your app has 30 seconds to perform any tasks and call the provided completion handler</blockquote><p>An elegant explanation, but one might easily overlook the detail the system wakes your app in the background. This means that if your app is in an unattached (killed) state your app will be woken up upon receiving a silent notification, and it will call the following delegate methods of AppDelegate;</p><ol><li>willFinishLaunchingWithOptions</li><li>didFinishLaunchingWithOptions</li></ol><p>Let’s try this on debugger via setting “Launch” type from “Automatically” to “Wait for the executable to be launched” and set a breakpoint to first line of `didFinishLaunchingWithOptions``</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2a2O3rbbuxy-uFB4-oezPg.jpeg" /><figcaption>Debugging didFinishLaunchingWithOptions on a silent notification launch</figcaption></figure><p>That’s the point where everything goes wrong in my case. I initialize my coordinator and a ViewController responsible for fetching API resources and also <strong>sending analytic events and attributes. </strong>I highly depend on silent notifications to push minor updates, especially for fetching assets. But that creates a problem in calculating daily active user charts, because at that point analytics tools already logged their first events, especially session events.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/946/1*aXXzkL4IgnC4-zRJ66ijOw.png" /><figcaption>Peaks in this DAU graph exactly matches the silent notifications timestamps</figcaption></figure><p>Accepting the reality that these are not really active users, but faulty app initialization took some courage 😢</p><h3>Solution</h3><p>Simmer down for a second if you have the same problem, embrace the fault, and take a look for Apple’s developer documentation on “<a href="https://developer.apple.com/documentation/uikit/app_and_environment/responding_to_the_launch_of_your_app">Responding to the Launch of Your App</a>”</p><p>On “didFinishLaunchingWithOptions”, articles state that you should initialize your data structures, verify required resources, prepare notifications, check for launch options for routing, and perform any one-time setups such as initializing 3&#39;rd party SDKs. Also warns you not to run long tasks here. But doesn’t mention anything about the UI.</p><p>Let us roll back to the <a href="https://developer.apple.com/documentation/uikit/app_and_environment/managing_your_app_s_life_cycle">main article</a> and check out the overall lifecycle. There is a great <a href="https://developer.apple.com/documentation/uikit/app_and_environment/scenes/preparing_your_ui_to_run_in_the_foreground">link</a> to “Preparing Your UI to Run in the Foreground”. It explains when you should prepare your UI for both SceneDelegate and AppDelegate approaches. So Apple suggests we prepare the UI in “applicationDidBecomeActive” method. This method is called when your application transitions from killed or background state to foreground state. So I ended up changing how I launch my app to this;</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8ae4cd66505e2d645a0a1e66810dda12/href">https://medium.com/media/8ae4cd66505e2d645a0a1e66810dda12/href</a></iframe><p>Silent notifications no more trigger my coordinator like a regular launch, and doing this also consumes fewer system resources where you really need a background fetch. Hope you find it helpful, any comments or questions are welcome</p><p><a href="https://twitter.com/camilker1">JavaScript is not available.</a></p><h3>Sources</h3><ul><li><a href="https://developer.apple.com/documentation/uikit/app_and_environment/managing_your_app_s_life_cycle">S1</a></li><li><a href="https://developer.apple.com/documentation/uikit/uiapplicationdelegate">S2</a></li><li><a href="https://developer.apple.com/documentation/uikit/app_and_environment/responding_to_the_launch_of_your_app">S3</a></li><li><a href="https://www.donnywals.com/updating-your-apps-with-silent-push-notifications/">And this great article about silent notifications</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fb5f50acf350" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Probabilistic Retention Cohort SQL on Snowflake]]></title>
            <link>https://medium.com/rakam-io/probabilistic-retention-cohort-sql-on-snowflake-22ab8db1d773?source=rss-131cf0ebbd42------2</link>
            <guid isPermaLink="false">https://medium.com/p/22ab8db1d773</guid>
            <category><![CDATA[snowflake]]></category>
            <category><![CDATA[rakamio]]></category>
            <category><![CDATA[retention]]></category>
            <dc:creator><![CDATA[ilker cam]]></dc:creator>
            <pubDate>Mon, 24 Feb 2020 15:55:41 GMT</pubDate>
            <atom:updated>2020-02-24T15:55:41.048Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uNFwmYTASlXlU4DdW5nAqg.png" /></figure><blockquote><strong>TL;DR</strong>: Retention cohorts are expensive to generate, especially if you’re working on a database lacking analytic functions. A deterministic approach is; joining ‘first action’ to ‘returning action’ steps and counting the distinct connectors (user or device identifiers). Joining huge amounts of data should be avoided when working on big data. A better solution is to estimate the intersection of steps using the HyperLogLog algorithm, lets practice this on Snowflake’s HLL functions.</blockquote><h3><strong>About Snowflake</strong></h3><p>Snowflake is a cloud warehouse where you forget about maintenance, and focus on your query. They have a variety of analytic functions, a great console, <a href="https://docs.snowflake.net/manuals/user-guide/conns-drivers.html">programmable interfaces</a> (npm package, JDBC driver). Loading data into Snowflake is easy for one time (COPY command) or continuously (Snowpipe). Truncated some table accidentally? They do <a href="https://docs.snowflake.net/manuals/user-guide/data-time-travel.html">time-travel</a> ⌛️</p><h3><strong>1. The MinHash Approach</strong></h3><p>The similarity of the two sets can be calculated using the <strong>Jaccard Index. </strong>It is formulated as;</p><blockquote>J(A,B) = |A ∩ B| / |A∪ B|</blockquote><p>Although calculating the union and intersection can be an expensive operation over millions of rows. Minhash is an efficient solution that inputs your data into <strong>K </strong>distinct hash functions. For example, you choose <strong>K=1000 </strong>for your user identifier column with one million rows. Minhash will describe that set with 1000 dim.</p><p>A similarity index might sound a solution for plotting a retention cohort. Let’s see why we can’t use this solution, the similarity indexes on given cases are as follows;</p><ul><li>If A and B are the same = 1.</li><li>If A and B are completely different = 0</li><li>If A is a superset of B and <strong>2.|A| = |B| </strong>= 0.5</li></ul><p>To Exemplify the last case; in typical retention, we have the first step as first_install and session_start as the returning step. Which would be a vanilla retention query that you can find most of the platforms dealing with user event data. Session start event is the superset of first install events. Every user must install the app to start the session. Even though all the users are still there on day 0 for the session start, the Jaccard Index is not a solution that considers the overlaps. For more information and an example about MinHash on Snowflake checkout <a href="https://docs.snowflake.net/manuals/user-guide/querying-approximate-similarity.html">here</a>.</p><h3><strong>2. Cardinality Estimation</strong></h3><p>The size of a set of elements is defined as cardinality in mathematics. In SQL we simply use the count(distinct row) function to calculate the number of distinctive rows. Imagine we’re filtering the set over some conditions, or even joining rows from another table. We might do the filtering on a column that is not clustered/partitioned, and your queries will start getting greedy on resources. The simplest <a href="https://docs.snowflake.net/manuals/sql-reference/functions/approx_count_distinct.html">solution</a> in Snowflake is to replace count distinct to approx_count_distinct , which uses HyperLogLog to estimate distinct cardinality. Let’s consider the following query;</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9b5e9fde9f95da0505be84eae5bf0844/href">https://medium.com/media/9b5e9fde9f95da0505be84eae5bf0844/href</a></iframe><p>Different sets grouped by each day (maybe a case of materialization). For a reason, you need to aggregate the total unique users via summing. Sorry, the sum you seek is randomized by a factor of how frequently duplicates appear in your inner queries group by statement. There must be a way to group by your set and aggregate the correct cardinality, right? Yes!, by breaking down approx_count_distinct and using a combination of HyperLogLog functions that Snowflake provides for us.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5f7d3962a6cf776cda796d452e2da6db/href">https://medium.com/media/5f7d3962a6cf776cda796d452e2da6db/href</a></iframe><p>The example above is very useful if you’re materializing cardinalities and aggregating later on. Also, check out <a href="https://docs.snowflake.net/manuals/sql-reference/functions/hll_export.html">HLL_EXPORT</a> and <a href="https://docs.snowflake.net/manuals/sql-reference/functions/hll_import.html">HLL_IMPORT</a> helpers to dump and import the binary representation into object format.</p><h3>3. Composing Retention Query using HLL Functions</h3><p>What is the number of users came back the next day? Assuming A is the first action and B the returning; Mathematically, the number of retained users can be described as the cardinality of intersection of A and B (|<strong>A⋂B|</strong>). Using HLL_COMBINE we can calculate the cardinality of unions, and substitute the intersections with;</p><pre>|<strong>A⋂B| = |A| + |B| - |A⋃B|</strong></pre><p>Let’s start with defining ‘first action’, ‘returning action’ and joining returning action day-by-day.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a37bfa87f4c96944bd2488f0bc0d37ad/href">https://medium.com/media/a37bfa87f4c96944bd2488f0bc0d37ad/href</a></iframe><p>The next_periods set joins the returning action by the date and calculates;</p><ul><li>The first appear date.</li><li>The next period (in days, or weeks depending on how you truncated).</li><li>Approximate count of first and returning action (via hll_estimate(fa/ra.hll_user)). To calculate intersection cardinality we need to subtract set cardinalities from the union cardinality.</li><li>Aggregates first and returning actions HyperLogLog estimation, we’ll need those later on when using hll_merge to calculate union of A and B.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gd_-FBXJevY1kBnUDzVokQ.png" /><figcaption>select * from next_periods to visualize the structure of the set.</figcaption></figure><p>To calculate the final form of the cohort, we will calculate the number of users for the same day (simply the first action) and union with merged and estimated HLL values of next_periods set.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e59ad0a4bd0e7352ea1eb60f93ff4197/href">https://medium.com/media/e59ad0a4bd0e7352ea1eb60f93ff4197/href</a></iframe><p>Since we’re sure that date and period pairs are unique. Taking a maximum of ra_user_count and fa_user_count is okay. hll_combine is an aggregation function thus we either take the maximum of those counts or do this inside a subquery. Since we’ve only one day in the first action, the query result will show retention over the same and next day only.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XAuYCPqOilkEStCtgcu_6w.png" /><figcaption>Query result for the retention cohort</figcaption></figure><h3><strong>About Rakam</strong></h3><p>At <a href="https://rakam.io/database/snowflake">rakam</a> we’ve spent hundreds of hours for the Snowflake integration. Providing efficient segmentation, funnel, retention and SQL interfaces.</p><p>You can start experimenting on our <a href="https://rakam.io/demo">demo</a> today, also connect your Snowflake cluster and try on your own data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=22ab8db1d773" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rakam-io/probabilistic-retention-cohort-sql-on-snowflake-22ab8db1d773">Probabilistic Retention Cohort SQL on Snowflake</a> was originally published in <a href="https://medium.com/rakam-io">rakam</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Firebase Retention Cohort Query on BigQuery]]></title>
            <link>https://medium.com/rakam-io/firebase-retention-cohort-query-on-bigquery-2e7ad94ad5c4?source=rss-131cf0ebbd42------2</link>
            <guid isPermaLink="false">https://medium.com/p/2e7ad94ad5c4</guid>
            <category><![CDATA[cohorot]]></category>
            <category><![CDATA[firebase]]></category>
            <category><![CDATA[retention]]></category>
            <category><![CDATA[hyperloglog]]></category>
            <category><![CDATA[bigquery]]></category>
            <dc:creator><![CDATA[ilker cam]]></dc:creator>
            <pubDate>Tue, 18 Feb 2020 09:45:46 GMT</pubDate>
            <atom:updated>2020-02-18T10:09:12.100Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0ZtcnD_0kOFj3lZL0B6-Xw.png" /><figcaption>A Weekly Retention Report, Exported on Firebase UI Console.</figcaption></figure><p>The cool features aside, Firebase is also a powerful analytics platform with built-in funnel and retention reports. You can filter your user segment on various dimensions such as platform, stream, demographics or custom user properties. Firebase can also attribute your installation marketing channels down to the ad-set and creatives. Although this powerful feature is <strong>free of charge </strong>there are some caveats such as Apple Search Ads and Facebook Ads that <strong>are not supported. </strong>If you’re focused more on marketing conversions, you can even filter your user segment on marketing data, if you’ve integrated your marketing channel.</p><blockquote><strong>TL;DR </strong>Composing an efficient retention query on BigQuery using probabilistic cardinality estimation function: HyperLogLog++, if you’re here for this jump <a href="#2c2e">here</a>. No idea about Firebase’s canonical data structure on BigQuery, <a href="https://medium.com/rakam-io/free-product-analytics-with-firebase-bigquery-rakam-f7dab7c392b4">here</a></blockquote><p>You can get answers to many questions with the builtin report that the UI provides you. But if you have the following needs, you need a bootstrap.</p><ul><li>Realtime data.</li><li>Ability to filter on all event attributes.</li><li>Filter the segment using external data (I.e: Marketing attributions from Adjust, S2S events from BigQuery).</li><li>Select first and returning action.</li></ul><p>Firebase only lets you create a retention report from the New User segment, also you can not select a returning action. For example, I’d like to see the retention cohort of <strong>All Users </strong>performed any event, come back and did <strong>Checkout </strong>event. For such a detail report, you’ll have to work on the SQL level. Luckily, Firebase can export your event data to your BigQuery project <strong>in real-time</strong>. Just navigate to <strong>Settings -&gt; Integrations -&gt; BigQuery</strong> and enable the BigQuery export feature.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9AO5k9W-DsLFHr1SnljpJQ.png" /><figcaption>BigQuery integration settings on Firebase UI</figcaption></figure><p>The BigQuery schema of Firebase is in Canonical form, a single table will be holding all your events as rows, user and event properties as an array of records in columns. For more details check out our previous <a href="https://medium.com/rakam-io/free-product-analytics-with-firebase-bigquery-rakam-f7dab7c392b4">post</a>. You have linked your Firebase data and got a basic understanding of how your data is stored in BigQuery. Now it is time to go in deeper.</p><h3>Composing Retention Report in SQL</h3><p>Let’s start with the simplest, the below query declares first and section. You may filter it on top-level properties (country, device, etc), or on user &amp; event parameters (you have it unnest it first.). Be sure to filter your segment with _TABLE_SUFFIX. Otherwise, it will be an expensive query.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9e4c2823bfe8f6fd3a6d89c2caf00062/href">https://medium.com/media/9e4c2823bfe8f6fd3a6d89c2caf00062/href</a></iframe><p>Once you filtered your first and returning action segments, it is time to generate the cohort. What need here is simply;</p><ul><li>First appear date.</li><li>Next period (in days, weeks or months depending on how you truncated the event_timestamp column).</li><li>The number of users, for (first appear date + next period).</li></ul><p>First appear date is trivial, just group by the whole first action on your date column and count your connector. To find the number of users in the next period, we have to join the returning action on the connector column.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/75f1ff0aeb45d91772442b6016a1bc09/href">https://medium.com/media/75f1ff0aeb45d91772442b6016a1bc09/href</a></iframe><p>If you have relatively small data or don’t care about the speed/cost optimization, the approach above should be the trick to get answers to all your retention queries. However, this is big data and why not use cool cardinality estimation features 😎</p><h3>Optimized Retention Query (using HyperLogLog)</h3><p>Calculating the exact cardinality of big sets can demand high resources, estimating the cardinalities using probabilistic estimators can show significant performance difference compared to the deterministic method. Bigquery has tons of aggregation functions, as for analytics; HyperLogLog++ functions can estimate the cardinality of a set using a probabilistic approach. Check more details for each HLL function <a href="https://cloud.google.com/bigquery/docs/reference/standard-sql/hll_functions">here</a>.</p><p>The above query’s estimation function can be altered to ‎<a href="https://cloud.google.com/bigquery/docs/reference/standard-sql/approximate_aggregate_functions#approx_count_distinct">APPROX_COUNT_DISTINCT</a> to use HyperLogLog estimation. But the join statement is still there, which uses a connector (a high cardinality column) as the join condition.</p><blockquote>Let’s ask the question: How do we calculate the number of users in the next period?</blockquote><p>The number of users who came back for the next period is basically, the cardinality of intersections of first and returning action for that period, |<strong>A⋂B|. </strong>Using HLL_COUNT.MERGE_PARTIAL, we can combine two sets of cardinality, |<strong>A⋃B|. </strong>And we also know that;</p><pre>|<strong>A⋂B| = |A| + |B| - |A⋃B|</strong></pre><p>Let’s start with filtering first and returning action, but this time instead of counting the connector, we use HLL_COUNT.INIT function. This will export a binary representation of the cardinality.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f72868dc4db3efef839d814e787c68ed/href">https://medium.com/media/f72868dc4db3efef839d814e787c68ed/href</a></iframe><p>If you take a peek to `AGG`, it will show you the binary representation of cardinalities for each day and next period, alongside with estimated cardinalities of first and returning action for that day and next period.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GTLs-w3n1OmZf8FTyyhuWg.png" /><figcaption>Select * From Agg -&gt; Each date + next period has a binary cardinality representation.</figcaption></figure><p>The next stage is to calculate the cardinality of intersections: |<strong>A⋂B| = |A| + |B|-A⋃B.</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ab07a50665a6ee6b4049085fa6e189bb/href">https://medium.com/media/ab07a50665a6ee6b4049085fa6e189bb/href</a></iframe><p>HLL_COUNT.MERGE_PARTIAL will combine (union) the cardinalities we grouped the query for, and the upper query will calculate the actual intersection via:</p><pre>(FA_USER_COUNT + RA_USER_COUNT) — HLL_COUNT.EXTRACT(SIM)</pre><p>The query can be easily materialized on the date column if you’d like to speed things more up. If you’re looking for an easier solution or more advanced query generation such as filtering your segment via joining another data source, we already handle all the steps and may more in <a href="https://rakam.io/demo?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=medium-firebase-retention">rakam</a>.</p><h3>About Rakam</h3><p>Model your data once and enable your non-technical users to get insights themselves. Can’t model data? Install one of our recipes to bootstrap your project, we support Firebase, Rakam-API, Snowplow, Segment, Tenjin, GA360 and much more weekly updated on this <a href="https://github.com/rakam-io/recipes">repository</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*IbV8LNi64ZceT_RO0vZwXw.png" /><figcaption>Visual retention cohort composer on <a href="https://rakam.io/?utm_source=medium&amp;utm_medium=blog&amp;utm_campaign=medium-firebase-retention">rakam.io</a></figcaption></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2e7ad94ad5c4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rakam-io/firebase-retention-cohort-query-on-bigquery-2e7ad94ad5c4">Firebase Retention Cohort Query on BigQuery</a> was originally published in <a href="https://medium.com/rakam-io">rakam</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Free Product Analytics with Firebase + BigQuery + Rakam]]></title>
            <link>https://medium.com/rakam-io/free-product-analytics-with-firebase-bigquery-rakam-f7dab7c392b4?source=rss-131cf0ebbd42------2</link>
            <guid isPermaLink="false">https://medium.com/p/f7dab7c392b4</guid>
            <category><![CDATA[firebase]]></category>
            <category><![CDATA[amplitude]]></category>
            <category><![CDATA[mixpanel]]></category>
            <category><![CDATA[bigquery]]></category>
            <category><![CDATA[product-analytics]]></category>
            <dc:creator><![CDATA[ilker cam]]></dc:creator>
            <pubDate>Tue, 04 Feb 2020 16:17:07 GMT</pubDate>
            <atom:updated>2020-02-04T16:17:07.362Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qeiqDhZx38JleSUwZ_2YHQ.png" /></figure><p>Many of us landed on Firebase due to it’s powerful and free services such as Crashlytics, Remote Config, Cloud Messaging, Analytics and many more. When it comes to understanding how your users behave, analytics plays a crucial role. Building a custom data pipeline may cost you in the early stages of your idea. So why not use Firebase Analytics? It is free and has a bunch of built-in reports and a dashboard showing the most critical metrics, like revenue, version adoption, active users and so on.</p><p>Hopefully, you crushed those early-stage numbers and reached enormous KPIs that you could not even dream of. You need to squeeze that data for more insights, understand more of your data. Aggregated metrics with few hours of cache are not sufficient for you anymore. You need answers to your Adhoc questions; complex funnel, retention and segmentation reports, and you need them in realtime. Now it is time to set up your brand new data pipeline and pay hundreds, if not thousands of dollars. Right?… Not necessarily?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/525/0*VnZ37hmOpI237dw8.jpg" /></figure><h3>Link Firebase to BigQuery</h3><p>To access unsampled data first, we have link our project to BigQuery. Doing that will allow you to query not Analytics but also A/B Testing, Crashlytics, Predictions and Cloud Messaging. To link your project simply follow the steps below;</p><ul><li>Navigate to the settings page in your project.</li><li>Click onto Integrations tab.</li><li>Click to link on the BigQuery card.</li></ul><p>For more visit <a href="https://support.google.com/firebase/answer/6318765?hl=en">here</a></p><h3>Firebase BigQuery Schema</h3><p>Firebase schema on BigQuery has its pros and cons. It is hard to compose a query but efficient if you filter on partition suffixes. The canonical form is great for ease of modeling. However, repeated records on user attributes and event parameters needs unnest operation. Get more info about the schema <a href="https://support.google.com/firebase/answer/7029846?hl=en">here</a>. Firebase also collects some well-known events by default, without any additional steps. They may differ from tracker to tracker, for more checkout also <a href="https://support.google.com/firebase/answer/6317485?hl=en">here</a>.</p><h4>1. Partitioned Tables</h4><p>There are two raw events table, both are stored in partitioned table format;</p><ul><li><strong>Historical Tables:</strong> Prefixed with events_ followed by a date, such as events_20200109</li><li><strong>Daily Table:</strong> Prefixed with events_intraday_ followed by today, such as events_intraday_20200110</li></ul><p>Good news is that you can use * to query all the events tables. But keep in mind that filtering the partition suffix will help you reduce your costs (a lot!!)</p><pre><strong>SELECT</strong> * <strong>FROM</strong> events_* <strong>WHERE</strong> _TABLE_SUFFIX <strong>BETWEEN</strong> &#39;20200107&#39; <strong>AND</strong> &#39;20200109&#39;</pre><p>add one more filter on event_type AND even_type = &#39;in_app_purchase&#39; and here are the revenue events for you!</p><p>Need real-time data, with historicals? Use UNION ALL</p><pre><strong>SELECT</strong> * <strong>FROM</strong> events_* <strong>WHERE</strong> _TABLE_SUFFIX <strong>BETWEEN</strong> &#39;20200107&#39; <strong>AND</strong> &#39;20200109&#39;<br><strong>UNION</strong> ALL<br><strong>SELECT</strong> * <strong>FROM</strong> events_intraday_* <strong>WHERE</strong> _TABLE_SUFFIX = &#39;20200110&#39;</pre><h4>2. User Properties and Event Parameters</h4><p>All the common attributes such as device properties, geo information are there for all events, and those custom parameters (like product_id in you in_app_purchase event) are stored as nested keys and values. If you’re not familiar with BigQuery repeated record types think these fields as an array of JSON object. Let’s assume a purchase event where we send the product_id and screen_name as event parameters;</p><pre>[<br>	{<br>		&quot;key&quot;: &quot;product_id&quot;,<br>		&quot;value&quot;: {<br>			&quot;int_value&quot;: null,<br>			&quot;float_value&quot;: null,<br>			&quot;double_value&quot;: null,<br>			&quot;string_value&quot;: &quot;consumables.oneYear&quot;<br>		}<br>	},<br>	{<br>		&quot;key&quot;: &quot;screen_name&quot;,<br>		&quot;value&quot;: {<br>			&quot;int_value&quot;: null,<br>			&quot;float_value&quot;: null,<br>			&quot;double_value&quot;: null,<br>			&quot;string_value&quot;: &quot;promotions.first_open&quot;<br>		}<br>	}<br>]</pre><p>How do we filter on the product_id? or event project it?</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/220/0*k_k3FiLZaiJxG5Ez.gif" /></figure><p>UNNEST IT!</p><pre><strong>SELECT</strong> <br>	(<strong>SELECT</strong> value.string_value <strong>FROM</strong> UNNEST(event_params) <strong>WHERE</strong> <strong>key</strong> = &#39;product_id&#39;) <strong>as</strong> product_id,<br>	(<strong>SELECT</strong> value.string_value <strong>FROM</strong> UNNEST(event_params) <strong>WHERE</strong> <strong>key</strong> = &#39;screen_name&#39;) <strong>as</strong> screen_name<br><strong>FROM</strong> events_*<br><strong>WHERE</strong> event_name = &#39;in_app_purchase&#39; <strong>AND</strong> _TABLE_SUFFIX <strong>BETWEEN</strong> &#39;20200107&#39; <strong>AND</strong> &#39;20200109&#39;</pre><p><strong>NOTE THAT</strong>: The same principle applies for user_properties. Keep in mind that this is an expensive operation and will increase your bytes-processed.</p><h3>Connecting your BigQuery to Rakam-BI</h3><p><a href="https://rakam.io/">Rakam</a> offers free service up to 3 team members with various bootstrapping options (recipes) for providers such as Firebase, Segment, Tenjin, Rakam-API and many on coming soon. If you’re ready to give it a try start with exploring the <a href="https://rakam.io/demo">demo </a>project.</p><h4>1. Create a service account</h4><ul><li>bigquery.datasets.get</li><li>bigquery.jobs.create</li><li>bigquery.tables.get</li><li>bigquery.tables.getData</li><li>bigquery.tables.list</li></ul><h4>2. Connecting your BigQuery Datasource</h4><p>Navigate to Connect new datasource and select Firebase or click <a href="https://app.rakam.io/connect/solution/firebase">here</a>. Get your project and dataset name (you can find them over the <a href="https://console.cloud.google.com/bigquery">BigQuery Console</a>). Your dataset name will start with analytics_</p><h4>3. Installing Firebase Recipe</h4><p>Once you connect your project, you will be directed to the recipe installation page. First, choose your database and schema (project and dataset). Firebase datasets start with analytics_ followed by your project identifier. To discover your event parameters and user properties Rakam will execute two SQL queries on your database. You can review this query if you like so, and click to Click to run Sql button. Lastly, if you are not sending in-app purchase data to Firebase, toggle the Install Revenue Metrics to off state.</p><p>Click on the Install button when the queries are finished. Rakam will now model your events, parameters and user properties into models. This might take a few minutes depending on the number of distinct events and properties you have. Note that both queries are scanning through your data over the last 15 days.</p><h4>4. Ready to Explore</h4><p>All your common custom Firebase events are now modeled. 5 minutes of your time is now what could take days to do.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*yQTznfkrr83oSrxb.png" /></figure><p>Now you have dashboards based on session and purchase segmentations. Navigate to your boards and explore your installed dashboards via this recipe. We’re adding more and more dashboards to the Firebase recipe every week. You can always go to the Recipes page and upgrade.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xwsXWzbRJk1b8FPg.png" /></figure><p>Explore more on Segmentation, Funnel and Retention reports <a href="https://player.vimeo.com/video/389043634">here</a> on the quick walkthrough video we prepared for you 🚀</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f7dab7c392b4" width="1" height="1" alt=""><hr><p><a href="https://medium.com/rakam-io/free-product-analytics-with-firebase-bigquery-rakam-f7dab7c392b4">Free Product Analytics with Firebase + BigQuery + Rakam</a> was originally published in <a href="https://medium.com/rakam-io">rakam</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Relay control from iOS Application [Swift] using NodeMCU]]></title>
            <link>https://medium.com/@cam.ilker/relay-control-from-ios-application-swift-using-nodemcu-1b0724a78a7b?source=rss-131cf0ebbd42------2</link>
            <guid isPermaLink="false">https://medium.com/p/1b0724a78a7b</guid>
            <category><![CDATA[nodemcu]]></category>
            <category><![CDATA[arduino]]></category>
            <dc:creator><![CDATA[ilker cam]]></dc:creator>
            <pubDate>Mon, 05 Jun 2017 14:56:55 GMT</pubDate>
            <atom:updated>2017-06-05T15:41:12.339Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*1WxLESRC3-jmdxRyPaXW-A.jpeg" /></figure><p>We have many stuff at home that we may want to turn on-off while we’re not home, or even we’re to lazy to get up from our confortable couches.<br>Lazy hobbiests and engineers work really hard to make their and others life easier. Let’s write an iOS application to control a relay circuit, so we can switch on and off lights 😄.</p><p>I had done similar project years ago when just met with Arduino. These days, there’re many boards out there with built-in internet access like NodeMCU. This board has a ESP8266 on it, we’re writing our programs to this chip which has a WiFi in it. Good news is that we can program it using the Arduino IDE with the same way that we know from Arduino boards.</p><p><strong>Section 0: Communication between iOS device and NodeMCU</strong></p><p>We need a lightweight, realtime and serverless medium to communicate between our devices. We don’t want to deal with three-tier architecture and concern about the network. Let’s use <a href="https://www.pubnub.com/">PubNub</a>, “a programmable network for developing realtime applications.”. It has over over 70 SDK’s for various of platforms and the documentation is very clear to understand.<br>We’ll using this service for bidirectional communication. We have only two use cases:</p><ol><li>Publish a message to a channel.</li><li>Subscribe to a channel, to retrive the messages.</li></ol><p><strong>Let’s dive in to the PubNub console:</strong></p><p>I didn’t thought that it would be that much easy. Register to the service, create an application, get your demo publish and subscribe keys, that’s it. We’re done with PubNub. It also has cool features like debug console, we’ll take a look once we finish programming NodeMCU.<br><em>We can use this Demo Keyset for development, I assume that you’re not concerned about a production ready usage.</em></p><p><strong>Section 1: Programming NodeMCU</strong></p><p>Before we start, you need to install three libraries;<br>1) ESP8266WiFi<br>2) <a href="http://pubnub arduino github">PubNub</a><br>3) <a href="https://github.com/bblanchon/ArduinoJson">ArduinoJson</a><br>4) <a href="https://github.com/schinken/SimpleTimer">SimpleTimer</a><br>You can install them from Arduino Ide &gt; Sketch &gt; Include Library &gt; Manage Libraries or Add .ZIP menu.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/342bb68eebfdf1c2caeea7538b225c28/href">https://medium.com/media/342bb68eebfdf1c2caeea7538b225c28/href</a></iframe><p>Check out the ESP8266 and WiFi (Shield) Support sections in PubNub’s arduino <a href="https://github.com/pubnub/arduino">repo</a> if you want to know why we defined the two macros.</p><p>Let’s setup the flow. We need to connect to the WiFi, PubNub service, setup pins and debug incoming &amp; outgoing messages through serial. NodeMCU has a weird pin numbers. I’m still not sure about it, you can check <a href="https://github.com/esp8266/Arduino/issues/584">this</a> issue thread.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/893adf61718988f39106006691d1759f/href">https://medium.com/media/893adf61718988f39106006691d1759f/href</a></iframe><p>I’ve used a simple JSON structure, an array of objects. Each object represents a pin that we use. “p” -&gt; pin number. “s” -&gt; state of the pin.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5f2653f883016b5aa50454c6ecdc2d94/href">https://medium.com/media/5f2653f883016b5aa50454c6ecdc2d94/href</a></iframe><p>We seek only two basic functionalities; Report pin status (On, Off), and change status.</p><ol><li>Publish Pin States: In out iOS application, we’ll need see if the pin’s are already on or off.</li><li>Handle Commands: iOS application will publish simple commands. Change pin state to high or low or request pin states (1).</li></ol><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/765b3126bcf43596a530ccd0b33697b7/href">https://medium.com/media/765b3126bcf43596a530ccd0b33697b7/href</a></iframe><p>ArduinoJson requires you to calculate the buffer size for encoding and decoding JSON strings. You can use the <a href="https://bblanchon.github.io/ArduinoJson/assistant/">Assistant</a> to do that. In order to use the same structure in the project I used “-1” pin number to request pin states from NodeMCU.</p><p>Let’s go to PubNub’s dashboard, select your application and go to the “Debug Console”. From there you can add client by simply typing the channel name which is “ProjectChannel” is my case. Type our json string to request the pin states {“p”: -1, “s”: false}. From the serial monitor you can see that NodeMCU got the request message from the “ProjectChannel” and sent the states of the used pins. You can also try to trigger a pin via {“p”: 1, “s”: true}. This will turn pin 1 to HIGH state.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/964/1*2CLbA8kGOF6os4hbg20zeQ.png" /></figure><p>I’ve connected some pins to the relay circuit. The circuit I’m using can operate with 5v input. I connected the pins VCC and GND of the relay circuit to Vin and Gnd pins in NodeMCU. Usually we use these pins to power NodeMCU, but we can also use it to get power from the USB.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GFF8tQ-g8oiy5se_ySgUkw.jpeg" /></figure><p>We’ve finished our work the circuit. Let’s build a very simple iOS application to control the pins.</p><p><strong>Section 2: iOS Application</strong></p><p>A POC application with a single view controller that has a table view inside. Each view cell has a switch and a label inside it. I used two libraries; <a href="https://github.com/Hearst-DD/ObjectMapper">ObjectMapper</a> and <a href="https://github.com/pubnub/objective-c">PubNub</a>. If your familiar with cocoapods you can simply add them to your pod file;</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/787376f529c2f13f857da70da2b7b7b0/href">https://medium.com/media/787376f529c2f13f857da70da2b7b7b0/href</a></iframe><p>Configuration of PubNub client is very easy. Put your publish and subscribe keys, set delegate to the view controller that your presenting, implement the delegates you require and subscribe to a channel.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d0186dee9890f9dc52702050cf7e2e06/href">https://medium.com/media/d0186dee9890f9dc52702050cf7e2e06/href</a></iframe><p>When we recieve a message we need to parse it. You can just check the dictionary keys or implement a mappable class using the ObjectMapper.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77620ce4d314ef8c94d8e0201106556f/href">https://medium.com/media/77620ce4d314ef8c94d8e0201106556f/href</a></iframe><p>So when we recieve a message from the published channel, we’ll have an array of device instances. Let’s implement the delegate “UITableViewDataSource”</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/135c6fd213adef7b7238aff9e84305bb/href">https://medium.com/media/135c6fd213adef7b7238aff9e84305bb/href</a></iframe><p>We’re drawing the cells in cellForRowAt indexPath delegate. We’re passing a closure to “OnSwitchStateChange” to publish the device instance with the new state of the pin.</p><p>We use a costume cell class to define the pin number label and switch. The closure is passed to this cell instance and executed the the state change action of the switch.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/903c8afe948c9a89e94e96cb607e3870/href">https://medium.com/media/903c8afe948c9a89e94e96cb607e3870/href</a></iframe><p>When you run the application and hit the refresh button, the tableview will be populated with the incoming message from NodeMCU.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/750/1*6Mt3PK6_33pA7UXlj0l5Eg.png" /></figure><p>Try to switch some pins now.</p><p>Congrats 🤡</p><p>Source: <a href="https://github.com/ilkerc/nodemcu-relay-ios">https://github.com/ilkerc/nodemcu-relay-ios</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=1b0724a78a7b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>