<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Anatolii Shulhat on Medium]]></title>
        <description><![CDATA[Stories by Anatolii Shulhat on Medium]]></description>
        <link>https://medium.com/@w.anatolii?source=rss-4fa1a98ff31d------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*71sQPC70hB8neI0wqAybvg.jpeg</url>
            <title>Stories by Anatolii Shulhat on Medium</title>
            <link>https://medium.com/@w.anatolii?source=rss-4fa1a98ff31d------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 06:59:22 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@w.anatolii/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Server-side cookie via Cloud Run and Cloudflare]]></title>
            <link>https://medium.com/@w.anatolii/server-side-cookie-via-cloud-run-and-cloudflare-b75aa317db75?source=rss-4fa1a98ff31d------2</link>
            <guid isPermaLink="false">https://medium.com/p/b75aa317db75</guid>
            <category><![CDATA[web-analytics]]></category>
            <category><![CDATA[digital-analytics]]></category>
            <category><![CDATA[google-tag-manager]]></category>
            <category><![CDATA[google-cloud-run]]></category>
            <category><![CDATA[cloudflare]]></category>
            <dc:creator><![CDATA[Anatolii Shulhat]]></dc:creator>
            <pubDate>Sat, 04 Oct 2025 17:19:12 GMT</pubDate>
            <atom:updated>2025-10-04T18:48:31.930Z</atom:updated>
            <content:encoded><![CDATA[<h3>Server-side cookie DIY</h3><p>Let’s say you were inspired by the idea of building your own custom collector and analytics data pipeline without using any third-party libraries, as described here: <a href="https://medium.com/@w.anatolii/how-to-collect-custom-events-in-bigquery-without-ga4-ae4b914fd16c">https://medium.com/@w.anatolii/how-to-collect-custom-events-in-bigquery-without-ga4-ae4b914fd16c</a>.</p><p>If you’ve worked with GTM server-side GA4 setups or more advanced analytics tools like Snowplow (I really like their sp cookie concept), you know that you can set a server-side cookie that’s more robust and less prone to manipulation on the client side compared to a regular browser cookie.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oW2J8nSDdx2qr1Q953x5Sw.jpeg" /></figure><p>So, how can we achieve the same setup with a custom solution described in my previous article?</p><h3>Enriching the pipeline</h3><ol><li>Let’s add the ss_temp_user_id column to the BQ table.</li></ol><pre>ALTER TABLE `GCP-PROJECT-ID.custom_analytics.page_views`<br>ADD COLUMN ss_temp_user_id STRING OPTIONS(description=&quot;Server-side temporary user identifier, analogous to temp_user_id on the frontend.&quot;);</pre><p>2. Let’s edit our Cloud Run collector and deploy a new version.</p><pre>const express = require(&#39;express&#39;);<br>const { v4: uuidv4 } = require(&#39;uuid&#39;);<br>const { PubSub } = require(&#39;@google-cloud/pubsub&#39;);<br>const cookieParser = require(&#39;cookie-parser&#39;);<br><br>const app = express();<br>const pubsub = new PubSub();<br>const TOPIC = process.env.PUBSUB_TOPIC;<br>const PORT = process.env.PORT || 8080;<br><br>app.use(express.text({ limit: &#39;128kb&#39; }));<br>app.use(cookieParser());<br><br>app.use((req, res, next) =&gt; {<br>  const origin = req.get(&#39;Origin&#39;);<br>  if (isAllowedOrigin(origin)) {<br>    res.set(&#39;Access-Control-Allow-Origin&#39;, origin);<br>    res.set(&#39;Access-Control-Allow-Credentials&#39;, &#39;true&#39;);<br>  }<br>  if (req.method === &#39;OPTIONS&#39;) return res.sendStatus(204);<br>  next();<br>});<br><br>function getClientIp(req) {<br>  const xfwd = req.headers[&#39;x-forwarded-for&#39;];<br>  return xfwd ? xfwd.split(&#39;,&#39;)[0].trim() : req.ip || req.socket.remoteAddress || &#39;&#39;;<br>}<br><br>function isAllowedOrigin(origin) {<br>  if (!origin) return false;<br>  try {<br>    const url = new URL(origin);<br>    return url.protocol === &#39;https:&#39; &amp;&amp; (url.hostname.endsWith(&#39;.example.com&#39;) || url.hostname === &#39;example.com&#39;);<br>  } catch {<br>    return false;<br>  }<br>}<br><br>app.post(&#39;/echoclap&#39;, async (req, res) =&gt; {<br>  try {<br>    const origin = req.get(&#39;Origin&#39;);<br>    if (!isAllowedOrigin(origin)) return res.status(403).send(&#39;Forbidden&#39;);<br><br>    let ssUserId = req.cookies.ss_temp_user_id || uuidv4();<br>    res.cookie(&#39;ss_temp_user_id&#39;, ssUserId, {<br>      httpOnly: true,<br>      secure: true,<br>      sameSite: &#39;Lax&#39;,<br>      path: &#39;/&#39;,<br>      domain: &#39;.example.com&#39;,<br>      maxAge: 365 * 24 * 60 * 60 * 1000,<br>    });<br><br>    const body = JSON.parse(req.body);<br>    const enriched = {<br>      ...body,<br>      event_id: uuidv4(),<br>      ss_temp_user_id: ssUserId,<br>      cloud_run_timestamp: new Date().toISOString(),<br>      http_ip_address: getClientIp(req),<br>      http_useragent: req.get(&#39;User-Agent&#39;) || &#39;&#39;,<br>      http_cookie: req.get(&#39;Cookie&#39;) || &#39;&#39;,<br>      http_referrer: req.get(&#39;Referer&#39;) || &#39;&#39;,<br>      http_language: req.get(&#39;Accept-Language&#39;) || &#39;&#39;<br>    };<br><br>    await pubsub.topic(TOPIC).publishMessage({ data: Buffer.from(JSON.stringify(enriched)) });<br><br>    res.status(204).send();<br>  } catch (err) {<br>    console.error(&#39;Pub/Sub publish error:&#39;, err);<br>    res.status(500).send(&#39;Internal Server Error&#39;);<br>  }<br>});<br><br>app.listen(PORT, () =&gt; console.log(`Listening on port ${PORT}`));</pre><p>3. package.json should be edited as well.</p><pre>{<br>  &quot;name&quot;: &quot;page-tracker&quot;,<br>  &quot;version&quot;: &quot;1.0.0&quot;,<br>  &quot;main&quot;: &quot;index.js&quot;,<br>  &quot;dependencies&quot;: {<br>    &quot;express&quot;: &quot;^4.18.2&quot;,<br>    &quot;@google-cloud/pubsub&quot;: &quot;^4.0.0&quot;,<br>    &quot;uuid&quot;: &quot;^9.0.0&quot;,<br>    &quot;cookie-parser&quot;: &quot;^1.4.6&quot;<br>  }<br>}</pre><p>4. In my previous article, I used a Load Balancer and added an A record in my domain hosting setup. In my case, I used the Zone Editor in cPanel hosting. In this guide, I propose skipping that step and making it first-party via Cloudflare and a Worker.</p><p>I highly appreciate this article, as it explores the idea of using a Worker for me: <a href="https://www.teamsimmer.com/2025/08/28/same-origin-deployment-with-server-side-google-tag-manager/">https://www.teamsimmer.com/2025/08/28/same-origin-deployment-with-server-side-google-tag-manager/</a>. It covers a slightly different topic, but it can be useful if you want to dive into different first-party or same-origin analytics script setups.</p><p>So, coming back to the DIY custom server-side cookie setup:</p><ul><li>Make sure you proxy your traffic through Cloudflare DNS.</li><li>Create a Worker in the Compute menu (I initially created a predefined “Hello World” worker script and modified it later).</li><li>Don’t forget to change https://page-tracker-XXXXXXXXXXXX.europe-west3.run.app/echoclap to your collector endpoint.</li></ul><pre>addEventListener(&#39;fetch&#39;, event =&gt; {<br>  event.respondWith(handleRequest(event.request))<br>})<br><br>async function handleRequest(request) {<br>  try {<br>    const url = new URL(request.url)<br>    console.log(&#39;Worker got request:&#39;, url.pathname, request.method)<br><br>    if (url.pathname.startsWith(&#39;/echoclap&#39;) &amp;&amp; request.method === &#39;POST&#39;) {<br>      const bodyText = await request.text()<br>      console.log(&#39;Request body:&#39;, bodyText)<br><br>      const response = await fetch(&#39;https://page-tracker-XXXXXXXXXXXX.europe-west3.run.app/echoclap&#39;, {<br>        method: &#39;POST&#39;,<br>        headers: request.headers,<br>        body: bodyText<br>      })<br><br>      return new Response(await response.text(), {<br>        status: response.status,<br>        headers: response.headers<br>      })<br>    }<br><br>    if (url.pathname.startsWith(&#39;/echoclap&#39;) &amp;&amp; request.method === &#39;OPTIONS&#39;) {<br>      return new Response(null, {<br>        status: 204,<br>        headers: {<br>          &#39;Access-Control-Allow-Origin&#39;: &#39;*&#39;,<br>          &#39;Access-Control-Allow-Methods&#39;: &#39;POST, OPTIONS&#39;,<br>          &#39;Access-Control-Allow-Headers&#39;: &#39;Content-Type&#39;,<br>        }<br>      })<br>    }<br><br>    return new Response(&#39;Not handled by Worker&#39;, { status: 404 })<br>  } catch (err) {<br>    return new Response(&#39;Worker error: &#39; + err.message, { status: 500 })<br>  }<br>}</pre><ul><li>Make sure you have added the route as example.com/echoclap.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pfl7WFeGfK9Q5ycTLrhiyg.png" /></figure><p>5. In your frontend pixel that sends POST requests to the collector, please change the endpoint to:</p><pre>var TRACKER_ENDPOINT = &#39;https://example.com/echoclap&#39;;</pre><h3>What makes this solution valuable?</h3><p>You can test this in Safari and see that the cookie isn’t limited to 7 days. But if you use the setup from my earlier article — the one with the A Record and Load Balancer — you’ll notice that the server-side cookie is capped at 7 days. That really surprised me.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5j_LUUkwxL9W2_qprtpnog.jpeg" /></figure><p>This happens because modern browsers like Safari, Chrome, and Firefox on iOS, as well as desktop Safari, now limit how long some cookies can live. This applies to server-side cookies in some cases as well — in fact, server-side cookies will be capped, and you can notice this by checking them in the browser. The lifetime of client-side cookies might not appear to change at first, but they will be deleted within 7 days.</p><p>This makes it harder to track a user’s journey over time, especially if they don’t visit your site frequently. Hello, ITP. Not every first-party setup will protect your server-side cookies from ITP manipulation — so make sure to test it.</p><p>For details: <a href="https://webkit.org/tracking-prevention/#cname-and-third-party-ip-address-cloaking-defense">https://webkit.org/tracking-prevention/#cname-and-third-party-ip-address-cloaking-defense</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b75aa317db75" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to collect custom events in BigQuery without GA4]]></title>
            <link>https://medium.com/@w.anatolii/how-to-collect-custom-events-in-bigquery-without-ga4-ae4b914fd16c?source=rss-4fa1a98ff31d------2</link>
            <guid isPermaLink="false">https://medium.com/p/ae4b914fd16c</guid>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[google-analytics-4]]></category>
            <category><![CDATA[digital-analytics]]></category>
            <category><![CDATA[web-analytics]]></category>
            <category><![CDATA[bigquery]]></category>
            <dc:creator><![CDATA[Anatolii Shulhat]]></dc:creator>
            <pubDate>Mon, 25 Aug 2025 15:37:01 GMT</pubDate>
            <atom:updated>2025-08-25T15:37:01.378Z</atom:updated>
            <content:encoded><![CDATA[<h3>Intro — Web Analytics DIY</h3><p>This story is more of a weekend fun for me. A way to push myself with a challenge that also helps to learn something new in digital analytics.</p><p>I’ve been curious about how Google Cloud Platform services can be used to build analytics solutions.</p><blockquote>At some point, I asked myself: do I really need the GA4 library with its third-party logic and limits just to collect basic website data and custom events in BigQuery?</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FJ6wZt-j76CKQ_BNYHefuA.jpeg" /></figure><p>GA4 does have a powerful feature: exporting raw data to BigQuery. From there, you can model almost anything a business or technical mind can imagine.</p><p>I see BigQuery (and tools like Dataform) as the real key to unlocking GA4’s potential.</p><p>So I thought — why not try building my own tracker? A lightweight script that sends data directly into BigQuery.</p><p>We live in an incredible time: with ChatGPT and a free weekend, you can create things you never imagined possible.</p><p>Of course, I wished ChatGPT could give me a perfect, ready-to-use solution. But as of August 2025, that’s not how it works.</p><p>Instead, over two weekends, I iterated, refined prompts, tested, and validated each part — until I had something working.</p><p>I hope walking you through this little journey will be useful.</p><p>In this post, I’ll share how I built a custom tracker to collect basic page views.</p><h3>Why does it matter, and how can it be used?</h3><p>Before diving into the technical details, I want to share my vision for how this architectural solution can be used and why it’s effective.</p><h4>Simple and brand-free</h4><p>This is a custom solution that can be easily deployed anywhere. It’s perfect if you need something basic and lightweight that can track website traffic and basic events.</p><h4><strong>Full control over your data</strong></h4><p>It allows seamless collection of digital analytics data without relying on any third-party scripts or libraries. You control everything. This is a huge advantage if you’re a developer and want to fully understand what your tracker is doing and what data it collects. From privacy to page speed optimization, schema, and limits — you control everything, with the only boundaries set by cloud services.</p><h4>Foundation for a larger build</h4><p>As I mentioned earlier, this is only an architectural experiment. I am a digital analyst, and I’m looking for different approaches to collect web analytics data. So I think, it can serve as a foundation for something more tailored to your needs. For example, you could use it to build a robust user identity solution. It allows for the collection of any combination of user ids, cookies, URL parameters, user agents, IP addresses, and more. In addition, it can act as a useful instrument for capturing critical data when your main backend tools are limited in handling high-load behavioral traffic. Not every backend team permits collecting everything, and in such cases, having a custom collector — independent of mainstream digital analytics platforms or development backend services — can be a valuable alternative.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zgWz3yDWqVbC-4gI10EKig.jpeg" /></figure><h3>What risks are not covered in the article?</h3><h4>Costs of GCP Services</h4><p>GA4 offers a free version, which is a huge advantage. Of course, you’re essentially paying with your website visitors’ data to use it at no cost. In the case of a custom solution like this one, you’ll need to pay for each service you use. I haven’t estimated the costs or tested this setup on high-traffic websites yet. I’ll cover this aspect in future articles once I see how it performs in practice for some time.</p><h4>Security</h4><p>This is quite a technical topic, and if you apply it to your projects, it’s better to double-check with DevOps or DataOps specialists who can advise on making it more reliable for real-world websites. I tried to follow Google’s default recommendations and challenge ChatGPT with tricky questions to explore different aspects, but building a cloud solution is still new to me. If you use GCP services, making it scalable and secure isn’t too hard, but it still requires expertise. My architectural setup should be quite flexible.</p><h3>Architecture</h3><p>The draft setup to track page views on a high-traffic site using GCP includes:</p><h4>Frontend</h4><ul><li><strong>JS script</strong> (added via GTM) that sends POST requests.</li></ul><h4>Backend</h4><ul><li><strong>Cloud Run</strong> to receive and enrich requests.</li><li><strong>Load Balancer</strong> to expose Cloud Run on my own domain.</li><li><strong>Pub/Sub</strong> to pass data from Cloud Run to BigQuery.</li><li><strong>BigQuery</strong> to store events.</li></ul><h3>Prepare CLI and GCP APIs</h3><p>You will use the CLI a lot, so I recommend visiting <a href="https://cloud.google.com/sdk/docs/install">https://cloud.google.com/sdk/docs/install</a> and configuring it for your laptop.</p><p>Once installation and authorization are successful, run the following code:</p><pre>gcloud services enable run.googleapis.com pubsub.googleapis.com bigquery.googleapis.com</pre><p>This is the first important step: enabling all the required APIs for the current project.</p><h3>Create BigQuery page_views table</h3><p>I would start from creating BigQuery table with all needed fields.</p><pre>-- 1. Ensure dataset exists<br>CREATE SCHEMA IF NOT EXISTS `GCP-PROJECT-ID.custom_analytics`;<br><br>-- 2. Ensure table exists<br>CREATE TABLE IF NOT EXISTS `GCP-PROJECT-ID.custom_analytics.page_views` (<br>  event_id STRING OPTIONS(description=&quot;A server-side UUID that can be used as the main key for the event-level table. Taken from: Cloud Run backend.&quot;),<br>  event_name STRING OPTIONS(description=&quot;The name of the event, defined by the business. Taken from: the JavaScript code running in the browser.&quot;),<br>  user_id STRING OPTIONS(description=&quot;A business identifier for the user, analogous to user_id in GA4 or Snowplow. Taken from: the JavaScript code running in the browser.&quot;),<br>  temp_user_id STRING OPTIONS(description=&quot;A temporary cookie-based identifier for the user, analogous to user_pseudo_id in GA4 or domain_userid in Snowplow. Taken from: a JavaScript-set cookie in the browser.&quot;),<br>  browser_timestamp TIMESTAMP OPTIONS(description=&quot;Timestamp generated by the JS script on the frontend in the browser when the script fires. Taken from: Date.now().&quot;),<br>  cloud_run_timestamp TIMESTAMP OPTIONS(description=&quot;Timestamp generated by the Cloud Run service on the backend. Taken from: server-side system clock.&quot;),<br>  location_hostname STRING OPTIONS(description=&quot;The domain name of the page, e.g., example.com. Taken from: window.location.hostname in the browser.&quot;),<br>  location_href STRING OPTIONS(description=&quot;The full URL of the page the user is on. Taken from: window.location.href.&quot;),<br>  location_pathname STRING OPTIONS(description=&quot;The path part of the URL, e.g., /about or /products/shoes. Taken from: window.location.pathname.&quot;),<br>  location_search STRING OPTIONS(description=&quot;The query string from the URL (everything after the ?). Taken from: window.location.search.&quot;),<br>  document_title STRING OPTIONS(description=&quot;The page title as shown in the browser tab. Taken from: document.title.&quot;),<br>  document_referrer STRING OPTIONS(description=&quot;The URL of the previous page that brought the user here. Taken from: document.referrer.&quot;),<br>  http_referrer STRING OPTIONS(description=&quot;The origin of the HTTP request as seen by the server. Taken from: Referer HTTP header.&quot;),<br>  document_cookie STRING OPTIONS(description=&quot;Cookies readable via JavaScript in the browser. Taken from: document.cookie.&quot;),<br>  http_cookie STRING OPTIONS(description=&quot;Cookies sent automatically in the HTTP request. Taken from: Cookie HTTP header.&quot;),<br>  navigator_useragent STRING OPTIONS(description=&quot;Browser details (device, OS, browser version) reported by JS. Taken from: navigator.userAgent.&quot;),<br>  http_useragent STRING OPTIONS(description=&quot;Browser details reported at HTTP request level. Taken from: User-Agent header.&quot;),<br>  navigator_language STRING OPTIONS(description=&quot;The language setting from the user&#39;s browser. Taken from: navigator.language.&quot;),<br>  http_language STRING OPTIONS(description=&quot;Preferred language sent automatically in HTTP request. Taken from: Accept-Language header.&quot;),<br>  http_ip_address STRING OPTIONS(description=&quot;User&#39;s IP address as seen from the server. Taken from: source IP of incoming HTTP request.&quot;)<br>);</pre><blockquote>I used technical naming conventions for the columns so I can clearly understand which fields come from parsing <strong>browser data</strong> or HTTP request headers.</blockquote><p>If needed, you can use more user-friendly or business conventions, such as page_url or page_path, and skip some duplicated fields if you decide on a single source of truth.</p><p>After running the script above, you will have a BigQuery table with descriptions for each field, serving as a vocabulary reference.</p><blockquote>The “Taken from” mentions can be valuable for quickly understanding the technical source of the data and for debugging.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n034rqb7GFNVOPSVADe4aw.png" /></figure><h3>Create Pub/Sub logic</h3><p>Now, let’s create a new Pub/Sub topic called <strong>page-views-topic</strong>.</p><p>This will serve as an endpoint for our backend to send request data from Cloud Run.</p><p>This topic is like a channel where all messages from Cloud Run will be published.</p><pre>gcloud pubsub topics create page-views-topic</pre><p>Other services (subscribers) can then read messages from this topic. In my case, I want BigQuery to act as a destination where all message data will be stored.</p><p>So the next step is to create a subscription <strong>page-views-sub</strong> as follows:</p><pre>gcloud pubsub subscriptions create page-views-sub \<br>  --topic=page-views-topic \<br>  --bigquery-table=GCP-PROJECT-ID:custom_analytics.page_views \<br>  --ack-deadline=600 \<br>  --use-table-schema</pre><p>I used the CLI and terminal to manage cloud services, but this can also be done easily via the <a href="https://console.cloud.google.com/cloudpubsub/topic/list">Cloud Console</a>.</p><p>I mostly used the default settings and fields. You can check the previous code snippets for details if it’s not clear which option to choose.</p><p>I rely on a JSON payload, partly defined in the frontend script and partly in the Cloud Run script. The request body fields correspond to BigQuery column names and types, which allows me to quickly collect data without spending much time on schema design or additional enrichment logic.</p><h3>Deploy Cloud Run app</h3><p>Now, let’s focus on the main backend app, which will receive requests from the website and send them to Pub/Sub for future storage in BigQuery.</p><p>I would propose to create the folder <strong>page_tracker</strong> on your laptop. It will have 3 files.</p><h4><strong>Create index.js as file #1</strong></h4><p>It has the main backend logic: handles POST requests, enriches data, and sends it to Pub/Sub.</p><p>From a technical digital analytics perspective, I would view this as a pseudo custom GTM Server-Side container, where you can apply different enrichments, configure the client as needed, and implement various routing logic.</p><p>The potential for customization is huge and depends on business requirements and development capacity.</p><p>Adjust the code a bit for your domain.</p><pre>const express = require(&#39;express&#39;);<br>const { v4: uuidv4 } = require(&#39;uuid&#39;);<br>const { PubSub } = require(&#39;@google-cloud/pubsub&#39;);<br><br>const app = express();<br>app.use(express.text({ limit: &#39;128kb&#39; }));<br><br>const TOPIC = process.env.PUBSUB_TOPIC;<br>const pubsub = new PubSub();<br><br>function getClientIp(req) {<br>  const xfwd = req.headers[&#39;x-forwarded-for&#39;];<br>  return xfwd ? xfwd.split(&#39;,&#39;)[0].trim() : req.ip || req.socket.remoteAddress || &#39;&#39;;<br>}<br><br>function isAllowedOrigin(origin) {<br>  if (!origin) return false;<br>  try {<br>    const url = new URL(origin);<br>    return url.protocol === &#39;https:&#39; &amp;&amp; (url.hostname.endsWith(&#39;.example.com&#39;) || url.hostname === &#39;example.com&#39;);<br>  } catch {<br>    return false;<br>  }<br>}<br><br>app.use((req, res, next) =&gt; {<br>  const origin = req.get(&#39;Origin&#39;);<br>  if (isAllowedOrigin(origin)) {<br>    res.set(&#39;Access-Control-Allow-Origin&#39;, origin);<br>    res.set(&#39;Access-Control-Allow-Credentials&#39;, &#39;true&#39;);<br>  }<br>  if (req.method === &#39;OPTIONS&#39;) return res.sendStatus(204);<br>  next();<br>});<br><br>app.post(&#39;/echoclap&#39;, async (req, res) =&gt; {<br>  try {<br>    if (!isAllowedOrigin(req.get(&#39;Origin&#39;))) return res.status(403).send(&#39;Forbidden&#39;);<br><br>    const body = JSON.parse(req.body); // parse text/plain as JSON<br><br>    const enriched = {<br>      ...body,<br>      event_id: uuidv4(),<br>      cloud_run_timestamp: new Date().toISOString(),<br>      http_ip_address: getClientIp(req),<br>      http_useragent: req.get(&#39;User-Agent&#39;) || &#39;&#39;,<br>      http_cookie: req.get(&#39;Cookie&#39;) || &#39;&#39;,<br>      http_referrer: req.get(&#39;Referer&#39;) || &#39;&#39;,<br>      http_language: req.get(&#39;Accept-Language&#39;) || &#39;&#39;<br>    };<br><br>    await pubsub.topic(TOPIC).publishMessage({ data: Buffer.from(JSON.stringify(enriched)) });<br><br>    res.status(204).send();<br>  } catch (err) {<br>    console.error(&#39;Pub/Sub publish error:&#39;, err);<br>    res.status(500).send(&#39;Internal Server Error&#39;);<br>  }<br>});<br><br>const PORT = process.env.PORT || 8080;<br>app.listen(PORT, () =&gt; console.log(`Listening on port ${PORT}`));2. </pre><h4><strong>Create package.json</strong> as file #2</h4><p>It lists all dependencies so the app can run inside the container.</p><pre>{<br>  &quot;name&quot;: &quot;page-tracker&quot;,<br>  &quot;version&quot;: &quot;1.0.0&quot;,<br>  &quot;main&quot;: &quot;index.js&quot;,<br>  &quot;dependencies&quot;: {<br>    &quot;express&quot;: &quot;^4.18.2&quot;,<br>    &quot;@google-cloud/pubsub&quot;: &quot;^4.0.0&quot;,<br>    &quot;uuid&quot;: &quot;^9.0.0&quot;<br>  }<br>}</pre><h4><strong>Create Dockerfile </strong>as file #3</h4><p>It defines the container: base image, dependencies, and how to start the app in the GCP.</p><pre>FROM node:18-slim<br>WORKDIR /app<br>COPY package*.json ./<br>RUN npm install --omit=dev<br>COPY . .<br>EXPOSE 8080<br>CMD [&quot;node&quot;,&quot;index.js&quot;]</pre><blockquote>Together, these 3 files let Cloud Run run a fully containerized Node.js service in GCP.</blockquote><p>So, you just need to put all three files together in the page_tracker folder, open a terminal in that folder, and run the code.</p><pre>gcloud run deploy page-tracker \<br>  --source . \<br>  --region europe-west3 \<br>  --platform managed \<br>  --allow-unauthenticated \<br>  --set-env-vars PUBSUB_TOPIC=page-views-topic,REQUIRE_KEY=false \<br>  --project GCP-PROJECT-ID</pre><p>Now, you have a Cloud Run page_tracker instance that can be viewed here: <a href="https://console.cloud.google.com/run">https://console.cloud.google.com/run</a>.</p><p>It will have a URL like https://page-tracker-************.europe-west3.run.app, so it’s time to make it the same domain as your website, i.e., a first-party setup.</p><h3>Make it first-party via Load Balancer</h3><h4>Configure IP Address and A record</h4><p>First, you need to create a GCP-based external IP address.</p><p>This IP will be used by your load balancer to route traffic from your domain to Cloud Run.</p><p>You can create it in the GCP Console or via CLI:</p><pre>gcloud compute addresses create analytics-lb-ip \<br>  --ip-version=IPV4 \<br>  --global</pre><p>Next, add this IP as an A record in your domain hosting setup. In my case, I used the Zone Editor in cPanel hosting.</p><p><strong>I created an A record for </strong><strong>analytics.example.com and pointed it to the IP address created earlier.</strong></p><p>Be aware that it takes some time for the setup to propagate.</p><p>analytics.example.com/echoclap will be used as a tracker endpoint in our JS script on the frontend later.</p><h4>Create SSL certificate</h4><p>To secure HTTPS traffic for this domain, let’s create a GCP SSL certificate. It will encrypt all communication between users’ browsers and your Google Cloud Load Balancer.</p><p>You can do this here: <a href="https://console.cloud.google.com/security/ccm/list/lbCertificates">https://console.cloud.google.com/security/ccm/list/lbCertificates</a> or again with the help of the CLI</p><pre>gcloud compute ssl-certificates create custom-analytics-lb-cert \<br>    --domains=analytics.example.com \<br>    --global</pre><p>Make sure it has the status <strong>Active</strong> — this may take some time.</p><h4>Configure Load balancing</h4><ol><li>Let’s create a <strong>Network Endpoint Group </strong>in GCP.</li></ol><p>A NEG allows serverless backends, like Cloud Run, to be used directly behind a Google Cloud Load Balancer. In short, the NEG directs the load balancer to your serverless service.</p><p>You can create it here: <a href="https://console.cloud.google.com/compute/networkendpointgroups/list">https://console.cloud.google.com/compute/networkendpointgroups/list</a> or use the CLI:</p><pre>gcloud compute network-endpoint-groups create custom-analytics-lb-neg \<br>  --region=europe-west3 \<br>  --network-endpoint-type=serverless \<br>  --cloud-run-service=page-tracker \<br>  --project=GCP-PROJECT-ID</pre><p>2. For additional security, I have created a security policy for Google Cloud Armor.</p><pre>gcloud compute security-policies create default-security-policy-for-custom-analytics-lb-be \<br>    --description &quot;Default security policy for: custom-analytics-lb-be&quot; \<br>    --project=GCP-PROJECT-ID</pre><p>3. Let’s set up the Load Balancer with backend and frontend configurations. For me, the CLI documentation in this step was a bit confusing, and I found it much more comfortable to use the Google Cloud Platform UI. So, I’m sharing some screenshots that will help you follow a similar setup.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z_fMDataUwZzQ3tR5SdJ7w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*793NtVLdQ0lz2JDLmyDHLw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VoN7x6-eJlYMMczDfNxr1Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SWcja5f-PwzLJnSSj2eKEg.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5U7F28DXFbGXZXAptv45yQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3n_LVOMAqYebAVLHw02wtQ.png" /></figure><p>After creation, you will obtain something like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-EVaoHFpIuyHJmc673Qo-A.png" /></figure><p>The GCP setup is now complete. Once all hosting configurations and SSL have propagated, it’s a good time to test everything and send requests from your website using a tag.</p><h3><strong>Publish JS that sends POST requests</strong></h3><p>In my case, I used a <strong>Custom HTML tag</strong> combined with an <strong>Initialization trigger</strong> in Google Tag Manager.</p><p>That setup was sufficient for me to run it as an experiment on my website.</p><p>You can ask your developer to integrate it directly into your codebase, adding any custom logic you need for a more reliable implementation.</p><pre>&lt;script&gt;<br>(function() {<br>  var TRACKER_ENDPOINT = &#39;https://analytics.example.com/echoclap&#39;;<br>  var COOKIE_NAME = &#39;temp_user_id&#39;;<br>  var COOKIE_TTL_DAYS = 365 * 5;<br><br>  function uuidv4() {<br>    return &#39;xxxxxxxx-xxxx-4xxx-yxxx-xxxxxxxxxxxx&#39;.replace(/[xy]/g, function(c) {<br>      var r = Math.floor(Math.random() * 16);<br>      return (c === &#39;x&#39; ? r : (r &amp; 0x3 | 0x8)).toString(16);<br>    });<br>  }<br><br>  function getRootDomain() {<br>    var parts = location.hostname.split(&#39;.&#39;);<br>    if (parts.length &lt;= 2) return location.hostname;<br>    return &#39;.&#39; + parts.slice(-2).join(&#39;.&#39;);<br>  }<br><br>  function getCookie(name) {<br>    var match = document.cookie.match(&#39;(^|;)\\s*&#39; + name + &#39;=([^;]+)&#39;);<br>    return match ? decodeURIComponent(match[2]) : null;<br>  }<br><br>  function setCookie(name, val, days) {<br>    var domain = getRootDomain();<br>    var exp = new Date(Date.now() + days * 864e5).toUTCString();<br>    document.cookie = name + &#39;=&#39; + encodeURIComponent(val) + &#39;; path=/; domain=&#39; + domain + &#39;; expires=&#39; + exp + &#39;; SameSite=Lax&#39;;<br>  }<br><br>  function sendEvent() {<br>    var temp_user_id = getCookie(COOKIE_NAME);<br>    if (!temp_user_id) {<br>      temp_user_id = uuidv4();<br>      setCookie(COOKIE_NAME, temp_user_id, COOKIE_TTL_DAYS);<br>    }<br><br>    var payload = {<br>      event_name: &#39;page_view&#39;,<br>      temp_user_id: temp_user_id,<br>      browser_timestamp: new Date().toISOString(),<br>      location_hostname: location.hostname,<br>      location_href: location.href,<br>      location_pathname: location.pathname,<br>      location_search: location.search,<br>      document_title: document.title,<br>      document_referrer: document.referrer,<br>      document_cookie: document.cookie,<br>      navigator_useragent: navigator.userAgent,<br>      navigator_language: navigator.language<br>    };<br><br>    fetch(TRACKER_ENDPOINT, {<br>      method: &#39;POST&#39;,<br>      headers: { &#39;Content-Type&#39;: &#39;text/plain&#39; },<br>      body: JSON.stringify(payload),<br>      credentials: &#39;include&#39;<br>    }).catch(function(err) {<br>      if (window.console) console.error(&#39;Analytics send error:&#39;, err);<br>    });<br>  }<br><br>  sendEvent();<br>})();<br>&lt;/script&gt;</pre><p>This script sets a special cookie temp_user_id, similar to _ga in GA4, and sends a POST request with browser-related data in the payload. You can add any fields you need, but remember to adjust your BigQuery table accordingly.</p><p>If everything was done correctly, you will see a POST request containing your payload with all the data scraped from your browser.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eJqY3yoWDo8afDkhx_-vqw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gS9ZUWQCvciPBxtZUhKLAg.png" /></figure><p>And of course, some data will appear in BigQuery.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YPFLvBWrAPNqCBM-GhxSEw.png" /></figure><blockquote>If something isn’t working or you run into errors, check out <a href="https://console.cloud.google.com/logs">Google Cloud Logs</a>. It’s extremely helpful for troubleshooting issues with any Google service. You can even share logs directly with Gemini in the GCP cabinet, which makes debugging much easier.</blockquote><p>Hope you found this article insightful. Feel free to connect with me on <a href="https://www.linkedin.com/in/wanatolii/">LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ae4b914fd16c" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to inject script for best performance in GTM]]></title>
            <link>https://medium.com/@w.anatolii/how-to-inject-script-for-best-performance-in-gtm-5896078f3169?source=rss-4fa1a98ff31d------2</link>
            <guid isPermaLink="false">https://medium.com/p/5896078f3169</guid>
            <category><![CDATA[template]]></category>
            <category><![CDATA[performance]]></category>
            <category><![CDATA[script]]></category>
            <category><![CDATA[tag-manager]]></category>
            <category><![CDATA[google-tag-manager]]></category>
            <dc:creator><![CDATA[Anatolii Shulhat]]></dc:creator>
            <pubDate>Thu, 16 Jan 2025 00:49:01 GMT</pubDate>
            <atom:updated>2025-01-16T09:24:45.572Z</atom:updated>
            <content:encoded><![CDATA[<h3>Intro</h3><p>When you start working with GTM heavily, you will eventually find that using Custom HTML for all tags is not always the best idea.</p><p>When you fire a Custom HTML tag, your browser performs a series of non-obvious actions that will affect performance and sometimes even how the website page is compiled.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*15_IGBlpdpK1N4KDH2K3eA.jpeg" /></figure><p>This can be especially problematic for SPAs — when, with each launch of such a tag, we continuously add new code that piles up and negatively impacts performance.</p><p>It is a huge topic with lots of nuances. Here’s a great guide with details: <a href="https://www.simoahava.com/analytics/custom-html-tag-guide-google-tag-manager/">https://www.simoahava.com/analytics/custom-html-tag-guide-google-tag-manager/</a>.</p><p>Here, I plan to focus on just one use case — how to best add a custom script using GTM without using Custom HTML internal tag.</p><h3>Adding a custom script</h3><p>In digital analytics work, it is very common to add scripts with a JS library to a website pages.</p><p>A client comes and asks to add something like this:</p><pre>&lt;script async src=&quot;https://www.googletagmanager.com/gtag/js?id=G-1&quot;&gt;&lt;/script&gt;</pre><p>In this case, it is the GA4 library, but in reality, it could be any script link from your vendor.</p><p>When there is no built-in tag from the vendor, people typically go and add it via Custom HTML — which is not the good solution.</p><h4>GTM Custom Template</h4><p>The best practice is to use the in-built Custom Template builder of Google Tag Manager.</p><p>It is a feature that allows you to create a tag using an isolated environment with built-in security rules and a specific JavaScript.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0PDGOEoZT0ct6Q4wnR5c1g.jpeg" /></figure><p>Custom templates allow you to use powerful APIs that are convenient to work with and obviously help address most simple to moderately complex digital analytics tasks.</p><p>Moreover, this kind of custom tag created by you is much easier to debug with the GTM Tag Assistant.</p><p>More information can be found here: <a href="https://developers.google.com/tag-platform/tag-manager/templates">https://developers.google.com/tag-platform/tag-manager/templates</a>.</p><h4>injectScript API instead Custom HTML</h4><p>To add a script to a page, you need to use the injectScript API.</p><p><a href="https://developers.google.com/tag-platform/tag-manager/templates/api#injectscript">Google has pretty good documentation</a>; however, if you want to obtain an already created custom template, you can read the article further. I will share it in the end.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9kN3UhtMNYVOAX2Yudh-wg.png" /></figure><p>As you can see, it has fairly straightforward parameters, but it should be sufficient for most relatively simple cases.</p><blockquote>The biggest downside for me right now is the inability to add a custom attribute to the script. This would be a really useful feature for working with OneTrust and other CMS systems that have built-in auto-blocking.</blockquote><p>So, you are not able to add something with additional attributes using injectScript API, like:</p><pre>&lt;script async data-ot-ignore src=&quot;https://www.googletagmanager.com/gtag/js?id=G-1&quot;&gt;&lt;/script&gt;</pre><p>I used Custom HTML tag for that.</p><p>Regarding the injectScript API, I have created a Custom Template for my casual daily routines.</p><p>It should not be a big challenge to modify it a bit and adjust it for any vendor. I hope it will be valuable to you as well.</p><h3>Script injector tag</h3><p>If you do not want to spend time building your own template, you can import mine from <a href="https://github.com/shulhat/script-injector">https://github.com/shulhat/script-injector</a>.</p><p>When you add it via the Custom Template Builder, you will be able to add a tag that will look like this.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Uor5O7rtUEtH-9CAmWDOzg.png" /></figure><blockquote>An important point is that you will need <strong>to modify the Permissions</strong> for your injected URL script.</blockquote><blockquote>Unfortunately, this must always be adjusted manually.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WU_imnHFU0FrHU5j1o2t0w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5NL6HQLJ0erueHkk2Q247Q.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ob-Zzasb7CX10EPpjNvVIw.png" /></figure><p>If you did everything correctly, you should see something like this in the Elements section of Chrome DevTools:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RDuPRxsJ8HmWhw9eO67pfw.png" /></figure><p>In the Tag Assistant, you will see the tag successfully fired.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NDXKAxCnHv5cYc8MN-h4fg.png" /></figure><p>The GA4 library here is just an example — I believe you will be able to adjust it for any other vendor.</p><p>In my daily practice, I often need to add a script with parameters to the URL, so I customized it to meet this need.</p><h4>Download the Script injector</h4><p>I have added the template to the community <a href="https://tagmanager.google.com/gallery/#/">https://tagmanager.google.com/gallery/#/</a>—hope Google approves it soon.</p><p>Until then, you can find it here: <a href="https://github.com/shulhat/script-injector">https://github.com/shulhat/script-injector</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5896078f3169" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Meta init: How to send user data to Facebook?]]></title>
            <link>https://medium.com/@w.anatolii/meta-init-how-to-send-user-data-to-facebook-568a9ceac971?source=rss-4fa1a98ff31d------2</link>
            <guid isPermaLink="false">https://medium.com/p/568a9ceac971</guid>
            <category><![CDATA[web-analytics]]></category>
            <category><![CDATA[facebook]]></category>
            <category><![CDATA[meta]]></category>
            <category><![CDATA[digital-marketing]]></category>
            <category><![CDATA[tag-manager]]></category>
            <dc:creator><![CDATA[Anatolii Shulhat]]></dc:creator>
            <pubDate>Tue, 31 Dec 2024 10:08:25 GMT</pubDate>
            <atom:updated>2025-01-10T19:45:44.840Z</atom:updated>
            <content:encoded><![CDATA[<h3>Intro</h3><p>Recently, I faced an issue with sending user data via the Meta Pixel.</p><p>There is a setup through GTM for several Facebook events, but the stakeholder says that the user data is not coming through.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g1C6GQHHpjbzABfH-CTf0g.jpeg" /></figure><p>I noticed that an unofficial Facebook tag template made by Simo was being used.</p><p>I saw that in Tag Assistant, all the variables were present — the tag displayed them, but in the request in the network tab, I could see that the user parameters were missing for some of the events.</p><p>Something was definitely wrong here. So, my first thought was that the template had become outdated.</p><p>After playing around with the Pixel, reading the documentation, analyzing the template code, and doing a few local experiments, I realized that the problem was not with the template itself, but with the expectation of how the data should be sent.</p><p>In my case, the problem was complicated by the fact that I had a Single Page Application and a funnel where user data appears and gets supplemented from event to event until the final conversion.</p><blockquote>Most likely, if your user data is not being sent along with the event but you are sure that the data exists on the page, the issue is likely in how the init call works in your setup.</blockquote><p>And this is exactly what this article will be about.</p><h3>What is wrong with user data and init?</h3><p>The problem can be briefly explained with the following code.</p><p>Let’s assume we have a Single Page Application with 3 forms that a user can fill out: for example, a registration form, a begin checkout form, and a final purchase form.</p><p>If you run the following dummy script on a single page, you will simulate a similar user journey.</p><pre>&lt;!-- Meta Pixel Code --&gt;<br>&lt;script&gt;<br>  <br>// Step 1<br>// Fire basic pixel code with an empty init and PageView event<br>  <br>  <br>!function(f,b,e,v,n,t,s)<br>{if(f.fbq)return;n=f.fbq=function(){n.callMethod?<br>n.callMethod.apply(n,arguments):n.queue.push(arguments)};<br>if(!f._fbq)f._fbq=n;n.push=n;n.loaded=!0;n.version=&#39;2.0&#39;;<br>n.queue=[];t=b.createElement(e);t.async=!0;<br>t.src=v;s=b.getElementsByTagName(e)[0];<br>s.parentNode.insertBefore(t,s)}(window, document,&#39;script&#39;,<br>&#39;https://connect.facebook.net/en_US/fbevents.js&#39;);<br>fbq(&#39;init&#39;, &#39;123456789&#39;);<br>fbq(&#39;track&#39;, &#39;PageView&#39;);<br>&lt;/script&gt;<br>&lt;noscript&gt;&lt;img height=&quot;1&quot; width=&quot;1&quot; style=&quot;display:none&quot;<br>src=&quot;https://www.facebook.com/tr?id=123456789&amp;ev=PageView&amp;noscript=1&quot;<br>/&gt;&lt;/noscript&gt;<br>&lt;!-- End Meta Pixel Code --&gt;<br><br>&lt;script&gt;<br><br>// Step 2<br>// A user finishes sign-up and we have some info about them<br><br>setTimeout(function() {<br>    fbq(&#39;init&#39;, &#39;123456789&#39;, {<br>      em: &#39;digital_analyst@email.com&#39;,<br>      fn: &#39;anatolii&#39;,<br>      ln: &#39;shulhat&#39;<br>    });<br>}, 1000);<br><br>setTimeout(function() {<br>    fbq(&#39;track&#39;, &#39;CompleteRegistration&#39;);<br>}, 2000);<br><br>// Step 3<br>// A user starts checkout and we have some new info about them<br><br>setTimeout(function() {<br>    fbq(&#39;init&#39;, &#39;123456789&#39;, {<br>      em: &#39;digital_analyst@email.com&#39;,<br>      fn: &#39;anatolii&#39;,<br>      ln: &#39;shulhat&#39;,<br>      db: &#39;22021994&#39;,<br>      country: &#39;usa&#39;<br>    });<br>}, 3000);<br><br>setTimeout(function() {<br>    fbq(&#39;track&#39;, &#39;InitiateCheckout&#39;);<br>}, 4000);<br><br>// Step 4<br>// A user finishes purchase and we have full possible info about them<br><br>setTimeout(function() {<br>    fbq(&#39;init&#39;, &#39;123456789&#39;, {<br>      em: &#39;digital_analyst@email.com&#39;,<br>      fn: &#39;anatolii&#39;,<br>      ln: &#39;shulhat&#39;,<br>      db: &#39;22021994&#39;,<br>      country: &#39;usa&#39;,<br>      st: &#39;ny&#39;,<br>      ct: &#39;newyork&#39;,<br>      zp: &#39;10002&#39;<br>    });<br>}, 5000);<br><br>setTimeout(function() {<br>    fbq(&#39;track&#39;, &#39;Purchase&#39;);<br>}, 6000);<br>&lt;/script&gt;</pre><p>I expected to see each event accompanied by a new set of user data — at least, the code seemed to align with my expectations.</p><p>However, the reality is different. Meta will not receive the events with all the defined user parameters.</p><blockquote>If you go to the <strong>Network</strong> tab in your browser and filter the requests using tr/?id=, you will be able to inspect which events are being sent to Facebook and which parameters are included in the request.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Os5KNzLVQDTjFigOilnH7g.png" /></figure><p>All user data parameters will be defined here as ud[].</p><p>Looks like we do not have all user data here.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*h-9dLjSqKZwOzROu1P72Fw.jpeg" /></figure><p>The outcome is a real surprise — and a nightmare for SPA owners — especially if you expect to send user data in a smooth, continuous flow to Meta.</p><h3>Meta Advanced Matching</h3><p>Let’s start from the beginning and discuss why this happens.</p><p>To correctly send user data, the first thing you need to do is read Meta’s documentation on <strong>Advanced Matching</strong>:<br><a href="https://developers.facebook.com/docs/meta-pixel/advanced/advanced-matching/">https://developers.facebook.com/docs/meta-pixel/advanced/advanced-matching/</a>.</p><h4>The tricky init call</h4><p>A key point is the use of fbq(&#39;init&#39;). This is where the process of transmitting user data begins, as this call is responsible for sending user data to be processed by Facebook&#39;s library.</p><p>Usually, the init call is empty, with no parameters added for the base pixel, unless you need to send something with the P<strong>ageView</strong> event.</p><pre>&lt;!-- Meta Pixel Code --&gt;<br>&lt;script&gt;<br>!function(f,b,e,v,n,t,s)<br>{if(f.fbq)return;n=f.fbq=function(){n.callMethod?<br>n.callMethod.apply(n,arguments):n.queue.push(arguments)};<br>if(!f._fbq)f._fbq=n;n.push=n;n.loaded=!0;n.version=&#39;2.0&#39;;<br>n.queue=[];t=b.createElement(e);t.async=!0;<br>t.src=v;s=b.getElementsByTagName(e)[0];<br>s.parentNode.insertBefore(t,s)}(window, document,&#39;script&#39;,<br>&#39;https://connect.facebook.net/en_US/fbevents.js&#39;);<br>fbq(&#39;init&#39;, {{Meta Pixel ID}});<br>fbq(&#39;track&#39;, &#39;PageView&#39;);<br>&lt;/script&gt;<br>&lt;noscript&gt;&lt;img height=&quot;1&quot; width=&quot;1&quot; style=&quot;display:none&quot;<br>src=&quot;https://www.facebook.com/tr?id={{Meta Pixel ID}}&amp;ev=PageView&amp;noscript=1&quot;<br>/&gt;&lt;/noscript&gt;<br>&lt;!-- End Meta Pixel Code --&gt;</pre><p>Typically, a second init call is made when user data parameters become available. For example, when a registration form is submitted, or when a user_data event is present in the data layer after a user logs in. It could also be any other flag that indicates, “Okay, user data is now on the page, and we can scrape it and send it.”</p><pre>fbq(&#39;init&#39;, &#39;123456789&#39;, {<br>  em: &#39;digital_analyst@email.com&#39;,<br>  fn: &#39;anatolii&#39;,    <br>  ln: &#39;shulhat&#39;<br>});</pre><p>The trickiest part comes when you fire a non-empty init call with user parameters (usually this is the second init on the page).</p><blockquote>All subsequent page events will carry the user parameters defined in that second init call — it is the second init on the page, or the first non-empty init call.</blockquote><p>If you send something like the following script on the same page, and this could be the second non-empty init call, it will not make any difference, because the Facebook Pixel will not pick up these parameters.</p><pre>fbq(&#39;init&#39;, &#39;123456789&#39;, {<br>      em: &#39;digital_analyst@email.com&#39;,<br>      fn: &#39;anatolii&#39;,<br>      ln: &#39;shulhat&#39;,<br>      db: &#39;22021994&#39;,<br>      country: &#39;usa&#39;,<br>      st: &#39;ny&#39;,<br>      ct: &#39;newyork&#39;,<br>      zp: &#39;10002&#39;<br>    });</pre><p>It does not seem very obvious.</p><p>Especially, this is not obvious if you are using third-party templates and do not have full control over the code execution process.</p><p>In Tag Assistant, everything might look fine, but Meta will not actually receive these user parameters.</p><p>So, how can this be fixed?</p><h3>How to send all user data to Meta?</h3><p>If you perform reverse engineering on the Facebook Pixel code, you can find that user data is stored in the object:</p><p>window.fbq.instance.pixelsByID[123456789].userData,</p><p>where 123456789 is your Meta Pixel ID.</p><p>You can easily verify this through the browser’s console.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sJhJgMvS98OpdUCB-BVqBw.png" /></figure><p>If we can populate this object, then we will likely be able to send better data to Meta.</p><p>Let’s go back to my initial dummy script that simulates the user journey within the page or SPA, and modify it a bit for the sake of the experiment.</p><pre>&lt;!-- Meta Pixel Code --&gt;<br>&lt;script&gt;<br>  <br>// Step 1<br>// Fire basic pixel code with an empty init and PageView event<br>  <br>!function(f,b,e,v,n,t,s)<br>{if(f.fbq)return;n=f.fbq=function(){n.callMethod?<br>n.callMethod.apply(n,arguments):n.queue.push(arguments)};<br>if(!f._fbq)f._fbq=n;n.push=n;n.loaded=!0;n.version=&#39;2.0&#39;;<br>n.queue=[];t=b.createElement(e);t.async=!0;<br>t.src=v;s=b.getElementsByTagName(e)[0];<br>s.parentNode.insertBefore(t,s)}(window, document,&#39;script&#39;,<br>&#39;https://connect.facebook.net/en_US/fbevents.js&#39;);<br>fbq(&#39;init&#39;, &#39;123456789&#39;);<br>fbq(&#39;track&#39;, &#39;PageView&#39;);<br>&lt;/script&gt;<br>&lt;noscript&gt;&lt;img height=&quot;1&quot; width=&quot;1&quot; style=&quot;display:none&quot;<br>src=&quot;https://www.facebook.com/tr?id=123456789&amp;ev=PageView&amp;noscript=1&quot;<br>/&gt;&lt;/noscript&gt;<br>&lt;!-- End Meta Pixel Code --&gt;<br><br>&lt;script&gt;<br><br>// Step 2<br>// A user finishes sign-up and we have some info about them<br><br>setTimeout(function() {<br>    fbq(&#39;init&#39;, &#39;123456789&#39;, {<br>      em: &#39;digital_analyst@email.com&#39;,<br>      fn: &#39;anatolii&#39;,<br>      ln: &#39;shulhat&#39;<br>    });<br>}, 1000);<br><br>setTimeout(function() {<br>    fbq(&#39;track&#39;, &#39;CompleteRegistration&#39;);<br>}, 2000);<br><br>// Step 3<br>// A user starts checkout and we have some new info about them<br><br>setTimeout(function() {<br>    window.fbq.instance.pixelsByID[123456789].userData.db = &#39;22021994&#39;;<br>    window.fbq.instance.pixelsByID[123456789].userData.country = &#39;usa&#39;;<br>}, 3000);<br><br>setTimeout(function() {<br>    fbq(&#39;track&#39;, &#39;InitiateCheckout&#39;);<br>}, 4000);<br><br>// Step 4<br>// A user finishes purchase and we have full possible info about them<br><br>setTimeout(function() {<br>    window.fbq.instance.pixelsByID[123456789].userData.st = &#39;ny&#39;;<br>    window.fbq.instance.pixelsByID[123456789].userData.ct = &#39;newyork&#39;;<br>    window.fbq.instance.pixelsByID[123456789].userData.zp = &#39;10002&#39;;<br>}, 5000);<br><br>setTimeout(function() {<br>    fbq(&#39;track&#39;, &#39;Purchase&#39;);<br>}, 6000);<br><br>&lt;/script&gt;</pre><p>After executing this code, you can check in the <strong>Network</strong> tab to see that the final event on the page now includes the user parameters that we intended to send to Meta.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RX5UAXkFpHWHqGkTrE_qWw.png" /></figure><p>Overall, this approach allows us to achieve what we need.</p><p>If you are working with the dataLayer and want to automate a bit the tagging process, you can use an approach like the following:</p><ol><li>Create a <strong>Custom HTML tag</strong> and, using <strong>Tag Sequencing</strong>, make sure it runs as a setup tag before each event.</li></ol><pre>&lt;script&gt;<br>  // Flag to enable/disable logs<br>  var logEnabled = true;<br><br>  // Extracting the values from the data layer<br>  var email = {{DLV - em}};<br>  var firstName = {{DLV - fn}};<br>  var lastName = {{DLV - ln}};<br>  var phone = {{DLV - ph}};<br>  var dateOfBirth = {{DLV - db}};<br>  var state = {{DLV - st}};<br>  var city = {{DLV - ct}};<br>  var zip_code = {{DLV - zp}};<br>  var country = {{DLV - country}};<br>  var externalId = {{DLV - external_id}};<br>  var gender = {{DLV - ge}};<br><br>  // Function to log only if logEnabled is true<br>  function customLogGtm(message, value) {<br>    if (logEnabled) {<br>      console.log(&quot;GTM script QA: &quot;, message, value);<br>    }<br>  }<br>  <br>  // Logging each variable to the console<br>  customLogGtm(&quot;email:&quot;, email);<br>  customLogGtm(&quot;firstName:&quot;, firstName);<br>  customLogGtm(&quot;lastName:&quot;, lastName);<br>  customLogGtm(&quot;phone:&quot;, phone);<br>  customLogGtm(&quot;dateOfBirth:&quot;, dateOfBirth);<br>  customLogGtm(&quot;state:&quot;, state);<br>  customLogGtm(&quot;city:&quot;, city);<br>  customLogGtm(&quot;zip_code:&quot;, zip_code);<br>  customLogGtm(&quot;country:&quot;, country);<br>  customLogGtm(&quot;externalId:&quot;, externalId);<br>  customLogGtm(&quot;gender:&quot;, gender);<br><br>  // Define a Facebook Pixel ID<br>  var definedPixelIds = {{Facebook Pixel Lookup table}};<br>  customLogGtm(&quot;Defined Pixel IDs:&quot;, definedPixelIds);<br><br>  // Loop through the fbq instance and modify the user data for each defined Pixel ID<br>  for (var pixelId in window.fbq.instance.pixelsByID) {<br>    if (window.fbq.instance.pixelsByID.hasOwnProperty(pixelId)) {<br>      var pixelObj = window.fbq.instance.pixelsByID[pixelId];<br><br>      // Only process if the pixelId is in the definedPixelIds array<br>      if (definedPixelIds.indexOf(pixelId) !== -1) {<br>        if (!pixelObj.userData) {<br>          pixelObj.userData = {};<br>        }<br>        var pixelUserData = pixelObj.userData;<br><br>        // Set the value if undefined, null, or empty string<br>        if (!pixelUserData.em &amp;&amp; email) pixelUserData.em = email;<br>        if (!pixelUserData.fn &amp;&amp; firstName) pixelUserData.fn = firstName;<br>        if (!pixelUserData.ln &amp;&amp; lastName) pixelUserData.ln = lastName;<br>        if (!pixelUserData.ph &amp;&amp; phone) pixelUserData.ph = phone;<br>        if (!pixelUserData.db &amp;&amp; dateOfBirth) pixelUserData.db = dateOfBirth;<br>        if (!pixelUserData.st &amp;&amp; state) pixelUserData.st = state;<br>        if (!pixelUserData.ct &amp;&amp; city) pixelUserData.ct = city;<br>        if (!pixelUserData.zp &amp;&amp; zip_code) pixelUserData.zp = zip_code;<br>        if (!pixelUserData.country &amp;&amp; country) pixelUserData.country = country;<br>        if (!pixelUserData.external_id &amp;&amp; externalId) pixelUserData.external_id = externalId;<br>        if (!pixelUserData.ge &amp;&amp; gender) pixelUserData.ge = gender;<br><br>        // Log user data after setting it<br>        customLogGtm(&quot;User data after setting:&quot;, pixelUserData);<br>      }<br>    }<br>  }<br>&lt;/script&gt;</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*chYFDTAMCrsJ-KgSSskTew.png" /></figure><p>2. Before the <strong>Hard Set</strong>, make sure the standard init call runs first.</p><pre> fbq(&#39;init&#39;, &#39;123456789&#39;, {<br>      em: &#39;digital_analyst@email.com&#39;,<br>      fn: &#39;anatolii&#39;,<br>      ln: &#39;shulhat&#39;,<br>      db: &#39;22021994&#39;,<br>      country: &#39;usa&#39;,<br>      st: &#39;ny&#39;,<br>      ct: &#39;newyork&#39;,<br>      zp: &#39;10002&#39;<br>    });</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5WqE-vO4ISYEFaU9tTA2jQ.png" /></figure><p>With this setup, you can be confident that user data is updated and sent to Meta before each event.</p><p>Here’s how the ideal process could work:</p><ol><li>The user takes an action (e.g., submits a form) on the website.</li><li>User data is pushed to the <strong>dataLayer</strong>. (Ask your dev team to ensure this is properly implemented.)</li><li>The standard init call is fired.</li><li>The <strong>Hard Set custom script</strong> checks whether the Facebook Pixel has all the available user data.</li><li>If the data layer contains the data but the Facebook Pixel object on the page does not, the script performs a <strong>hard set</strong> to populate the missing values.</li><li>Finally, the event, now enriched with the correct user data, is sent to Meta.</li></ol><p>Do not forget to disable logs after you are sure everything is working as expected.</p><h3>Normilizing and hashing</h3><p>When you have set up user data transmission through the init call and examine the request after initializing the user data, you will notice that the values will be automatically hashed by the Pixel using SHA-256.</p><p>In fact, there are a few additional surprises when it comes to data normalization. Before the data is hashed, it is first normalized.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/896/1*rMIeEls9tKyU0gijLQ88Yw.jpeg" /></figure><p>It is great if you have read the documentation and control this process yourself, but Meta’s library will handle it for you if you happen to send user data in a way that does not follow their official recommendations.</p><p>For example, if you send:</p><p><strong>Anatolii Test</strong></p><p>The frontend Meta Pixel will normalize this to:</p><p><strong>anatoliitest</strong></p><p>Then, the Pixel will automatically hash this value, and you will see:</p><p><strong>5701966a9d7dfe09448a36c905ab347c5a4c8f242d8f1e11853cee3aac005ce2</strong></p><p>I highly recommend experimenting with hashing here: <a href="https://emn178.github.io/online-tools/sha256.html">SHA-256 Online Tool</a> and reading the documentation: <a href="https://developers.facebook.com/docs/marketing-api/conversions-api/parameters/customer-information-parameters/">Facebook Marketing API Documentation</a>.</p><p>If you are sending data from both the backend and frontend and using deduplication, this could be a serious issue.</p><p>I hope this article and my findings prove valuable to you.</p><p>This is a good foundation for a new GTM Meta custom template suited for SPAs and complex user data sending cases. I hope to code it eventually, but for now, it is just custom HTML notes.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=568a9ceac971" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to extract and consolidate Click IDs, Query Parameters & Cookies using GTM]]></title>
            <link>https://medium.com/@w.anatolii/how-to-extract-and-consolidate-click-ids-query-parameters-cookies-using-gtm-ddc2390a5261?source=rss-4fa1a98ff31d------2</link>
            <guid isPermaLink="false">https://medium.com/p/ddc2390a5261</guid>
            <category><![CDATA[performance-marketing]]></category>
            <category><![CDATA[google-tag-manager]]></category>
            <category><![CDATA[server-side-tracking]]></category>
            <category><![CDATA[apics]]></category>
            <category><![CDATA[cookies]]></category>
            <dc:creator><![CDATA[Anatolii Shulhat]]></dc:creator>
            <pubDate>Sun, 15 Dec 2024 11:46:19 GMT</pubDate>
            <atom:updated>2024-12-15T11:46:19.729Z</atom:updated>
            <content:encoded><![CDATA[<h3>Intro</h3><p>More and more often, I have to work with CAPI and set up conversion tracking. These days, businesses increasingly try to use server-side or backend data sending to improve marketing accuracy and efficiency.</p><p>When working with CAPI from different marketing vendors, it’s often necessary to send click IDs and cookie values along with conversions to associate them with a specific user.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DfKtI6pWjv7jItdf_Ikeqg.jpeg" /><figcaption>Cookies</figcaption></figure><p>For example, if we look at Snapchat’s documentation, we can see that they require sending the parameter sc_click_id, which should contain the value from ?ScCid= available in the browser’s query parameter. This value is added to the page URL when a user clicks on a Snapchat ad.</p><p>In addition to the click ID, they also ask to include sc_cookie1, which holds the value of a first-party cookie _scid set under your domain. This serves as a kind of user ID or browser ID associated with a pseudonymous user.</p><p>If we look at Meta’s CAPI, we can see that Facebook also asks for something similar — the fbc parameter as the click ID, which can be retrieved by parsing the fbclid query parameter or by parsing the _fbc cookie if the pixel is added.</p><p>As for the browser ID, or the user ID — Facebook requests the addition of the fbp parameter, which is generated by the pixel through the _fbp cookie.</p><blockquote>In general, a similar framework exists with most performance marketing advertisers. They all require either both click and user values, or at least one of them, if you want to use CAPI for sending conversions. This is necessary for proper user stitching — one of the factors for high-quality data collection.</blockquote><p>And if the data collection setup is complex — <strong>with various events, page views not being tracked, limited or legacy backend, no GTM server-side container, or any other specific use case</strong> — you might want to simplify the process and have all the necessary data in a structured format on the frontend. This way, it would be easier to parse them using a GTM and send them whenever needed.</p><p>In this article, I would like to propose a solution that will allow parsing and consolidating click IDs and cookie values by creating new cookies with a clear structure available on the frontend.</p><blockquote>These new cookies can then be saved and used on the backend or server-side when necessary to send high-quality CAPI requests.</blockquote><p>My goal here is to cover just part of the data collection and consolidation process — the sending of this data is best handled with your developers.</p><h3>Custom GTM template: parse query params and cookies</h3><p>I decided to handle parsing and consolidation through the creation of a GTM custom template.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X96lFiLyQAyVuRqy_l3Kiw.png" /><figcaption>A GTM custom template tag to parse query parameters and cookie values.</figcaption></figure><p>My goal was to create a tag where you provide a list of query parameter names and a list of cookie names, and it reads their values, <strong>creating new cookies with a clear key-value pair structure for future use.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*806qXLnkWNX23qLE1qiDBQ.png" /><figcaption>How cookies will look in the browser after a successful tag is fired</figcaption></figure><h4>Key-value pairs to store Click IDs and User IDs in cookies</h4><p>As I mentioned, almost all marketing vendors work with two entities — clicks and users (or browsers). Therefore, there should be at least two cookies: one for consolidating the click ID from query parameters, and another for consolidating the user or browser ID from other cookies, which are typically created by pixels.</p><p>A long time ago, there was a popular script among web analysts for creating the Initial Traffic Source cookie from Luna Metrics. If you’re interested in revisiting it, you can check it out here: <a href="https://www.analyticsmania.com/post/track-the-initial-traffic-source-of-the-visitor-gtm/">https://www.analyticsmania.com/post/track-the-initial-traffic-source-of-the-visitor-gtm/</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NjETF5dN_7BRFZaCjumLnA.png" /><figcaption>Legacy ITS script by Lunametrics</figcaption></figure><p>This script was used on many websites and had a peculiar logic that was difficult to modify if you wanted to add something new. However, <strong>this script stores all values in a fairly straightforward structure using key-value pairs separated by |, which has served as the foundation for my approach.</strong></p><p>So, in the middle of journey, I knew that there should be cookies in the form of</p><blockquote>c_ids for clicks</blockquote><p>and</p><blockquote>u_ids for users or browsers</blockquote><p>and</p><blockquote>the values like ScCid=test123|fbclid=test123|twclid=Xtest</blockquote><p>Of course, marketing pixels do not have a perfect consistent logic, and it’s impossible to tailor everything for every case. Sometimes, there might be cases where clicks are recorded in u_ids. For example, the _fbc cookie from Meta. This cookie is for a consistent click ID, but it’s extracted from a different cookie. However, in most cases, cookies will contain entities related to the user, which is why this naming convention is used.</p><h4>First and last click attribution</h4><p>During the work, I also came up with the idea of adding attribution.</p><p>If it’s important for the business to have only the first ad touch in the browser, we can create an initial group of cookies. Every time the user clicks an ad, the click ID or user ID value will be recorded in this group and permanently tied to the browser. Only new vendor values can be added to these cookies. If the vendor’s click is already present in the cookie, it should not be updated.</p><p>If we want to work with the last-click attribution model and only record the most recent relevant touch, we need to use the “last” group. In this case, each time a marketing vendor’s click and user values will be overwritten. For the most classic and accurate data collection, this is the best option if you’re sending the click in near real-time.</p><p>So, we should have, at the end, cookies like this:</p><blockquote>initial_c_ids, last_c_ids for clicks</blockquote><p>and</p><blockquote>initial_u_ids, last_u_ids for users or browsers</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*806qXLnkWNX23qLE1qiDBQ.png" /><figcaption>How cookies will look in the browser after a successful tag is fired</figcaption></figure><h4>QA logs to debug the script</h4><p>Well, where would we be without logs in the console? It’s essential to understand what’s happening and why. It’s a convenient mechanism for the transparency of the template’s operation and the QA process.</p><p>Every script added to the site must work transparently and have visibility tools. As each new logic is added, it becomes more challenging to understand what’s going on, so to save time, it’s always helpful to have native logs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iqOKjtfB31T40eXUw2Eu8g.png" /></figure><h4>CAPI cookies lifetime</h4><p>Different businesses have different windows for sending conversions. Sometimes, there’s complex business logic for working with affiliate partners. Therefore, it would be great to have the ability to set the expiration time for cookie data.</p><p>In my template, this option is available.</p><h4>How to work with a custom template</h4><p>The configuration will entirely depend on your GTM container and frontend setup and may vary.</p><p>I would recommend importing and creating it with the “All Page Views” trigger as the default; however, if you have your own use cases, you can tailor it to your needs.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*X96lFiLyQAyVuRqy_l3Kiw.png" /><figcaption>A GTM custom template tag to parse query parameters and cookie values.</figcaption></figure><p>I have noticed that some pixel cookies require time to be updated in the browser before they can be scraped by my custom template.</p><p>If it is important for you to consolidate cookies immediately, you can create an additional custom HTML tag with something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AHkGTkhM45mx7kBHXnn9tw.png" /><figcaption>Extra script to adjust cookie update timing</figcaption></figure><blockquote>&lt;script&gt;<br> setTimeout(function() {<br> window.dataLayer = window.dataLayer || [];<br> window.dataLayer.push({<br> event: ‘update_consolidated_ids’,<br> });<br> }, 1000); // 1000 ms = 1 second delay<br>&lt;/script&gt;</blockquote><p>Just adjust the timing as needed and create a Custom Event trigger to fire my previously described custom template. This should update the u_ids cookies more accurately. Test it to ensure it works as expected.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VHeE9IGc3zGLlOlxsNEJYA.png" /><figcaption>A new potential trigger</figcaption></figure><h3>Resources</h3><p>You can download the template here: <a href="https://github.com/shulhat/clickid-cookie-consolidator">https://github.com/shulhat/clickid-cookie-consolidator</a>.</p><p>I have submitted it as a community template and am waiting for Google’s approval.</p><p>I will add a link to the gallery entry here as soon as Google reviews it and publishes it globally.</p><p>In my opinion, the script is not limited to working with CAPI. I think it will also be useful for various affiliate tracking, where the storage and consolidation of query parameters are important. I hope this template will help simplify the routine of consolidating various parameters and cookies.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ddc2390a5261" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Days Since Last Session in GA4 with GTM]]></title>
            <link>https://medium.com/@w.anatolii/days-since-last-session-in-ga4-with-gtm-e75b58c6ad42?source=rss-4fa1a98ff31d------2</link>
            <guid isPermaLink="false">https://medium.com/p/e75b58c6ad42</guid>
            <category><![CDATA[google-analytics]]></category>
            <category><![CDATA[google-analytics-4]]></category>
            <category><![CDATA[google-tag-manager]]></category>
            <category><![CDATA[custom-templates]]></category>
            <category><![CDATA[tag]]></category>
            <dc:creator><![CDATA[Anatolii Shulhat]]></dc:creator>
            <pubDate>Wed, 31 Jul 2024 23:22:18 GMT</pubDate>
            <atom:updated>2024-08-01T00:39:39.602Z</atom:updated>
            <content:encoded><![CDATA[<h3>Time Since Last Session — Intro</h3><p>It’s been a while since I wrote my previous article, which means I’ve accumulated some new content.</p><p>I am currently working on a project, the main goal of which is to migrate from Adobe Analytics to GA4.</p><p>To my surprise, I found that GA4 does not have a built-in dimension that would show how much time has passed since the previous session.</p><p><strong>In Adobe, this is Days since last visit.</strong></p><p><strong>In Universal Analytics, it is Days since last session.</strong></p><p><strong>And GA4 asks us to use BigQuery and build this dimension ourselves.</strong></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cY6MjHqv_wcwEIKNdN0FzQ.png" /></figure><p>However, I often encounter projects that do not have the ability to use and actively support BQ, and therefore do not see the point in setting it up. There are projects that want quick wins, but not a complex data setup.</p><p>They often use GA4 UI only.</p><p>So I started thinking about how this could be done with client-side GTM and GA4 tag.</p><p>There must definitely be a way.</p><p>After playing around with HTML tags and cookies, I decided that all this needs to be thrown into a template.</p><p>Especially since there is a cool opportunity to pull the current session ID through the gtag API, which somewhat simplifies the dance with reading and setting cookies.</p><h3>Custom Template Output</h3><p>I decided that the most convenient approach would be to create a tag manager template that would push the time since the last session to the data layer.</p><p>It would be great if every moment the user views a page, I could see an event in the data layer with data about the current and previous session — theirs IDs and sequential numbers:</p><blockquote>current_session_id</blockquote><blockquote>previous_session_id</blockquote><blockquote>current_session_number</blockquote><blockquote>previous_session_number</blockquote><p>Also, I need a series of metrics in different time units, showing how much time has passed between the start of the sessions.</p><blockquote>mins_since_last_session</blockquote><blockquote>days_since_last_session</blockquote><blockquote>weeks_since_last_session</blockquote><blockquote>monthes_since_last_session</blockquote><p>The most useful part for me is the ability to quickly obtain similar recency groups as in Adobe Analytics.</p><blockquote>adobe_recency_buckets</blockquote><p>And as a bonus for people who are nostalgic for Universal Analytics, I want to add similar days since last session buckets.</p><blockquote>universal_recency_buckets</blockquote><p>While I was working on the template, I realized that it would be good to have some options for debugging how the script gets and sets the cookie.</p><blockquote>Disable QA dataLayer events</blockquote><p>It would also be beneficial to have the ability to control whether it sends empty fields or specific text in the case when there is no information about the previous session.</p><blockquote>No empty string in the main event</blockquote><p>As a result, I created the following tag where you can control which fields you write into the data layer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sa42ywfmAuTtwEkjbrCctg.png" /></figure><p>The result of this custom template is the following data layer custom event:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8ueqL-g1Ymn8b95MA_lDQA.png" /></figure><h3>How does the tag work?</h3><p>The most important thing that we need in order to generate all the fields described above is just two variables — the IDs of the previous and current session.</p><h4>What is a session ID in GA4?</h4><p>Essentially, it’s the timestamp of the beginning of the session.</p><p>The difference between two timestamps is what we need to create the required dimension.</p><p>Google allows the use of gtag API to retrieve the current session, but the previous session, if you don’t have BQ and GTM SS — can’t be simply obtained.</p><p>Therefore, we need to create a logic that would store the current session in a cookie and, at the initiation of a new session, would store it as the previous one and store the current session again.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1023/1*tgS6FyI9g83Z0L5JXcCTrg.png" /></figure><blockquote>Essentially, we always need to keep the current and previous sessions recorded in cookies.</blockquote><h4>Let’s look at what the script does</h4><ol><li>It retrieves the ID and number of the current session through the gtag API.</li><li>It reads a custom cookie that is supposed to store both the previous and current sessions.</li><li>If there is no previous and current session recorded — it records the current live one.</li><li>If the session saved as current does not match the live session returned by the gtag API — we shuffle the sessions. The one that was saved as current becomes the previous, and the new live session is saved as current.</li><li>Other conditions most likely mean that the user is still on the same session as before, thus there’s nothing to overwrite in the cookies.</li><li>Once the script has worked with the values in the cookies, it calculates the difference between the two session timestamps and converts the seconds into the necessary time values, or buckets, that are familiar in other analytics systems.</li><li>The final step should be an event with all the above-described parameters, which would allow creating a dimension about the time difference between sessions.</li></ol><blockquote>Look for <strong>since_last_session_data</strong> event in the dataLayer.</blockquote><blockquote>Create a custom event trigger to build your setup.</blockquote><h4>Additional options in the template</h4><p>The template works through cookies.</p><blockquote>This means that if the user is new, there is no information about them.</blockquote><p>We can leave all the respective fields empty or indicate that there is no data.</p><p>Personally, I find it more convenient to have the fields pre-filled, but someone else might find it more useful to have empty fields.</p><p>If you enable this field</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/278/1*tYIPHsILyAdtOmSVWeHuYA.png" /></figure><p>then all empty variables will be filled with the value ‘No previous session data’.</p><p>Another aspect is debugging.</p><p>It was important for me to test several workflows to ensure that the correct actions are taken at each stage.</p><p>If you are also interested in conducting QA work, I recommend enabling the corresponding option in the script.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/244/1*Q3WNiM_omYTTYhXqXjSkAQ.png" /></figure><p>You can get 6 additional events at the data level to be sure that everything is working okay.</p><ol><li>dlPush({<br> ‘event’: ‘get_cookies_with_session_data_qa’,<br> ‘storedPreviousSessionID’: storedPreviousSessionID,<br> ‘storedCurrentSessionID’: storedCurrentSessionID,<br> ‘liveCurrentSessionID’: liveCurrentSessionID<br> });</li><li>dlPush({<br> ‘event’: ‘store_cookies_with_session_data_qa’,<br> ‘store_type’: ‘no_previous_session_id_stored’,<br> ‘storedPreviousSessionID’: storedPreviousSessionID,<br> ‘storedCurrentSessionID’: storedCurrentSessionID,<br> ‘liveCurrentSessionID’: liveCurrentSessionID<br> });</li><li>dlPush({<br> ‘event’: ‘store_cookies_with_session_data_qa’,<br> ‘store_type’: ‘change_previously_stored_session_ids’,<br> ‘storedPreviousSessionID’: storedPreviousSessionID,<br> ‘storedCurrentSessionID’: storedCurrentSessionID,<br> ‘currentSessionId’: liveCurrentSessionID<br> });</li><li>dlPush({<br> ‘event’: ‘no_cookies_storing_qa’,<br> ‘store_type’: ‘the_same_session_is_running’,<br> ‘storedPreviousSessionID’: storedPreviousSessionID,<br> ‘storedCurrentSessionID’: storedCurrentSessionID,<br> ‘currentSessionId’: liveCurrentSessionID<br> });</li><li>dlPush({<br> event: ‘live_current_session_id_gtag_qa’,<br> gtagApiQA: liveCurrentSessionDataGtagObject<br> });</li><li>dlPush({<br> ‘event’: ‘seconds_since_last_session_qa’,<br> ‘timeDiff’: secondsSinceLastSession,<br> ‘currentSessionID’: currentSessionID,<br> ‘previousSessionID’: previousSessionID<br> });</li></ol><p>Most likely, once I understand that the template is working well enough and the data collection satisfies me, I will remove this part.</p><p>For now, as it is beta testing for me, I will leave the possibility to test some things as part of the functionality.</p><h3>Adobe Analytics Mapping</h3><p>In the adobe_recency_buckets field, the following values will be recorded:</p><ol><li>“Less than 1 day”</li><li>“1 to 3 days”</li><li>“3 to 7 days”</li><li>“7 to 14 days”</li><li>“14 days to 1 month”</li><li>“Longer than 1 month”</li></ol><p><a href="https://experienceleague.adobe.com/en/docs/analytics/components/dimensions/return-frequency">https://experienceleague.adobe.com/en/docs/analytics/components/dimensions/return-frequency</a>.</p><h3>Universal Analytics Mapping</h3><p>I have taken the following time distance buckets between sessions to map the logic of the previous Google Analytics version.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5Wf4v35UeUaP1Dm6u0JE3g.png" /></figure><p>Measured in days.</p><ol><li>“0”</li><li>“1”</li><li>“2”</li><li>“3”</li><li>“4”</li><li>“5”</li><li>“6”</li><li>“7”</li><li>“8–14”</li><li>“15–30”</li><li>“31–60”</li><li>“61–120”</li><li>“121–364”</li><li>“365+”</li></ol><p>For fans of previous analytics, in my opinion, this is a good solution to bring back this useful dimension.</p><h3>Custom Logic for Time Buckets</h3><p>The issue of cardinality in GA4 has not been eliminated, so I think most of GTM developers will want to create their own custom buckets based on the websites.</p><p>If you need your own bucket logic — I highly recommend using the required time unit from the data layer and grouping it independently by creating a RegEx Table variable.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*39LgpNfm5z82gmak8ixcnw.png" /></figure><p>RegEx will allow you to formulate any groups.</p><h3>How to setup the data collection?</h3><p>I propose to add it as a cleanup tag in Tag Sequences for GA4 config.</p><p>It should be fired after a Google Analytics 4 tag has completely fired; otherwise, the gtag GET API will not do anything.</p><p>I was inspired by <a href="https://www.simoahava.com/custom-templates/gtag-get-api/">https://www.simoahava.com/custom-templates/gtag-get-api/</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gn91VEw-smY13aq01GRIJg.png" /></figure><p>The custom template tag that does the main trick can be added like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*WyQxnI4LzCmEe3NdT7_Zag.png" /></figure><p>Where Page View event tag is being done like:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QYi9LRcEsVnHLpeLWvwInA.png" /></figure><p>This allows for a page view event with the new session data layer parameter to be captured.</p><p>In the simplest setup, you could have the following tags added:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fbZXpXBUuHQR1vwFuoji3A.png" /></figure><p>Be extremely careful with automatically collected events fired by gtag. I recommend disabling them in the GA4 data stream settings and controlling their firing in GTM.</p><h3>Resources</h3><p>You can download the template here: <a href="https://github.com/shulhat/time-since-last-session-gtag">https://github.com/shulhat/time-since-last-session-gtag</a>.</p><p>I have submitted it as a community template and am waiting for Google’s approval.</p><p>Since this is my first attempt at creating and sharing a GTM custom template, I do not know how much time it might take.</p><p>I will add a link to the gallery entry here as soon as Google reviews it and publishes it globally.</p><p>I plan to optimize it after some testing. Since I am not a JS developer, the code is far from perfection. But I hope it will help you anyway.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e75b58c6ad42" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Turn Off OneTrust Auto-Blocking to free up Google Tag Manager]]></title>
            <link>https://medium.com/@w.anatolii/how-to-turn-off-onetrust-auto-blocking-to-free-up-google-tag-manager-582860d40c7a?source=rss-4fa1a98ff31d------2</link>
            <guid isPermaLink="false">https://medium.com/p/582860d40c7a</guid>
            <category><![CDATA[gdpr]]></category>
            <category><![CDATA[consent-mode]]></category>
            <category><![CDATA[google-tag-manager]]></category>
            <category><![CDATA[onetrust]]></category>
            <category><![CDATA[google-analytics-4]]></category>
            <dc:creator><![CDATA[Anatolii Shulhat]]></dc:creator>
            <pubDate>Sun, 17 Dec 2023 12:48:25 GMT</pubDate>
            <atom:updated>2023-12-17T12:52:24.928Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to turn off OneTrust auto-blocking to free up Google Tag Manager</h3><h3>Advanced consent mode task and odd GTM script</h3><p>On one of the recent projects, I needed to configure Advanced Google Consent Mode. For this task, as a web analyst, I find it more convenient to work with Google Tag Manager, as it allows me to significantly reduce the load on the development team.</p><blockquote>As you know, Advanced Google Consent Mode requires tags, and in my case, Google Tag Manager, be executed each time a page loads without any blockers.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*lE7j77OszDV2sVIXeqgZkA.png" /></figure><p>I noticed that on the project website, Google Tag Manager does not function correctly if the visitor is new and has not yet interacted with the banner. To get the script to start working correctly at this stage and send the tags, it was necessary to accept consent for the Performance Cookies group through the OT banner.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HnG2LCRn9A2VmZ5oJ1pZqw.jpeg" /></figure><p>My first thought, as someone who didn’t set up OT on the site from scratch, was obviously something hardcoded by the developers. And perhaps it is necessary to write a technical task and ask them to change the logic.</p><p>Shortly, it is necessary to free up GTM from the dependency on consent for the OT performance cookies group. It was my thoughts then.</p><p><strong>I was somewhat surprised to learn that the reason is not the code written by the developers, but OneTrust’s cookie auto-blocking technology.</strong></p><h3><strong>OneTrust auto-blocking feature</strong></h3><p>At start OneTrust account owner most likely will use the OT tool to scan the entire site to find existing cookies and tracking scripts. OT will allow cataloging the scan results based on its huge database of possible cookies (near 11 million of cookies). Then, the OT account owner will set up the consent banner, choose the consent model for different countries if needed, and add the script to the site.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gBnPRbUIabVcWptby65vHw.png" /></figure><p>Next, the the owner will start choosing how to block groups of cookies identified in previous iterations. It will be possible to use more custom JavaScript wrappers, or use the GTM — customizing triggers for elements being sent to the data layer by OT.</p><blockquote>However, the simplest way for the OT account owner is to enable Auto-blocking feature.</blockquote><blockquote>It disables all tracking scripts until the user consents to the use of a certain group of cookies.</blockquote><p>For the web analyst who wants to set up Advanced Consent Mode — <strong>such auto-blocking is a problem</strong>, because Google Tag Manager doesn’t work as it should.</p><h4>What does auto-blocking look like at the code level of a rendered website?</h4><p>Most likely, when auto-blocking is enabled, if you go into the browser’s developer mode and look in the Elements tab at the scripts related to GTM — you will see something like this:</p><blockquote>&lt;script async=”” type=”text/plain” src=”<a href="https://www.googletagmanager.com/gtm.js?id=GTM-NLR6H6W&amp;amp;gtm_auth=KDsBauO9BfS2PwvcY1n5yg&amp;amp;gtm_preview=env-2&amp;amp;gtm_cookies_win=x">https://www.googletagmanager.com/gtm.js?id=GTM-XXXXXXX&amp;amp;gtm_auth=xxxxxxxxxxxxxxxxxxxxxx&amp;amp;gtm_preview=env-x&amp;amp;gtm_cookies_win=x</a>&quot; class=”optanon-category-C0002 “&gt;&lt;/script&gt;</blockquote><p><strong>Please, do not confuse with the code that was added from the container. It is not the same:</strong></p><blockquote>&lt;script&gt;(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({‘gtm.start’:new Date().getTime(),event:’gtm.js’});var f=d.getElementsByTagName(s)[0],j=d.createElement(s),dl=l!=’dataLayer’?’&amp;l=’+l:’’;j.async=true;j.src=’<a href="https://www.googletagmanager.com/gtm.js?id=&#39;+i+dl+">https://www.googletagmanager.com/gtm.js?id=&#39;+i+dl+</a> ‘&amp;gtm_auth=<a href="https://www.googletagmanager.com/gtm.js?id=GTM-NLR6H6W&amp;amp;gtm_auth=KDsBauO9BfS2PwvcY1n5yg&amp;amp;gtm_preview=env-2&amp;amp;gtm_cookies_win=x">xxxxxxxxxxxxxxxxxxxxxx</a>&amp;gtm_preview=env-x&amp;gtm_cookies_win=x’;f.parentNode.insertBefore(j,f);})(window,document,’script’,’dataLayer’,’<a href="https://www.googletagmanager.com/gtm.js?id=GTM-NLR6H6W&amp;amp;gtm_auth=KDsBauO9BfS2PwvcY1n5yg&amp;amp;gtm_preview=env-2&amp;amp;gtm_cookies_win=x">XXXXXXX</a>’);&lt;/script&gt;</blockquote><p><strong>Currently, we are only discussing the first script mentioned above</strong>, because it is the result of the website rendering. It contains the reason why GTM does not work.</p><ul><li>First, it has <strong><em>type=”text/plain”</em></strong>. When you see <strong><em>type=”text/plain”</em></strong> in a script tag within an HTML document, it indicates that the content of the script should be treated as plain text, not as executable code.</li><li>Secondly, it has <strong><em>class=”optanon-category-C0002&quot;</em></strong> inside. This means that OT will only allow this script to run if the user consents to a group of cookies defined as C0002. In my case, this is the Performance cookie group.</li></ul><p>On a website without auto-blocking, the script will always have the type <strong><em>“text/javascript”</em></strong>, and in the class, all groups except C0001 will be absent — or <strong>there may be no such class at all.</strong></p><h3>How to resolve OneTrust blocking issues and free up GTM</h3><p>To prevent the auto-blocking of Google Tag Manager, <strong>we need to add one of the following elements to the GTM snippet you installed on the website:</strong></p><blockquote>j.setAttributeNode(d.createAttribute(‘data-ot-ignore’));</blockquote><blockquote>OR</blockquote><blockquote>j.setAttribute(‘class’,’optanon-category-C0001&#39;);</blockquote><p>C0001 is a critical group of cookies without which a website could not work correctly. It is allowed by default according to the OT settings.</p><p>Your final correct GTM snippet will look like:</p><blockquote>&lt;! — Google Tag Manager → <br>&lt;script&gt;(function (w, d, s, l, i) { w[l] = w[l] || []; w[l].push({ ‘gtm.start’: new Date().getTime(), event: ‘gtm.js’ }); <br>var f = d.getElementsByTagName(s)[0], j = d.createElement(s), dl = l != ‘dataLayer’ ? ‘&amp;l=’ + l : ‘’; <strong>j.setAttributeNode(d.createAttribute(‘data-ot-ignore’)); </strong><br>j.async = true;j.src = ‘<a href="https://www.googletagmanager.com/gtm.js?id=&#39;">https://www.googletagmanager.com/gtm.js?id=&#39;</a> + i + dl; <br>f.parentNode.insertBefore(j, f); })(window, document, ‘script’, ‘dataLayer’, ‘GTM-XXXXXXX’); <br>&lt;/script&gt;</blockquote><p>Then, we will be able to work with the GTM container and block all tags inside it depending on your vision. Also, it enables us to complete the setup of Google Advanced Consent Mode.</p><p><strong>Be aware that the script above does not have any environment variables. It is better to copy and modify your own from the GTM account according to the rule I described.</strong></p><p>If you want official OneTrust sources for the solution I described:</p><ol><li><a href="https://my.onetrust.com/articles/en_US/Knowledge/UUID-71e7d0a8-03d8-e272-e683-72cadded2ecf">https://my.onetrust.com/articles/en_US/Knowledge/UUID-71e7d0a8-03d8-e272-e683-72cadded2ecf</a>. It worked some time ago, but it seems broken now. Probably, owners will fix it soon.</li><li>Login and visit <a href="https://developer.onetrust.com/onetrust/docs/google-tag-manager#important-notes-about-google-tag-manager-integration">https://developer.onetrust.com/onetrust/docs/google-tag-manager#important-notes-about-google-tag-manager-integration</a>.</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=582860d40c7a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Google Consent Mode v2 — Prepare your marketing for 2024]]></title>
            <link>https://medium.com/@w.anatolii/google-consent-mode-v2-prepare-your-marketing-for-2024-e209746f95db?source=rss-4fa1a98ff31d------2</link>
            <guid isPermaLink="false">https://medium.com/p/e209746f95db</guid>
            <category><![CDATA[google-analytics-4]]></category>
            <category><![CDATA[consent-mode]]></category>
            <category><![CDATA[google-ads]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[gdpr]]></category>
            <dc:creator><![CDATA[Anatolii Shulhat]]></dc:creator>
            <pubDate>Sun, 03 Dec 2023 22:19:26 GMT</pubDate>
            <atom:updated>2023-12-04T16:17:44.887Z</atom:updated>
            <content:encoded><![CDATA[<h3>Google Consent Mode v2 — Prepare your marketing for 2024</h3><h3><strong>Introduction</strong></h3><p>In my latest series of blog articles, I have delved into the topic of Google’s Consent Mode. On November 28, 2023, Google hosted an online session to present significant enhancements to this feature, along with linked functionalities in their internal tools. The effects of these changes will be felt in the spring of 2024. Collectively, this has been referred to as Consent Mode v2.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rfR787UqhElXDMfnKlGtZw.png" /></figure><p>The previous beta version of Google Consent Mode was considered rather exotic and usually be necessitated by the specific requirements of the project and preparation for a cookieless future. However, I think Consent Mode v2 will most likely go mainstream, because it influences marketing tools much more deeper. Especially it is actual for projects that have users from the EEA and use Google Ads.</p><h4>What did Google talk about in their presentation of Consent Mode v2 updates?</h4><ol><li>We were shown a brief overview of the digital advertising tools developemnt in the context of increasing privacy regulation.</li><li>Google explained what Consent Mode is, how it works, and what will be in the updated version.</li><li>There was a comparison of the Basic and Advanced Consent Modes.</li><li>Google briefly talked about Consent Mode modeling in Google Ads without details.</li><li>Different implementation paths for Consent Mode were shown.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oj_VdgI4oNtL0plJnmv63g.png" /></figure><p>If you didn’t attend the presentation, I’ll try to briefly summarize the main content, my thoughts and the action items list that need to be planned for the first few months of 2024.</p><p><a href="https://www.youtube.com/watch?v=IcNS8ATkGo0">Also, you can watch the session recording here on the Google Youtube channel.</a></p><h3>Context that led to this update</h3><h4>Privacy regulation and technical implications</h4><p>Mobile device identifiers and 3rd party cookies are legacy things that are going away at the moment. Users want more control over how their data is used.</p><p>If businesses want to continue measure and drive awareness, intent, and sales goals, they will need to have quality consented data and tools that allow not only to measure but also to fill in the missing data.</p><p>In short, if you want to get the maximum result from marketing in 2024 and you are a high-roller of Google instruments, you definitly will need to set up Google Consent Mode. This feature is the technological response to the development of privacy regulation in the world.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z5mtOTngzy4jDsWl3pgZQQ.jpeg" /></figure><p>One of the important digital marketing regulations will come into force in March 2024. Chrome will stop supporting 3P cookies, and businesses will need to start collecting additional consent for the use of cookies for certain marketing activities.</p><h3><strong>What is Consent Mode?</strong></h3><p>I would say it is a mapping of consents obtained for different types of cookie usage (when users click on the CMP banner on a website) with each tag being sent to Google by adding “deny” or “allow” values to special parameters (analytics_storage and ad_storage in beta Google Consent Mode).</p><p>It allows Google services to respect user choices and use consented data to enable a variety of related internal features.</p><p><a href="https://medium.com/@w.anatolii/consent-mode-with-google-analytics-4-and-tag-manager-how-to-collect-data-and-respect-users-db7c41457567">I did the overview of beta Consent Mode and its implementation here.</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nQt_XnD_Lh7rnwVtNRrRcw.jpeg" /></figure><h3>Consent Mode Types</h3><p>Google presented two types of consent mode implementation. You will probably choose between them based on your legal requirements and desired output.</p><h4>Advanced Consent Mode</h4><p>To my surprise, Google says that advanced implementation should be more familiar to us.</p><p>Ha.</p><p>I would say most of the projects I worked on had basic consent mode implementation.</p><p>The unique feature of the advanced mode is that the tag will be triggered before the user interacts with the CMP banner.</p><p>In other words, Advanced Consent Mode means that tags will be fired even if a user rejects giving consent for cookie usage.</p><p>However, all tags will adjust their behavior according to the user’s choice: a tag will not write cookies or will not enable ad personalization depending on a parameter and its value.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iYgTP9LtHMJ9rlNdX0xRaw.jpeg" /></figure><blockquote>This type of implementation will enable the collection of cookieless pings and provide more accurate data modeling to fill gaps from unconsented users. It is a highly debatable question whether these pings are allowed by ePrivacy in the EU, and most experts prefer to avoid them. I would advise speaking with your legal department, explaining all the technical nuances to them before choosing a Consent Mode type.</blockquote><h4>Basic Consent Mode</h4><p>Basic implementation means that the Google tag will not be activated until the user consents to the use of cookies. If the user refuses to give consent, no tags will be triggered, and no information will be stored.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ah_-w0N6gXk9pwxhl3sNbg.jpeg" /></figure><h3><strong>How to implement Consent Mode on the website?</strong></h3><p>There are three possible ways to implement Google Consent Mode on the website.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ThPCE9IMhG3QM94XU0I6Xg.jpeg" /></figure><ol><li>As you can see, the easiest way is just to use the recommended and integrated CMPs (Consent Management Platforms) from Google’s documentation.</li><li>A setup of moderate complexity can be done via GTM (Google Tag Manager) — and <a href="https://medium.com/@w.anatolii/google-consent-mode-with-onetrust-and-tag-manager-ea39b717d0e6">I have my own case described here</a>.</li><li>Also, I have seen customized consent mode setups done by developers, which represent the highest level of complexity. In this scenario, web analysts have almost no control.</li></ol><p>You need to know Consent Mode parameters and their logic to implement it correctly.</p><h3>Parameters in Consent Mode v2</h3><h4>What have been changed with v2?</h4><p>Google has introduced two new parameters that we can use to communicate users’ consents.</p><p>In fact, this is the biggest change in Consent Mode to date. The previous version of the feature focused on collecting consented data. <strong>The new version of Consent Mode has been upgraded to control the usage of consented data.</strong></p><p>As you can see from the following screenshot, it involves controlling data usage in Ads.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nUEy5eGAoCMAGxho8K9Jjg.jpeg" /></figure><h4>How do parameters influence tags in different implementations?</h4><p>As you can see from the following screenshot, new parameters do not have a serious advantage in advanced consent mode. Any implementation will require users’ positive consent to use data for any advertising purpose.</p><p>However, there is still a powerful benefit in advanced implementation, such as sending and collecting cookieless hits. This allows us to use aggregated data in Google tools.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NBlhUgraVvCXlplRNy7-ZA.jpeg" /></figure><h3>Consent Mode Modelling</h3><p>If you have added a CMP (Consent Management Platform) banner to your site and configured it correctly, you will face performance tracking issues. You will partially or completely lose data from users who do not consent to the use of cookies.</p><p>As a result, conversions are lost, and the optimization of campaigns worsens.</p><blockquote>The modeling in Consent Mode helps overcome this problem by filling the gaps with a help of machine learning algorithms (Google says with a help of AI — in line with tech fashion trends).</blockquote><p>During the presentation, Google particularly focused on modeling within Google Ads conversions.</p><p><strong>It is important to mention that GA4 behavioral and conversions modelling (for users who decline cookie consent) will work only with Advanced Consent Mode.</strong></p><p><strong>Google Ads Conversions modelling is enabled across basic and advanced types both.</strong></p><h4>Advanced Consent Mode Modelling</h4><p>During the advanced implementation of modeling, Google’s algorithms can use observable consented user journeys and extrapolate patterns onto aggregated, cookieless data that lack personal identifiers. Google utilizes signals such as Device Type, Conversion Type, Country, Time of Day, and Browser Type as indicators to make the modeling more accurate.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VgNWV9qZEPLW0DPFheuiZQ.jpeg" /></figure><h4>Basic Consent Mode Modelling</h4><p>Basic implementation also allows for conversion modeling in Google Ads. However, its accuracy is lower. In this case, we do not collect any cookieless hits, so the modeling is based only on data from users who have consented to cookie usage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2_g0tPuGp0wk1YINWqFs9A.jpeg" /></figure><p>The accuracy of such an approach is questionable; nevertheless, it is still better than not having Consent Mode at all.</p><p>For comparison and summary, the following table was provided during the presentation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TJQD0QYcywTBGDCiufrpzg.jpeg" /></figure><p>For accuracy, Google recommends using the advanced implementation. The basic version is better than not having a consent mode at all.</p><h3>What if a website does not have Consent Mode implemented?</h3><p>If we do not send consent information to Google with tags from March 2024, we will not be able to capture new EEA users in the audience lists within the GA4 Audiences feature and share them with Ads. This will make it impossible to do remarketing in Google Ads, Floodlight, and Display.</p><h3>Important action items for website owners</h3><ol><li>Check whether you have EEA traffic.</li><li>Collect user concent for EEA users (via OneTrust, Cookiebots or other Consent Management Platforms).</li><li>If you are using Universal Analytics 360 and want to continue using remarkting audiences, conversions exporting and reliable bidding optimizations, migrate to Google Analytics 4.</li><li>Share collected CMP consent signals with Google tools. In other words, make correct setup of Consent Mode v2.</li><li>If you are sending offline data to Google via API, add consent signals as well.</li></ol><h3>Post Scriptum</h3><p>If you’re interested in this topic and want to delve into web analytics community discussions, I recommend following <a href="https://www.linkedin.com/in/simoahava">Simo Ahava on LinkedIn</a>. Great discussions often arise under his posts, and there’s a lot to learn. Just search for Consent Mode topic.</p><p>I am also glad to see new professional connections on <a href="https://www.linkedin.com/in/wanatolii/">Anatolii Shulhat LinkedIn</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e209746f95db" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to use attribution models in Google Analytics 4?]]></title>
            <link>https://medium.com/@w.anatolii/how-to-use-attribution-models-in-google-analytics-4-0cd4fcaef06e?source=rss-4fa1a98ff31d------2</link>
            <guid isPermaLink="false">https://medium.com/p/0cd4fcaef06e</guid>
            <category><![CDATA[attribution]]></category>
            <category><![CDATA[marketing]]></category>
            <category><![CDATA[google-analytics-4]]></category>
            <category><![CDATA[attribution-modeling]]></category>
            <category><![CDATA[google-analytics]]></category>
            <dc:creator><![CDATA[Anatolii Shulhat]]></dc:creator>
            <pubDate>Thu, 09 Nov 2023 12:27:30 GMT</pubDate>
            <atom:updated>2023-11-15T10:32:04.101Z</atom:updated>
            <content:encoded><![CDATA[<h3>Introduction</h3><p>Recently, a client inquired about the benefits of using data-driven attribution instead of the last-click model in GA4. I did not want to delve into the topic solely by comparing the models mechanically, as there is already plenty of text online. Instead, I was intrigued by how businesses could better comprehend and leverage the attribution models in GA4 to their advantage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*j5_nZE36LmHLeI10" /></figure><h3>Attribution models in GA4</h3><p>In essence, attribution is always about crediting the outcome of activities to a particular entity. In the realm of digital marketing and web analytics, it involves assigning value to various interactions that a user has with a product throughout their journey.</p><p>Google Analytics 4 was fundamentally conceived as a tool that could analyse a user’s complete journey across different devices and platforms within its properties. Consequently, a significant emphasis was placed on enhancing its attribution capabilities — the range of available models was streamlined, but their overall quality was significantly enhanced.</p><p>By the end of 2023, Google Analytics 4 properties are set to have the Data-Driven reporting attribution model enabled by default.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*AIorf2JXJvgwM6lkLjobtQ.png" /></figure><p>As observed, Google marks it as the recommended model.</p><p>It truly is the most sophisticated attribution model which, in the predecessor to GA4, Universal Analytics (UA), was exclusively available with the premium 360 version starting at an annual cost of $150,000.</p><p>To comprehend why the data-driven model stands out, let’s examine the range of models that are currently available.</p><h3>1. Google paid channels Last click</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/48/0*O8aF7K7r5SEgM3yk" /></figure><p>The model assigns 100% of the value to the last Google paid channel that the user clicked on before the conversion.</p><h4>Pros</h4><p>This model is elegantly straightforward, showcasing all conversions where Google Ads was involved. It also identifies the precise advertisement that led to conversion. If the channel funnel is very simple, user intent is clear, and the end goal is reached swiftly and impulsively, this model could be suitable.</p><h4>Cons</h4><p>In scenarios with a complex channel mix and thoughtful purchases, this model, to my mind, unjustly ignores other traffic sources, benefiting only the Google Ads source by artificially boosting its performance. From a business utility perspective, we lose the visibility of other channels’ results and fail to recognize their value.</p><h3>2. Paid and organic last click</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/48/0*O8aF7K7r5SEgM3yk" /></figure><p>The model attributes the entire value to the last organic or paid channel that the user engaged with before conversion.</p><h4>Pros</h4><p>This model appears more comprehensive than its predecessor, as it does not attribute all value to only one possible traffic source. Due to its last-click basis, it remains incredibly straightforward and suits businesses with a simple mix of channels, where conversions happen quickly, and the significance of the final session is well-established. Historically, this has been the industry norm, as seen in affiliate marketing where the last affiliate to refer a client is commonly rewarded.</p><h4>Cons</h4><p>However, the oversimplification of focusing solely on the last interaction overlooks the initial stages of the customer journey. Even with a limited channel mix, initial user interactions can subtly shape the eventual conversion. Overlooking the start of the funnel could result in a skewed marketing optimization that overvalues the final touchpoint.</p><h3>3. Data-driven attribution</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/48/0*JADswkXnG8KhiCbf" /></figure><p>The data-driven model allocates conversion value across channels based on the data from the account, with Google’s algorithms assessing each click’s contribution and modelling a unique distribution for interaction points. This method allows for a nuanced understanding of each channel’s role in the conversion journey.</p><h4>Pros</h4><p>Unlike previous models that might seem like guesses, data-driven attribution is grounded in data, offering insights into the effectiveness of each touchpoint. This model serves as a best starting tool to gauge which advertisements are more likely to result in conversions. <a href="https://www.kaushik.net/avinash/marketing-analytics-attribution-is-not-incrementality/">Web analytics father Avinash Kaushik also endorses data-driven attribution</a>. His critiques of other models, available on his blog, are enlightening and <a href="https://www.kaushik.net/avinash/multi-channel-attribution-modeling-good-bad-ugly-models/?utm_source=newsletter&amp;utm_medium=email&amp;utm_campaign=tinyletter#lastadwords">often entertaining</a>.</p><p>Considering the advanced nature of this model within Google Analytics 4, one might question the necessity of discussing its limitations.</p><h4>Cons</h4><p>However, it is important to recognize that the model’s efficacy is data-dependent: the greater the data volume, the more refined the training and outcomes. The mechanics of the model are a black box, making it impossible to transparently analyse or replicate the value distribution processes in BigQuery with raw data. Additionally, GA4’s attribution models vary across different reports, which isn’t always transparent, leading to potential confusion regarding data accuracy.</p><blockquote>The models’ descriptions notably omit direct channels. Logically, this raises the next question.</blockquote><h3>What is the role of direct traffic in GA4 attribution models?</h3><p>Remember that in GA4, all attribution models disregard direct traffic unless the conversion journey is composed solely of direct interactions.</p><p>This presents a substantial blind spot, particularly for businesses with complex conversion paths that span multiple sessions over an extended period.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1bE_Mw3fBq_oEfvs" /></figure><p>My view is that this stems from the reliance on cookie-based attribution, which is becoming less effective in a landscape of multiple devices, varied conversion windows, and heightened privacy measures.</p><p>Tracking the full journey of a user from point A to point B is increasingly challenging as individuals use numerous gadgets and as technology providers impose stricter tracking limitations.</p><p>For those keen on this subject, follow my channel — I aim to delve into the topic of cookieless attribution in an upcoming article.</p><h3>Reports and attribution models</h3><p>I have mentioned that there is a certain confusion with reports and the attribution models used within them.</p><h4>User acquisition report</h4><p>Take, for example, the User acquisition report, where primary dimensions are prefixed with ‘First user’. In this case, Data-driven attribution is not applied, and instead, we are looking at a kind of user-based first click attribution model. If you want to see which marketing channels and which advertisements brought the user to the site and from what the first contact occurred — this report is what you should examine.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pRXZ_rA5BZki84K2iDubGA.png" /></figure><h4>Traffic acquisition report</h4><p>If we look at the Traffic acquisition report, where primary dimensions have the ‘Session’ prefix, we encounter the last-non-direct attribution for sessions. Direct remains only for users who have no other touchpoints.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*byIvt74zcPDDm9ZdveyNxQ.png" /></figure><h4>What is about data-driven attribution report?</h4><p>Data-driven attribution report you can find it in the Advertising section → Performance → All channels. It cannot be customized which is the biggest disadvantage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RH19HOzmPj2zryooXzXMuw.png" /></figure><p>If you want to use standard Reports section and customize the dashboard, GA4 does not have default report for data-driven attribution; however, we can create one.</p><p>But first, let’s understand the dimensions and metrics we can utilize.</p><p>I reiterate that in the digital world, attribution is the assignment of value for a conversion to various touchpoints with the product throughout the user’s journey. Touchpoints can include any dimension — traffic channel, device, landing page, source, etc. If we are building a model from raw data, the selection of dimensions can be limited only by the analyst’s imagination and the amount of data available.</p><p>When dealing with Google Analytics 4, our choice of product touchpoints is somewhat limited. If we consider Reports and Explorations, we can work with data-driven attribution within such event-scoped dimensions:</p><ul><li>Campaign / Campaign ID</li><li>Default channel group</li><li>Medium</li><li>Source</li><li>Source / medium</li><li>Source platform</li><li>Google Ads account name</li><li>Google Ads ad group name</li><li>Google Ads ad network type</li><li>Google Ads keyword text</li><li>Google Ads query</li></ul><p>If the property is integrated with Search Ads 360, we will have additional options:</p><ul><li>SA360 engine account ID / name / type</li><li>SA360 ad group name</li><li>SA360 keyword text</li><li>SA360 query</li></ul><p>It has already been noted that attribution dimensions are event-scoped. Currently, with the above-mentioned dimensions, only the following metrics can be utilized:</p><ul><li>Conversions</li><li>Total revenue</li><li>Purchase revenue</li><li>Total ad revenue.</li></ul><p>Within these metrics, the value of channels will be reassigned according to data-driven algorithms.</p><p>If you are interested in sessions or users, you should refer to the standard Acquisition reports, but remember that data-driven attribution is not present there.</p><h4>How to create a data-driven attribution report?</h4><ol><li>Head to the Library and click ‘Create new report’ → ‘Create detail report’, then add all dimensions from the Attribution category to the Primary Dimensions list.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Eu1SpbgQ5AZ2QDh8zMIr6g.png" /></figure><p>2. Next, add the event-based metrics that work with attribution. Click ‘Apply’. Save the report, name it, and then return to the Library section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*CQfZF06JcoK6aAEo6rNB4g.png" /></figure><p>3. Add the newly created detail report to your published collection of reports.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*XAMamFRlcnHDpxqO2gE1aQ.png" /></figure><p>As a result, the Life Cycle collection (or any other of your choice) can be enhanced with a new useful report.</p><blockquote>Similarly, <strong>data-driven attribution can be constructed using Explorations</strong>, but it is important to remember that there is limited Data Retention, and the <strong>maximum possible Date Range is the last 14 months.</strong></blockquote><h3>Applying attribution</h3><p>Attribution is always the retrospective distribution of conversions among all possible sources. An attribution model is a handy tool when it comes to determining the overall marketing performance by channel at the end of the month, allocating bonuses, and planning operational and marketing management.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ByUJDI0KaGENG-eC" /></figure><p>However, one should not treat built attribution models as something that will precisely tell you where to invest to increase conversions, ROAS, or reduce CPA.</p><p>In my opinion, the data-driven model can be a convenient starting point for analysing incrementality, whose goal is precisely to determine the impact of each channel on the total number of conversions.</p><p>One might think for years that a particular paid source is bringing in the main conversions, then turn it off and realize that the same customers still arrive and purchase, just through different means.</p><p>The attribution model shows the value, or rather the distribution of value, of your existing channels and specific data. The attribution model does not indicate that this is the best combination of channels, or that the most valuable channel will bring more conversions if money is added to it. Although such a case cannot be excluded. An attribution model will not show this with accuracy but experiments with campaigns and their analysis might.</p><p>For a deeper dive into the topic, I recommend reading the article <a href="https://www.kaushik.net/avinash/marketing-analytics-attribution-is-not-incrementality/">Marketing Analytics: Attribution Is Not Incrementality — Occam’s Razor by Avinash Kaushik</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0cd4fcaef06e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to export Universal Google Analytics data]]></title>
            <link>https://medium.com/@w.anatolii/how-to-export-universal-google-analytics-data-d37752f20f0d?source=rss-4fa1a98ff31d------2</link>
            <guid isPermaLink="false">https://medium.com/p/d37752f20f0d</guid>
            <category><![CDATA[google-analytics-tips]]></category>
            <category><![CDATA[google-api]]></category>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[r]]></category>
            <category><![CDATA[google-analytics]]></category>
            <dc:creator><![CDATA[Anatolii Shulhat]]></dc:creator>
            <pubDate>Tue, 07 Nov 2023 16:20:36 GMT</pubDate>
            <atom:updated>2023-11-09T12:29:44.439Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to export Universal Google Analytics data via API with R</h3><h3>Introduction</h3><p>As we navigate through 2023 and into 2024, web analysts encounter two important points on the timeline:</p><ol><li>The handy, free Google Analytics Universal (UA) stopped processing new data on July 1, 2023.</li><li>The ability to access historical data via the Google Analytics Universal (UA) interface and API will be discontinued for all users after July 1, 2024.</li></ol><p>For many, the transition to Google Analytics 4 or to other platforms has smoothly passed. Our current focus shifting towards adaptation and mastery of these new tools.</p><p>However, the impending challenge for businesses, particularly those reliant on the no-cost Google Analytics Universal (UA), is exporting and archiving historical data and reports before mentioned Sunset.</p><p>I want to share my experience, which should help you discover the best practices for exporting years of data from legacy Google Analytics. The article provides a practical example of a project that archives almost all possible reports via API with R.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*jE2xJxoXmiRCx-kR" /></figure><h3>Ways to archive historical UA data</h3><ol><li>If you have Google Analytics 360, you can save your data with BigQuery. <a href="https://support.google.com/analytics/answer/3416092?sjid=3696582300335443795-EU#zippy=%2Cin-this-article">Follow this guide.</a></li><li>Utilizing the Google Analytics interface allows exporting of reports in various formats such as CSV, TSV, Excel, Sheets, and PDF. This is a viable option if <strong>you are dealing with a limited data set and need to save just a few reports</strong>. This approach can become a repetitive, time-consuming task.</li><li>The guide by Google recommends using the <a href="https://developers.google.com/analytics/solutions/google-analytics-spreadsheet-add-on">Google Sheets add-on for Analytics</a>. Currently, I am unable to install this add-on because of an error that is under discussion in the “<a href="https://groups.google.com/g/google-analytics-spreadsheet-add-on/c/dT7496g9Fe0">Google blocks access to the Google Analytics add-on</a>” thread. If you have installed it before this problem occurred and <strong>your report does not exceed 5000 rows</strong>, it could serve as an efficient solution with the capability to automate the data transfer from Sheets to BigQuery. Be aware, though, that <strong>there could be issues if you are dealing with a huge amount of data and a broad date range.</strong></li><li>If you are dealing with a large dataset, numerous reports, and various dimension combinations, it might be worthwhile to explore the use of the <a href="https://developers.google.com/analytics/devguides/reporting/core/v4">Google Analytics Reporting API</a>. Google’s official documentation provides guidance on interfacing with the API via Java, PHP, JavaScript, and Python. As for myself, I prefer working with R, which is why I find <a href="https://code.markedmondson.me/googleAnalyticsR/">Mark Edmondson’s googleAnalyticsR library</a> particularly useful, as it allows for the automation of nearly anything with a rather legible syntax. <strong>It is crucial to keep in mind the daily limit of 50,000 requests per project, as well as a limit of 10,000 requests per day for each specific View.</strong></li></ol><p>I am working with a project on the free version of Google Analytics Universal (UA) that <strong>has a numerous dimensions and metrics in use, with data from more than 7 million users dating back to 2016</strong>. I have opted for a custom approach using the Google Analytics Reporting API, and I would like to present my solution to you, which has significantly streamlined the process of data archiving and reports exporting. God, save the automatization!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*hKjJC_tM6P12jhRU" /></figure><h3>Preparation for UA data export</h3><p>Before starting the data export process, it is essential to identify what specific data the business needs to preserve.</p><blockquote>You should prioritize saving data that are crucial for future annual comparisons.</blockquote><p>Attempting to save all possible data for a large project is nearly an impossible task due to the enormous effort required.</p><p>Overcoming all the API limits is possible, but is the data worth the effort and time spent on complex engineering?</p><p>If the data will aid in more effective decision-making — yes, and that is where the focus should be.</p><p>Think about next entities:</p><ol><li>Primary dimension. Each Google Analytics report has it. For example, Country, Age or Product Name.</li><li>Secondary dimensions. A lot of Google Analytics reports can be enriched by the additional context adding a new dimension to a primary one.</li><li>Time dimensions. Google Analytics reports are always aggregated. Usually, you see metrics constructed within date range or time series context.</li><li>Metrics. We are here because of them. What numbers you want to export? There are users, sessions, conversion rates, etc.</li><li>Segments. Do you use segments regularly? Will you need segmented reports in the future?</li></ol><p>My archiving project case is based on previous entities except segments. If you want to automate your export routines, we need to organize all mentioned things into structured files. You can find more details in the following section of the article.</p><h3>Project structure</h3><p>The first file we need to create should include details on all reports, their titles, and special flags for script operations. Titles can be formulated based on standard conventions (see the screenshot).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bhSdvCMddWHQ2uJVZSji_w.png" /></figure><h4>1. reports.csv</h4><p>For the future R script to work properly, the file must include the following columns:</p><ul><li>Report</li></ul><p>In the column, you should specify the items you wish to archive, such as Audience Demographics Age, Audience Demographics Gender, Audience Interests Overview, and so on.</p><ul><li>Skip</li></ul><p>While preparing, you might be tempted to export more reports than are actually necessary. With TRUE and FALSE values, you can manage the script’s functionality — to include or exclude a specific report from the process. If all the reports you are adding are to be downloaded, then set the value to FALSE for each.</p><ul><li>Report Type</li></ul><p>This column is necessary for managing various processing scenarios for the final queries to the Google Analytics Reporting API. In my situation, some reports include secondary dimensions, others do not, and there are also reports where certain metrics cannot be combined. These instances will have distinct processing rules. If you are utilizing my script, the values need to be specified as: only_primary_dimensions, with_secondary_dimensions, separate_metrics.</p><ul><li>Priority</li></ul><p>If you have a substantial amount of data and wish to control the download priorities, you should add ordinal numbers to the column.</p><ul><li>Filter Dimension</li><li>Filter Dimension Values</li></ul><p>Some reports will include filters, such as for mobile data or PPC traffic. For these, we will need the dimension name and the specific value that should be filtered.</p><blockquote>For an example of the reports file, please refer to <a href="https://docs.google.com/spreadsheets/d/1e4JBuLAqso_XG8gvEMcwYLiYlSOTH7fffdy8H5pF92s/edit?usp=sharing">my Spreadsheet</a>. To ensure the script functions properly, maintain a copy in csv format.</blockquote><h4>2. primary_dimensions.csv</h4><p>Every Google Analytics report is based on a primary dimension, which could be either a single entity or a combination thereof.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OCD1j9wF-lSnCfwwvN4QMA.png" /></figure><p>The file contains the following columns:</p><ul><li>Primary Dimensions</li></ul><p>The dimension’s name is taken from the Google Analytics UI — for the reader’s convenience.</p><ul><li>Primary Dimensions API</li></ul><p>The relevant dimension value for interaction with the Reporting API. Mapping is performed using the <a href="https://ga-dev-tools.google/dimensions-metrics-explorer/">UA Dimensions &amp; Metrics Explorer</a>.</p><ul><li>Skip</li></ul><p>The same principle applies as with the reports file — to conveniently control what to include or exclude. If all the dimensions you are adding should be downloaded, then insert the value FALSE in every entry.</p><ul><li>Report</li></ul><p>The key to link with the corresponding report. Such links will be necessary when the script compiles the request body for the API.</p><blockquote><a href="https://docs.google.com/spreadsheets/d/1_PUpYVjaOGQsALTzBmD5UU6Ekyq2Wh4jo4PbTRPgQMY/edit?usp=sharing">Examples of primary_dimensions can be found here</a>. To ensure the script functions correctly, maintain a copy in the csv format. Be careful with Custom Dimensions and Content Groups — since they differ from one project to another, they may require modification when necessary.</blockquote><h4>3. secondary_dimensions.csv</h4><p>For those reports that utilize secondary dimensions, it is necessary to prepare appropriate lists.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LRyVwA5vpgh393Nt2fdSVw.png" /></figure><p>The file may contain the following columns:</p><ul><li>Secondary Dimensions</li></ul><p>The dimension’s name is taken from the Google Analytics UI — for the reader’s convenience.</p><ul><li>Secondary Dimensions API</li></ul><p>The relevant dimension value for interaction with the Reporting API. Mapping is performed using the <a href="https://ga-dev-tools.google/dimensions-metrics-explorer/">UA Dimensions &amp; Metrics Explorer</a>.</p><ul><li>Skip</li></ul><p>The same principle applies as with the reports file — to conveniently control what to include or exclude. If all the dimensions you are adding should be downloaded, then insert the value FALSE in every entry.</p><ul><li>Report</li></ul><p>The key to link with the corresponding report. Such links will be necessary when the script compiles the request body for the API.</p><blockquote><a href="https://docs.google.com/spreadsheets/d/1blO_ZVJRIgIxRtlIu8cB_4ohJNtxRevX5b1IH5QSUjs/edit?usp=sharing">Examples of secondary_dimensions can be found here</a>. To ensure the script functions correctly, maintain a copy in the csv format. Be careful with Custom Dimensions and Content Groups — since they differ from one project to another, they may require modification when necessary.</blockquote><h4>4. time_dimensions.csv</h4><p>To properly build time series graphs, it is necessary to include time dimensions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cyYcfqeZ6MHrF57bChBUcg.png" /></figure><ul><li>Time Dimensions</li></ul><p>The dimension’s name is taken from the Google Analytics UI — for the reader’s convenience.</p><ul><li>Time Dimensions API</li></ul><p>The relevant dimension value for interaction with the Reporting API. Mapping is performed using the <a href="https://ga-dev-tools.google/dimensions-metrics-explorer/">UA Dimensions &amp; Metrics Explorer</a>.</p><ul><li><strong>Type of metric</strong></li></ul><p>In Google Analytics, we typically work with aggregated data that revolves around time-based dimensions. Essentially, metrics can be categorized into event-based and others, which include aggregated, proportional, and user-based metrics. For instance, when we have daily numbers of page views, reconstructing monthly or weekly trends from this data is straightforward. However, adding up daily unique users does not give an accurate monthly total since same users can visit on multiple days. This logic also applies to average daily metrics. By categorizing metrics in this manner and considering the multitude of possible combinations, we can more efficiently use our time and API resources. This is because we can limit the extraction of some metrics to only the daily dimension when it is sufficient for our analysis.</p><blockquote><a href="https://docs.google.com/spreadsheets/d/1dQofqPrmj25TOSdsTMtF4b8qAqKCPdPRtUS4TJCOFKE/edit?usp=sharing">Here is an example of the time_dimensions</a>. To ensure the script functions correctly, maintain a copy solely in the CSV format.</blockquote><h4>5. metrics.csv</h4><p>For each report intended for export, it is recommended to compile a corresponding list of metrics.</p><ul><li>Metric</li></ul><p>The metric’s name is taken from the Google Analytics UI — for the reader’s convenience.</p><ul><li>Metric API</li></ul><p>The relevant metric value for interaction with the Reporting API. Mapping is performed using the <a href="https://ga-dev-tools.google/dimensions-metrics-explorer/">UA Dimensions &amp; Metrics Explorer</a>.</p><ul><li>Type of metric</li></ul><p>In Google Analytics, we typically work with aggregated data that revolves around time-based dimensions. Essentially, metrics can be categorized into event-based and others, which include aggregated, proportional, and user-based metrics. For instance, when we have daily numbers of page views, reconstructing monthly or weekly trends from this data is straightforward. However, adding up daily unique users does not give an accurate monthly total since same users can visit on multiple days. This logic also applies to average daily metrics. By categorizing metrics in this manner and considering the multitude of possible combinations, we can more efficiently use our time and API resources. This is because we can limit the extraction of some metrics to only the daily dimension when it is sufficient for our analysis.</p><ul><li>Skip</li></ul><p>The same principle applies as with the reports file — to conveniently control what to include or exclude. If all the dimensions you are adding should be downloaded, then insert the value FALSE in every entry.</p><ul><li>Report</li></ul><p>The key to link with the corresponding report. Such links will be necessary when the script compiles the request body for the API.</p><blockquote><a href="https://docs.google.com/spreadsheets/d/15PFqkHRaceI1VeSr9_JT6H3TOpc4n-6_CnVsro0pHow/edit?usp=sharing">Here is an example of the metrics</a>. To ensure the script functions correctly, maintain a copy solely in the CSV format.</blockquote><h4>6. gaua_export.R</h4><p>The R script is used to formulate requests to the Google Analytics Reporting API and to save the export results. I recommend reviewing it in sections.</p><h3>Main script</h3><h4>Install libraries</h4><p>For our tasks, we will utilize the modern R libraries from the <a href="https://www.tidyverse.org/">Tidyverse</a> suite, alongside the <a href="https://code.markedmondson.me/googleAnalyticsR/index.html">googleAnalyticsR package by Mark Edmondson</a>, complemented by the <a href="https://www.rdocumentation.org/packages/googleAuthR/versions/2.0.0">googleAuthR package</a> to facilitate an easy connection to Google services.</p><pre># List of required packages<br>required_packages &lt;- c(&quot;googleAnalyticsR&quot;, &quot;tidyverse&quot;, &quot;googleAuthR&quot;)<br><br># Function to check and install any missing packages<br>install_and_load_packages &lt;- function(packages) {<br>  for (pkg in packages) {<br>    if (!require(pkg, character.only = TRUE)) {<br>      install.packages(pkg)<br>      library(pkg, character.only = TRUE)<br>    }<br>  }<br>}<br><br># Call the function with the vector of package names<br>install_and_load_packages(required_packages)</pre><h4>Authentication</h4><p>To operate the script effectively, it is essential to create your own Google Project. Utilizing the default project provided within the library could lead to maxing out the API call quota, as it is shared among all users. To circumvent this, you will need to establish a new Google Project and activate the Analytics APIs. Additionally, you have the option to establish a service account and load a key to enable background authentication. For comprehensive guidance, consult the <a href="https://code.markedmondson.me/googleAnalyticsR/reference/ga_auth.html">Authenticate with Google Analytics OAuth2</a> section.</p><pre># Set the file path for the Google Project JSON file that contains your client ID.<br>googleAuthR::gar_set_client(&quot;c:/data/gcp_client.json&quot;) <br><br># Authenticate with Google Analytics using the set client details.<br># This will open a browser window for you to log in if you are not already authenticated.<br>ga_auth()</pre><h4>Uploading of prepared files</h4><p>At this point, the script reads all files that were prepared in the “Preparation” phase. These files are required to construct the API requests.</p><pre># Define the directory path where you will save data and from which you will read data.<br>dir_path &lt;- &quot;c:/data/&quot;<br><br># Read the &#39;reports.csv&#39; file, <br># filter out rows where Skip is &quot;FALSE&quot;, <br># and drop the &#39;Skip&#39; column from the dataset<br>reports_all &lt;- read_csv(paste0(dir_path, &#39;reports.csv&#39;), show_col_types = FALSE) %&gt;% <br>  filter(Skip == &quot;FALSE&quot;) %&gt;% <br>  select(-Skip)<br><br># Read the &#39;metrics.csv&#39; file,<br># operations as above<br>metrics_all &lt;- read_csv(paste0(dir_path, &#39;metrics.csv&#39;), show_col_types = FALSE) %&gt;% <br>  filter(Skip == &quot;FALSE&quot;) %&gt;% <br>  select(-Skip)<br><br># Read the &#39;primary_dimensions.csv&#39; file, applying similar data cleaning<br># operations as above.<br>primary_dimensions_all &lt;- read_csv(paste0(dir_path, &#39;primary_dimensions.csv&#39;), show_col_types = FALSE) %&gt;% <br>  filter(Skip == &quot;FALSE&quot;) %&gt;% <br>  select(-Skip)<br><br># Read the &#39;secondary_dimensions.csv&#39; file<br># operations as above.<br>secondary_dimensions_all &lt;- read_csv(paste0(dir_path, &#39;secondary_dimensions.csv&#39;)) %&gt;% <br>  filter(Skip == &quot;FALSE&quot;) %&gt;% <br>  select(-Skip)<br><br># Read the &#39;time_dimensions.csv&#39; file<br>time_dimensions_all &lt;- read_csv(paste0(dir_path, &#39;time_dimensions.csv&#39;))</pre><h4>Creation of table for queries</h4><p>At this stage, the script generates a comprehensive table using the files prepared during the previous phase. Each row of this table effectively represents a potential API request. The columns of this table will be utilized for exporting data.</p><pre># Loop through all reports<br>for (report_type_number in 1:n_distinct(reports_all$`Report`)) {<br>  <br>  # Transformations if the report type is &quot;separate_metrics&quot;<br>  if (reports_all$`Report Type`[report_type_number] == &quot;separate_metrics&quot;) {<br>    transformed_table_1 &lt;- reports_all[report_type_number, ] %&gt;%<br>      left_join(metrics_all, by = &#39;Report&#39;) %&gt;%<br>      left_join(primary_dimensions_all %&gt;% select(-`Type of dimension`), by = &#39;Report&#39;) %&gt;%<br>      mutate(path_file = paste0(dir_path, Report, &#39;/&#39;, `Type of metric`, &#39;/&#39;, `Primary Dimensions`, &#39;/&#39;, `Metric`, &#39;.csv&#39;)) %&gt;%<br>      mutate(path_folder = paste0(dir_path, Report, &#39;/&#39;, `Type of metric`, &#39;/&#39;, `Primary Dimensions`)) %&gt;%<br>      mutate(`Dimension API Combined` = paste0(`Primary Dimensions API`)) %&gt;%<br>      mutate(`Dimension Combined` = paste0(`Primary Dimensions`))<br>  }<br>  <br>  # Transformations if the report type is &quot;with_secondary_dimensions&quot;<br>  if (reports_all$`Report Type`[report_type_number] == &quot;with_secondary_dimensions&quot;) {<br>    metrics_combined &lt;- metrics_all %&gt;%<br>      filter(Report == reports_all$`Report`[report_type_number]) %&gt;%<br>      left_join(time_dimensions_all, by = &#39;Type of metric&#39;) %&gt;%<br>      group_by(`Type of metric`, `Time Dimensions`) %&gt;%<br>      mutate(Metric = paste(Metric, collapse = &quot;,&quot;)) %&gt;%<br>      mutate(`Metric API` = paste(`Metric API`, collapse = &quot;,&quot;)) %&gt;%<br>      ungroup() %&gt;%<br>      select(-Report) %&gt;%<br>      distinct() %&gt;%<br>      bind_rows(metrics_all %&gt;%<br>                 filter(Report == reports_all$`Report`[report_type_number]) %&gt;%<br>                 group_by(`Type of metric`) %&gt;%<br>                 mutate(Metric = paste(Metric, collapse = &quot;,&quot;)) %&gt;%<br>                 mutate(`Metric API` = paste(`Metric API`, collapse = &quot;,&quot;)) %&gt;%<br>                 ungroup() %&gt;%<br>                 select(-Report) %&gt;%<br>                 distinct()) %&gt;%<br>      filter(!( `Type of metric` == &#39;Event based&#39; &amp; is.na(`Time Dimensions`))) %&gt;%<br>      mutate(`Time Dimensions` = if_else(is.na(`Time Dimensions`), &#39;No Secondary Time Dimension&#39;, `Time Dimensions`)) %&gt;%<br>      mutate(`Time Dimensions API` = if_else(is.na(`Time Dimensions API`), &#39;No Secondary Time Dimension&#39;, `Time Dimensions API`))<br>    <br>    transformed_table_1 &lt;- reports_all[report_type_number, ] %&gt;%<br>      left_join(primary_dimensions_all %&gt;% select(-`Type of dimension`), by = &#39;Report&#39;) %&gt;%<br>      cross_join(secondary_dimensions_all %&gt;% filter(Report == reports_all$`Report`[report_type_number]) %&gt;% select(-Report)) %&gt;%<br>      cross_join(metrics_combined) %&gt;%<br>      mutate(path_file = paste0(dir_path, Report, &#39;/&#39;, `Type of metric`, &#39;/&#39;, `Time Dimensions`, &#39;/&#39;, `Primary Dimensions`, &#39;,&#39;, `Secondary Dimensions`, &#39;.csv&#39;)) %&gt;%<br>      mutate(path_folder = paste0(dir_path, Report, &#39;/&#39;, `Type of metric`, &#39;/&#39;, `Time Dimensions`)) %&gt;%<br>      mutate(`Dimension API Combined` = paste0(`Time Dimensions API`, &#39;,&#39;, `Primary Dimensions API`, &#39;,&#39;, `Secondary Dimensions API`)) %&gt;%<br>      mutate(`Dimension API Combined` = gsub(&#39;No Secondary Time Dimension,&#39;, &#39;&#39;, `Dimension API Combined`)) %&gt;%<br>      mutate(`Dimension Combined` = paste0(`Time Dimensions`, &#39;,&#39;, `Primary Dimensions`, &#39;,&#39;, `Secondary Dimensions`)) %&gt;%<br>      mutate(`Dimension Combined` = gsub(&#39;No Secondary Time Dimension,&#39;, &#39;&#39;, `Dimension Combined`))<br>  }<br>  <br>  # Transformations if the report type is &quot;only_primary_dimensions&quot;<br>  if (reports_all$`Report Type`[report_type_number] == &quot;only_primary_dimensions&quot;) {<br>    metrics_combined &lt;- metrics_all %&gt;%<br>      filter(Report == reports_all$`Report`[report_type_number]) %&gt;%<br>      left_join(time_dimensions_all, by = &#39;Type of metric&#39;) %&gt;%<br>      group_by(`Type of metric`, `Time Dimensions`) %&gt;%<br>      mutate(Metric = paste(Metric, collapse = &quot;,&quot;)) %&gt;%<br>      mutate(`Metric API` = paste(`Metric API`, collapse = &quot;,&quot;)) %&gt;%<br>      ungroup() %&gt;%<br>      select(-Report) %&gt;%<br>      distinct() %&gt;%<br>      bind_rows(metrics_all %&gt;%<br>                 filter(Report == reports_all$`Report`[report_type_number]) %&gt;%<br>                 group_by(`Type of metric`) %&gt;%<br>                 mutate(Metric = paste(Metric, collapse = &quot;,&quot;)) %&gt;%<br>                 mutate(`Metric API` = paste(`Metric API`, collapse = &quot;,&quot;)) %&gt;%<br>                 ungroup() %&gt;%<br>                 select(-Report) %&gt;%<br>                 distinct()) %&gt;%<br>      filter(!( `Type of metric` == &#39;Event based&#39; &amp; is.na(`Time Dimensions`))) %&gt;%<br>      mutate(`Time Dimensions` = if_else(is.na(`Time Dimensions`), &#39;No Secondary Time Dimension&#39;, `Time Dimensions`)) %&gt;%<br>      mutate(`Time Dimensions API` = if_else(is.na(`Time Dimensions API`), &#39;No Secondary Time Dimension&#39;, `Time Dimensions API`))<br>    <br>    transformed_table_1 &lt;- reports_all[report_type_number, ] %&gt;%<br>      left_join(primary_dimensions_all %&gt;% select(-`Type of dimension`), by = &#39;Report&#39;) %&gt;%<br>      cross_join(metrics_combined) %&gt;%<br>      mutate(path_file = paste0(dir_path, Report, &#39;/&#39;, `Type of metric`, &#39;/&#39;, `Time Dimensions`, &#39;/&#39;, `Primary Dimensions`, &#39;.csv&#39;)) %&gt;%<br>      mutate(path_folder = paste0(dir_path, Report, &#39;/&#39;, `Type of metric`, &#39;/&#39;, `Time Dimensions`)) %&gt;%<br>      mutate(`Dimension API Combined` = paste0(`Time Dimensions API`, &#39;,&#39;, `Primary Dimensions API`)) %&gt;%<br>      mutate(`Dimension API Combined` = gsub(&#39;No Secondary Time Dimension,&#39;, &#39;&#39;, `Dimension API Combined`)) %&gt;%<br>      mutate(`Dimension Combined` = paste0(`Time Dimensions`, &#39;,&#39;, `Primary Dimensions`)) %&gt;%<br>      mutate(`Dimension Combined` = gsub(&#39;No Secondary Time Dimension,&#39;, &#39;&#39;, `Dimension Combined`))<br>  }<br>  <br>  # Combine transformed tables for different report types<br>  if (report_type_number == 1) {<br>    transformed_table_2 &lt;- transformed_table_1<br>  } else {<br>    transformed_table_2 &lt;- bind_rows(transformed_table_2, transformed_table_1)<br>  }<br>}<br><br># Some operations due to the API limit in 10 metrics per request<br># Filter rows based on the count of metrics in Metric API <br>transformed_table_3 &lt;- transformed_table_2 %&gt;%<br>  mutate(count_metrics = str_count(`Metric API`, &quot;,&quot;) + 1) %&gt;%<br>  filter(count_metrics &gt; 10) <br><br>transformed_table_4 &lt;- transformed_table_2 %&gt;%<br>  mutate(count_metrics = str_count(`Metric API`, &quot;,&quot;) + 1) %&gt;%<br>  filter(count_metrics &lt;= 10) <br><br># Determine the maximum number of metrics<br>max_metrics &lt;- max(transformed_table_3$count_metrics)<br><br># Break the metrics into groups of 10<br>break_into_tens &lt;- function(n) {<br>  tens &lt;- floor(n / 10)<br>  remainder &lt;- n %% 10<br>  result &lt;- c(rep(10, tens))<br>  if (remainder &gt; 0) {<br>    result &lt;- c(result, remainder)<br>  }<br>  return(result)<br>}<br><br>metrics_tens_groups &lt;- break_into_tens(max_metrics)<br>  <br># Function to get a chunk of metrics based on chunk number<br>get_metric_chunk &lt;- function(metric_str, chunk_num) {<br>  values &lt;- unlist(strsplit(metric_str, &quot;,&quot;))<br>  start &lt;- (chunk_num - 1) * 10 + 1<br>  end &lt;- min(chunk_num * 10, length(values))<br><br>  if (start &gt; length(values)) {<br>    return(&quot;&quot;)<br>  }<br>  <br>  chunk &lt;- values[start:end]<br>  return(paste(chunk, collapse = &quot;,&quot;))<br>}<br><br># Loop through metric chunks and create separate tables<br>for (metrics_tens_groups_number in 1:length(metrics_tens_groups)) {<br>  transformed_table_3 &lt;- transformed_table_3 %&gt;%<br>    rowwise() %&gt;%<br>    mutate(metric_part = metrics_tens_groups_number, <br>           chuncked_metric_api = get_metric_chunk(`Metric API`, metrics_tens_groups_number)) %&gt;%<br>    mutate(metric_part = metrics_tens_groups_number, <br>           chuncked_metric = get_metric_chunk(`Metric`, metrics_tens_groups_number))<br>  <br>  if (metrics_tens_groups_number == 1) {<br>    transformed_table_5 &lt;- transformed_table_3<br>  } else {<br>    transformed_table_5 &lt;- bind_rows(transformed_table_5, transformed_table_3)<br>  }<br>}<br><br># Combine transformed tables and finalize the data<br>transfromed_table_7 &lt;- bind_rows(transformed_table_4, transformed_table_5) %&gt;%<br>  select(-count_metrics) %&gt;%<br>  mutate(Metric = if_else(is.na(chuncked_metric), Metric, chuncked_metric)) %&gt;% <br>  mutate(`Metric API` = if_else(is.na(chuncked_metric_api), `Metric API`, chuncked_metric_api)) %&gt;%<br>  select(-chuncked_metric, -chuncked_metric_api) %&gt;%<br>  mutate(metric_part2 = paste0(&quot;metric_part_&quot;, metric_part, &quot;.csv&quot;)) %&gt;%<br>  mutate(path_file = if_else(is.na(metric_part), path_file, gsub(&quot;.csv$&quot;, &quot;&quot;, path_file))) %&gt;%<br>  mutate(path_file = if_else(is.na(metric_part), path_file, paste0(path_file, &#39;_&#39;, metric_part2))) %&gt;%<br>  select(-metric_part2, -metric_part)<br>}</pre><h4>Create folders and dictionary</h4><p>If you are managing numerous reports, a dictionary is essential for navigating all the data effectively. Moreover, this section of the script is responsible for creating directories to store all the files.</p><pre># Write a vocabulary file<br>write_csv(<br>  transfromed_table_7 %&gt;%<br>    select(<br>      path_folder, path_file, Metric, `Primary Dimensions`,<br>      `Secondary Dimensions`, `Time Dimensions`, `Dimension Combined`<br>    ),<br>  paste0(dir_path, &#39;/vocabulary.csv&#39;)<br>)<br><br># Create a list of distinct folder paths<br>folders_list &lt;- transfromed_table_7 %&gt;%<br>  select(path_folder) %&gt;%<br>  distinct()<br><br># Loop through each unique folder path in folders_list<br>for (folder_number in 1:n_distinct(folders_list$path_folder)) {<br>  <br>  # Check if the folder does not exist<br>  if (!file.exists(folders_list$path_folder[folder_number])) {<br>    <br>    # Create the folder if it does not exist, including parent directories<br>    dir.create(folders_list$path_folder[folder_number], recursive = TRUE)<br>  }<br>}</pre><h4>Progress tracking</h4><p>If you are dealing with a substantial volume of data and numerous reports, it is evident that the script will be executed multiple times. I suggest maintaining a variable to track the completion percentage of the reports from the overall list.</p><pre># Create a data frame &#39;file_list&#39; from &#39;transfromed_table_7&#39;<br>file_list &lt;- transfromed_table_7 %&gt;%<br>  select(path_file) %&gt;%<br>  distinct() %&gt;%<br>  mutate(status = ifelse(file.exists(path_file), 1, 0))<br><br># Calculate the archiving status as a percentage<br>archiving_status &lt;- paste0(<br>  round(sum(file_list$status) / n_distinct(file_list$path_file) * 100, digits = 4),<br>  &quot;%&quot;<br>)<br><br># Now &#39;archiving_status&#39; contains the percentage of existing files in &#39;path_file&#39;<br># You can use &#39;archiving_status&#39; as needed in your further analysis or reporting.</pre><h4>Requests to Google Analytics</h4><p>The core of the script is here. It sets up the options for querying the API, tracks any errors that occur, logs the outcomes of operations, and allows for customization and multiple executions as required.</p><pre># Define the Google Analytics View ID that you want to retrieve data for.<br>ga_id &lt;- 1<br><br># Specify the date range for the Google Analytics data you are interested in.<br>date_range &lt;- c(&quot;2012-01-01&quot;, &quot;2023-08-01&quot;)<br><br># Loop over each row in the transformed_table_7 data frame.<br>for (google_report_number in 1:nrow(transfromed_table_7)) {<br><br>  # Check if the file does not already exist in the specified path.<br>  if (!file.exists(transfromed_table_7$path_file[google_report_number])) {<br><br>    # Print a message indicating that the file is about to be exported.<br>    print(paste(&quot;File is going to be exported:&quot;, transfromed_table_7$path_file[google_report_number]))<br>    <br>    # Initialize the list of arguments for the google_analytics function call.<br>    ga_args &lt;- list(<br>        ga_id, # Presumably defined earlier; this is the Google Analytics ID.<br>        date_range = date_range, # Presumably defined earlier; the date range for the report.<br>        # Extract the metrics from the current row&#39;s &#39;Metric API&#39; column.<br>        metrics = unlist(strsplit(transfromed_table_7$`Metric API`[google_report_number], &quot;,&quot;)),<br>        # Extract the dimensions from the current row&#39;s &#39;Dimension API Combined&#39; column.<br>        dimensions = unlist(strsplit(transfromed_table_7$`Dimension API Combined`[google_report_number], &quot;,&quot;)),<br>        anti_sample = TRUE  # Argument to control anti-sampling in Google Analytics queries.<br>)<br><br># Add dimension filters to the argument list if they exist for the current report.<br>if (!is.na(transfromed_table_7$`Filter Dimension`[google_report_number])) {<br>  ga_args$dim_filters &lt;- filter_clause_ga4(list(dim_filter(<br>  dimension = unlist(strsplit(transfromed_table_7$`Filter Dimension`[google_report_number], &quot;,&quot;)), <br>  operator = &quot;IN_LIST&quot;,<br>  caseSensitive = TRUE,<br>  expressions = unlist(strsplit(transfromed_table_7$`Filter Dimension Values`[google_report_number], &quot;,&quot;))<br>)))<br>}<br><br>     # Try to retrieve data from Google Analytics, and handle any errors that may occur.<br>    data &lt;- tryCatch({<br><br>      # Execute the Google Analytics query with the specified arguments<br>      do.call(google_analytics, ga_args)<br><br>      # Print an error message if the data retrieval fails.<br><br>    }, error = function(e) {<br>      cat(&quot;Error in retrieving data from Google Analytics:&quot;, e$message, &quot;\n&quot;)<br>      stop(&quot;Stopping execution due to error.&quot;)<br>    })<br>    <br>    # Check if no data is returned from the Google Analytics query. <br>    if (is.null(data)) {<br><br>      # Create an empty data frame with a specific message.<br>      data &lt;- data.frame(&#39;no_possible_values_for_this_combination_of_dimensions&#39;)<br>      <br>      # Write the empty data frame to a CSV file in the specified path.<br>      write_csv(data, transfromed_table_7$path_file[google_report_number])<br>    } else {<br><br>      # Write the retrieved data to a CSV file in the specified path.<br>      write_csv(data, transfromed_table_7$path_file[google_report_number])<br>    }<br>    <br>    # Print a message indicating that the file has been created.<br>    print(paste(&quot;File is being created:&quot;, transfromed_table_7$path_file[google_report_number]))<br>    <br>    # Update the file list to include the new file&#39;s status.<br>    file_list &lt;- transfromed_table_7 %&gt;% <br>  select(path_file) %&gt;% <br>  distinct() %&gt;%<br>  mutate(status = ifelse(file.exists(path_file), 1, 0))<br><br>  } else {<br>    # If the report file already exists, print a message indicating so. <br>    print(paste0(&quot;Already exist: &quot;, transfromed_table_7$path_file[google_report_number]))<br>  }<br>}</pre><p>If configured properly, the resulting console output should resemble the following. Given that we have enabled the non-sampling mode (a very advantageous API feature), there will be a multitude of API calls. Notice that with each file creation, a corresponding notification will be generated to inform us.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qVPd7PdcuDY68GSHo6otBg.png" /></figure><p>Ultimately, if you exceed the API limit, you will immediately see a 403 or 429 error in the console, indicating that you have reached the daily quota.</p><blockquote>You may accumulate folders filled with numerous CSV files, which can then be imported into Sheets or another Google Storage platform. Afterward, you can proceed to visualize the data if necessary.</blockquote><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*bAb7b9BT9wdDNX16" /></figure><h3>Post scriptum</h3><p>If you find yourself needing to use segments (I did not have such a need), <a href="https://code.markedmondson.me/googleAnalyticsR/articles/v4.html#segments-v4">you can learn more about it here</a>. This information will help to enhance the script mentioned above and facilitate the automation of segment usage. Should any questions arise, feel free to write to me — I will be glad to respond.</p><p>Your archiving requirements will likely differ. I would be keen to learn about your approaches. The possibilities with the API are extensive. For instance, it can easily facilitate the basic download of hit &amp; session data from Universal Analytics. The main blocker is the daily limit of 10,000 users. For those who are interested, <a href="https://code.markedmondson.me/googleAnalyticsR/articles/user-activity.html">take a look at the User Activity API.</a></p><p>If you are not using R, challenge ChatGPT with my script pieces as prompts. It will greatly simplify the process of rewriting in another language.</p><p>If you find value in this article and it helps your work, I would deeply appreciate your applause.</p><p>Such support inspires me to create more valuable content.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d37752f20f0d" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>