<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Weiming Chen on Medium]]></title>
        <description><![CDATA[Stories by Weiming Chen on Medium]]></description>
        <link>https://medium.com/@weimingchenzero?source=rss-7d453923e23------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*7fYazD9C67mbu2Ic4ZVQ1Q.jpeg</url>
            <title>Stories by Weiming Chen on Medium</title>
            <link>https://medium.com/@weimingchenzero?source=rss-7d453923e23------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 16:41:48 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@weimingchenzero/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Comprehensive Research Report: Modern Technologies and Techniques for Mobile App Data Scraping]]></title>
            <link>https://weimingchenzero.medium.com/comprehensive-research-report-modern-technologies-and-techniques-for-mobile-app-data-scraping-c5bb78100acd?source=rss-7d453923e23------2</link>
            <guid isPermaLink="false">https://medium.com/p/c5bb78100acd</guid>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Weiming Chen]]></dc:creator>
            <pubDate>Tue, 03 Jun 2025 05:15:22 GMT</pubDate>
            <atom:updated>2025-06-03T05:15:22.633Z</atom:updated>
            <content:encoded><![CDATA[<h4>Introduction</h4><p>This report provides an in-depth analysis of modern technologies and techniques used for data scraping from native mobile applications (iOS and Android). It focuses on methods designed to bypass common anti-scraping measures and offers a comparative evaluation of their effectiveness, scalability, and maintainability. The research covers reverse engineering, automation frameworks, OCR, proxy rotation, ML/AI solutions, and crucial ethical and legal considerations.</p><p>— -</p><h4>1. Reverse Engineering Techniques for Mobile App APIs</h4><h4>Overview</h4><p>This document captures key information on reverse engineering techniques for mobile app APIs, focusing on methods to bypass anti-scraping measures. The research covers tools, tactics, and procedures for both Android and iOS platforms.</p><h4>Key Tools for Reverse Engineering</h4><h4>Android Tools</h4><ul><li><strong>APKTool</strong>: Open-source tool for decompiling, modifying, and recompiling Android applications</li><li><strong>JADX</strong>: Java decompiler that converts bytecode into readable Java source code</li><li><strong>HTTP Toolkit</strong>: Open-source tool for intercepting and analyzing HTTP(S) traffic</li></ul><h4>iOS Tools</h4><ul><li><strong>IDA Pro</strong>: Powerful disassembler for iOS applications</li><li><strong>Hopper</strong>: Disassembler that converts binary code to human-readable assembly</li><li><strong>Ghidra</strong>: Open-source software reverse engineering tool developed by NSA</li><li><strong>Radare2</strong>: Open-source disassembler and debugger</li></ul><h4>Cross-Platform Tools</h4><ul><li><strong>Frida</strong>: Dynamic instrumentation framework for both Android and iOS that allows injecting JavaScript code into running apps</li></ul><h4>Tactics for Reverse Engineering</h4><h4>Static Analysis</h4><ul><li>Examining app code without execution</li><li>Identifying hard-coded data, third-party libraries, and potential security flaws</li><li>Inspecting source code or disassembled code</li></ul><h4>Dynamic Analysis</h4><ul><li>Running the app while monitoring behavior</li><li>Intercepting function calls</li><li>Inspecting memory and network traffic</li></ul><h4>Decompilation</h4><ul><li>Converting compiled code to high-level programming language</li><li>Understanding app source code, logic, and structure</li><li>Identifying vulnerabilities and security issues</li></ul><h4>Hooking and Instrumentation</h4><ul><li>Intercepting function calls</li><li>Modifying or augmenting app behavior</li><li>Gaining deeper insights into functionality</li></ul><h4>Procedures for API Reverse Engineering</h4><ol><li><strong>Environment Setup</strong></li><li>Install relevant tools</li><li>Configure virtualized or physical devices</li><li><strong>App Acquisition</strong></li><li>Download from official app stores</li><li>Extract APK from Android device</li><li>Acquire iOS app from Apple device</li><li><strong>Decompilation and Code Review</strong></li><li>Use appropriate tools based on platform</li><li>Review extracted source code</li><li>Look for API endpoints, authentication methods</li><li><strong>Traffic Interception</strong></li><li>Set up proxy to intercept HTTPS traffic</li><li>Install custom certificates</li><li>Bypass certificate pinning if necessary</li><li><strong>Certificate Pinning Bypass Techniques</strong></li><li>Using older Android versions (Android 6 or earlier)</li><li>Using emulators with system-wide certificates</li><li>Runtime manipulation with Frida</li><li>Code modification in APK before installation</li></ol><h4>Challenges in Modern Mobile App Reverse Engineering</h4><h4>Android Challenges</h4><ul><li>Recent Android versions don’t trust user-provided certificates by default</li><li>Certificate pinning implementations in apps</li><li>Need for rooted devices or emulators to add system-wide certificates</li></ul><h4>Certificate Pinning Workarounds</h4><ul><li>Using Android 6 emulator (less restrictive with certificates)</li><li>Finding older app versions compatible with Android 6</li><li>Using APKMirror to locate older APK versions</li><li>Matching architecture for APK with emulator configuration</li></ul><h4>Practical Example: Bypassing Certificate Pinning</h4><ol><li>Set up Android 6 on an emulator</li><li>Find a compatible version of the target app</li><li>Install the app on the emulator</li><li>Set up HTTP Toolkit to intercept traffic</li><li>Start the app and observe unencrypted API traffic</li></ol><h4>Benefits of API Reverse Engineering</h4><ul><li>Deeper understanding of API capabilities</li><li>More efficient troubleshooting of integration issues</li><li>Improved application performance</li><li>Accelerated development cycles</li><li>Legacy system integration</li><li>Security analysis and vulnerability identification</li></ul><p>— -</p><h4>2. Automation Tools and Frameworks for Mobile App Scraping</h4><h4>Overview</h4><p>This section provides detailed information on automation tools and frameworks used for data scraping from native mobile applications, with a focus on their capabilities, strengths, and limitations.</p><h4>Key Automation Frameworks</h4><h4>1. Appium</h4><p>Appium is a popular open-source framework for automating mobile applications across platforms.</p><p><strong>Key Features:</strong></p><ul><li><strong>Cross-platform compatibility</strong>: Works with iOS, Android, and Windows applications</li><li><strong>Language support</strong>: Supports multiple programming languages (Java, JavaScript, Python, Ruby, C#)</li><li><strong>No app modification</strong>: Doesn’t require modifying the app’s source code</li><li><strong>WebDriver protocol</strong>: Uses the same API as Selenium WebDriver</li><li><strong>Application types</strong>: Supports native, hybrid, and mobile web applications</li><li><strong>Framework integration</strong>: Works with React Native, Xamarin, and Flutter apps</li></ul><p><strong>Strengths for Data Scraping:</strong></p><ul><li>Ability to automate UI interactions to navigate through apps</li><li>Can extract text and data from screen elements</li><li>Supports screenshot capture for visual data extraction</li><li>Can interact with hidden or dynamic elements</li></ul><p><strong>Limitations:</strong></p><ul><li>Slower execution compared to native frameworks</li><li>May struggle with complex animations and gestures</li><li>Requires device-specific setup and configuration</li><li>Test stability issues with complex applications</li></ul><h4>2. Frida</h4><p>Frida is a dynamic instrumentation toolkit that allows developers to inject JavaScript into native apps on Windows, macOS, iOS, Android, and Linux.</p><p><strong>Key Features:</strong></p><ul><li><strong>Runtime manipulation</strong>: Injects custom JavaScript into running processes</li><li><strong>Hooking capabilities</strong>: Can intercept function calls and modify their behavior</li><li><strong>Cross-platform</strong>: Works across multiple operating systems</li><li><strong>Scriptable</strong>: Provides a powerful scripting API</li><li><strong>Non-invasive</strong>: Doesn’t require app modification or recompilation</li></ul><p><strong>Strengths for Data Scraping:</strong></p><ul><li>Can access internal app data structures directly</li><li>Ability to bypass security measures like certificate pinning</li><li>Can hook into network calls to intercept API responses</li><li>Enables extraction of encrypted or protected data</li><li>Can automate complex interactions within the app</li></ul><p><strong>Limitations:</strong></p><ul><li>Steep learning curve</li><li>Requires understanding of app internals</li><li>Scripts may break with app updates</li><li>Potential legal and ethical concerns</li></ul><h4>3. XCUITest (iOS)</h4><p>Apple’s native UI testing framework for iOS applications.</p><p><strong>Key Features:</strong></p><ul><li><strong>Native integration</strong>: Built and maintained by Apple</li><li><strong>Swift/Objective-C</strong>: Uses the same languages as iOS development</li><li><strong>Xcode integration</strong>: Seamless integration with Apple’s development environment</li><li><strong>Performance</strong>: Fast execution compared to cross-platform tools</li><li><strong>Device features</strong>: Access to iOS-specific features and capabilities</li></ul><p><strong>Strengths for Data Scraping:</strong></p><ul><li>High performance on iOS devices</li><li>Reliable element identification</li><li>Access to iOS accessibility features</li><li>Stable automation for complex iOS interfaces</li></ul><p><strong>Limitations:</strong></p><ul><li>iOS only, not cross-platform</li><li>Requires Apple developer tools and macOS</li><li>Limited to what Apple allows through public APIs</li></ul><h4>4. Espresso (Android)</h4><p>Google’s native UI testing framework for Android applications.</p><p><strong>Key Features:</strong></p><ul><li><strong>Native integration</strong>: Built and maintained by Google</li><li><strong>Java/Kotlin</strong>: Uses Android’s primary development languages</li><li><strong>Android Studio integration</strong>: Seamless integration with Android’s IDE</li><li><strong>Performance</strong>: Fast and reliable execution</li><li><strong>Synchronization</strong>: Automatically waits for UI events to complete</li></ul><p><strong>Strengths for Data Scraping:</strong></p><ul><li>High performance on Android devices</li><li>Reliable element identification</li><li>Deep integration with Android’s accessibility features</li><li>Stable for complex Android interfaces</li></ul><p><strong>Limitations:</strong></p><ul><li>Android only, not cross-platform</li><li>Limited to what Google allows through public APIs</li><li>Requires Android development knowledge</li></ul><h4>Comparative Analysis for Data Scraping</h4><h4>Cross-Platform Capabilities</h4><ul><li><strong>Best</strong>: Appium, Frida</li><li><strong>Limited</strong>: XCUITest (iOS only), Espresso (Android only)</li></ul><h4>Depth of Access to App Data</h4><ul><li><strong>Deepest</strong>: Frida (can access internal app structures)</li><li><strong>Moderate</strong>: Appium (UI level access)</li><li><strong>Limited</strong>: XCUITest, Espresso (primarily UI level)</li></ul><h4>Ease of Implementation</h4><ul><li><strong>Easiest</strong>: Espresso (for Android)</li><li><strong>Moderate</strong>: Appium, XCUITest (for iOS)</li><li><strong>Most Complex</strong>: Frida (requires deeper technical knowledge)</li></ul><h4>Stability and Maintenance</h4><ul><li><strong>Most Stable</strong>: XCUITest (iOS), Espresso (Android)</li><li><strong>Moderate</strong>: Appium</li><li><strong>Variable</strong>: Frida (depends on app complexity and updates)</li></ul><h4>Security Bypass Capabilities</h4><ul><li><strong>Strongest</strong>: Frida</li><li><strong>Moderate</strong>: Appium with plugins</li><li><strong>Limited</strong>: XCUITest, Espresso</li></ul><h4>Integration with Other Scraping Techniques</h4><h4>With Reverse Engineering</h4><ul><li>Frida provides the strongest integration with reverse engineering efforts</li><li>Can be used to implement findings from static analysis</li><li>Enables dynamic modification of app behavior based on discovered API endpoints</li></ul><h4>With OCR Solutions</h4><ul><li>Appium can be combined with OCR libraries to extract text from non-accessible elements</li><li>Screenshots captured through automation frameworks can be processed with OCR tools</li></ul><h4>With Proxy/IP Rotation</h4><ul><li>Automation frameworks can be configured to use proxies</li><li>Can programmatically change proxy settings during execution</li></ul><h4>With ML/AI Solutions</h4><ul><li>Automation scripts can incorporate ML models for intelligent navigation</li><li>Can use computer vision APIs to identify and interact with visual elements</li></ul><p>— -</p><h4>3. OCR for Image-Based Data Extraction from Mobile Apps</h4><h4>Overview</h4><p>This section provides comprehensive information on Optical Character Recognition (OCR) techniques for extracting data from mobile applications, particularly when dealing with image-based content that cannot be accessed through traditional API methods.</p><h4>Understanding OCR Technology</h4><h4>Core Components of OCR</h4><ol><li><strong>Image Preprocessing</strong></li><li>Enhancing image quality for better text recognition</li><li>Techniques include noise reduction, binarization, and normalization</li><li>Critical for handling variable quality of mobile screenshots</li><li><strong>Text Detection</strong></li><li>Identifying regions containing text within images</li><li>Detecting lines, words, individual characters, and special symbols</li><li>Crucial for isolating relevant data from complex mobile interfaces</li><li><strong>Character Recognition</strong></li><li>Converting detected characters into digital text</li><li>Uses algorithms and machine learning models trained on various fonts and styles</li><li>Core functionality that determines OCR accuracy</li><li><strong>Post-processing</strong></li><li>Error correction and text reconstruction</li><li>Using language models to improve accuracy and context recognition</li><li>Formatting output text for further processing</li></ol><h4>Challenges of OCR in Mobile App Scraping</h4><h4>Image Quality and Variability</h4><ul><li>Mobile screenshots vary in quality, resolution, and lighting conditions</li><li>Blurry images and low-resolution captures significantly impact OCR accuracy</li><li>Preprocessing techniques are essential to enhance image clarity</li></ul><h4>Diverse Text Styles and Formats</h4><ul><li>Mobile apps use various fonts, styles, and text layouts</li><li>Custom fonts and stylized text can be difficult for standard OCR engines</li><li>Training on diverse datasets improves recognition of different text styles</li></ul><h4>Multilingual Text Recognition</h4><ul><li>Global apps require support for multiple languages and scripts</li><li>Complex scripts and unique character sets present additional challenges</li><li>Language detection algorithms help identify and switch to appropriate models</li></ul><h4>OCR Implementation for Mobile App Scraping</h4><h4>Open-Source OCR SDKs</h4><p><strong>Tesseract OCR</strong></p><ul><li>Leading open-source OCR engine with strong text recognition capabilities</li><li>Platform-specific wrappers: Tesseract4Android for Android and TesseractOCRiOS for iOS</li><li>Latest version (5.5.0) offers improved accuracy and performance</li><li>Supports over 100 languages and can be trained for custom fonts</li></ul><h4>Cloud-Based OCR Services</h4><p><strong>Google Cloud Vision API</strong></p><ul><li>Powerful OCR capabilities with high accuracy</li><li>Supports multiple languages and text styles</li><li>Handles complex layouts and mixed content</li><li>Provides structured data extraction</li></ul><p><strong>Microsoft Azure Computer Vision</strong></p><ul><li>Advanced OCR with layout analysis</li><li>Good performance with mobile screenshots</li><li>Supports multiple languages and handwritten text</li><li>Offers both synchronous and asynchronous processing</li></ul><p><strong>Amazon Textract</strong></p><ul><li>Specialized in extracting structured data from documents</li><li>Can identify form fields and tables</li><li>Useful for processing app screens with structured information</li><li>Provides confidence scores for extracted text</li></ul><h4>Optimizing OCR for Mobile App Scraping</h4><h4>Image Preprocessing Techniques</h4><ul><li>Convert images to grayscale for better contrast</li><li>Apply adaptive thresholding or binarization</li><li>Use Gaussian blur to reduce noise</li><li>Ensure optimal resolution (300–400 DPI recommended)</li><li>Crop images to focus on relevant text areas</li></ul><h4>Performance Optimization</h4><ul><li>Reuse OCR engine instances for multiple images</li><li>Implement proper resource management</li><li>Cache results for frequently processed screens</li><li>Use background threading for OCR operations</li><li>Consider batch processing for multiple screenshots</li></ul><h4>Accuracy Improvement Strategies</h4><ul><li>Train custom models for specific app fonts and layouts</li><li>Use context-aware post-processing</li><li>Implement dictionary-based correction for domain-specific terms</li><li>Combine results from multiple OCR engines for verification</li><li>Apply regular expressions to validate extracted data formats</li></ul><h4>Integration with Other Scraping Techniques</h4><h4>Combining OCR with UI Automation</h4><ul><li>Use automation frameworks to navigate through apps</li><li>Capture screenshots at specific points for OCR processing</li><li>Extract data that cannot be accessed through UI element properties</li><li>Create workflows that combine direct UI interaction and OCR</li></ul><h4>Hybrid Approaches</h4><ul><li>Use API scraping when possible for structured data</li><li>Fall back to OCR for image-based content</li><li>Combine results from multiple sources for complete data extraction</li><li>Validate OCR results against known patterns or expected formats</li></ul><h4>Practical Applications in Mobile App Scraping</h4><h4>Financial Data Extraction</h4><ul><li>Capturing transaction details from banking apps</li><li>Extracting investment information from brokerage apps</li><li>Reading receipt and invoice data from payment apps</li></ul><h4>E-commerce and Retail</h4><ul><li>Extracting product details and pricing information</li><li>Capturing promotional offers and discounts</li><li>Reading reviews and ratings</li></ul><h4>Social Media and Content</h4><ul><li>Extracting post content and engagement metrics</li><li>Reading user profiles and activity data</li><li>Capturing trending topics and hashtags</li></ul><p>— -</p><h4>4. Proxy Rotation and IP Masking Strategies for Mobile App Scraping</h4><h4>Overview</h4><p>This section provides comprehensive information on proxy rotation and IP masking strategies specifically for mobile app data scraping, focusing on techniques to bypass anti-scraping measures and maintain uninterrupted data collection.</p><h4>Understanding IP Rotation</h4><h4>What is IP Rotation?</h4><p>IP rotation involves switching between multiple IP addresses to distribute outgoing traffic across different machines. In the context of mobile app scraping, this technique helps hide the original IP address, bypass rate limits, access geo-restricted content, and evade anti-bot detection systems.</p><h4>Benefits of IP Rotation for Mobile App Scraping</h4><ul><li><strong>Enhanced anonymity</strong>: Constantly changing IP addresses makes it difficult for mobile apps to identify and profile scraping activities</li><li><strong>Improved security</strong>: Protects against potential threats by making it harder to target specific devices</li><li><strong>Bypass rate limits</strong>: Distributes requests across multiple IPs to avoid triggering rate-limiting mechanisms</li><li><strong>Access geo-restricted content</strong>: Changes apparent location to access region-locked features or content</li><li><strong>Avoid IP bans</strong>: Prevents permanent blocking by distributing traffic across multiple addresses</li><li><strong>Improved connectivity</strong>: Circumvents intentional speed throttling by some service providers</li></ul><h4>Types of IPs for Rotation in Mobile App Scraping</h4><h4>Datacenter IPs</h4><ul><li><strong>Description</strong>: IPs provided by servers owned by cloud hosting providers</li><li><strong>Advantages</strong>: Inexpensive, fast, and available in large quantities</li><li><strong>Disadvantages</strong>: Easily detectable by sophisticated anti-scraping systems</li><li><strong>Use case</strong>: Best for non-sensitive scraping tasks where detection is less concerning</li></ul><h4>ISP IPs</h4><ul><li><strong>Description</strong>: Server-hosted IPs registered with internet providers</li><li><strong>Advantages</strong>: More natural appearance than datacenter IPs, less likely to be blocked</li><li><strong>Disadvantages</strong>: More expensive than datacenter IPs, limited availability</li><li><strong>Use case</strong>: Medium-sensitivity scraping tasks requiring better detection avoidance</li></ul><h4>Residential IPs</h4><ul><li><strong>Description</strong>: IP addresses assigned to regular internet users by network providers</li><li><strong>Advantages</strong>: Appear as genuine user traffic, high success rate against anti-scraping measures</li><li><strong>Disadvantages</strong>: Expensive, potentially slower than datacenter options</li><li><strong>Use case</strong>: High-sensitivity scraping tasks requiring maximum anonymity</li></ul><h4>Mobile IPs</h4><ul><li><strong>Description</strong>: IP addresses assigned to cellular networks</li><li><strong>Advantages</strong>: Mimic mobile device characteristics, change frequently as devices move between cell towers</li><li><strong>Disadvantages</strong>: Most expensive option, can have variable connection quality</li><li><strong>Use case</strong>: Mobile app-specific scraping where appearing as a genuine mobile user is critical</li></ul><h4>Implementation Strategies for Mobile App Scraping</h4><h4>Proxy Pools and Management</h4><ul><li><strong>Creating proxy pools</strong>: Gathering and organizing multiple proxies from various sources</li><li><strong>Proxy verification</strong>: Testing proxies for functionality, speed, and anonymity</li><li><strong>Proxy categorization</strong>: Organizing proxies by type, location, and performance metrics</li><li><strong>Rotation algorithms</strong>: Implementing intelligent rotation based on success rates and response times</li></ul><h4>Session Management</h4><ul><li><strong>Session persistence</strong>: Maintaining consistent sessions across IP changes when required</li><li><strong>Cookie and header management</strong>: Preserving necessary authentication data between requests</li><li><strong>Device fingerprinting mitigation</strong>: Techniques to maintain consistent device profiles despite IP changes</li></ul><h4>Rotation Patterns</h4><ul><li><strong>Time-based rotation</strong>: Changing IPs at fixed time intervals</li><li><strong>Request-based rotation</strong>: Rotating after a specific number of requests</li><li><strong>Failure-based rotation</strong>: Changing IPs when encountering errors or blocks</li><li><strong>Random rotation</strong>: Unpredictable IP changes to avoid pattern detection</li><li><strong>Intelligent rotation</strong>: Algorithm-based decisions considering success rates and block patterns</li></ul><h4>Advanced Techniques for Mobile App Scraping</h4><h4>Mobile-Specific Proxy Configuration</h4><ul><li><strong>Mobile user agent rotation</strong>: Cycling through realistic mobile user agents alongside IP rotation</li><li><strong>Mobile carrier emulation</strong>: Using proxies that mimic specific mobile carriers’ network characteristics</li><li><strong>Location-based rotation</strong>: Changing apparent geographic location to match target app’s expectations</li></ul><h4>Handling Mobile App Authentication</h4><ul><li><strong>Token preservation</strong>: Maintaining authentication tokens across proxy rotations</li><li><strong>Device ID consistency</strong>: Ensuring consistent device identification despite IP changes</li><li><strong>Session management</strong>: Techniques for maintaining logged-in sessions during proxy rotation</li></ul><h4>Best Practices for Effective IP Rotation</h4><h4>Technical Optimization</h4><ul><li><strong>Intelligent rotation timing</strong>: Adjust rotation frequency based on app’s rate limiting patterns</li><li><strong>Proxy health monitoring</strong>: Continuously verify proxy performance and remove problematic IPs</li><li><strong>Request throttling</strong>: Implement delays between requests to mimic human behavior</li><li><strong>Header consistency</strong>: Maintain consistent headers across IP changes when appropriate</li></ul><h4>Risk Mitigation</h4><ul><li><strong>Gradual scaling</strong>: Start with low request volumes and gradually increase</li><li><strong>Distributed scraping</strong>: Spread requests across multiple source machines</li><li><strong>Fallback mechanisms</strong>: Implement alternative data collection methods when proxies fail</li><li><strong>Monitoring and alerting</strong>: Set up systems to detect blocking or unusual response patterns</li></ul><p>— -</p><h4>5. Modern ML/AI/GenAI Solutions for Mobile App Data Scraping</h4><h4>Overview</h4><p>This section explores the cutting-edge machine learning (ML), artificial intelligence (AI), and generative AI (GenAI) solutions for mobile app data scraping, focusing on methods that can bypass common anti-scraping measures and enhance data extraction capabilities.</p><h4>AI-Powered Scraping Approaches</h4><h4>Agentic AI for Automated Scraping</h4><p>Agentic AI refers to autonomous AI systems that can perform complex tasks with minimal human intervention. In the context of mobile app scraping, these agents can:</p><ul><li><strong>Autonomously navigate</strong> through mobile app interfaces</li><li><strong>Identify and extract</strong> relevant data elements</li><li><strong>Adapt to changes</strong> in app structure or anti-scraping measures</li><li><strong>Self-correct</strong> when encountering errors or unexpected scenarios</li></ul><h4>Computer Vision for UI Element Recognition</h4><p>Computer vision techniques have become essential for mobile app scraping, especially when dealing with apps that use non-standard UI elements or image-based content:</p><p><strong>Deep Learning Models for Element Detection</strong></p><ul><li>Object Detection Networks like YOLO or Faster R-CNN can identify UI elements like buttons, text fields, and images within app screenshots</li><li>Instance Segmentation techniques can separate individual UI elements even when they overlap</li><li>UI Element Classification models trained to recognize specific types of UI components across different app designs</li></ul><p><strong>Visual Relationship Understanding</strong></p><ul><li>Spatial Relationship Analysis for understanding the layout and relationships between UI elements</li><li>Action Prediction for determining which elements are interactive and what actions can be performed</li><li>Visual Hierarchy Mapping for determining the hierarchical structure of UI elements</li></ul><h4>Natural Language Processing for Content Extraction</h4><p>NLP techniques are crucial for understanding and extracting meaningful information from text within mobile apps:</p><p><strong>Text Extraction and Understanding</strong></p><ul><li>Named Entity Recognition (NER) for identifying and classifying key information like names, prices, dates, and locations</li><li>Sentiment Analysis for determining sentiment in user reviews, comments, or other text content</li><li>Topic Modeling for categorizing content into relevant topics or themes</li><li>Text Summarization for creating concise summaries of longer text content</li></ul><h4>GenAI Applications in Mobile App Scraping</h4><h4>LLM-Based Code Generation</h4><p>Large Language Models (LLMs) can generate custom scraping code tailored to specific mobile apps:</p><p><strong>Automated Script Creation</strong></p><ul><li>App-Specific Code Generation for creating extraction scripts customized for each target app</li><li>Code Optimization for refining scripts for efficiency and detection avoidance</li><li>Multi-platform Support for generating code for different mobile platforms (iOS, Android)</li></ul><p><strong>Self-Improving Code</strong></p><ul><li>Iterative Refinement where LLMs can analyze failed attempts and improve code accordingly</li><li>Error Handling generation for various scenarios</li><li>Performance Optimization by continuously improving code efficiency based on results</li></ul><h4>Synthetic Data Generation</h4><p>GenAI can create synthetic data to train scraping models or test scraping systems:</p><p><strong>Training Data Creation</strong></p><ul><li>UI Mock Generation for creating synthetic mobile app interfaces for training computer vision models</li><li>Variation Generation for producing multiple variations of UI elements to improve model robustness</li><li>Edge Case Simulation for generating unusual or rare scenarios to improve model handling</li></ul><h4>ML/AI for Anti-Scraping Bypass</h4><h4>Fingerprinting Evasion</h4><p>Machine learning models can help evade device fingerprinting techniques:</p><p><strong>Browser and Device Fingerprint Manipulation</strong></p><ul><li>Feature Randomization by intelligently randomizing browser and device characteristics</li><li>Consistency Preservation by ensuring all fingerprint elements remain consistent</li><li>Realistic Variation by generating variations that mimic real device diversity</li></ul><p><strong>Behavioral Fingerprinting Evasion</strong></p><ul><li>Human-like Interaction by simulating realistic mouse movements, typing patterns, and navigation</li><li>Timing Naturalization by adding natural variations in timing between actions</li><li>Context-Aware Behavior by adjusting behavior based on the context within the app</li></ul><h4>CAPTCHA and Challenge Solving</h4><p>AI systems can overcome various verification challenges:</p><p><strong>Visual CAPTCHA Solving</strong></p><ul><li>Image Recognition for identifying objects, text, or patterns in visual CAPTCHAs</li><li>Puzzle Solving for completing puzzle-based challenges</li><li>Semantic Understanding for solving CAPTCHAs that require understanding context or meaning</li></ul><h4>Integration with Other Scraping Techniques</h4><h4>Hybrid Approaches</h4><p>Combining ML/AI with traditional scraping techniques creates powerful hybrid solutions:</p><p><strong>AI-Enhanced Traditional Scraping</strong></p><ul><li>Intelligent Retry Logic using ML to determine optimal retry strategies</li><li>Dynamic Selector Adjustment by automatically updating selectors when app structure changes</li><li>Priority-Based Extraction using AI to prioritize the most valuable data for extraction</li></ul><p><strong>Multi-Method Orchestration</strong></p><ul><li>Method Selection by intelligently choosing the best scraping method for each target</li><li>Fallback Chains by creating sophisticated fallback mechanisms when primary methods fail</li><li>Parallel Processing by coordinating multiple scraping approaches simultaneously</li></ul><h4>Challenges and Limitations</h4><h4>Ethical and Legal Boundaries</h4><p>AI-powered scraping must navigate complex ethical and legal considerations:</p><ul><li><strong>Terms of Service Compliance</strong>: Many apps explicitly prohibit automated data extraction</li><li><strong>Data Privacy Regulations</strong>: Compliance with GDPR, CCPA, and other privacy laws</li><li><strong>Intellectual Property Rights</strong>: Respecting copyright and other IP protections</li><li><strong>Competitive Intelligence Boundaries</strong>: Understanding legal limits on competitor analysis</li></ul><h4>Technical Challenges</h4><p>Even advanced AI systems face significant technical hurdles:</p><ul><li><strong>App Update Adaptation</strong>: Rapidly adapting to frequent app updates</li><li><strong>Anti-AI Measures</strong>: Dealing with countermeasures specifically designed to detect AI</li><li><strong>Resource Intensity</strong>: Managing the computational resources required for AI-powered scraping</li><li><strong>Accuracy and Reliability</strong>: Ensuring consistent data quality across different app versions</li></ul><p>— -</p><h4>6. Comparative Analysis of Mobile App Data Scraping Techniques</h4><p>This section provides a comprehensive comparative analysis of various mobile app data scraping techniques, evaluating their effectiveness, scalability, and maintainability based on extensive research and practical implementation experience.</p><h4>Evaluation Criteria</h4><h4>Effectiveness</h4><ul><li><strong>Data Access Depth</strong>: How deeply the technique can access app data (surface-level UI vs. internal API data)</li><li><strong>Accuracy</strong>: How accurately the technique extracts the intended data</li><li><strong>Bypass Capability</strong>: Ability to bypass anti-scraping measures</li><li><strong>Data Completeness</strong>: Ability to extract all required data types (text, images, structured data)</li></ul><h4>Scalability</h4><ul><li><strong>Performance</strong>: Speed and resource efficiency</li><li><strong>Parallelization</strong>: Ability to scale horizontally</li><li><strong>Stability</strong>: Consistency of results across multiple runs</li><li><strong>Volume Handling</strong>: Ability to handle large data volumes</li></ul><h4>Maintainability</h4><ul><li><strong>Code Complexity</strong>: Complexity of implementation and maintenance</li><li><strong>Resilience to App Changes</strong>: How well the technique handles app updates</li><li><strong>Debugging Ease</strong>: Ease of troubleshooting and fixing issues</li><li><strong>Skill Requirements</strong>: Technical expertise needed to implement and maintain</li></ul><h4>Comparative Analysis Table</h4><p>Technique</p><p>Effectiveness</p><p>Scalability</p><p>Maintainability</p><p>Best Use Cases</p><p>Limitations</p><p><strong>Reverse Engineering</strong></p><p>High for API data access; Excellent bypass capability</p><p>Medium; Resource-efficient but complex to parallelize</p><p>Low; Highly sensitive to app updates</p><p>Apps with stable internal APIs; Security research</p><p>Requires deep technical expertise; Legal risks; Breaks with app updates</p><p><strong>Automation Frameworks</strong></p><p>Medium-High for UI data; Limited for encrypted data</p><p>Medium; Good parallelization but resource-intensive</p><p>Medium; Moderately resilient to minor UI changes</p><p>UI-based data extraction; End-to-end testing</p><p>Slow performance; Breaks with major UI changes; Resource-intensive</p><p><strong>OCR</strong></p><p>Medium for visible text; Low for complex layouts</p><p>Low-Medium; Computationally intensive</p><p>High; Independent of app internals</p><p>Image-based data; Text in graphics; Screenshots</p><p>Low accuracy for complex layouts; Cannot access non-visible data</p><p><strong>Proxy Rotation</strong></p><p>N/A (supportive technique)</p><p>High; Excellent for distributed scraping</p><p>High; Simple to maintain</p><p>Avoiding IP bans; Large-scale scraping</p><p>Not a standalone technique; Requires proxy infrastructure</p><p><strong>ML/AI/GenAI</strong></p><p>Medium-High; Adaptable to various data types</p><p>Medium; Computationally intensive</p><p>Medium-High; Self-adapting to minor changes</p><p>Complex layouts; Unstructured data; Dynamic content</p><p>High computational cost; Potential accuracy issues; Requires training</p><h4>Detailed Analysis by Technique</h4><h4>1. Reverse Engineering Techniques</h4><p><strong>Effectiveness: 8/10</strong></p><ul><li>Direct access to raw API data</li><li>Bypasses UI limitations and restrictions</li><li>Can access encrypted or hidden data</li><li>Typically faster than UI-based approaches</li><li>May not capture UI-specific elements or visual data</li></ul><p><strong>Scalability: 7/10</strong></p><ul><li>Low resource usage per request</li><li>Fast execution compared to UI automation</li><li>Complex session management for parallelization</li><li>May trigger rate limiting or anti-bot measures at scale</li></ul><p><strong>Maintainability: 4/10</strong></p><ul><li>Highly sensitive to app updates</li><li>Requires deep technical expertise</li><li>Time-consuming to debug and update</li><li>Clear data structures once APIs are understood</li></ul><h4>2. Automation Frameworks</h4><p><strong>Effectiveness: 7/10</strong></p><ul><li>Access to all visible UI elements</li><li>Can extract data in the context of the UI</li><li>Works across different app versions if UI is stable</li><li>Cannot access encrypted API data directly</li><li>May miss data not visible in the current view</li></ul><p><strong>Scalability: 6/10</strong></p><ul><li>Can be parallelized across multiple devices/emulators</li><li>High resource usage per instance</li><li>Slower execution speed</li><li>Device farm management complexity</li></ul><p><strong>Maintainability: 6/10</strong></p><ul><li>More resilient to minor app updates than reverse engineering</li><li>Easier to understand and debug with visual feedback</li><li>Breaks with significant UI changes</li><li>Complex synchronization and waiting logic</li></ul><h4>3. OCR for Image-Based Data</h4><p><strong>Effectiveness: 6/10</strong></p><ul><li>Works regardless of app’s internal structure</li><li>Can extract text from images and graphics</li><li>Limited to visible text in the current view</li><li>Lower accuracy, especially with complex layouts</li></ul><p><strong>Scalability: 5/10</strong></p><ul><li>Can be parallelized across multiple processors</li><li>High computational requirements</li><li>Slower processing speed</li><li>Resource-intensive for high accuracy</li></ul><p><strong>Maintainability: 8/10</strong></p><ul><li>Highly resilient to app updates</li><li>Simple implementation for basic use cases</li><li>Independent of app’s internal structure</li><li>Requires tuning for optimal accuracy</li></ul><h4>4. Proxy Rotation and IP Masking</h4><p><strong>Effectiveness: 8/10</strong> (as a supportive technique)</p><ul><li>Effectively bypasses IP-based rate limiting</li><li>Enables longer scraping sessions</li><li>Can simulate different geographic locations</li><li>Does not directly extract data</li></ul><p><strong>Scalability: 9/10</strong></p><ul><li>Excellent for large-scale operations</li><li>Distributes load across multiple IPs</li><li>Reduces per-IP request frequency</li><li>Varying proxy performance and reliability</li></ul><p><strong>Maintainability: 8/10</strong></p><ul><li>Simple core logic</li><li>Independent of app internals</li><li>Stable across app updates</li><li>Requires monitoring proxy health and performance</li></ul><h4>5. ML/AI/GenAI Solutions</h4><p><strong>Effectiveness: 7/10</strong></p><ul><li>Adaptable to various data formats and layouts</li><li>Can understand context and semantics</li><li>Handles unstructured data well</li><li>May lack precision for highly structured data</li></ul><p><strong>Scalability: 6/10</strong></p><ul><li>Can process complex inputs systematically</li><li>Parallelizable with sufficient computing resources</li><li>High computational requirements</li><li>Expensive for large-scale operations</li></ul><p><strong>Maintainability: 7/10</strong></p><ul><li>Adaptable to minor app changes without code updates</li><li>Generalizable across similar apps</li><li>May require retraining for major changes</li><li>Complex debugging when issues occur</li></ul><h4>Hybrid Approaches and Best Practices</h4><h4>Effective Hybrid Strategies</h4><p>The most robust mobile app scraping solutions often combine multiple techniques to leverage their respective strengths while mitigating weaknesses:</p><ol><li><strong>Reverse Engineering + Proxy Rotation</strong>: Combines the data access depth of reverse engineering with the anti-detection capabilities of proxy rotation.</li><li><strong>Automation + OCR</strong>: Uses automation frameworks to navigate the app and OCR to extract data from complex UI elements or images.</li><li><strong>Reverse Engineering + ML/AI</strong>: Uses reverse engineering to access raw data and ML/AI to process and extract insights from unstructured responses.</li><li><strong>Comprehensive Hybrid Approach</strong>: Implements a decision tree that selects the most appropriate technique based on the specific data being targeted:</li><li>Use reverse engineering for structured API data</li><li>Use automation for interactive elements</li><li>Use OCR for image-based content</li><li>Apply ML/AI for complex or unstructured data</li><li>Implement proxy rotation across all methods</li></ol><h4>Best Practices for Scalability</h4><ol><li><strong>Distributed Architecture</strong>: Implement a distributed scraping architecture with centralized coordination.</li><li><strong>Queue-Based Processing</strong>: Use message queues to manage scraping tasks and handle retries.</li><li><strong>Incremental Scraping</strong>: Prioritize incremental data updates over full rescans when possible.</li><li><strong>Resource Pooling</strong>: Implement resource pools for proxies, devices, and computing resources.</li><li><strong>Caching Strategy</strong>: Cache intermediate results to avoid redundant processing.</li></ol><h4>Best Practices for Maintainability</h4><ol><li><strong>Modular Design</strong>: Separate scraping logic, data processing, and infrastructure concerns.</li><li><strong>Configuration-Driven</strong>: Make scraping behavior configurable without code changes.</li><li><strong>Comprehensive Logging</strong>: Implement detailed logging for debugging and monitoring.</li><li><strong>Automated Testing</strong>: Create tests that verify scraping functionality against app changes.</li><li><strong>Version Control</strong>: Track app versions and corresponding scraper configurations.</li></ol><h4>Conclusion and Recommendations</h4><h4>When to Use Each Technique</h4><p><strong>Reverse Engineering</strong> is most appropriate when:</p><ul><li>Deep access to structured API data is required</li><li>Performance and efficiency are critical</li><li>Target app has a stable internal API</li><li>Technical expertise is available for maintenance</li></ul><p><strong>Automation Frameworks</strong> are most appropriate when:</p><ul><li>UI-based data extraction is sufficient</li><li>Visual context is important</li><li>Moderate scale is acceptable</li><li>App updates frequently but with minor UI changes</li></ul><p><strong>OCR</strong> is most appropriate when:</p><ul><li>Only visible text data is needed</li><li>Target app has strong anti-scraping measures</li><li>Maintenance simplicity is prioritized</li><li>Dealing with image-based content</li></ul><p><strong>Proxy Rotation</strong> should be implemented when:</p><ul><li>Large-scale scraping is required</li><li>Target app implements IP-based rate limiting</li><li>Long-running scraping operations are needed</li><li>Geographic distribution is beneficial</li></ul><p><strong>ML/AI/GenAI Solutions</strong> are most appropriate when:</p><ul><li>Dealing with unstructured or complex data</li><li>Extracting insights rather than raw data</li><li>Target app has inconsistent layouts</li><li>Adaptability to changes is prioritized</li></ul><p>— -</p><h4>7. Ethical and Legal Considerations for Mobile App Data Scraping</h4><h4>Overview</h4><p>This section provides a comprehensive overview of the ethical and legal considerations related to mobile app data scraping, including compliance with major regulations such as GDPR and CCPA, as well as best practices for responsible data collection.</p><h4>Legal Framework</h4><h4>Key Regulations</h4><p><strong>1. General Data Protection Regulation (GDPR)</strong></p><ul><li><strong>Personal Data Protection</strong>: GDPR defines personal data broadly, covering any information relating to an identified or identifiable natural person.</li><li><strong>Legal Basis Requirement</strong>: Any collection of personal data requires a legal basis, such as consent, legitimate interest, or contractual necessity.</li><li><strong>Data Subject Rights</strong>: Individuals have rights to access, rectify, erase, and port their personal data.</li><li><strong>Territorial Scope</strong>: Applies to organizations outside the EU that process EU residents’ data.</li><li><strong>Penalties</strong>: Violations can result in fines up to €20 million or 4% of global annual revenue, whichever is higher.</li></ul><p><strong>GDPR Implications for Mobile App Scraping:</strong></p><ul><li>Scraping personal data from mobile apps used by EU residents falls under GDPR jurisdiction</li><li>Obtaining valid consent for data collection is challenging in scraping scenarios</li><li>Privacy by design principles must be incorporated into scraping tools</li><li>Data minimization is required — only collect what is necessary</li><li>Scraped data must be stored securely and deleted when no longer needed</li></ul><p><strong>2. California Consumer Privacy Act (CCPA) and California Privacy Rights Act (CPRA)</strong></p><ul><li><strong>Consumer Rights</strong>: California residents have rights to know what personal information is collected, delete their data, opt-out of sales, and non-discrimination.</li><li><strong>Business Obligations</strong>: Businesses must disclose data practices and provide mechanisms for consumers to exercise their rights.</li><li><strong>Scope</strong>: Applies to for-profit businesses that meet certain thresholds and collect California residents’ personal information.</li><li><strong>Penalties</strong>: Violations can result in civil penalties of up to $7,500 per intentional violation.</li></ul><p><strong>CCPA/CPRA Implications for Mobile App Scraping:</strong></p><ul><li>Requires disclosure of all categories of personal information collected</li><li>Consumers must be able to opt out of having their data collected</li><li>Businesses must be able to delete scraped data upon request</li><li>Scraped data cannot be “sold” without providing an opt-out option</li></ul><p><strong>3. Computer Fraud and Abuse Act (CFAA)</strong></p><ul><li><strong>Unauthorized Access</strong>: Prohibits accessing a computer without authorization or exceeding authorized access.</li><li><strong>Scope</strong>: Applies to protected computers, which includes virtually any computer connected to the internet.</li><li><strong>Penalties</strong>: Violations can result in criminal penalties including imprisonment and fines.</li></ul><p><strong>CFAA Implications for Mobile App Scraping:</strong></p><ul><li>Circumventing technical barriers (like authentication) may violate the CFAA</li><li>Violating Terms of Service may potentially be considered “exceeding authorized access”</li><li>Recent court decisions have narrowed the scope of CFAA, but legal risk remains</li></ul><p><strong>4. Digital Millennium Copyright Act (DMCA)</strong></p><ul><li><strong>Anti-Circumvention</strong>: Prohibits circumventing technological measures that control access to copyrighted works.</li><li><strong>Safe Harbor</strong>: Provides safe harbor for online service providers from copyright infringement liability.</li><li><strong>Penalties</strong>: Violations can result in civil and criminal penalties.</li></ul><p><strong>DMCA Implications for Mobile App Scraping:</strong></p><ul><li>Bypassing app security measures may violate anti-circumvention provisions</li><li>Scraping copyrighted content from apps may constitute copyright infringement</li><li>Reverse engineering apps may implicate DMCA concerns</li></ul><p><strong>5. Terms of Service (ToS) and End User License Agreements (EULAs)</strong></p><ul><li><strong>Contractual Obligations</strong>: ToS and EULAs create binding contractual obligations between users and app providers.</li><li><strong>Scraping Prohibitions</strong>: Many explicitly prohibit automated data collection or scraping.</li><li><strong>API Usage Restrictions</strong>: May specify limitations on API usage and data access.</li><li><strong>Enforcement</strong>: Violations can lead to account termination, legal action, and potential breach of contract claims.</li></ul><p><strong>ToS/EULA Implications for Mobile App Scraping:</strong></p><ul><li>Violating ToS may constitute breach of contract</li><li>Many apps explicitly prohibit automated data collection</li><li>API rate limits and usage restrictions are legally binding</li><li>Circumventing technical measures may violate both ToS and laws like CFAA/DMCA</li></ul><h4>Ethical Considerations</h4><h4>1. Privacy Impact</h4><ul><li><strong>User Expectation</strong>: Users typically do not expect their in-app data to be collected by third parties.</li><li><strong>Contextual Integrity</strong>: Data collected in one context (app usage) is repurposed for another without user awareness.</li><li><strong>Sensitive Information</strong>: Mobile apps often contain highly sensitive personal information (location, health data, communications).</li><li><strong>Secondary Use</strong>: Scraped data may be used for purposes users never consented to.</li></ul><h4>2. Informed Consent</h4><ul><li><strong>Transparency</strong>: Users should know what data is being collected and how it will be used.</li><li><strong>Voluntary</strong>: Consent should be freely given, not coerced.</li><li><strong>Specific</strong>: Consent should be specific to particular data collection and uses.</li><li><strong>Revocable</strong>: Users should be able to withdraw consent.</li></ul><p><strong>Challenge in Scraping Context</strong>: Mobile app scraping typically occurs without the knowledge or consent of users, making it difficult to meet ethical standards for informed consent.</p><h4>3. Data Minimization</h4><ul><li><strong>Necessity</strong>: Only collect data that is necessary for the intended purpose.</li><li><strong>Proportionality</strong>: The scope of data collection should be proportional to the legitimate need.</li><li><strong>Retention Limits</strong>: Data should be retained only as long as necessary.</li><li><strong>Anonymization</strong>: Personal identifiers should be removed when possible.</li></ul><h4>4. Impact on App Providers</h4><ul><li><strong>Server Load</strong>: Intensive scraping can increase server load and costs.</li><li><strong>Competitive Disadvantage</strong>: Scrapers may free-ride on the app provider’s investment.</li><li><strong>Revenue Impact</strong>: May undermine subscription or advertising-based business models.</li><li><strong>User Trust</strong>: Data breaches or misuse of scraped data can damage user trust in the app.</li></ul><h4>Best Practices for Ethical and Legal Compliance</h4><h4>1. Legal Compliance Framework</h4><ul><li><strong>Legal Assessment</strong>: Conduct a thorough legal analysis before initiating any scraping project.</li><li><strong>Jurisdiction Mapping</strong>: Identify which laws apply based on the location of users, app providers, and your organization.</li><li><strong>Documentation</strong>: Maintain documentation of legal compliance measures.</li><li><strong>Regular Review</strong>: Update compliance measures as laws and regulations evolve.</li><li><strong>Legal Counsel</strong>: Consult with specialized legal counsel for complex scraping projects.</li></ul><h4>2. Respect for Technical Measures</h4><ul><li><strong>Robots.txt Equivalent</strong>: Respect any mechanisms that signal scraping restrictions.</li><li><strong>Rate Limiting</strong>: Implement reasonable rate limits to avoid server impact.</li><li><strong>Identification</strong>: Consider identifying your scraper through user-agent strings or other means.</li><li><strong>API Preference</strong>: Use official APIs when available instead of scraping.</li><li><strong>Authentication Respect</strong>: Do not circumvent authentication mechanisms.</li></ul><h4>3. Data Protection Measures</h4><ul><li><strong>Security Controls</strong>: Apply appropriate technical and organizational security measures.</li><li><strong>Access Controls</strong>: Limit access to scraped data on a need-to-know basis.</li><li><strong>Encryption</strong>: Encrypt scraped data both in transit and at rest.</li><li><strong>Data Minimization</strong>: Only collect and retain necessary data.</li><li><strong>Anonymization</strong>: Anonymize or pseudonymize personal data when possible.</li><li><strong>Deletion Protocols</strong>: Establish protocols for secure data deletion.</li></ul><h4>4. Transparency and Notice</h4><ul><li><strong>Privacy Policy</strong>: Maintain a clear privacy policy describing scraping activities.</li><li><strong>Notification Mechanisms</strong>: Where feasible, notify affected individuals about data collection.</li><li><strong>Opt-Out Mechanisms</strong>: Provide mechanisms for individuals to opt out of data collection.</li><li><strong>Contact Information</strong>: Make it easy for individuals to contact you with concerns.</li><li><strong>Purpose Limitation</strong>: Clearly define and limit the purposes for which scraped data will be used.</li></ul><h4>5. Ethical Decision Framework</h4><ul><li><strong>Necessity Test</strong>: Is scraping necessary to achieve the objective?</li><li><strong>Proportionality Test</strong>: Is the scope of scraping proportional to the legitimate need?</li><li><strong>Harm Assessment</strong>: What potential harms could result, and how can they be mitigated?</li><li><strong>Alternatives Analysis</strong>: Are there less invasive alternatives to achieve the same goal?</li><li><strong>Benefit Analysis</strong>: Does the benefit justify the potential privacy impact?</li><li><strong>Stakeholder Consideration</strong>: How does scraping affect all stakeholders (users, app providers, society)?</li></ul><h4>Conclusion</h4><p>Mobile app data scraping exists in a complex legal and ethical landscape. While there are legitimate use cases for such activities, they must be balanced against legal compliance requirements, ethical considerations, and respect for user privacy and app provider interests.</p><p>Organizations engaging in mobile app scraping should:</p><ol><li><strong>Conduct thorough legal analysis</strong> before initiating scraping projects</li><li><strong>Implement robust data protection measures</strong> for any collected data</li><li><strong>Consider the ethical implications</strong> of scraping activities</li><li><strong>Respect technical measures</strong> implemented by app providers</li><li><strong>Be transparent</strong> about data collection practices</li><li><strong>Minimize data collection</strong> to what is strictly necessary</li><li><strong>Establish clear data lifecycle policies</strong> including secure deletion</li><li><strong>Stay informed</strong> about evolving legal and regulatory requirements</li></ol><p>By following these guidelines and implementing appropriate compliance measures, organizations can mitigate the legal and ethical risks associated with mobile app data scraping while still achieving legitimate business objectives.</p><p>— -</p><h4>8. Code Examples and Implementation References</h4><p>This section provides practical code examples for implementing various mobile app scraping techniques. These examples are intended to demonstrate the concepts discussed in the previous sections and provide a starting point for implementation.</p><h4>Code Examples for Reverse Engineering</h4><pre># Example: Using Frida to hook into an Android app&#39;s network calls<br>import frida<br>import sys<br><br># JavaScript to inject into the target process<br>frida_script = &quot;&quot;&quot;<br>Java.perform(function() {<br>    // Target the OkHttp library&#39;s network calls<br>    var OkHttpClient = Java.use(&#39;okhttp3.OkHttpClient&#39;);<br>    var Request = Java.use(&#39;okhttp3.Request&#39;);<br>    var Response = Java.use(&#39;okhttp3.Response&#39;);<br>    <br>    // Hook the newCall method to intercept requests<br>    OkHttpClient.newCall.implementation = function(request) {<br>        console.log(&#39;\\n[+] Intercepted OkHttp request&#39;);<br>        <br>        // Extract request details<br>        var url = request.url().toString();<br>        console.log(&#39;[+] URL: &#39; + url);<br>        <br>        // Extract headers<br>        var headers = request.headers();<br>        var headerCount = headers.size();<br>        console.log(&#39;[+] Headers:&#39;);<br>        for (var i = 0; i &lt; headerCount; i++) {<br>            var name = headers.name(i);<br>            var value = headers.value(i);<br>            console.log(&#39;    &#39; + name + &#39;: &#39; + value);<br>        }<br>        <br>        // Extract request body if present<br>        var requestBody = request.body();<br>        if (requestBody) {<br>            try {<br>                var Buffer = Java.use(&#39;okio.Buffer&#39;);<br>                var buffer = Buffer.$new();<br>                requestBody.writeTo(buffer);<br>                var requestBodyString = buffer.readUtf8();<br>                console.log(&#39;[+] Request Body: &#39; + requestBodyString);<br>            } catch (e) {<br>                console.log(&#39;[!] Error reading request body: &#39; + e);<br>            }<br>        }<br>        <br>        // Call the original method and get the response<br>        var call = this.newCall.call(this, request);<br>        <br>        // Return the call object<br>        return call;<br>    };<br>    <br>    // Hook the execute method to intercept responses<br>    Response.newBuilder.implementation = function() {<br>        console.log(&#39;\\n[+] Intercepted OkHttp response&#39;);<br>        <br>        // Get the response object<br>        var response = this;<br>        <br>        // Extract response details<br>        var statusCode = response.code();<br>        var message = response.message();<br>        console.log(&#39;[+] Status: &#39; + statusCode + &#39; &#39; + message);<br>        <br>        // Extract response headers<br>        var headers = response.headers();<br>        var headerCount = headers.size();<br>        console.log(&#39;[+] Headers:&#39;);<br>        for (var i = 0; i &lt; headerCount; i++) {<br>            var name = headers.name(i);<br>            var value = headers.value(i);<br>            console.log(&#39;    &#39; + name + &#39;: &#39; + value);<br>        }<br>        <br>        // Extract response body<br>        try {<br>            var responseBody = response.body();<br>            if (responseBody) {<br>                var bodyString = responseBody.string();<br>                console.log(&#39;[+] Response Body: &#39; + bodyString);<br>                <br>                // Create a new body with the same content to avoid consuming the original<br>                var ResponseBody = Java.use(&#39;okhttp3.ResponseBody&#39;);<br>                var MediaType = Java.use(&#39;okhttp3.MediaType&#39;);<br>                var newBody = ResponseBody.create(MediaType.parse(responseBody.contentType().toString()), bodyString);<br>                <br>                // Replace the original body<br>                response = response.newBuilder().body(newBody).build();<br>            }<br>        } catch (e) {<br>            console.log(&#39;[!] Error reading response body: &#39; + e);<br>        }<br>        <br>        // Call the original method<br>        var builder = this.newBuilder.call(this);<br>        <br>        return builder;<br>    };<br>});<br>&quot;&quot;&quot;<br><br>def on_message(message, data):<br>    if message[&#39;type&#39;] == &#39;send&#39;:<br>        print(&quot;[*] {0}&quot;.format(message[&#39;payload&#39;]))<br>    else:<br>        print(message)<br><br># Connect to the device<br>device = frida.get_usb_device()<br><br># Spawn the target app<br>pid = device.spawn([&quot;com.example.targetapp&quot;])<br>session = device.attach(pid)<br><br># Create a script and load the JavaScript<br>script = session.create_script(frida_script)<br>script.on(&#39;message&#39;, on_message)<br>script.load()<br><br># Resume the app<br>device.resume(pid)<br><br># Keep the script running<br>sys.stdin.read()</pre><h4>Code Examples for Automation Frameworks</h4><pre># Example: Using Appium to automate and scrape data from a mobile app<br>from appium import webdriver<br>from appium.webdriver.common.appiumby import AppiumBy<br>import time<br>import json<br>import os<br><br>class MobileAppScraper:<br>    def __init__(self, platform=&quot;android&quot;):<br>        &quot;&quot;&quot;<br>        Initialize the mobile app scraper<br>        <br>        Args:<br>            platform (str): &#39;android&#39; or &#39;ios&#39;<br>        &quot;&quot;&quot;<br>        self.platform = platform<br>        self.driver = None<br>        self.results = []<br>        <br>        # Create output directory<br>        os.makedirs(&quot;scraped_data&quot;, exist_ok=True)<br>    <br>    def setup_driver(self):<br>        &quot;&quot;&quot;<br>        Set up the Appium driver based on platform<br>        &quot;&quot;&quot;<br>        if self.platform.lower() == &quot;android&quot;:<br>            desired_caps = {<br>                &#39;platformName&#39;: &#39;Android&#39;,<br>                &#39;deviceName&#39;: &#39;Android Device&#39;,<br>                &#39;automationName&#39;: &#39;UiAutomator2&#39;,<br>                &#39;appPackage&#39;: &#39;com.example.targetapp&#39;,<br>                &#39;appActivity&#39;: &#39;com.example.targetapp.MainActivity&#39;,<br>                &#39;noReset&#39;: True<br>            }<br>        else:  # iOS<br>            desired_caps = {<br>                &#39;platformName&#39;: &#39;iOS&#39;,<br>                &#39;deviceName&#39;: &#39;iPhone Simulator&#39;,<br>                &#39;automationName&#39;: &#39;XCUITest&#39;,<br>                &#39;bundleId&#39;: &#39;com.example.targetapp&#39;,<br>                &#39;noReset&#39;: True<br>            }<br>        <br>        self.driver = webdriver.Remote(&#39;http://localhost:4723/wd/hub&#39;, desired_caps)<br>        print(f&quot;{self.platform} driver initialized&quot;)<br>    <br>    def navigate_to_section(self, section_name):<br>        &quot;&quot;&quot;<br>        Navigate to a specific section in the app<br>        <br>        Args:<br>            section_name (str): Name of the section to navigate to<br>        &quot;&quot;&quot;<br>        try:<br>            # Wait for app to load<br>            time.sleep(3)<br>            <br>            # Find and click on the menu button<br>            menu_button = self.driver.find_element(AppiumBy.ACCESSIBILITY_ID, &quot;menu&quot;)<br>            menu_button.click()<br>            time.sleep(1)<br>            <br>            # Find and click on the section<br>            section = self.driver.find_element(AppiumBy.XPATH, f&quot;//android.widget.TextView[@text=&#39;{section_name}&#39;]&quot;)<br>            section.click()<br>            time.sleep(2)<br>            <br>            print(f&quot;Navigated to section: {section_name}&quot;)<br>            return True<br>        except Exception as e:<br>            print(f&quot;Error navigating to section {section_name}: {str(e)}&quot;)<br>            return False<br>    <br>    def extract_list_items(self, max_items=20):<br>        &quot;&quot;&quot;<br>        Extract items from a list view<br>        <br>        Args:<br>            max_items (int): Maximum number of items to extract<br>            <br>        Returns:<br>            list: Extracted items<br>        &quot;&quot;&quot;<br>        items = []<br>        <br>        try:<br>            # Find list container<br>            list_container = self.driver.find_element(AppiumBy.ID, &quot;itemsRecyclerView&quot;)<br>            <br>            # Extract visible items<br>            list_items = self.driver.find_elements(AppiumBy.XPATH, &quot;//android.widget.LinearLayout[contains(@resource-id, &#39;item_container&#39;)]&quot;)<br>            <br>            # Process each item<br>            for i, item in enumerate(list_items):<br>                if i &gt;= max_items:<br>                    break<br>                <br>                try:<br>                    # Extract item details<br>                    title = item.find_element(AppiumBy.ID, &quot;itemTitle&quot;).text<br>                    description = item.find_element(AppiumBy.ID, &quot;itemDescription&quot;).text<br>                    <br>                    # Try to extract price if available<br>                    try:<br>                        price = item.find_element(AppiumBy.ID, &quot;itemPrice&quot;).text<br>                    except:<br>                        price = &quot;N/A&quot;<br>                    <br>                    # Add to results<br>                    items.append({<br>                        &quot;title&quot;: title,<br>                        &quot;description&quot;: description,<br>                        &quot;price&quot;: price<br>                    })<br>                    <br>                    print(f&quot;Extracted item {i+1}: {title}&quot;)<br>                except Exception as e:<br>                    print(f&quot;Error extracting item {i+1}: {str(e)}&quot;)<br>            <br>            # Scroll to see more items if needed<br>            if len(list_items) &lt; max_items:<br>                self.driver.swipe(500, 1000, 500, 300, 1000)  # Swipe up<br>                time.sleep(1)<br>                <br>                # Extract more items after scrolling<br>                more_items = self.driver.find_elements(AppiumBy.XPATH, &quot;//android.widget.LinearLayout[contains(@resource-id, &#39;item_container&#39;)]&quot;)<br>                <br>                for i, item in enumerate(more_items):<br>                    if len(items) &gt;= max_items:<br>                        break<br>                    <br>                    # Skip already processed items<br>                    if i &lt; len(list_items):<br>                        continue<br>                    <br>                    try:<br>                        # Extract item details<br>                        title = item.find_element(AppiumBy.ID, &quot;itemTitle&quot;).text<br>                        description = item.find_element(AppiumBy.ID, &quot;itemDescription&quot;).text<br>                        <br>                        # Try to extract price if available<br>                        try:<br>                            price = item.find_element(AppiumBy.ID, &quot;itemPrice&quot;).text<br>                        except:<br>                            price = &quot;N/A&quot;<br>                        <br>                        # Add to results<br>                        items.append({<br>                            &quot;title&quot;: title,<br>                            &quot;description&quot;: description,<br>                            &quot;price&quot;: price<br>                        })<br>                        <br>                        print(f&quot;Extracted item {len(items)}: {title}&quot;)<br>                    except Exception as e:<br>                        print(f&quot;Error extracting item after scroll: {str(e)}&quot;)<br>        <br>        except Exception as e:<br>            print(f&quot;Error extracting list items: {str(e)}&quot;)<br>        <br>        return items<br>    <br>    def extract_item_details(self, item_index=0):<br>        &quot;&quot;&quot;<br>        Extract detailed information about a specific item<br>        <br>        Args:<br>            item_index (int): Index of the item to extract details from<br>            <br>        Returns:<br>            dict: Detailed item information<br>        &quot;&quot;&quot;<br>        try:<br>            # Find and click on the item<br>            items = self.driver.find_elements(AppiumBy.XPATH, &quot;//android.widget.LinearLayout[contains(@resource-id, &#39;item_container&#39;)]&quot;)<br>            <br>            if item_index &gt;= len(items):<br>                print(f&quot;Item index {item_index} out of range (only {len(items)} items available)&quot;)<br>                return None<br>            <br>            items[item_index].click()<br>            time.sleep(2)<br>            <br>            # Extract detailed information<br>            details = {}<br>            <br>            # Extract title<br>            try:<br>                details[&quot;title&quot;] = self.driver.find_element(AppiumBy.ID, &quot;detailTitle&quot;).text<br>            except:<br>                details[&quot;title&quot;] = &quot;N/A&quot;<br>            <br>            # Extract description<br>            try:<br>                details[&quot;description&quot;] = self.driver.find_element(AppiumBy.ID, &quot;detailDescription&quot;).text<br>            except:<br>                details[&quot;description&quot;] = &quot;N/A&quot;<br>            <br>            # Extract price<br>            try:<br>                details[&quot;price&quot;] = self.driver.find_element(AppiumBy.ID, &quot;detailPrice&quot;).text<br>            except:<br>                details[&quot;price&quot;] = &quot;N/A&quot;<br>            <br>            # Extract additional information<br>            try:<br>                info_elements = self.driver.find_elements(AppiumBy.XPATH, &quot;//android.widget.LinearLayout[contains(@resource-id, &#39;info_item&#39;)]&quot;)<br>                <br>                additional_info = {}<br>                for info in info_elements:<br>                    try:<br>                        key = info.find_element(AppiumBy.ID, &quot;infoKey&quot;).text<br>                        value = info.find_element(AppiumBy.ID, &quot;infoValue&quot;).text<br>                        additional_info[key] = value<br>                    except:<br>                        pass<br>                <br>                details[&quot;additional_info&quot;] = additional_info<br>            except:<br>                details[&quot;additional_info&quot;] = {}<br>            <br>            # Take a screenshot<br>            screenshot_path = f&quot;scraped_data/item_{item_index}.png&quot;<br>            self.driver.get_screenshot_as_file(screenshot_path)<br>            details[&quot;screenshot&quot;] = screenshot_path<br>            <br>            print(f&quot;Extracted details for item: {details[&#39;title&#39;]}&quot;)<br>            return details<br>            <br>        except Exception as e:<br>            print(f&quot;Error extracting item details: {str(e)}&quot;)<br>            return None<br>    <br>    def scrape_section(self, section_name, max_items=20, extract_details=True):<br>        &quot;&quot;&quot;<br>        Scrape a specific section of the app<br>        <br>        Args:<br>            section_name (str): Name of the section to scrape<br>            max_items (int): Maximum number of items to extract<br>            extract_details (bool): Whether to extract detailed information for each item<br>            <br>        Returns:<br>            dict: Scraped data<br>        &quot;&quot;&quot;<br>        result = {<br>            &quot;section&quot;: section_name,<br>            &quot;timestamp&quot;: time.strftime(&quot;%Y-%m-%d %H:%M:%S&quot;),<br>            &quot;items&quot;: [],<br>            &quot;details&quot;: []<br>        }<br>        <br>        # Navigate to the section<br>        if not self.navigate_to_section(section_name):<br>            return result<br>        <br>        # Extract list items<br>        items = self.extract_list_items(max_items)<br>        result[&quot;items&quot;] = items<br>        <br>        # Extract detailed information if requested<br>        if extract_details and items:<br>            # Go back to the list view<br>            self.driver.back()<br>            time.sleep(1)<br>            <br>            # Extract details for each item (up to 5 items to avoid excessive time)<br>            for i in range(min(5, len(items))):<br>                detail = self.extract_item_details(i)<br>                if detail:<br>                    result[&quot;details&quot;].append(detail)<br>                <br>                # Go back to the list view<br>                self.driver.back()<br>                time.sleep(1)<br>        <br>        # Save results to file<br>        filename = f&quot;scraped_data/{section_name.lower().replace(&#39; &#39;, &#39;_&#39;)}.json&quot;<br>        with open(filename, &#39;w&#39;) as f:<br>            json.dump(result, f, indent=2)<br>        <br>        print(f&quot;Saved scraped data to {filename}&quot;)<br>        return result<br>    <br>    def scrape_app(self, sections=None):<br>        &quot;&quot;&quot;<br>        Scrape multiple sections of the app<br>        <br>        Args:<br>            sections (list): List of section names to scrape<br>        &quot;&quot;&quot;<br>        if not sections:<br>            sections = [&quot;Products&quot;, &quot;Categories&quot;, &quot;Featured&quot;]<br>        <br>        try:<br>            self.setup_driver()<br>            <br>            for section in sections:<br>                print(f&quot;\nScraping section: {section}&quot;)<br>                result = self.scrape_section(section)<br>                self.results.append(result)<br>            <br>            # Save combined results<br>            with open(&quot;scraped_data/all_results.json&quot;, &#39;w&#39;) as f:<br>                json.dump(self.results, f, indent=2)<br>            <br>            print(&quot;\nScraping completed successfully&quot;)<br>            <br>        except Exception as e:<br>            print(f&quot;Error during scraping: {str(e)}&quot;)<br>        finally:<br>            if self.driver:<br>                self.driver.quit()<br>                print(&quot;Driver closed&quot;)<br><br># Example usage<br>if __name__ == &quot;__main__&quot;:<br>    scraper = MobileAppScraper(platform=&quot;android&quot;)<br>    scraper.scrape_app(sections=[&quot;Products&quot;, &quot;Categories&quot;, &quot;Featured&quot;])</pre><h4>Code Examples for OCR Implementation</h4><pre># Example: Using Tesseract OCR to extract text from mobile app screenshots<br>import cv2<br>import pytesseract<br>import numpy as np<br>from PIL import Image<br>import os<br>import json<br>import time<br><br>class MobileAppOCR:<br>    def __init__(self, tesseract_path=None):<br>        &quot;&quot;&quot;<br>        Initialize the OCR processor<br>        <br>        Args:<br>            tesseract_path (str): Path to Tesseract executable<br>        &quot;&quot;&quot;<br>        # Set Tesseract path if provided<br>        if tesseract_path:<br>            pytesseract.pytesseract.tesseract_cmd = tesseract_path<br>        <br>        # Create output directory<br>        os.makedirs(&quot;ocr_results&quot;, exist_ok=True)<br>    <br>    def preprocess_image(self, image_path):<br>        &quot;&quot;&quot;<br>        Preprocess the image for better OCR results<br>        <br>        Args:<br>            image_path (str): Path to the image<br>            <br>        Returns:<br>            numpy.ndarray: Preprocessed image<br>        &quot;&quot;&quot;<br>        # Read the image<br>        img = cv2.imread(image_path)<br>        <br>        # Convert to grayscale<br>        gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)<br>        <br>        # Apply thresholding<br>        _, thresh = cv2.threshold(gray, 150, 255, cv2.THRESH_BINARY)<br>        <br>        # Apply Gaussian blur<br>        blur = cv2.GaussianBlur(thresh, (3, 3), 0)<br>        <br>        return blur<br>    <br>    def detect_text_regions(self, image):<br>        &quot;&quot;&quot;<br>        Detect regions containing text in the image<br>        <br>        Args:<br>            image (numpy.ndarray): Input image<br>            <br>        Returns:<br>            list: List of bounding boxes for text regions<br>        &quot;&quot;&quot;<br>        # Create MSER detector<br>        mser = cv2.MSER_create()<br>        <br>        # Detect regions<br>        regions, _ = mser.detectRegions(image)<br>        <br>        # Convert regions to bounding boxes<br>        boxes = []<br>        for region in regions:<br>            x, y, w, h = cv2.boundingRect(region)<br>            <br>            # Filter out very small regions<br>            if w &lt; 10 or h &lt; 10 or w &gt; image.shape[1] * 0.9:<br>                continue<br>                <br>            boxes.append((x, y, x+w, y+h))<br>        <br>        # Merge overlapping boxes<br>        merged_boxes = self._merge_boxes(boxes)<br>        <br>        return merged_boxes<br>    <br>    def _merge_boxes(self, boxes, threshold=10):<br>        &quot;&quot;&quot;<br>        Merge overlapping bounding boxes<br>        <br>        Args:<br>            boxes (list): List of bounding boxes (x1, y1, x2, y2)<br>            threshold (int): Distance threshold for merging<br>            <br>        Returns:<br>            list: Merged bounding boxes<br>        &quot;&quot;&quot;<br>        if not boxes:<br>            return []<br>            <br>        # Sort boxes by x coordinate<br>        boxes = sorted(boxes, key=lambda box: box[0])<br>        <br>        merged_boxes = [boxes[0]]<br>        <br>        for box in boxes[1:]:<br>            prev_box = merged_boxes[-1]<br>            <br>            # Check if boxes are close enough to merge<br>            if (box[0] &lt;= prev_box[2] + threshold and <br>                box[1] &lt;= prev_box[3] + threshold and <br>                prev_box[0] &lt;= box[2] + threshold and <br>                prev_box[1] &lt;= box[3] + threshold):<br>                # Merge boxes<br>                merged_boxes[-1] = (<br>                    min(prev_box[0], box[0]),<br>                    min(prev_box[1], box[1]),<br>                    max(prev_box[2], box[2]),<br>                    max(prev_box[3], box[3])<br>                )<br>            else:<br>                merged_boxes.append(box)<br>        <br>        return merged_boxes<br>    <br>    def extract_text_from_image(self, image_path, lang=&#39;eng&#39;):<br>        &quot;&quot;&quot;<br>        Extract text from an image<br>        <br>        Args:<br>            image_path (str): Path to the image<br>            lang (str): Language for OCR<br>            <br>        Returns:<br>            dict: Extracted text and regions<br>        &quot;&quot;&quot;<br>        # Preprocess the image<br>        preprocessed = self.preprocess_image(image_path)<br>        <br>        # Detect text regions<br>        regions = self.detect_text_regions(preprocessed)<br>        <br>        # Extract text from each region<br>        region_texts = []<br>        <br>        original_img = cv2.imread(image_path)<br>        visualization_img = original_img.copy()<br>        <br>        for i, (x1, y1, x2, y2) in enumerate(regions):<br>            # Extract region<br>            region_img = preprocessed[y1:y2, x1:x2]<br>            <br>            # Skip if region is too small<br>            if region_img.size == 0:<br>                continue<br>            <br>            # Extract text<br>            text = pytesseract.image_to_string(region_img, lang=lang).strip()<br>            <br>            # Skip empty results<br>            if not text:<br>                continue<br>            <br>            # Add to results<br>            region_texts.append({<br>                &quot;region_id&quot;: i,<br>                &quot;position&quot;: {&quot;x1&quot;: x1, &quot;y1&quot;: y1, &quot;x2&quot;: x2, &quot;y2&quot;: y2},<br>                &quot;text&quot;: text<br>            })<br>            <br>            # Draw rectangle on visualization image<br>            cv2.rectangle(visualization_img, (x1, y1), (x2, y2), (0, 255, 0), 2)<br>            cv2.putText(visualization_img, f&quot;{i}&quot;, (x1, y1-5), cv2.FONT_HERSHEY_SIMPLEX, 0.5, (0, 0, 255), 2)<br>        <br>        # Also perform full image OCR<br>        full_text = pytesseract.image_to_string(preprocessed, lang=lang).strip()<br>        <br>        # Save visualization image<br>        base_name = os.path.basename(image_path).split(&#39;.&#39;)[0]<br>        vis_path = f&quot;ocr_results/{base_name}_regions.jpg&quot;<br>        cv2.imwrite(vis_path, visualization_img)<br>        <br>        result = {<br>            &quot;image_path&quot;: image_path,<br>            &quot;full_text&quot;: full_text,<br>            &quot;regions&quot;: region_texts,<br>            &quot;visualization&quot;: vis_path<br>        }<br>        <br>        return result<br>    <br>    def extract_structured_data(self, ocr_result, patterns=None):<br>        &quot;&quot;&quot;<br>        Extract structured data from OCR results using patterns<br>        <br>        Args:<br>            ocr_result (dict): OCR result from extract_text_from_image<br>            patterns (dict): Dictionary of regex patterns for data extraction<br>            <br>        Returns:<br>            dict: Extracted structured data<br>        &quot;&quot;&quot;<br>        import re<br>        <br>        if not patterns:<br>            patterns = {<br>                &quot;price&quot;: r&#39;\$\s*(\d+(?:\.\d{2})?)&#39;,<br>                &quot;email&quot;: r&#39;[a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}&#39;,<br>                &quot;phone&quot;: r&#39;(?:\+\d{1,2}\s?)?\(?\d{3}\)?[\s.-]?\d{3}[\s.-]?\d{4}&#39;,<br>                &quot;url&quot;: r&#39;https?://(?:www\.)?[-a-zA-Z0-9@:%._\+~#=]{1,256}\.[a-zA-Z0-9()]{1,6}\b(?:[-a-zA-Z0-9()@:%_\+.~#?&amp;//=]*)&#39;<br>            }<br>        <br>        structured_data = {}<br>        <br>        # Search in full text<br>        full_text = ocr_result[&quot;full_text&quot;]<br>        for key, pattern in patterns.items():<br>            matches = re.findall(pattern, full_text)<br>            if matches:<br>                structured_data[key] = matches<br>        <br>        # Search in individual regions<br>        region_data = []<br>        for region in ocr_result[&quot;regions&quot;]:<br>            region_result = {&quot;region_id&quot;: region[&quot;region_id&quot;], &quot;position&quot;: region[&quot;position&quot;], &quot;matches&quot;: {}}<br>            <br>            for key, pattern in patterns.items():<br>                matches = re.findall(pattern, region[&quot;text&quot;])<br>                if matches:<br>                    region_result[&quot;matches&quot;][key] = matches<br>            <br>            if region_result[&quot;matches&quot;]:<br>                region_data.append(region_result)<br>        <br>        if region_data:<br>            structured_data[&quot;regions&quot;] = region_data<br>        <br>        return structured_data<br>    <br>    def process_screenshot_batch(self, screenshot_dir, output_file=None):<br>        &quot;&quot;&quot;<br>        Process a batch of screenshots<br>        <br>        Args:<br>            screenshot_dir (str): Directory containing screenshots<br>            output_file (str): Path to output JSON file<br>            <br>        Returns:<br>            list: Results for all processed screenshots<br>        &quot;&quot;&quot;<br>        results = []<br>        <br>        # Get all image files<br>        image_files = [f for f in os.listdir(screenshot_dir) if f.lower().endswith((&#39;.png&#39;, &#39;.jpg&#39;, &#39;.jpeg&#39;))]<br>        <br>        for i, image_file in enumerate(image_files):<br>            print(f&quot;Processing image {i+1}/{len(image_files)}: {image_file}&quot;)<br>            <br>            image_path = os.path.join(screenshot_dir, image_file)<br>            <br>            # Extract text<br>            ocr_result = self.extract_text_from_image(image_path)<br>            <br>            # Extract structured data<br>            structured_data = self.extract_structured_data(ocr_result)<br>            <br>            # Combine results<br>            combined_result = {<br>                &quot;filename&quot;: image_file,<br>                &quot;ocr_result&quot;: ocr_result,<br>                &quot;structured_data&quot;: structured_data<br>            }<br>            <br>            results.append(combined_result)<br>            <br>            # Save individual result<br>            with open(f&quot;ocr_results/{image_file.split(&#39;.&#39;)[0]}_result.json&quot;, &#39;w&#39;) as f:<br>                json.dump(combined_result, f, indent=2)<br>        <br>        # Save combined results if output file specified<br>        if output_file:<br>            with open(output_file, &#39;w&#39;) as f:<br>                json.dump(results, f, indent=2)<br>        <br>        return results<br><br># Example usage<br>if __name__ == &quot;__main__&quot;:<br>    # Initialize OCR processor<br>    ocr = MobileAppOCR(tesseract_path=r&quot;C:\Program Files\Tesseract-OCR\tesseract.exe&quot;)  # Adjust path as needed<br>    <br>    # Process a batch of screenshots<br>    results = ocr.process_screenshot_batch(&quot;app_screenshots&quot;, &quot;ocr_results/all_results.json&quot;)<br>    <br>    print(f&quot;Processed {len(results)} screenshots&quot;)</pre><h4>Code Examples for Proxy Rotation</h4><pre># Example: Advanced proxy rotation for mobile app API requests<br>import requests<br>import random<br>import json<br>import time<br>from typing import Dict, List, Any, Optional<br><br>class MobileAppProxyRotator:<br>    &quot;&quot;&quot;<br>    A class to handle proxy rotation for mobile app API requests<br>    &quot;&quot;&quot;<br>    def __init__(self, proxy_file: str):<br>        &quot;&quot;&quot;<br>        Initialize the MobileAppProxyRotator<br>        <br>        Args:<br>            proxy_file (str): Path to the file containing proxies<br>        &quot;&quot;&quot;<br>        self.proxies = self._load_proxies(proxy_file)<br>        self.user_agents = [<br>            &quot;Mozilla/5.0 (iPhone; CPU iPhone OS 14_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148&quot;,<br>            &quot;Mozilla/5.0 (iPad; CPU OS 14_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148&quot;,<br>            &quot;Mozilla/5.0 (Linux; Android 11; SM-G991B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.120 Mobile Safari/537.36&quot;,<br>            &quot;Mozilla/5.0 (Linux; Android 10; SM-G981B) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.162 Mobile Safari/537.36&quot;,<br>            &quot;Mozilla/5.0 (iPhone; CPU iPhone OS 14_6 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) CriOS/91.0.4472.80 Mobile/15E148 Safari/604.1&quot;<br>        ]<br>        <br>    def _load_proxies(self, proxy_file: str) -&gt; List[str]:<br>        &quot;&quot;&quot;<br>        Load proxies from a file<br>        <br>        Args:<br>            proxy_file (str): Path to the file containing proxies<br>            <br>        Returns:<br>            List[str]: List of proxy addresses<br>        &quot;&quot;&quot;<br>        try:<br>            with open(proxy_file, &quot;r&quot;) as f:<br>                return [line.strip() for line in f if line.strip()]<br>        except Exception as e:<br>            print(f&quot;Error loading proxies: {str(e)}&quot;)<br>            return []<br>    <br>    def get_random_user_agent(self) -&gt; str:<br>        &quot;&quot;&quot;<br>        Get a random mobile user agent<br>        <br>        Returns:<br>            str: Random user agent string<br>        &quot;&quot;&quot;<br>        return random.choice(self.user_agents)<br>    <br>    def get_random_proxy(self) -&gt; Dict[str, str]:<br>        &quot;&quot;&quot;<br>        Get a random proxy<br>        <br>        Returns:<br>            Dict[str, str]: Proxy dictionary for requests<br>        &quot;&quot;&quot;<br>        if not self.proxies:<br>            raise Exception(&quot;No proxies available&quot;)<br>        <br>        proxy = random.choice(self.proxies)<br>        return {<br>            &quot;http&quot;: f&quot;http://{proxy}&quot;,<br>            &quot;https&quot;: f&quot;http://{proxy}&quot;<br>        }<br>    <br>    def make_api_request(self, url: str, method: str = &quot;GET&quot;, headers: Optional[Dict[str, str]] = None, <br>                        data: Optional[Dict[str, Any]] = None, max_retries: int = 3) -&gt; Optional[requests.Response]:<br>        &quot;&quot;&quot;<br>        Make an API request with proxy rotation<br>        <br>        Args:<br>            url (str): API endpoint URL<br>            method (str): HTTP method (GET, POST, etc.)<br>            headers (Dict[str, str]): HTTP headers<br>            data (Dict[str, Any]): Request data<br>            max_retries (int): Maximum number of retries<br>            <br>        Returns:<br>            requests.Response: Response object if successful, None otherwise<br>        &quot;&quot;&quot;<br>        if headers is None:<br>            headers = {}<br>        <br>        # Add mobile user agent if not specified<br>        if &quot;User-Agent&quot; not in headers:<br>            headers[&quot;User-Agent&quot;] = self.get_random_user_agent()<br>        <br>        # Add common mobile app headers<br>        if &quot;Accept&quot; not in headers:<br>            headers[&quot;Accept&quot;] = &quot;application/json&quot;<br>        if &quot;Accept-Language&quot; not in headers:<br>            headers[&quot;Accept-Language&quot;] = &quot;en-US,en;q=0.9&quot;<br>        if &quot;Connection&quot; not in headers:<br>            headers[&quot;Connection&quot;] = &quot;keep-alive&quot;<br>        <br>        attempts = 0<br>        <br>        while attempts &lt; max_retries:<br>            try:<br>                proxy = self.get_random_proxy()<br>                <br>                if method.upper() == &quot;GET&quot;:<br>                    response = requests.get(<br>                        url,<br>                        headers=headers,<br>                        proxies=proxy,<br>                        timeout=10<br>                    )<br>                elif method.upper() == &quot;POST&quot;:<br>                    response = requests.post(<br>                        url,<br>                        headers=headers,<br>                        json=data,<br>                        proxies=proxy,<br>                        timeout=10<br>                    )<br>                else:<br>                    print(f&quot;Unsupported method: {method}&quot;)<br>                    return None<br>                <br>                if response.status_code == 200:<br>                    return response<br>                elif response.status_code == 429:  # Too Many Requests<br>                    print(f&quot;Rate limited (429) - Proxy: {proxy} - Retrying...&quot;)<br>                    time.sleep(2 * (attempts + 1))  # Exponential backoff<br>                else:<br>                    print(f&quot;Request failed with status code: {response.status_code} - Proxy: {proxy}&quot;)<br>            <br>            except Exception as e:<br>                print(f&quot;Request failed - Proxy: {proxy} - Error: {str(e)}&quot;)<br>            <br>            attempts += 1<br>            time.sleep(1)  # Wait before retrying<br>        <br>        print(f&quot;Failed after {max_retries} attempts&quot;)<br>        return None<br><br># Example usage<br>if __name__ == &quot;__main__&quot;:<br>    rotator = MobileAppProxyRotator(&quot;proxies_list.txt&quot;)<br>    <br>    # Example API endpoint (replace with actual mobile app API)<br>    api_url = &quot;https://api.example.com/products&quot;<br>    <br>    # Custom headers to mimic a mobile app<br>    headers = {<br>        &quot;X-App-Version&quot;: &quot;1.2.3&quot;,<br>        &quot;X-Device-ID&quot;: &quot;sample-device-id-12345&quot;,<br>        &quot;X-Platform&quot;: &quot;Android&quot;<br>    }<br>    <br>    # Make API request<br>    response = rotator.make_api_request(api_url, headers=headers)<br>    <br>    if response:<br>        print(f&quot;API Request Successful&quot;)<br>        print(f&quot;Response: {json.dumps(response.json(), indent=2)}&quot;)<br>    else:<br>        print(&quot;API Request Failed&quot;)</pre><h4>Code Examples for ML/AI/GenAI Solutions</h4><pre># Example: Using Gemini AI for mobile app data extraction<br>import google.generativeai as genai<br>import base64<br>from PIL import Image<br>import io<br>import json<br>import os<br><br># Configure Gemini API<br>GEMINI_API_KEY = &quot;YOUR_GEMINI_API_KEY&quot;  # Replace with your actual API key<br>genai.configure(api_key=GEMINI_API_KEY)<br><br>def extract_data_from_screenshot(image_path, extraction_schema):<br>    &quot;&quot;&quot;<br>    Extract structured data from a mobile app screenshot using Gemini Vision<br>    <br>    Args:<br>        image_path (str): Path to the screenshot image<br>        extraction_schema (dict): Schema defining the data to extract<br>        <br>    Returns:<br>        dict: Extracted structured data<br>    &quot;&quot;&quot;<br>    # Load the image<br>    image = Image.open(image_path)<br>    <br>    # Prepare the image for Gemini<br>    # Convert to bytes<br>    image_bytes = io.BytesIO()<br>    image.save(image_bytes, format=image.format or &#39;PNG&#39;)<br>    image_bytes = image_bytes.getvalue()<br>    <br>    # Prepare prompt for Gemini<br>    schema_str = json.dumps(extraction_schema, indent=2)<br>    prompt = f&quot;&quot;&quot;<br>    Extract the following information from this mobile app screenshot according to the schema below.<br>    Return the data as a valid JSON object matching the schema.<br>    <br>    SCHEMA:<br>    {schema_str}<br>    <br>    Return ONLY the JSON object with the extracted data, nothing else.<br>    &quot;&quot;&quot;<br>    <br>    # Generate response from Gemini Vision<br>    model = genai.GenerativeModel(&#39;gemini-pro-vision&#39;)<br>    response = model.generate_content([prompt, image_bytes])<br>    <br>    # Parse the response to extract the JSON<br>    try:<br>        # Extract JSON from the response<br>        response_text = response.text<br>        # Find JSON in the response (it might be wrapped in markdown code blocks)<br>        if &quot;```json&quot; in response_text:<br>            json_str = response_text.split(&quot;```json&quot;)[1].split(&quot;```&quot;)[0].strip()<br>        elif &quot;```&quot; in response_text:<br>            json_str = response_text.split(&quot;```&quot;)[1].strip()<br>        else:<br>            json_str = response_text.strip()<br>            <br>        extracted_data = json.loads(json_str)<br>        return extracted_data<br>    except Exception as e:<br>        print(f&quot;Error parsing Gemini response: {str(e)}&quot;)<br>        print(f&quot;Raw response: {response.text}&quot;)<br>        return None<br><br># Example usage for mobile app screenshots<br>def analyze_mobile_app_screens(screenshots_folder):<br>    &quot;&quot;&quot;<br>    Analyze multiple mobile app screenshots and extract data<br>    <br>    Args:<br>        screenshots_folder (str): Path to folder containing screenshots<br>        <br>    Returns:<br>        list: Extracted data from each screenshot<br>    &quot;&quot;&quot;<br>    # Define the extraction schema<br>    screen_schema = {<br>        &quot;screen_type&quot;: &quot;string&quot;,  # e.g., &quot;product_listing&quot;, &quot;product_detail&quot;, &quot;checkout&quot;<br>        &quot;elements&quot;: [<br>            {<br>                &quot;type&quot;: &quot;string&quot;,  # e.g., &quot;button&quot;, &quot;text&quot;, &quot;image&quot;<br>                &quot;content&quot;: &quot;string&quot;,<br>                &quot;position&quot;: &quot;string&quot;  # e.g., &quot;top&quot;, &quot;bottom&quot;, &quot;center&quot;<br>            }<br>        ],<br>        &quot;main_content&quot;: {<br>            &quot;title&quot;: &quot;string&quot;,<br>            &quot;description&quot;: &quot;string&quot;,<br>            &quot;price&quot;: &quot;string&quot;<br>        },<br>        &quot;navigation_options&quot;: [&quot;string&quot;]<br>    }<br>    <br>    results = []<br>    <br>    # Process each screenshot in the folder<br>    for filename in os.listdir(screenshots_folder):<br>        if filename.endswith((&#39;.png&#39;, &#39;.jpg&#39;, &#39;.jpeg&#39;)):<br>            image_path = os.path.join(screenshots_folder, filename)<br>            print(f&quot;Processing {filename}...&quot;)<br>            <br>            # Extract data using Gemini Vision<br>            screen_data = extract_data_from_screenshot(image_path, screen_schema)<br>            <br>            if screen_data:<br>                screen_data[&quot;filename&quot;] = filename<br>                results.append(screen_data)<br>            else:<br>                print(f&quot;Failed to extract data from {filename}&quot;)<br>    <br>    return results<br><br># Example usage<br>if __name__ == &quot;__main__&quot;:<br>    screenshots_folder = &quot;mobile_app_screenshots&quot;<br>    results = analyze_mobile_app_screens(screenshots_folder)<br>    <br>    # Save results to JSON file<br>    with open(&quot;app_screen_analysis.json&quot;, &quot;w&quot;) as f:<br>        json.dump(results, f, indent=2)<br>    <br>    print(f&quot;Analyzed {len(results)} screenshots and saved results to app_screen_analysis.json&quot;)</pre><p>— -</p><h4>9. Conclusion and Recommendations</h4><p>This comprehensive research report has explored modern technologies and techniques for data scraping from native mobile applications, with a focus on methods that bypass common anti-scraping measures. Through detailed analysis of reverse engineering techniques, automation frameworks, OCR solutions, proxy rotation strategies, and ML/AI/GenAI approaches, we have provided a thorough understanding of the capabilities, limitations, and best practices for each method.</p><h4>Key Findings</h4><ol><li><strong>No Single Perfect Solution</strong>: Each scraping technique has distinct strengths and weaknesses, making them suitable for different scenarios. The most effective approach often involves combining multiple techniques in a hybrid strategy.</li><li><strong>Trade-offs Between Techniques</strong>: There are clear trade-offs between effectiveness, scalability, and maintainability across different scraping methods:</li><li>Reverse engineering provides the deepest data access but is highly sensitive to app updates</li><li>Automation frameworks offer good UI-level access but are resource-intensive</li><li>OCR is resilient to app changes but limited to visible text</li><li>Proxy rotation is essential for large-scale operations but is not a standalone technique</li><li>ML/AI solutions are adaptable but computationally expensive</li><li><strong>Legal and Ethical Considerations</strong>: Mobile app scraping exists in a complex legal and ethical landscape that requires careful navigation to ensure compliance with regulations like GDPR and CCPA, as well as respect for terms of service and intellectual property rights.</li></ol><h4>Recommendations</h4><ol><li><strong>Start with the Simplest Approach</strong>: Begin with the least complex technique that meets your requirements, then add sophistication as needed.</li><li><strong>Implement Hybrid Strategies</strong>: Combine multiple techniques to leverage their respective strengths while mitigating weaknesses:</li><li>Use reverse engineering for structured API data</li><li>Use automation for interactive elements</li><li>Use OCR for image-based content</li><li>Apply ML/AI for complex or unstructured data</li><li>Implement proxy rotation across all methods</li><li><strong>Prioritize Maintainability</strong>: Design scraping systems with maintainability in mind, using modular architecture, configuration-driven behavior, and comprehensive logging.</li><li><strong>Respect Legal and Ethical Boundaries</strong>: Conduct thorough legal analysis before initiating scraping projects, implement robust data protection measures, and consider the ethical implications of scraping activities.</li><li><strong>Plan for Resilience</strong>: Implement fallback mechanisms, error handling, and monitoring to ensure scraping operations can adapt to changes in target apps.</li><li><strong>Consider Cost-Effectiveness</strong>: Evaluate the total cost of ownership for each approach, including development, infrastructure, and maintenance costs.</li></ol><p>By carefully selecting and implementing the appropriate techniques based on specific requirements, organizations can build effective, scalable, and maintainable mobile app data scraping solutions while navigating the complex legal and ethical landscape.</p><p>— -</p><h4>References</h4><ol><li>Corellium. (2025). “Mobile Reverse Engineering TTPs.” <a href="https://www.corellium.com/blog/mobile-reverse-engineering-ttps">https://www.corellium.com/blog/mobile-reverse-engineering-ttps</a></li><li>Apriorit. (2024). “Reverse Engineering an API Guide.” <a href="https://www.apriorit.com/dev-blog/reverse-engineering-an-api-guide">https://www.apriorit.com/dev-blog/reverse-engineering-an-api-guide</a></li><li>Data Dive. (2024). “Reverse Engineer Android App API.” <a href="https://data-dive.com/reverse-engineer-android-app-api/">https://data-dive.com/reverse-engineer-android-app-api/</a></li><li>BrowserStack. (2025). “Mobile Application Testing Frameworks.” <a href="https://www.browserstack.com/guide/mobile-application-testing-frameworks">https://www.browserstack.com/guide/mobile-application-testing-frameworks</a></li><li>pCloudy. (2024). “How to Perform OCR Testing in Mobile Apps.” <a href="https://pcloudy.medium.com/how-to-perform-ocr-testing-in-mobile-apps-402111e95447">https://pcloudy.medium.com/how-to-perform-ocr-testing-in-mobile-apps-402111e95447</a></li><li>Transloadit. (2025). “Implementing OCR in Android and iOS Apps with Open-Source SDKs.” <a href="https://transloadit.com/devtips/implementing-ocr-in-android-and-ios-apps-with-open-source-sdks/">https://transloadit.com/devtips/implementing-ocr-in-android-and-ios-apps-with-open-source-sdks/</a></li><li>ZenRows. (2025). “IP Rotation Scraping.” <a href="https://www.zenrows.com/blog/ip-rotation-scraping">https://www.zenrows.com/blog/ip-rotation-scraping</a></li><li>Confluent. (2025). “Real-Time Web Scraping.” <a href="https://www.confluent.io/blog/real-time-web-scraping/">https://www.confluent.io/blog/real-time-web-scraping/</a></li><li>Human Security. (2024). “Controlling AI-Driven Content Scraping with Human.” <a href="https://www.humansecurity.com/learn/blog/controlling-ai-driven-content-scraping-with-human/">https://www.humansecurity.com/learn/blog/controlling-ai-driven-content-scraping-with-human/</a></li><li>PromptCloud. (2025). “Legal Considerations in Web Scraping: Navigating GDPR, CCPA, and Beyond.” <a href="https://www.promptcloud.com/blog/legal-considerations-in-web-scraping-navigating-gdpr-ccpa-and-beyond/">https://www.promptcloud.com/blog/legal-considerations-in-web-scraping-navigating-gdpr-ccpa-and-beyond/</a></li><li>Apify. (2024). “Is Web Scraping Legal?” <a href="https://blog.apify.com/is-web-scraping-legal/">https://blog.apify.com/is-web-scraping-legal/</a></li><li>Bright Data. (2025). “Web Scraping with Gemini AI in Python — Step-by-Step Guide.” <a href="https://brightdata.com/blog/web-data/web-scraping-with-gemini">https://brightdata.com/blog/web-data/web-scraping-with-gemini</a></li><li>Praveen Govindaraj. (2024). “Web scraping Made Simple with GenAI.” <a href="https://praveengovindaraj.com/web-scraping-made-simple-with-genai-3911d1a46cbc">https://praveengovindaraj.com/web-scraping-made-simple-with-genai-3911d1a46cbc</a></li><li>Miquido. (2025). “What is AI Data Scraping?” <a href="https://www.miquido.com/ai-glossary/what-is-ai-data-scraping/">https://www.miquido.com/ai-glossary/what-is-ai-data-scraping/</a></li><li>PromptCloud. (2025). “Mobile App Scraping to Extract Data from iOS &amp; Android Apps.” <a href="https://www.promptcloud.com/blog/guide-to-mobile-app-scraping/">https://www.promptcloud.com/blog/guide-to-mobile-app-scraping/</a></li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c5bb78100acd" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Very powerful prompt: “Explain it with gradually increasing complexity.”]]></title>
            <link>https://weimingchenzero.medium.com/very-powerful-prompt-explain-it-with-gradually-increasing-complexity-ab91d97c099a?source=rss-7d453923e23------2</link>
            <guid isPermaLink="false">https://medium.com/p/ab91d97c099a</guid>
            <dc:creator><![CDATA[Weiming Chen]]></dc:creator>
            <pubDate>Thu, 27 Jun 2024 15:06:40 GMT</pubDate>
            <atom:updated>2024-06-27T15:06:40.958Z</atom:updated>
            <content:encoded><![CDATA[<blockquote>Very powerful prompt: “Explain it with gradually increasing complexity.”</blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ab91d97c099a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Pitch script]]></title>
            <link>https://weimingchenzero.medium.com/pitch-script-25a66cf3ef30?source=rss-7d453923e23------2</link>
            <guid isPermaLink="false">https://medium.com/p/25a66cf3ef30</guid>
            <dc:creator><![CDATA[Weiming Chen]]></dc:creator>
            <pubDate>Tue, 25 Jun 2024 13:53:39 GMT</pubDate>
            <atom:updated>2024-06-25T13:53:39.043Z</atom:updated>
            <content:encoded><![CDATA[<p>Pitch script</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*EAC9OO06WfKkS7lI.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=25a66cf3ef30" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Engineering release note]]></title>
            <link>https://weimingchenzero.medium.com/engineering-release-note-cdce34b78f77?source=rss-7d453923e23------2</link>
            <guid isPermaLink="false">https://medium.com/p/cdce34b78f77</guid>
            <dc:creator><![CDATA[Weiming Chen]]></dc:creator>
            <pubDate>Tue, 25 Jun 2024 13:31:06 GMT</pubDate>
            <atom:updated>2024-06-25T13:31:06.912Z</atom:updated>
            <content:encoded><![CDATA[<p>Engineering release note</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DKCEPkR7EIP9TMsq.png" /></figure><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cdce34b78f77" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Applied Data Science with Churn Problem.]]></title>
            <link>https://weimingchenzero.medium.com/applied-data-science-with-churn-problem-a778821577a2?source=rss-7d453923e23------2</link>
            <guid isPermaLink="false">https://medium.com/p/a778821577a2</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[churn]]></category>
            <category><![CDATA[xgboost]]></category>
            <category><![CDATA[churn-prediction]]></category>
            <dc:creator><![CDATA[Weiming Chen]]></dc:creator>
            <pubDate>Thu, 30 Dec 2021 05:12:47 GMT</pubDate>
            <atom:updated>2021-12-31T07:48:40.316Z</atom:updated>
            <content:encoded><![CDATA[<h3>Applied Data Science with Churn Problem. Building Predictive Machine Learning Model for Consumer Business Industry</h3><h4>Intro</h4><p>In the consumer product/service industry, the churn problem is crucial for business. A churn problem is a general way of describing customers switching to another or canceling out — for example, telecommunication phone plans, gaming platforms, entertainment streaming platforms. However, every business defines the Churn problem differently concerning each’s business model nature. For example, some may define churn to paid subscribers only based on MRR (monthly recurring revenue). On the other hand, for some utterly free service, these businesses may define churn on activity-based, that observe on user’s recency activity, who stop using the service within an allowable time gap. (e.g Twitter, Instagram).</p><p>In addition, for companies that offer a freemium business model (free tier + paid tier), users have options to cancel membership or closeout account; businesses may define churn on an overall cancellation based.</p><h4>Project Description</h4><h4>Project Overview</h4><p>Sparkify is a simulated music streaming service similar to a Spotify-like business. I will work through churn problem solving based on Sparkify’s event log dataset in this post. The goal is to measure the churn, design metrics for forecasting churn, and build a predictive machine learning model.</p><h4>Project Problem</h4><p>Churn is often an overlooked issue for the consumer industry. By convention, Stakeholders, Customer Success Team often emphasizes acquiring new users, allocating a large amount of resource and energy for competing for new clients. However, the unit cost to getting a new customer is relatively more expensive than keeping an existing customer. Furthermore, once users switch to a competitor, it is even harder to get them back. Data scientists spend efforts on forecasting users’ churn probability in a lead time — extract pattern within churn users. Even find out latent factors associated with the product and or pricing model.</p><p>For the Sparkify project, the problem is measuring the overall churn rate, designing analytic metrics, and building a classification model to predict churn.</p><h4>About This Post</h4><p>The post will focus more on applied data science on the churn, the design process, and methodology. For full detailed implementation, I host my end to end source code for this project here:</p><p>Link on my GitHub.</p><p><a href="https://github.com/gilzero/project_churn_spark">https://github.com/gilzero/project_churn_spark</a></p><p>Technologies Used for this project</p><ul><li>Python 3.9</li><li>Jupyter</li><li>Pandas</li><li>Numpy</li><li>Plotly</li><li>Matplotlib</li><li>Scikit-learn</li><li>XGBoost</li><li><em>Apache Spark (Databricks) for the full set</em></li></ul><p>—</p><h4>Data Exploration</h4><p>It comes with a mini size and full-size event datasets. For this post, I worked on the mini-size set locally. It contains 286,500 event logs for 225 distinct users.</p><p>Raw data attributes:</p><ul><li>ts: timestamp of event log</li><li>userId: account ID</li><li>sessionId: each browsing session ID</li><li>page: all kinds of pages user encountered for the event</li><li>auth: the authentication status of user for the event</li><li>method: HTTP REST method</li><li>status: HTTP request code</li><li>level: indicate user’s tier, free-tier or paid-tier</li><li>itemInSession: the cumulative count number of items in the session</li><li>location: user’s location (U.S based)</li><li>userAgent: browsing hardware / OS</li><li>lastName</li><li>firstName</li><li>registration: timestamp of user’s registration date time</li><li>artist: artist name of the song listen</li><li>song: song name</li><li>length: playback duration of a song play event</li></ul><p>Example of data instances:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*moyHbXlhIJCKAqYA6D76eA.png" /></figure><p>By checking the max and min timestamp of the event, it shows the dataset is about a two-month interval sliced event logs.</p><p>There are some instances of missing userId, indicating anonymous users. I will trim these instances.</p><p>Consumer products usually follow the weekly cycle of human behavior. So, out of curiosity, let’s start with checking out what days of week users use the service to stream music.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*v4sQYvxCDAVS9jW9VpfE9g.png" /></figure><p>I initially thought users tend to use streaming entertainment on the weekend than on weekdays because people might be busy at work or studying during weekdays. Still, the dataset reveals it is people listen more on a weekday. It is a reminder that a quick visualization can help you justify any preconception or misconception.</p><p>One of the most crucial attributes in the raw log is ‘page’.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*aF9H7ttW3qSg2aB-XWJisA.png" /></figure><p>Within values of the ‘page’ attribute, the dominant event is ‘NextSong’, when the user clicks ‘NextSong’, indicating a song play count.</p><p>There is no description of attributes and values for the dataset, but I would infer them as such events:</p><ul><li>NextPage: indicate a song play event</li><li>Home: home page visited</li><li>Thumbs Up: a ‘Like’ button hit for a song</li><li>Add to Playlist: user added a song to his/her playlists</li><li>Add Friend: user hit Add Friend social button</li><li>Roll Advert: user experienced an advertisement during song listening</li><li>Login: user login action</li><li>Logout: user log out action</li><li>Downgrade: user viewed the Downgrade page.</li><li>Help: user viewed the Help page</li><li>Settings: user viewed the Settings page</li><li>About: user viewed the About Us page</li><li>Upgrade: user view the Upgrade page</li><li>Save Settings: user adjusted settings</li><li>Error: user encounter an error</li><li>Submit Upgrade: user submitted an upgrade (nice :-) )</li><li>Submit Downgrade: user submitted a downgrade, down-sell :-(</li><li>Cancel: user viewed the Cancel page.</li><li>Cancellation Confirmation: user submit and confirm the cancellation. (Oh no!)</li><li>Register: user registration page.</li><li>Submit Registration: new user entered.</li></ul><p>There are all the possible activity events for users. Here is the bottom line: for consumer service, we should focus primarily on engagement and utilization-related events, things like song play, like, dislike, playlist. In the next section, we need to pay attention to these primary behavioral metrics.</p><p>User experience related to page browsing events such as ad listened, home page view, help page, adjust the setting, etc. These might not be important as those of engagement/utilization metrics.</p><h4>Measuring Churn</h4><p>Measuring churn can be tricky and confusing if you don’t stick to your specific business model and nature. For churn measuring in this project, we have possibly related events in the log such as Submit Downgrade, Cancellation Confirmation. But how to identify? Should I count Downgrade as churn in this case? Here is the logic: It is a freemium business model. Both free and paid tier users can cancel. But only paid users can downgrade. I consider Downgrade as Down-Sell for this scenario. Users that downgrade would continue to use the service. They are still retained. If a business is measuring churn based on MRR (Monthly Recurring Revenue) formula, since Downgrade is a Down-sell means possibly reducing the MRR, if that were the case, I would consider Downgrade as churn. However, based on the limited information of the dataset, we do not have pricing data. I decided to measure churn as <strong>standard count based</strong>. I will ignore Downgrade as churn. <strong>So the indicator for churn is ‘Cancellation Confirmation.’ Users that finally confirm cancellation are the users who churn.</strong></p><p>After some calculation for churn measurement:</p><ul><li>Number of all distinct users: 225</li><li>Number of paid users: 165</li><li>Number of free users: 60</li><li>Number of Churn Users: 52</li><li>Number of Retain Users: 173</li><li><strong>Churn rate within all users: 0.23</strong></li><li>Churn rate within free-tier users: 0.35</li><li>Churn rate within paid-tier users: 0.19</li></ul><p>Churn Rate: 23%. (Number of Total Churn Users / Number of All Users)</p><p>It also shows that paid-tier users have more ‘loyalty’ than free-tier users. (19% vs 35%)</p><h4>Feature Engineering (Metric Design)</h4><p>Now the fun part: Feature Engineering. (Or in business terms: Metric Design). Feature Engineering is a more adopted term in the data science/data engineering field. They might be interchangeable that both are about generating analytic features/metrics for building a model. There are no strict rules. Some believe it is a combination of art and science. Exciting? Let’s get started!</p><h4>Feature: Tenure Days</h4><p>Tenure means the number of days or length the users have been using the service. For this dataset, the formula would be</p><p>Tenure = latest event date time — user registration date time</p><p>Visualization check:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*7_BWwyktkwP_Qxah2A4Ulg.png" /></figure><p>churn users generally have fewer tenure days than retain/non-churn users. (57 days vs. 86 days)</p><p>Hypothesis: churn users have fewer tenure days. Metric: tenure</p><h4>Feature: UserAgent</h4><p>The raw UserAgent example instance like “Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/37.0.2062.103 Safari/537.36”. It gives us specific information such as browser, device os, hardware. I will try to generalize to the device to check if the device might be a factor.</p><p>Apply a custom transformation, create a ‘device’ column</p><p>Also, compute the percentage of churn for each device group.</p><p>Stats screenshots</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*g_-TlLPk_xyC1e1pHsPYPg.png" /></figure><p>Visualization of device churn check</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*vfX-795Qk5aq2aQjCTUwKg.png" /></figure><p>It seems that the Linux user has more churn percentage. However, the instances of Linux are limited. At this point, not sure is by random chance or not. It will become more clear if we have more Linux user data. For the mini set, I will include it in training anyway. (Note that there is a risk of over-fitting due to the few instances)</p><p>Hypothesis: Users using the Linux version of the product are more likely to churn. Metric: is_linux</p><h4>Feature: Location</h4><p>Raw event contains users’ location information in city and state abbreviation code. They are too specific. I will generalize them by transforming them into four central U.S regions and comparing percentages.</p><p>Stats results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*en8UNF8Q4u-r28AXJ5uHpA.png" /></figure><p>Visualization check</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*ss40oF3lMSWNgtsJskDMzw.png" /></figure><p>As the plot shows: they are somewhat proportional. There is no typical evidence that region is a factor of churn. Ignore location.</p><h4>Feature: Gender</h4><p>Gender is another categorical data available in the dataset. Is gender a matter to churn? Let’s find out:</p><p>Compute the percentage comparison.</p><p>Stats:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*LbZArJ1Q2VYkWMm7p9rEtA.png" /></figure><p>Visualization Check:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*2ziLKCMUNHa_ZCMBr8BoUA.png" /></figure><p>Another way of grouping visualization check:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*_HUbAciO-JeASy3xSraorA.png" /></figure><p>The difference is noticeable.</p><p>Hypothesis: males are relatively more likely to churn than females. Metric: is_male</p><h4>Feature: Level</h4><p>Is there any difference in churn rate for free-tier users vs. paid-tier users?</p><p>Stats</p><p>35% vs. 18%</p><p>Visualization Check</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*63POeFY4d1H9buegQ1RVRg.png" /></figure><p>There is a clear difference.</p><p>Hypothesis: Free-tier user churn at a more significant rate. Paid-tier users have more ‘loyalty’ for the service. Metric: is_free</p><h4>(Features: Aggregations of all kinds of ‘page’ events.)</h4><p>Next, I will iterate to calculate aggregation for different events, lay out the stats, and decide whether to establish a metric hypothesis if necessary. (Since the aggregation are pretty straightforward, I will not visualize every single of it for saving some space for this post)</p><h4>Feature: Song Play Count</h4><ul><li>Churn-Users: 699.88 (avg)</li><li>Retained-Users: 1108.17 (avg)</li></ul><p>Retain users averagely play significantly more songs than churn users.</p><p>Hypothesis: churn users play fewer songs. This is a primary metric. Metric: n_play</p><h4>Feature: Like Count</h4><ul><li>Churn-Users: 35.75 (avg)</li><li>Retained-Users: 61.80 (avg)</li></ul><p>Hypothesis: retain user hit ‘Like’ more. Metric: n_like</p><h4>Feature: Dislike Count</h4><ul><li>Churn-Users: 9.54 (avg)</li><li>Retained-Users: 11.85 (avg)</li></ul><p>Not much difference. But we will test it out. It may relate to engagement. Metric: n_dislike</p><h4>Feature: Add to Playlist Count</h4><ul><li>Churn-Users: 19.96 (avg)</li><li>Retained-Users: 31.72 (avg)</li></ul><p>Hypothesis: churn users add fewer songs to the playlist. Metric: n_addtolist</p><h4>Feature: Add Friend Count</h4><ul><li>Churn-Users: 12.23 (avg)</li><li>Retained-Users: 21.04 (avg)</li></ul><p>Hypothesis: churn users add fewer friend. Metric: n_friend</p><h4>Feature: Home Page Viewing Count</h4><ul><li>Churn-Users: 32.15 (avg)</li><li>Retained-Users: 48.61 (avg)</li></ul><p>There is a noticeable difference also. Statistically speaking, churn users visit the homepage less. Retain users more frequently to check the home page. However, is homepage visit matter to engagement or service utilization? I am reluctant to use it. I decide to ignore it. I will explain the decision more below.</p><h4>Feature: Advertisement Listened Count</h4><ul><li>Churn-Users: 18.60 (avg)</li><li>Retained-Users: 17.14 (avg)</li></ul><p>Not much difference. Ignore it.</p><h4>Feature: Downgrade Page Viewing Count</h4><ul><li>Churn-Users: 6.48 (avg)</li><li>Retained-Users: 9.93 (avg)</li></ul><p>It shows some difference, really not that much, and not related to engagement. Ignore it. I will also explain the decision below.</p><h4>Feature: Submit Downgrade Action Count</h4><ul><li>Churn-Users: 0.17 (avg)</li><li>Retained-Users: 0.31 (avg)</li></ul><p>It is a noticeable difference — even double. But I would not select it, which will be explained below.</p><h4>Feature: Help Page Viewing Count</h4><ul><li>Churn-Users: 4.59 (avg)</li><li>Retained-Users: 7.02 (avg)</li></ul><p>Not much different at scale. Ignore it.</p><p>Before continuing, let’s pause a minute to explain some decisions I made for the aggregation data.</p><p>For consumer product/service, the bottom line for feature engineering (metrics design) here is: WE SHOULD FOCUS ON USER BEHAVIORAL EVENTS, specifically on:</p><ul><li>Engagement, and</li><li>Utilization</li></ul><p>Some people might argue that statistically speaking, those stats showing noticeable difference should also be considered.</p><p>Theoretically, yes. But I would like to emphasize that metrics should be explainable and meaningful for our specific problem and purpose. In this case, we draw our attention span on metrics that to engagement and utilization. Including all kinds of statistically sound metrics may score higher in the model. However, it might risk over-fitting by feeding too many nuances to Machine Learning training to memorize things. To recall that, our goal for Machine Learning, building a predictive model, is trying to extract and <strong>generalize</strong> the pattern as much as possible, not to overfit or memorize. Please take a look at our example. Let’s say number_of_downgrade_action shows a relationship to churn rate, but we need to think this way: what are the behaviors causes downgrade. Do Home page or Help page viewing matter to song playing engagement? Sometimes, it is not a good idea to overthink. Our goal is to generalize core product engagement and utilization here. Let’s move on.</p><p>Quickly wrap up some decisions so far:</p><ul><li>Categorical data (Nominal Data). is_linux, is_male, is_free</li><li>Discrete data aggregation, select relate to user engagement and service utilization, and show some evidence — things like song, like, dislike, playlist, session, items in session, tenure days.</li><li>Continuous data, for example, playback duration length.</li></ul><p>With some computation, I have calculated the following features (metrics) for <strong>each</strong> user.</p><ul><li>n_play</li><li>n_session</li><li>n_item</li><li>n_addtolist</li><li>n_addfriend</li><li>n_like</li><li>n_dislike</li><li>n_distinct</li><li>tenure</li><li>total_length</li><li>ratio_length_per_session</li><li>ratio_addtolist_per_play</li><li>ratio_like_per_play</li><li>ratio_dislike_per_play</li><li>change_perc_n_play</li><li>change_perc_total_length</li></ul><p>For details of calculation and formula, refer to my source code host <a href="https://github.com/gilzero/project_churn_spark">here</a>. The meaning of each metric should be self-explainable with its variable name. First, however, I want to elaborate on ratiometric and change metrics.</p><p>Ratiometric is a metric design from two or more standalone metrics. It often indicates a unit cost, unit usage, or unit engagement. Y per X. Interestingly, some standalone metric themselves does not show a relationship to the churn, but it reveals a trend when embedded to a ratiometric. You will see detailed evidence next.</p><p>A quick look at a ratiometric stats:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/393/1*sIRqYDR18jwD1jwaDIeLZA.png" /></figure><p>And for Change Percentage metric is a measure of momentum shifting. It is a kind of design in many different industries. E.g., the Stock market measuring PE ratio change. Coffee Roasting metric RoR (Rate of Rise) measuring the dynamic temperature changing. In this case, I generate a change percentage of the song played and a change percentage of playback duration in two different recency periods. The design is to try to see whether the momentum of change has any relationship to churn.</p><p>A quick look of a change percentage metric stats:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*AVsURRobFkbPCsmByaguEQ.png" /></figure><h4>Create a flattened analytic dataset</h4><p>Once the features are computed, let’s create an analytic dataset by flattening the metrics.</p><p>Here is a quick look at the analytic dataset I created:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*Ms4F499kx3dAf-q18KJFaw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*I9e6c6njRWBDSIIB8WFs3g.png" /></figure><p>Also, don’t forget to concatenate categorical features transformed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*nK-fDDQbmwiR3Omcxeoa2Q.png" /></figure><p>And the outcome label columns is ‘is_churn’</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*9VbwKxh8vyY_St4sZtNElQ.png" /></figure><p>Then fill the missing data.</p><h4>Metric Cohort Analysis</h4><p>Next, let’s conduct a metric cohort analysis. For each feature (metric), divide into ten cohorts based on actual values, and compare each cohort’s mean value to the churn rate against each cohort. Then, plot them in a line chart to analyze the relationship with churn. We will also check the skewness stats.</p><p>The first run attempt of metric cohort analysis. Results:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*6ckErKqlsvswf025hvYmig.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*VUqERrC9lzls_gYmgOil2g.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*s2-eXlfn4Z38uqQhYQnrkQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*9M4-4sI0afERPa6luiel2g.png" /></figure><p>Our first run of metric cohort analysis isn’t too bad but certainly noisy. Don’t worry, and we will work on the refinement next. Some metrics do show a relationship with churn, while some are volatile. Let’s take away the high skewed metrics.</p><h4>Skewness Check</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1005/1*uGNPqbIhuvQHuaTWnQceTg.png" /></figure><p>Bottom line: high skewed metrics can be unreliable/problematic.</p><p>A skewed metric around zero indicates it is approach evenly distributed around the mean. More evenly cohort group values. Suppose a skewness is too high. It may result in a long tail. For example, right long-tail shape skew, most metric values are before the mean value.</p><h4>Data Normalization</h4><p><strong>The strategy here is to remove skewed metrics and normalize the remaining metrics data values.</strong> Then, conduct another round of cohort analysis to see how it goes.</p><p>I use MinMaxScaler() to transform the analytic data to scaled values.</p><p>A quick look here:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*HWdPeINnKT4jhYenkO33HQ.png" /></figure><h4>Refinement</h4><p>With reselected and normalized data, run another Metric Cohort Analysis. This time I also zoom out a bit to 6 cohorts. Here is the result: (2nd round)</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*IvQNop9akRgxS8XzjvvANQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*nbpXiSTF86ExTwNBUQG7WQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*ER7gPpOSpYbFulA2W_z6mw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*2LXYoU7xlCQvUMvb5lJVmw.png" /></figure><p>Visually look, so much improvement. Isn’t it! Most of the metrics show evident relationships with churn.</p><p>However, <em>n_dislike</em> and <em>ratio_like_per_day</em> are still not telling a story. Therefore, I will filter these two metrics out.</p><p>Be patient. Time to run the last round of Metrics Cohort Analysis to confirm. Let’s see the result:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*-IbPK1F1PAN7Jw0Ncysy6w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*UsvnYf5L7YR8FU1OUN4OoA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*kHl1wcHf1S9XQQXXiIafGQ.png" /></figure><p>Awesome! After all the effort on Feature Engineering with rounds of refinement, the metrics re-selected are all showing some degrees of relationship / pattern here.</p><p>Before we move to the next section, I mentioned above a reminder for ratiometric design. Have you noticed the interesting found?</p><p>Check the plot of <em>n_dislike</em> vs. <em>ratio_dislike_per_play</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*8aqnmEM2vZ-5yqSoKJIjRg.png" /><figcaption>n_dislike</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*sTk4o6yM81O8vn5lw7by8g.png" /><figcaption>ratio_dislike_per_play</figcaption></figure><p>That’s right. A standalone metric of the number of dislikes each user n_dislike does not show any clear pattern. However, when we design it into a ratio representing a unit of disengagement, <em>ratio_dislike_per_play</em>, the higher it is, the more likely the user churn.</p><h4>Machine Learning</h4><p>We have done some bits of Data Science + Data Engineering works above so far. Now, let’s move to a bit of the artificial intelligence part for the project: Machine Learning. This churn problem is a supervised classification model labeling a binary outcome: prediction of YES/NO (1/0). For this project, I will approach Logistic Regression and XGBoost (Decision Tree) models and compare their performance.</p><p>If we have adequately designed the dataset, it should be easy for ‘Machine’ to crunch the analytic data. So I will quickly lay out the declarative code and then explain validation, evaluation, and justification in detail.</p><h4>Logistic Regression</h4><p>Import libraries</p><pre>from sklearn.model_selection import train_test_split<br>from sklearn import linear_model<br>from sklearn.model_selection import GridSearchCV<br>from sklearn.metrics import f1_score</pre><p>Firstly, subset independent variable values (features set, the X) and dependent variable (outcome set, the y).</p><pre># Prepare independent variable (X) and dependent variable (y)<br><br># Independent variables values. the X. (dataset that without the user id, and outcome label columns)<br>X = df_normalized.iloc[:, 1:-1].values<br><br># Dependent variable. the y<br>y = df_normalized[&#39;is_churn&#39;].values</pre><p>Create training data, holdout test data split, then train the model.</p><pre># Create train set, test set<br>X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42, stratify=y)<br><br># Initialize and logistic_regression instance<br>log_reg = linear_model.LogisticRegression(penalty=&#39;l1&#39;, solver=&#39;liblinear&#39;, fit_intercept=True)<br><br># Train the model<br>log_reg.fit(X_train, y_train)</pre><p>Check result with train set and test set.</p><pre># Not use accuracy score, because mostly 0 value. unbalanced.<br># So use f1 score.<br><br>#f1 score on train set<br>print(f&quot;f1 score on train set: {f1_score(y_train, log_reg.predict(X_train), average=&#39;weighted&#39;)}&quot;)<br><br>#f1 score on test set<br>print(f&quot;f1 score on test set: {f1_score(y_test, log_reg.predict(X_test), average=&#39;weighted&#39;)}&quot;)</pre><ul><li>f1 score on train set: 0.6994369057509463</li><li>f1 score on test set: 0.7295742232451093</li></ul><h4>XGBoost</h4><p>“XGBoost initially started as a research project by Tianqi Chen as part of the Distributed (Deep) Machine Learning Community (DMLC) group. It became well known in the ML competition circles after its use in the winning solution of the Higgs Machine Learning Challenge.” (<a href="https://en.wikipedia.org/wiki/XGBoost">Wikipedia</a>)</p><p>Import library and train. Reuse the train test set.</p><pre>import xgboost<br><br># XGBoost classifier model instance<br>xgb_clf = xgboost.XGBRFClassifier(use_label_encoder=False)<br><br># Train the model.<br>xgb_clf.fit(X_train, y_train)</pre><pre># Train set f1 score.<br>f1_score(y_train, xgb_clf.predict(X_train), average=&#39;weighted&#39;)<br><br># Test set f1 score.<br>f1_score(y_test, xgb_clf.predict(X_test), average=&#39;weighted&#39;)</pre><ul><li>f1 score on train set: 0.8949914949914949</li><li><strong>f1 score on test set: 0.8222222222222222</strong></li></ul><p>XGBoost is outperforming Logistic Regression in this case. Let’s further conduct hyperparameter tuning to see if we can improve even more on the model. Use GridSearchCV</p><pre># GridSearch Cross Validation Tuning the model<br><br>param_grid = {<br>    &quot;max_depth&quot;: [3, 4, 5, 7],<br>    &quot;learning_rate&quot;: [0.1, 0.01, 0.05],<br>    &quot;gamma&quot;: [0, 0.25, 1],<br>    &quot;reg_lambda&quot;: [0, 1, 10],<br>    &quot;scale_pos_weight&quot;: [1, 3, 5],<br>    &quot;subsample&quot;: [0.8],<br>    &quot;colsample_bytree&quot;: [0.5],<br>}<br><br># XGBClassifier instance<br>xgb_cl = xgboost.XGBClassifier(objective=&quot;binary:logistic&quot;, use_label_encoder=False)<br><br># GridSearchCV instance<br>grid_cv = GridSearchCV(xgb_cl, param_grid, n_jobs=-1, cv=3, scoring=&quot;f1_weighted&quot;)<br><br># Train<br>grid_cv.fit(X_train, y_train)</pre><pre># f1 score on train set<br>f1_score(grid_cv.predict(X_train), y_train, average=&#39;weighted&#39;)<br><br># f1 score on test set<br>f1_score(grid_cv.predict(X_test), y_test, average=&#39;weighted&#39;)</pre><ul><li>f1 score on train set: 0.8637037037037038</li><li><strong>f1 score on test set: 0.8418679549114332</strong></li></ul><h4>Validation, Evaluation, Justification</h4><p>As the results show, the winner model is XGBoost with a test set f1 score: 84.2%. (After hyperparameter tuning). It is about 11% more accurate than Logistic Regression for this case.</p><p>Keep in mind that this sort of churn dataset is highly imbalanced. The majority of the observation is NotChurn, indicating the retained users. In other words, there will be much more observations of the ’0’ label than the ‘1’ label.</p><p>(Note that, for train test split process. The ‘stratify=y’ argument is important here. It ensures to maintain the proportion of values in split to the original dataset.)</p><p>What has been fed into this Logistic Regression training is biased. The logistic regression here might be over-optimistic because, with the imbalanced data it learned, it favors to predict ‘0’ as the outcome. It is not helpful for churn problems; we miss out on opportunities to detect churn.</p><p>Some mention that the <a href="https://web.stanford.edu/class/archive/cs/cs109/cs109.1178/lectureHandouts/220-logistic-regression.pdf">word ‘regression’ in Logistic Regression is confusing</a> because it is actually a classification by estimating the probability of an outcome. So it makes sense that for this case, it ‘lost’ to XGBoost as the labeling is not about estimating continuous value; it Is so much tricky to calculate coefficients for such churn problem.</p><p>In contrast, XGBoost is based on decision trees and random forest training. It creates different trees and has cleaver penalization of trees and correct errors of previous trees. It works excellent for classification churn problems.</p><p>The primary concern of the prediction/forecasting for churn is <strong>Type II Error</strong>: <strong>False Negative</strong>. Here is why: with a fire alarm analogy, a fire alarm does not ring the bell when a fire is happening, life-threatening. For a churn problem, a False Negative would be like, fail to identify churn when the churn happens, and then we would miss the best timing or opportunity to retain that user, miss out or underestimate. However, in another way around, if a model accidentally identifies a non-churn user as a churn, the marketing team send out a special offer to them, e.g., give out free extra usage, it makes the existing happy customer happier, from a business perspective, it may increase cost in a short-term. Still, it is not harmful as miss identifying real churn.</p><p>Generally, it is better not to conclude a detect than a detect (Type I Error), e.g., Medical Industry. But not always, for consumer churn problems, Type II errors are worse in this case because once we fail to detect churn when they leave, they DO go. They hardly turn back.</p><p>The aim of the model is not about the ability of successful labeling of ‘0’. So, for example, if 10 out of 100 users would churn, with a model predicting all ‘0’s, it still scores 90% because 90 out of 100 are ‘0’s based on the default accuracy score. Therefore the default accuracy score does not make sense. In that case, it missed out on all ten of ten churns and still got a 90+ score.</p><p>This is why I pick F1 Score as the evaluation metric. Because what we concern more is how many real churn that the model is able to identify. It takes harmonic mean of precision and recall rather than blindly default accuracy.</p><p>(For more details of formula, check <a href="https://en.wikipedia.org/wiki/Precision_and_recall">Precision and Recall</a>, <a href="https://en.wikipedia.org/wiki/F-score">F-score</a>. )</p><p>To run validation based on f1 score, simply pass holdout test set of train data to a model’s predict method, check the return value against holdout test set of outcome data, with f1 score method.</p><p>Besides, I give you the concrete example of why I am disappointed with Logistic Regression here that compares the predictions of the two modes just made:</p><p><strong>Logistic Regression prediction on test set:</strong> almost all ‘0’s for actual churn instances.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*lJMfhMhJMxMWu_aTUuouFw.png" /></figure><p><strong>XGBoost prediction on test set:</strong> Much better performance with most actual churn instances predicted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*q6Yb4bgsHgJurww2NqtUtg.png" /></figure><p>Number of actual churn instances is successfully predicted: XGBoost</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*oFEU1LP2KFC1t090nXADRg.png" /></figure><p>Let’s put the performance of model into a table for comparison:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*P0NOxnwNoUt7Vf7d3UBeFg.png" /></figure><p>As you can see, XGBoost scores 11% more than Logistic Regression, but know the difference in the number of real churns predicted is a vast difference — <strong>7 vs 1</strong>! So that is quite impressive for XGBoost for churn problem classification.</p><p>In contrast, even Logistic Regression scored a 0.73, looks acceptable. BUT the actual prediction is inferior.</p><p>For XGBoost grid search cross validation training, <em>scoring=”f1_weighted”</em> is specified, telling the training aim to maximize based on this score.</p><h4>Reflection &amp; Improvement</h4><p>This is a challenging and highly practical project to work on. I particularly enjoy the process of Feature Engineering (Metrics Design) part. However, there are no customer details and subscription details data, and only a raw event log is provided. Therefore, I need to invest more time and experiments on various metrics to develop a churn problem strategy and refinement iteration. I would say probably almost 90% of my time on this project is on Feature Engineering. About 5% of the time on Data cleaning, wrangling, preprocessing, transforming. The rest of 5% on Machine Learning models.</p><p>Further improvement can be made, including grouping correlated features, also known as Dimension Reduction. Before that, calculate a features correlation matrix to decide what metrics to filter out or group together. It will provide further advantages to generalize the model.</p><p>For the full size of 12GB data dataset, it is better to run with Apache Spark. Amazon AWS EMR is an option for setting up. However, I highly recommend using Databricks to work with it. I found it is much easier and quicker to set up the cluster and provide a cleaner interface. (Databricks is a higher abstract level application based on Apache Spark.) It also helps me auto-scale up and down the number of node workers. The methodology and general flow for data science and machine learning implementation are the same as the mini set as we work through above. We just need to use another set of data engineering tools for Databricks. It provides Pandas DataFrame equivalent libraries like PySpark, and SparkSQL to run SQL if you prefer the SQL way. For example:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VpUifTrC8xArFlCzY3WZGg.png" /></figure><p>And recently, Databricks team has been porting <a href="https://databricks.com/blog/2020/06/24/introducing-koalas-1-0.html">Koalas</a> to Databricks runtime, which means we can also do Pandas like syntax in Spark.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/955/1*NJaFOk0yW5Aw5OFohwzyqw.png" /></figure><p>I recommend anyone needs Apache Spark to try Databricks. It charges on useage based. (I have no affiliation with it.)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eMWjaxuWFZD_I5S3qYRqoQ.png" /></figure><h4>Reference &amp; Credit</h4><p>The snippet of <a href="https://github.com/carl24k/fight-churn/blob/master/fightchurn/listings/chap5/listing_5_3_metric_scores.py">Metric Cohort Plot</a> and <a href="https://github.com/carl24k/fight-churn/blob/master/fightchurn/listings/chap5/listing_5_2_dataset_stats.py">Skewness</a> reshape function is from Carl S. Gold, the author of ‘<a href="https://www.amazon.com/Fighting-Churn-Data-Carl-Gold/dp/161729652X">Fighting Churn with Data</a>’. I would also like to recommend his book ‘Fighting Churn with Data’. I enjoy reading his book, very knowledgeable, which provides systematic techniques for working with all kinds of subscription-based businesses. It is a great book that also guided me in approaching the project.</p><p>And, thanks to Udacity providing this raw event log dataset.</p><p>Thanks for reading!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a778821577a2" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Hypothesis Pitfalls on Excess Coffee Effects Research in University of South Australia]]></title>
            <link>https://weimingchenzero.medium.com/hypothesis-pitfalls-on-excess-coffee-effects-research-in-university-of-south-australia-557c937c073f?source=rss-7d453923e23------2</link>
            <guid isPermaLink="false">https://medium.com/p/557c937c073f</guid>
            <category><![CDATA[hypothesis-testing]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[brain]]></category>
            <category><![CDATA[coffee]]></category>
            <category><![CDATA[regression-analysis]]></category>
            <dc:creator><![CDATA[Weiming Chen]]></dc:creator>
            <pubDate>Thu, 18 Nov 2021 08:44:31 GMT</pubDate>
            <atom:updated>2022-01-28T05:27:48.053Z</atom:updated>
            <content:encoded><![CDATA[<p>I came across various news posts about the manifest of proof on high coffee consumption’s impact on the brain, heart. Then I traced back to the citing source, and it was referenced in a research article: <em>Kitty Pham, Anwar Mulugeta, Ang Zhou, John T. O’Brien, David J. Llewellyn &amp; Elina Hyppönen (2021) High coffee consumption, brain volume and risk of dementia and stroke, Nutritional Neuroscience, DOI: 10.1080/1028415X.2021.1945858</em></p><p><strong>The approach and declaration remain questionable for me as I consider it potentially falls to several hypothesis pitfalls.</strong></p><p>Let’s lay out the critical context in that research article:</p><p><a href="https://www.tandfonline.com/doi/full/10.1080/1028415X.2021.1945858">https://www.tandfonline.com/doi/full/10.1080/1028415X.2021.1945858</a></p><blockquote>Summary: People who drink six or more cups of coffee a day have a 53% increased risk of developing dementia and a higher risk of stroke.</blockquote><blockquote>Methods: We conducted prospective analyses of habitual coffee consumption on 398,646 UK Biobank participants (age 37–73 years), …. We examined the associations with brain volume using covariate adjusted linear regression, and with odds of dementia (4,333 incident cases) and stroke (6,181 incident cases) using logistic regression.</blockquote><blockquote>Results: … After full covariate adjustment, consumption of &gt;6 cups/day was associated with 53% higher odds of dementia compared to consumption of 1–2 cups/day (fully adjusted OR 1.53, 95% CI 1.28, 1.83) ….</blockquote><blockquote>Conclusion: High coffee consumption was associated with smaller total brain volumes and increased odds of dementia.</blockquote><h4>Here are what I think (3 major pitfalls it has)</h4><h4>Population vs Sampling</h4><p>Coffee is a worldwide drink. The conclusion inference represents the entire population. However, the population sampling is from UK Biobank participants. So we don’t know how random the sampling distribution is. e.g., the sample demographics, genders? Even ethnicity / genes (For example, western people are more allergen in peanut than Asian | Netflix Documentary, Rotten — The Peanut Problem <a href="https://www.imdb.com/title/tt7830586/">https://www.imdb.com/title/tt7830586/</a>). Properties of the sample? Is it generalizable, uncluster enough to produce a linear regression?</p><p>The sampling process potentially biases the results and systematically inflates the regression model from the start.</p><h4>Coffee vs. Caffeine (Multi Variables)</h4><p>Cups of coffee are not Amounts of Caffeine.</p><p>For example, a 6-ounce cup of coffee may contain twice the caffeine content than a 12-ounce cup of coffee.</p><p>It depends on the coffee varietal, coffee blend, etc.</p><p>(Think about robusta contains almost twice the caffeine as in arabica)</p><p>That implies an inadequate set of data variables on coffee and caffeine in the research.</p><p>It analyzes a multivariate study area that should acquire adequate variables on coffee and variables on caffeine, before concluding ‘cups’ of coffee. It is ambiguous.</p><h4>Correlation vs Causation</h4><p>Confounding Bias.</p><p>A common IceCream-Shark correlation pitfall.</p><p><a href="https://www.kdnuggets.com/2019/01/dr-data-ice-cream-linked-shark-attacks.html">Do sharks attack you more if you eat more ice cream?</a></p><p>Psychologically, Our minds are wired to believe such a type of causation from a storytelling perspective.</p><p>Hold on a second, what kind of people regularly drink 6+ cups of coffee? That is an insane amount of coffee consumption individually. But, 6+ cups of every type of drink is risky. Beers, cokes, red bulls, you name it. (Even coffee tasters in cupping sessions won’t swallow coffee liquid. They split out after slurp. )</p><p>Possibly, people with a high intensive workload, beyond-limit attention, and focus are more likely to drink 6+ cups of coffee/day. As a result, they could lack sleep, rest, exercise, appropriate diet, and mental stress.</p><p>There is a latent Confronting variable: lifestyle. Pertinent factors are potentially ignored in this research experiment.</p><blockquote>“Blind respect for authority is the greatest enemy of truth.”</blockquote><p>Albert Einstein</p><h4>In conclusion</h4><p>I don’t think the alternative hypothesis from this research article can reject the null. However, the way the research approach is not convincing, at least for me. To lay such declaration, the research article needs more persuasive evidence to draw association. The effect association may exist, but the approach is questionable.</p><p>The bottom line of that research paper, reminds that getting rest, hydrated when necessary, listen to your body, your internal biometrical algorithm won’t lie to you.</p><p>Does the research article draw a truth-worthy conclusion? Perhaps no. Even, it sent out an irresponsible impact. “Great power requires great responsibilities.” Especially true in scientific methodology. To extend, think about the illusion, misinterpretation and spread with social media (coffee: brain shrinker, heart attacker) will have a distinct impact on coffee farmer / producer wellbeing in Africa and/or other coffee farms producing regions.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=557c937c073f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Blockchain Understanding]]></title>
            <link>https://weimingchenzero.medium.com/blockchain-understanding-2bb3be61e928?source=rss-7d453923e23------2</link>
            <guid isPermaLink="false">https://medium.com/p/2bb3be61e928</guid>
            <category><![CDATA[blockchain]]></category>
            <category><![CDATA[data]]></category>
            <category><![CDATA[decentralization]]></category>
            <dc:creator><![CDATA[Weiming Chen]]></dc:creator>
            <pubDate>Mon, 08 Nov 2021 16:20:10 GMT</pubDate>
            <atom:updated>2021-11-17T15:59:16.325Z</atom:updated>
            <content:encoded><![CDATA[<p>After several technical, non-technical books, occasional random blog posts, videos about blockchain in the past, I still found blockchain misleading, confusing, unintuitive. It is not about cryptography, not about the distributed network, not about the database design, not about software architect, etc.</p><p>Literally, chains of blocks of data.</p><p>Today, HBR’s (Harvard business review)’s “Blockchain” first chapter gave me a precise “A-Ha” moment realization.</p><p>It states blockchain as a <strong>foundational</strong> technology, given a trajectory comparison of TCP/IP adoption — <strong>that made sense</strong>.</p><p>Before the TCP/IP protocol network, the connection between every two parties by wiring connects from the switchboard. (Remind me, Angelina Jolie, the Changeling (film 2008), her job (switchboard supervisor): answer the phone, click the switchboard routes as requested)</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0g-MND7EajYVJcbuY6QPgQ.jpeg" /></figure><p>Then TCP/IP was firstly adopted in private sectors: Universities, U.S Defense.</p><p>Every node (peers) using TCP/IP protocols forms a public network, data transmitted by sockets encoded and decoded. It is a <strong>foundational</strong> technology. They were firstly trying and benefiting inside sectors. All public gradually adopted it. Then boom. We now cannot imagine a world without the internet (on top of TCP/IP)</p><p>And the four grid frameworks, from <strong>single-use</strong> to <strong>transformative</strong> use.</p><p>Single-use and private sector (organization internal) use are immediately approachable at this stage. And there are already off-the-shelf solutions.</p><p><strong>Smart Contract</strong> kind of use is fascinating, but it is more under transformative type require all sorts of institutions working together to approach. So it still could be decade(s) to go. But it is ultimately the foundational transformation of society, business, day-to-day living.</p><p>Another book, “Mastering Blockchain,” has differentiated:</p><ul><li>Centralized</li><li>Distributed</li><li>Decentralized</li></ul><p><em>Distributed</em> and <em>Decentralized</em> are blurring interchangeably in software architecture. However, it is precisely a <strong><em>decentralized</em></strong>, peer-to-peer network formed with interested nodes in the blockchain field.</p><p>(Distributed is somehow still mean by a centrally planned interest. For example, a distributed system may still require a leader node to know how nodes are distributed. ).</p><p><strong>Blocks distributed</strong> and <strong>blocks chained</strong> are not the same thing. What is meant by ‘chained’ is more as sequential appended-only, timestamped-only chained. (So, what confused/misled me before were lots of illustrations showing data grids that link together, but instead should be chaining together. Since the linking together can only explain on the half part: the form of such peer-to-peer network but not the core idea behind ‘<strong>chains</strong>’. I believe you saw tons of such misleading graphics.)</p><p>Bitcoin is a type of blockchain application. A single-use type. As an alternative virtual payment.</p><p>The analogy of obtaining such money:</p><p>In the physical world, you need to proof-of-work to get money, e.g., spending your effort or knowledge to finish some tasks to earn money as pay. (Proof of work). In Bitcoin applying, the proof of work is somewhat the consuming computing resources/powers. (Mining).</p><p>The bottom line is that Bitcoin is an adoption / application of blockchain (Single-Purpose type), already phenomenal. Bitcoin is backed with blockchain.</p><p>Blockchain would be fun to explore the possibilities of real impact to bring in.</p><p>‘Plumbing’ services:</p><p>hardware, software, services that connect to use the network, and built on top of the network.</p><p>(Before, is the internet. Next, guess the blockchain.)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2bb3be61e928" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[DeepMind AlphaGo Documentary review.]]></title>
            <link>https://weimingchenzero.medium.com/deepmind-alphago-documentary-review-67d52cd566d0?source=rss-7d453923e23------2</link>
            <guid isPermaLink="false">https://medium.com/p/67d52cd566d0</guid>
            <category><![CDATA[deepmind]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[articifical-intelligence]]></category>
            <dc:creator><![CDATA[Weiming Chen]]></dc:creator>
            <pubDate>Thu, 21 Oct 2021 15:53:20 GMT</pubDate>
            <atom:updated>2021-10-22T06:01:01.879Z</atom:updated>
            <content:encoded><![CDATA[<p>Today, I watched the 2-hour DeepMind AlphaGo documentary. Astonishing.</p><p>The editor Cindy Lee did fantastic work on the editing and drama-style storytelling. I first heard this match-up news years ago. At that time, the news didn’t surprise me much. I (like many others) thought computer programs beating humans wasn’t fresh news. But after today’s watching, I have very complex feelings mixed on the story behind it. It was such a historical and milestone moment in computing / AI history.</p><p>First of all, I heart-fully respect the humbleness of Lee Sedol and the DeepMind AlphaGo team. I also admire the stillness of Lee. After three straight losses, which means already lost the series. (3 of 5). I cannot imagine the mental state shift and pressure on Lee. But he wasn’t going to give up. The 4th one, he came up the aka ‘God Move’ — the 78th move. AlphaGo console log showed that particular move was a 0.007% probability, in other words, the AlphaGo algorithm predicted one of a ten thousand (1 of 10000) likelihood its opponent would make that move, or put a human was extremely unlikely to take that move. However, Lee found and decided to take that option. It turned out to be the turning point of the 4th game. It confused AlphaGo. AlphaGo lost that 4th game.</p><p>In a high abstract view level of AlphaGo algorithm, it functions based on a kind of Minimax algorithm and layers of layers neural network deep learning of supervised machine learning. It later improvises its own creative and unique strategies that thousands of years of Go players could never think of before or may even consider as ‘silly.’ In other words of machine learning, AlphaGo discovered new patterns of winning rules by supervised learning on past practices, clustered new strategies that Go masters never could imaging before, it was beyond humankind capabilities and wisdom.</p><p>AlphaGo also gave me a lesson (I mean, not about the Go game). As human beings, we are always told to do this or that by what and how, accomplish this or that, and follow such a roadmap or that path. As a result, people inevitably fall into conventional. We should be able to be dragged out of the convention, discover, experiment, improvise, and be adaptive. But to be in the flow zone, you surely need to invest dedicated effort in training.</p><blockquote>“Be formless, be shapeless.”</blockquote><p>Later, Lee Sedol said that although no other people can beat him, there is an entity that he cannot win. Notice that he used the word ‘Entity’ rather than cold words like Machine, Computer, AI, or Program. It encompasses the message that he acknowledges Artificial Intelligence as a real thing, an actual entity rather than just another fancy software. It truly impacts humanity domain, shows his respect for digitalization.</p><p>Humanity (perhaps more precisely, Human Intelligence) drives us as human beings to where we are today so far in the world. Without a doubt, Artificial Intelligence will take our world to the next stage, accomplish things beyond human capabilities and capacities. Let’s take analogies as the Internet. Three decades ago, the Internet was rare, unaffordable, ungeneralized. Nowadays, it is so general, ubiquitous, like water and air. So think about years forward since this decade, Artificial Intelligence / Machine Learning to be well generalized, ubiquitously injected in our world paradigms, it to be invisible but everywhere in people’s domains. Everywhere.</p><p>AGI (Artificial General Intelligence) — the following fundamentals.</p><p>The “Entity”.</p><p>(My review of this documentary: 10/10.</p><p>Touching. Shocking. Astonishing. Educational. Smooth Tempo. Thought Provoking. Drama Style. Real.</p><p>Excellent machine learning documentary in the market.</p><p>Give credit to DeepMind and AlphaGo team, who made such possibilities. (Also, don’t forget Google / Alphabet giant as back yard proving unlimited computing resources and computing powers for DeepMind to conduct supervised training)</p><p>Of course, massive respect to Lee Sedol, who pushes the human limit.</p><p>At this moment, remind me of Bruce Lee’s quote:</p><blockquote>“There are no limits, only plateaus. ”</blockquote><p>AI can break human plateaus, drive our world to go beyond the next stage. It can be promising or toxic. What will likely be? You never know.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=67d52cd566d0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Solving Common Probability Problems with Python Pt.2 — Continuous Data]]></title>
            <link>https://weimingchenzero.medium.com/solving-common-probability-problems-with-python-pt-2-continuous-data-cdef36b98830?source=rss-7d453923e23------2</link>
            <guid isPermaLink="false">https://medium.com/p/cdef36b98830</guid>
            <category><![CDATA[probability]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Weiming Chen]]></dc:creator>
            <pubDate>Sun, 17 Oct 2021 09:40:06 GMT</pubDate>
            <atom:updated>2021-10-17T14:40:53.644Z</atom:updated>
            <content:encoded><![CDATA[<h3>Solving Common Probability Problems with Python Pt.2 — Continuous Data</h3><p><strong>Continuous Data</strong> measures scale units, e.g., height, weight, speed, time. The value can go ‘continuously,’ unlimited within any two numbers.</p><p>For example: 1.0000001 meters, 65.98574123213kg, 100.5kg, 100.999333kg. For every two variables, we can have infinite possible values in between. It is almost impossible to count each exact value accurately, but we can arrange them into intervals for better representation.</p><p>(In contrast to <strong>Discrete Data</strong>, examples of counting results of a coin toss, counting sale. We assure always have a discrete value, number of ‘head/tail,’ number of items sold, etc. )</p><p>The <strong>Continuous Probability Distribution</strong> ideally describes the probability that a value falls in a particular range. <strong>Continuous Probability Distribution</strong> can be <strong>Normal Distribution</strong> (<strong>Gaussian distribution</strong>) — the bell shape curve.</p><p><strong>Probability Density Function</strong> (PDF) and <strong>Cumulative Distribution Function</strong> (CDF) can be both worked to continuous data.</p><p>Probability Density Function (PDF) calculates a probability for an exact data point. However, if the dataset is large enough, the product density function value will approach very little towards zero. (Such as asking, “what is the probability of 58.573 kg weight in all women in an area?”) A single value of continuous data’s probability would not tell you much.</p><p>For continuous data, a more meaningful way to work on is the Cumulative Distribution Function (CDF) to determine the probability within an interval. (Change above question to check for weights such as 58–59KG range or 55–60kg range.)</p><p>For example, heights of 1.90 m — 2.10 m interval falls in 70% interval of NBA Players Heights distribution.</p><p>Subtract a CDF to get the gap.</p><pre>norm_dist.cdf(210) - norm_dist.cdf(190)<br><br># 0.71463211317374</pre><p>For a small dataset that has minimal data points, the pdf() method is OK.</p><p>It is more meaningful for a large dataset with countless continuous data to use cdf() to find out the probability between two values.</p><h4>Example</h4><p>Below is an example working with NBA Players’ heights.</p><h4>Question to answer:</h4><h4>What is the likelihood that an NBA player’s height falls within 2.10–2.20 m?</h4><pre><strong>import </strong>numpy <strong>as </strong>np<br><strong>import </strong>pandas <strong>as </strong>pd<br><strong>from </strong>scipy.stats <strong>import </strong>norm<br><strong>import </strong>matplotlib.pyplot <strong>as </strong>plt<br><br>df = pd.read_csv(<strong>&#39;files/Players.csv&#39;</strong>)<br><br><br>df[<strong>&#39;height&#39;</strong>].describe()<br><br><br><strong>&quot;&quot;&quot;<br>count    3921.000000<br>mean      198.704922<br>std         9.269761<br>min       160.000000<br>25%       190.000000<br>50%       198.000000<br>75%       206.000000<br>max       231.000000<br>Name: height, dtype: float64<br>&quot;&quot;&quot;<br><br><br></strong><em># Get a Normal Distribution instance for our height data.<br></em>dist = norm(df[<strong>&#39;height&#39;</strong>].mean(), df[<strong>&#39;height&#39;</strong>].std())<br><br><em># Calculate the interval probability<br></em>p = dist.cdf(220) - dist.cdf(210)<br><br><em># 0.10071770329577223</em></pre><p>Quickly check the distribution shape.</p><pre><em># x-axis: all height data points<br># y-axis: probability for each individual data point. (calculated by pdf())<br><br></em>plt.bar(df[<strong>&#39;height&#39;</strong>], df[<strong>&#39;height&#39;</strong>].apply(<strong>lambda </strong>x: dist.pdf(x)))<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*M-G4_4gDEko2EQEUeRvyOg.png" /></figure><p>Note that the area under the curve is always equal to 1, representing all data points’ probabilities.</p><p>The sliced part is the interval we want to find out. In this case, 10%. (<em>Apologies for my terrible hand drawing.</em>)</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*bbzodv23EOesBUDA5hix1g.png" /></figure><p>The likelihood that a NBA player between 2.10–2.20 m is 10%.</p><h4>Conclusion</h4><p>You can use the Product Density Function and Cumulative Distribution Function to work around with continuous data.</p><p>Continuous Probability Distributions can have the type of Normal Distribution (aka Gaussian Distribution) with a central tendency and bell curve style fit. We can use scipy.stats.norm to have a distribution instance by passing in mean and standard deviation and finding the likelihood within an interval.</p><p>For more references:</p><p><a href="https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.norm.html">scipy.stats.norm - SciPy v1.11.2 Manual</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cdef36b98830" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Inside Airbnb Boston 2021: Building a Machine Learning Price Predictor model for Boston Listings]]></title>
            <link>https://weimingchenzero.medium.com/inside-airbnb-boston-2021-building-a-machine-learning-price-predictor-model-for-boston-listings-d6bcbe13b0f9?source=rss-7d453923e23------2</link>
            <guid isPermaLink="false">https://medium.com/p/d6bcbe13b0f9</guid>
            <category><![CDATA[airbnb]]></category>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[boston]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Weiming Chen]]></dc:creator>
            <pubDate>Sat, 16 Oct 2021 11:02:36 GMT</pubDate>
            <atom:updated>2021-10-16T16:55:25.280Z</atom:updated>
            <content:encoded><![CDATA[<p>Inside Airbnb Boston 2021: Building a Machine Learning Price Predictor model for Boston Listings</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*V5LCwRMQ9Ws0OIBxKNZLCQ.jpeg" /><figcaption>Building in Boston. Taken @ 2019.12.12</figcaption></figure><h4>What is the article about?</h4><p>In this post, I will conduct a data mining effort by following CRISP-DM standards on the dataset of Airbnb Boston Listing 2021. Build a listing price estimator model with Machine Learning. Also, answer some business questions related to this dataset.</p><h4>Why should the reader learn this topic?</h4><ul><li>For people that are interested in how to build a price estimator or predictor model.</li><li>For Airbnb users (host, end-user, traveler) to better understand how to price their room, compare and select rooms within Boston, even negotiate with hosts.</li><li>For agency that wants to start Boston-based Airbnb business.</li><li>For real estate property people that want further discover patterns in the Boston property domain.</li></ul><h4>What are the prerequisites?</h4><p>This post is intended for a non-technical overview.</p><p>To keep it simple, I abstract away the technical parts from this post.</p><p>I host this project code in GitHub, you can checkout full Jupyter notebook here:</p><p><a href="https://github.com/gilzero/airbnb_boston_2021">GitHub - gilzero/airbnb_boston_2021</a></p><p>Data can be found here:</p><p><a href="http://insideairbnb.com/get-the-data.html">Inside Airbnb. Adding data to the debate.</a></p><p>Let’s get it stated.</p><h4>CRISP-DM</h4><p>To better organize the structure of this post, we follow this cycle.</p><ol><li>Business understanding</li><li>Data Understanding</li><li>Data Preparation</li><li>Modelling</li><li>Evaluation</li><li>Deployment</li></ol><h4>Business Understanding</h4><p>Let’s define our Questions for understanding of below:</p><p>1) What are the most popular property types in Boston listings 2021?</p><p>2) What are the most popular / in-demand neighborhood in Boston listings 2021?</p><p>3) What are the standard amenities host are offerings in Boston listings 2021?</p><p>4) How is the Distribution of Price Range in Boston listings 2021?</p><p>5) Boston Pricing Model. What are some top factors that contribute to price?</p><p>For example, suppose you were a property investor, by understanding these questions, you might be able to develop a strategy to offer rooms in high-demand neighborhoods with popular property types, make sure those essential amenities are provided. Set a reasonable price checked by the model. Assure a better Airbnb rental income rate.</p><h4>Data Understanding</h4><p>Covid 19 changes the tourism, rental industries dramatically. Things changed a lot and are more dynamic. So I tried to use the latest data available. At the time of writing, the dataset used in from Airbnb’s official September scraped batch. 2021-Sep</p><p>I wouldn’t combine past historical Boston data, especially pre-covid19.</p><h4>Data Preparation</h4><p>The dataset has 3123 instance of listings, with 74 variables.</p><p>There are some missing values in variables like</p><p>‘neighbourhood_group_cleansed’, ‘bathrooms’, ‘calendar_updated’, ‘license’, ‘neighborhood_overview’, ‘neighbourhood’, ‘host_about’, ‘review_scores_value’, ‘beds’, ‘bedrooms’…</p><p>Note that Airbnb does not encode bathroom numbers numerically, so the entire bathroom column is all empty value. Instead, Airbnb stores as string text in another column ‘bathrooms_text’ as a categorical variable, storing values such as: ‘1 bath’, ‘1.5 shared bath’, ‘half private bath’</p><p>I drop those columns that contain a significantly amount of missing/empty values. e.g(&gt; 75% missing)</p><p>For categorical columns, I fill the missing with the most common ones in that columns.</p><p>For some numeric columns, I fill the missing with median value rather than mean. (For example, we can’t have 1.37 beds, better to use the median in this case as an integer number) It would be more meaningful to fill with 0 for missing value as an initializer with a missing review rating score. It is probably not so fair to assign to ‘4.xx’ average rating for missing.</p><p>So for filling missing/empty data, always case by case depending on the attribute of that column, make sure what that variable represents.</p><p>Develop the fillna strategy for different variables would help to improve the model later we build on.</p><p>The dependent variable ‘price’ is stored as a U.S locale money format. e.g. ‘$1,234.00’. Before sending to computation, we need to transform to pure numeric format. ‘$1,234.00’ =&gt; 1234.00</p><p>There are quite a several categorical variables. One strategy is to use One-Hot Encoding. We generate new dummy columns for each variable’s value and encode with value 1 or 0 to mark the value.</p><ol><li>Subset numerics dataset.</li><li>Subset categorical dataset and flatten it to a one-hot set.</li><li>Re-combine numerics set and categorical (encoded) set.</li></ol><p>Flow:</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*fBcSkx9WqbAz7hSVpC_g9w.png" /></figure><p>(Image made with OmniGraffle)</p><p>Now we have a ready-to-roll cleaned dataset to feed to Machine Learning.</p><h4>Modelling</h4><p>Now we come to the exciting part: Modeling, Mining.</p><p>Before we start building the model, I quickly answer the above questions.</p><h4>Popular neighborhoods for Boston Listings 2021</h4><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*XIZXpGVlRwVCY6dhrgHT8w.png" /></figure><ol><li>Dorchester 418</li><li>Downtown 293</li><li>Roxbury 263</li><li>South End 235</li><li>Brighton 232</li><li>Jamaica Plain 215</li><li>Back Bay 195</li><li>East Boston 180</li><li>Allston 158</li><li>Beacon Hill 137</li><li>South Boston 135</li><li>Fenway 110</li><li>North End 80</li><li>Charlestown 70</li><li>Roslindale 56</li><li>Mission Hill 55</li><li>Bay Village 50</li><li>South Boston Waterfront 47</li><li>Chinatown 43</li><li>West End 38</li></ol><p>(“Location, Location, Location”)</p><p>— -</p><h4>Popular (high demand) property types for Boston Listings 2021</h4><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*VJD1rGStaI3SHU-rgqmZKg.png" /></figure><ol><li>Entire rental unit 1283</li><li>Private room in rental unit 526</li><li>Private room in residential home 378</li><li>Entire condominium (condo) 264</li><li>Entire serviced apartment 184</li><li>Entire residential home 126</li><li>Private room in condominium (condo) 71</li><li>Entire guest suite 47</li><li>Private room in townhouse 38</li><li>Room in boutique hotel 36</li></ol><p>(Entire unit and private rooms of all kinds are top demand, seems like entire unit are pretty popular in Boston while other cities favor rooms more.)</p><p>— -</p><h4>Common Essential Amenities</h4><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*13HbgWvXlKy3LE49xkGzMw.png" /></figure><ol><li>Smoke alarm 3017</li><li>Wifi 2996</li><li>Long term stays allowed 2960</li><li>Carbon monoxide alarm 2813</li><li>Heating 2806</li><li>Essentials 2804</li><li>Kitchen 2795</li><li>Hangers 2728</li><li>Hair dryer 2581</li><li>Air conditioning 2553</li><li>Iron 2505</li><li>Shampoo 2387</li><li>Hot water 2291</li><li>Microwave 2172</li><li>Refrigerator 2151</li><li>Washer 2097</li><li>Dedicated workspace 2096</li><li>Dryer 2048</li><li>Coffee maker 1991</li><li>Dishes and silverware 1916</li></ol><p>(Those security-related are on the top: Smoke alarm and Carbon monoxide alarm. Of course, safety first, as always. WIFI, Heating, Long-Term Stays, Heating, follows by Air Con, etc. I once visited Boston 2 years ago. I remember how freezing it was in the winter! Anyway, if you will be an Airbnb host in Boston, double-check your standard room stuff in the list might be beneficial. )</p><p>—</p><h4>Distribution of Boston Airbnb Listing Price 2021</h4><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*PpCpYT4PgSclFA5OSNegWA.png" /></figure><ul><li>Right skewed shape.</li><li>$60-$80 is the most offering range,</li><li>Overall, $80 — $160 are popular,</li><li>$180–220 is also quite common. Since Boston is such a higher standard city.</li><li>Listing offerings drop significantly after &gt; $220.</li></ul><p>—</p><p>Now, we enter the dragon: <strong>Machine Learning</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7SOQctqoFwlcKRepXfzh7Q.jpeg" /><figcaption>Lobster in a Boston restaurant (Union Oyster House). Taken @ 2019.12.12</figcaption></figure><p>Drop the instances that price &gt; $800 since they were rare in Boston Listings. Consider as an outlier, exceptional cases. Otherwise, the model will be confused with regularization.</p><blockquote>“Special cases aren’t special enough to break the rules.” (Zen of Python)</blockquote><p>We then need to split data into</p><ol><li>Independent Variables (All the variables that would impact Dependent Variable)</li><li>Dependent Variable (In this case, the ‘price’)</li></ol><p>For independent variables, we merge the numeric subset and categorical subset (One-Hot encoded).</p><p>For feature selections, I select only parts of variable in the numeric subset, e.g.:</p><p>‘accommodates’, ‘bedrooms’, ‘beds’, ‘number_of_reviews’, ‘review_scores_rating’. For example, some variables are related to REVIEW, so I picked an overall one for the review rating.</p><p>For feature selection of categorical variables, I selected ‘host_is_superhost’, ‘host_identity_verified’, ‘neighbourhood_cleansed’, ‘property_type’, ‘room_type’, ‘bathrooms_text’ (encoded versions)</p><p>(There are almost columns after all categorical variables flatten. If we use all of them for training, it might run into overfitting. So to better regularization, I only selected the above encoded categorical variables)</p><p>Choose a Linear Regression algorithm for this case. The imperative fitting is done by <a href="https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html">sklearn.linear<em>model LinearRegression</em></a><em>.</em></p><p>(For details of project code and libraries used, you can check my code of this project in GitHub)</p><h4>Evaluation</h4><p>After computation, we got a model fitted.</p><ul><li>R-squared for train data score: 0.62</li><li>R-squared for test data score: 0.60</li></ul><p>The model explains ~60% of the data variances. It could be a bare bone working model in real life.</p><p>However, for more accountability on predictability (aiming for R-squared &gt; 0.7), I might need to train on more Boston past data and experiment with different features selection combinations. (The model used 3000 instances/rows after cleaning and hot-one encoded categorical variables) I might aim for 6000 workings (cleaned) samples to train.</p><p>Below is the leading coefficients output from the model.</p><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*wr21Lv0y40JuKOTER6WqDA.png" /></figure><p>This table shows some factors to consider when estimating Boston Airbnb prices. Most from the top are bathroom numbers, neighborhoods, property types. Interestingly enough, I found bathroom numbers occurrences more than others in front positions. So it might reveal that people value bathroom quality, bathroom independence so much when considering an Airbnb option in Boston.</p><p>Anyway, the coefficients serves as a reference, remember ‘<a href="https://en.wikipedia.org/wiki/Correlation_does_not_imply_causation">Correlation does not imply causation</a>’.</p><h4>Deployment</h4><p>Boom! We have the first version of the Boston Airbnb Price Predictor model ready.</p><p>We can continuously improve the model by feeding more accurate data and experimenting with more feature selection and cleaning strategies, cycling the loop of CRISP-DM to enhance your business.</p><p>You can apply the same approach to other cities you favor.</p><p>e.g., Airbnb of New York, Airbnb of Tokyo, Airbnb of Shanghai, compare the nuances.</p><p>We haven’t deployed the model to the cloud yet. However, the model was tested and run locally. All technical parts and complete code I served in my GitHub.</p><p>For deployment like this one, I would recommend putting it on a Serverless FaaS for computing. FaaS is my recent favorite for deployment. Amazon Lambda could be a choice.</p><h4>Conclusion</h4><p>Thanks for reading.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bi4b8yOuwUXvA7zN-_EbTg.jpeg" /><figcaption>Street of Boston. Taken @ 2019.12.12</figcaption></figure><p>We have walked through a CRISP-DM cycle from understanding a business problem to running a real-life model by mining a real dataset.</p><p>I am certainly not satisfied with 0.6 R-squared. It would be an excellent opportunity to improve this model when I get more data and try more experiments.</p><p>If you have any questions, feel free to contact me.</p><p>Happy mining.</p><p>Alan Weiming Chen / Oct 16th 2021</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d6bcbe13b0f9" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>