<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Sejal Vaidya on Medium]]></title>
        <description><![CDATA[Stories by Sejal Vaidya on Medium]]></description>
        <link>https://medium.com/@sejalv?source=rss-3402a1d6b9bc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*13DIAbOWD1zoKZxloxeq_Q.jpeg</url>
            <title>Stories by Sejal Vaidya on Medium</title>
            <link>https://medium.com/@sejalv?source=rss-3402a1d6b9bc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 10:01:01 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sejalv/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Containerized ML deployment with AWS Lambda]]></title>
            <link>https://sejalv.medium.com/containerized-ml-deployment-with-aws-lambda-680540fb92f4?source=rss-3402a1d6b9bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/680540fb92f4</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[ecr]]></category>
            <category><![CDATA[docker]]></category>
            <dc:creator><![CDATA[Sejal Vaidya]]></dc:creator>
            <pubDate>Sun, 03 Jan 2021 17:17:06 GMT</pubDate>
            <atom:updated>2021-01-03T17:17:06.946Z</atom:updated>
            <content:encoded><![CDATA[<h4>This post will demonstrate how you can deploy a Machine Learning model on a Serverless API (AWS Lambda), using ECR with Docker as runtime.</h4><p>This is a companion article to the online workshop I conducted for <a href="https://datatalks.club/"><strong>DataTalks.Club</strong></a><strong>.</strong></p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F79B8AOKkpho%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D79B8AOKkpho&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F79B8AOKkpho%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/c148abbce88401385f2622da939d60a5/href">https://medium.com/media/c148abbce88401385f2622da939d60a5/href</a></iframe><p>The repository for the workshop can be found here: <a href="https://github.com/sejalv/serverless-ml-workshop">https://github.com/sejalv/serverless-ml-workshop</a></p><p>This article was originally published on <a href="https://datatalks.club/blog/ml-deployment-lambda.html">DataTalks.Club</a>.</p><h3>Background</h3><p>Deploying machine learning models into production can sometimes be a stumbling block for Data Science teams. A common mode of deployment is to find somewhere to host your models and expose them via APIs. In practice, this can make it easy for your end users to integrate your model outputs directly into their applications and business processes.</p><p>Moreover, Serverless Computing has gone long way towards abstracting away the complexity of deploying applications into production environments by largely doing away with the need to manually configure servers, load balancers, containers etc. Provision of such auto-scaling and fault-tolerant functionalities, is what can act as a leverage for Data Scientists for simple model deployments, without any operational overhead.</p><h3>New features in AWS</h3><p>As of 01.12.2020, re:Invent announced that AWS Lambda can now support up to <a href="https://aws.amazon.com/about-aws/whats-new/2020/12/aws-lambda-supports-10gb-memory-6-vcpu-cores-lambda-functions/"><strong>10 GB of memory and 6 vCPU cores</strong></a>. This means:</p><p><a href="https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/"><strong>Container tooling for workflows</strong></a><strong>:</strong></p><ul><li>You can now package and deploy Lambda functions as container images (Docker/OCI) of up to 10 GB in size and push those images to AWS ECR.</li></ul><p><a href="https://docs.aws.amazon.com/lambda/latest/dg/runtimes-images.html"><strong>Images</strong></a><strong>:</strong></p><ul><li>Provision of base images for all the supported Lambda runtimes (Python, Node.js, Java, .NET, Go, Ruby) to easily add your code and dependencies</li><li>Extendible base images for custom runtimes based on Amazon Linux to include runtime implementing the Lambda Runtime API</li><li>Deployment of your own arbitrary base images to Lambda, for example- images based on Alpine or Debian Linux.</li></ul><p><strong>Computation:</strong></p><ul><li>Proportional to memory, increased CPU power to 6vCPUs</li><li>Easily build and deploy compute-intensive workloads that rely on sizeable dependencies, such as use-cases in machine learning or high-performance computing to perform faster</li></ul><p>Many more updates related to AWS Lambda and other resources are available on the AWS blog or on <strong>re:Invent</strong> pages.</p><h3>Benefits of new updates in AWS Lambda</h3><ul><li>We can build larger and faster runtimes with 10GB memory (than the previous 250 MB limit), to serve heavier processes on AWS Lambda, such as ML models using libraries such as pandas, scikit-learn, and PyTorch.</li><li>We can now build and deploy the AWS Lambda function with a docker container as runtime using Amazon ECR, instead of the custom packaging with .zip files, and also overcome the limitations of Lambda Layers. All the libraries and the weights of the model can now be packaged and published to a docker registry or an ECR registry.</li><li>Using container images with version tags will give us some leverage on model versioning</li></ul><h3>Workflow</h3><p>The goal is to produce a simple build-train-deploy workflow that works in any CI/CD tool. In addition, we want to:</p><ul><li>Use a docker container, to build/serve anywhere containers can run</li><li>Expose a REST API so that others can consume it</li><li>Decouple the training and serving, to minimise the complexity and attack surface of the operational container</li></ul><p>The result is a Docker-based workflow that should look familiar to any software engineer.</p><p>Ref: <a href="https://winderresearch.com/a-simple-docker-based-workflow-for-deploying-a-machine-learning-model/">https://winderresearch.com/a-simple-docker-based-workflow-for-deploying-a-machine-learning-model/</a></p><p>To adapt this in a serverless context (along with the new updates from AWS), we will:</p><ul><li>Train and serialize a model inside a Docker container</li><li>Package the service using Docker and AWS ECR</li><li>Serve the model with serverless API (AWS Lambda)</li><li>Build, Test, and Deploy, with a CI/CD workflow using GitHub Actions</li></ul><p>In addition, we can also use the SAM (Serverless Application Model) framework, in order to use an infrastructure-as-code (IaC) style to set up and configure resources on AWS. This makes it more maintainable, reproducible, and allows version control over IaC.</p><h3>Setup</h3><p>You can look up the prerequisites and configuration needed to continue with the workshop <a href="https://github.com/sejalv/serverless-ml-workshop">here</a>.</p><p>We will use the <strong>SAM</strong> framework to develop our serverless application. SAM, or Serverless Application Model, is an open-source framework (provided by AWS) for provisioning AWS services via code, and resources such as functions, triggers, and APIs.</p><p><strong>Advantages of SAM</strong>:</p><ul><li>acts as a wrapper to define the project’s AWS CloudFormation stack</li><li>extends the AWS CLI to functionality for building and testing Lambda applications.</li><li>now uses Docker to run functions in an Amazon Linux environment that matches Lambda</li><li>emulates application’s build environment and API</li><li>IaC benefits like source control, parameterized deployment</li></ul><p>Once you have the prereqs installed, you can generate a new project using the SAM framework:</p><pre>$ sam init</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/728/0*eTeoMwW5T9lTqCXH.png" /></figure><p>The project structure generated as a result:</p><pre>|-- service                # name of the service/project, eg. sam-app<br>     |-- app.py            # lambda handler<br>     |-- Dockerfile        # Image<br>     |-- requirements.txt  # Dependencies<br>|-- tests<br>     |-- unit<br>          |--test_handler.py<br>|-- template.yaml          # SAM template configuration</pre><p>template.yaml : SAM pre-configures all the necessary info in this file for you to have a vanilla functioning app:</p><ul><li>PackageType: Image/Zip, filled during the sam init step</li><li>Metadata [Dockerfile, DockerContext, DockerTag]: filled during the sam init step</li></ul><h3>Containerization</h3><blockquote><em>There are many different ways that you can leverage Docker to deploy your models. My approach here involves training and serializing a model during the image build process. Specifically, we will </em>embed<em> the trained model into the Docker image. Then, whenever we want to run inference or anything, we simply need to run a container from that image, deserialize the model, and generate our predictions.</em></blockquote><blockquote><em>This architecture provides </em>simplicity<em>: since the only moving pieces are Docker and the model training code. It allows us to leverage Docker’s image tagging system to store and version control our models. It also allows us to use a container registry service, like Amazon’s </em><a href="https://aws.amazon.com/ecr/"><em>Elastic Container Registry</em></a><em>, to store and manage our models. Rather than worry about persisting individual model components, we can just store entire Docker images which contain all of the necessary model artifacts!</em></blockquote><p>Ref: <a href="https://mlinproduction.com/docker-for-ml-part-1/">MLinProduction’s Docker for Machine Learning series</a> by Luigi Patruno</p><p>As explained here, <strong>our deployment pipeline will be directly integrating the serialized model into the API.</strong> We choose this approach to leverage the large container memory provided to us, and because the scale of the model and our application is pretty small for this workshop.</p><p><strong>NOTE</strong>: However, in most production workflows, it is advisable to keep the training and deployment pipelines decoupled. More info in Conclusion.</p><h3>Training</h3><p>As the focus of this workshop is on deployment, we won’t concern ourselves with training an accurate model. But you can easily replace this with your own more complex model, or convert your Jupyter Notebook to create a train.py file.</p><p>The contents of this training script can be as simple as:</p><ol><li>Load and parse data</li><li>Clean and prepare data</li><li>Train model</li><li>Save model parameters results</li></ol><p><a href="https://github.com/sejalv/serverless-ml-workshop/blob/dev/service/train.py">Our training script</a> will load a training data set, prepare the data into split into train &amp; test sets, train the model, generate evaluation metrics for the model, and serialize both the model and the evaluation metrics to a specific location.</p><p>The Dockerfile (located inside the service directory, if you have used SAM) should look something like:</p><pre><strong>FROM</strong> public.ecr.aws/lambda/python:3.8 as base</pre><pre><strong>FROM</strong> base AS train<br><strong>COPY</strong> requirements.txt .<br><strong>RUN </strong>pip install <strong>-r</strong> requirements.txt<br><strong>ENV</strong> MODEL_LOCAL_PATH=pickled_model.pkl<br><strong>COPY</strong> train.py .<br><strong>RUN </strong>python3 train.py</pre><p>Where base is the Lambda base image pulled from <a href="https://gallery.ecr.aws/lambda/python">AWS ECR Public</a>. You can either use this or one from <a href="https://hub.docker.com/r/amazon/aws-lambda-python">DockerHub</a>, or build your custom one.</p><p>MODEL_LOCAL_PATH is the output file of your serialized model.</p><p>To train your model, all you need to do is: docker build -t &lt;myimage&gt; . and wait for it to finish, then you have a container with trained parameters or results ready to serve.</p><h3>Serving</h3><p><a href="http://app.py/">app.py</a> : This is where our Lambda code lives.</p><p>In a nutshell, this imports libraries, pulls your model from the image, parses the input request, classifies it, and returns a response.</p><p>The serving container should:</p><ol><li>Load saved model parameters/results</li><li>Instantiate REST API</li><li>Define routes and serve model</li></ol><p>The same <a href="https://github.com/sejalv/serverless-ml-workshop/blob/dev/service/Dockerfile">Dockerfile</a> should now look something like:</p><pre><em># multi-stage build</em></pre><pre><strong>FROM</strong> public.ecr.aws/lambda/python:3.8 as base</pre><pre><strong>FROM</strong> base AS train<br><strong>COPY</strong> requirements.txt .<br><strong>RUN </strong>pip install <strong>-r</strong> requirements.txt<br><strong>ENV</strong> MODEL_LOCAL_PATH=pickled_model.pkl<br><strong>COPY</strong> train.py .<br><strong>RUN </strong>python3 train.py</pre><pre><strong>FROM</strong> base<br><strong>RUN </strong>pip install scikit-learn==0.22.1<br><strong>ENV</strong> MODEL_LOCAL_PATH=pickled_model.pkl<br><strong>COPY</strong> --from=train ./var/task/pickled_model.pkl pickled_model.pkl<br><strong>COPY</strong> app.py ./</pre><pre><em># Command can be overwritten by providing a different command in the template directly.</em><br><strong>CMD</strong> [&quot;app.lambda_handler&quot;]</pre><p>This is a <a href="https://docs.docker.com/develop/develop-images/multistage-build/"><strong>Multi-stage build</strong></a> which incorporates both our training and serving containers. The serving container copies the training artefacts from the training build and pastes them into a location that is expected by the app.py file.</p><p>Ref: <a href="https://winderresearch.com/a-simple-docker-based-workflow-for-deploying-a-machine-learning-model/">https://winderresearch.com/a-simple-docker-based-workflow-for-deploying-a-machine-learning-model/</a></p><h3>Build, Test, and Deploy</h3><p>Before we begin, please ensure to set the environment variables to your desired values. Or you can directly copy the commands from the GitHub repo.</p><pre>export AWS_REGION=eu-central-1<br>export AWS_ACCOUNT=PUT_VALUE_HERE<br>ECR_REPO=PUT_VALUE_HERE<br>DOCKER_IMAGE=serverless-ml</pre><p>The Build-Test-Deploy flow can be done in two ways:</p><ol><li><strong>Using Docker</strong></li></ol><p>To <strong>build</strong> the container image locally:</p><pre>$ docker build <strong>-t</strong> <strong>${</strong>DOCKER_IMAGE<strong>}</strong> ./&lt;path-to-Dockerfile&gt;</pre><p>This will package all the dependencies, kick off the training, copy the model params/results over to your serving container, and serve an API endpoint.</p><p>To check if this is working, you can start the container image locally using the Lambda Runtime Interface Emulator, and <strong>test</strong> the function invocation with cURL:</p><pre>$ docker run <strong>-p</strong> 8080:8080 <strong>${</strong>DOCKER_IMAGE<strong>}</strong></pre><pre>$ curl <strong>-XPOST</strong> &quot;http://localhost:8080/2015-03-31/functions/function/invocations&quot; \<br>    <strong>-d</strong> &#39;{&quot;body&quot;: {&quot;data&quot;: &quot;.10&quot;}}&#39;</pre><p>Output:</p><pre>{&quot;statusCode&quot;: 200, &quot;body&quot;: &quot;{\&quot;prediction\&quot;: \&quot;1\&quot;}&quot;}</pre><p>Or run unit-tests using pytest on your local machine:</p><pre>$ docker [run|exec] <strong>${</strong>DOCKER_IMAGE<strong>}</strong> python <strong>-m</strong> pytest tests/ <strong>-v</strong></pre><p>Finally, to <strong>deploy</strong>, authenticate Docker with ECR, and push the image to the container registry with the latest tag:</p><pre>$ aws ecr get-login-password | \<br>    docker login <strong>--username</strong> AWS \<br>        <strong>--password-stdin</strong> <strong>${</strong>AWS_ACCOUNT<strong>}</strong>.dkr.ecr.<strong>${</strong>AWS_REGION<strong>}</strong>.amazonaws.com</pre><pre>$ docker push <strong>${</strong>AWS_ACCOUNT<strong>}</strong>.dkr.ecr.<strong>${</strong>AWS_REGION<strong>}</strong>.amazonaws.com/<strong>${</strong>ECR_REPO<strong>}</strong>:latest</pre><p><strong>2. Using SAM:</strong></p><p>To add an API endpoint to your Lambda Function, you can update the Events section in your template.yaml to:</p><pre><strong>Events</strong>:<br>    &lt;ServiceName&gt;:<br>        <strong>Type</strong>: Api<br>        <strong>Properties</strong>:<br>        <strong>Path</strong>: /classify  <em># or whatever endpoint name you like</em><br>        <strong>Method</strong>: post     <em># or get, based on your API</em></pre><p>Optionally, in the same file, you can also add to the ImageUri tag, the ECR repo to where your Docker Image would be pushed:</p><pre>&lt;project&gt;:<br>    <strong>Type</strong>: AWS::Serverless::Function<br>    <strong>Properties</strong>:<br>      <strong>FunctionName</strong>: &lt;service-name&gt;<br>      <strong>ImageUri</strong>: &lt;aws-account&gt;.dkr.ecr.&lt;aws-region&gt;.amazonaws.com/&lt;image-name&gt;:&lt;tag&gt;<br>      <strong>PackageType</strong>: Image</pre><p><strong>To build:</strong></p><pre>$ sam build</pre><p>The SAM CLI builds a docker image from the Dockerfile, and generates a <strong>CloudFormation stack</strong> in the .aws-sam/build directory to provision the AWS resources, based on the config defined in template.yaml file.</p><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/443/0*zSNnrTY46UEdnfWF.png" /></figure><p><strong>To test:</strong></p><pre>$ sam local start-api</pre><pre>$ curl <strong>-XPOST</strong> http://127.0.0.1:3000/classify \<br>    <strong>-H</strong> &#39;Content-Type: application/json&#39; \<br>    <strong>-d</strong> &#39;{&quot;data&quot;:&quot;.10&quot;}&#39;</pre><p>Output:</p><pre>{&quot;prediction&quot;: 1}</pre><p><strong>To deploy:</strong></p><pre>$ sam deploy <strong>--guided</strong></pre><p>The guided deployment will walk through all required parameters and will create a samconfig.toml for us, that caches the resource-related info.</p><p>Output:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2TeM6PwY7nXjQtlw.png" /></figure><h3>CI/CD</h3><p>The steps we executed manually above, can be integrated into a CI/CD pipeline for automatic build, test and push. For this, we’ll use GitHub Actions, but feel free to pick your choice of tool.</p><p><strong>Steps</strong></p><ol><li>Store your AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in <a href="https://github.com/sejalv/serverless-ml-workshop/settings/secrets/actions"><strong>GitHub Secrets</strong></a> of your repository.</li><li>Create the .github/workflows directory in the root folder of your project.</li><li>Now we add two workflows:</li></ol><ul><li><a href="https://github.com/sejalv/serverless-ml-workshop/blob/dev/.github/workflows/python-app.yml">tests.yml</a>: which will install Python dependencies, run tests and lints. This is based on the <a href="https://github.com/actions/starter-workflows/blob/e9e00b017736d3b3811cedf1ee2e8ceb3c48e3dd/ci/python-app.yml">Python Application template provided by GitHub Actions</a></li><li><a href="https://github.com/sejalv/serverless-ml-workshop/blob/dev/.github/workflows/deploy-sam.yml">deploy.yml</a>: which will build the image and deploy it to Cloud (ECR). This is a custom workflow built on Ubuntu environment and using bash to execute the same steps from the section above.</li></ul><h3>Cleanup</h3><p>To delete the sample application that you created, use the AWS CLI:</p><pre>$ aws cloudformation delete-stack <strong>--stack-name</strong> &lt;stack-name&gt;</pre><h3>Conclusion</h3><p>As seen here, deploying ML models as serverless functions is probably the fastest, simplest route to getting a stable, scalable ML API deployed. And the new AWS features on container integration provide for resilient deployments.</p><p>However, it should also be noted that serverless is not always a good fit, especially for large ML workloads in production.</p><p><strong>When to use serverless/Lambda in DS &amp; ML?</strong></p><ul><li>Ideal for building prototypes or new products that use lightweight libraries</li><li>When your model is small. A large model is going to result in slow response times.</li><li>Data vectors are small. Lambda comes with a 10GB memory ceiling. So if your input data is massive, it’s just going to crash.</li><li>Your model will be sporadically accessed. Lambda scales out (not up) to infinite and charges you per request. So it’s well suited to intermittent “on-the-fly” demand.</li><li>Low-cost is important. Lambda charges $0.20 per million requests.</li></ul><h3>Future Enhancements and Tradeoffs</h3><p><strong>Monitoring/tracing/alerting</strong></p><ul><li>If there are downstream services that need to be notified, we could add notifications to our Lambda function using SNS or CloudWatch too.</li></ul><p><strong>Workflow design alternatives</strong></p><ol><li>Use of storage for models</li></ol><ul><li>In terms of design, it makes sense to have decoupled deployment pipelines for training and serving, if the changes in models and the application code are not always in sync, and the deployment cycles need to vary. This is often the case in large production workloads.</li><li>In this case, the training script could run independently and push the serialized model to a storage like s3. And your service (Lambda) can then load the model from s3 into your application code.</li></ul><p>2. Use of Workers:</p><ul><li>Ref: <a href="https://medium.com/swlh/how-to-deploy-your-scikit-learn-model-to-aws-44aabb0efcb4">https://medium.com/swlh/how-to-deploy-your-scikit-learn-model-to-aws-44aabb0efcb4</a></li><li><a href="https://www.ritchievink.com/blog/2018/09/16/deploy-any-machine-learning-model-serverless-in-aws/">https://www.ritchievink.com/blog/2018/09/16/deploy-any-machine-learning-model-serverless-in-aws/</a></li></ul><p><strong>Sagemaker vs. Lambda</strong></p><ul><li>The pay-per-request model of Lambdas makes it great for AB testing and prototyping. But even when Lambda functions come with a 10GB memory ceiling, they are short-running in their processing capacity (they timeout after <em>max</em> 15 minutes). So if your model might be expensive to load or if the input data is very large, it’s better to opt for Sagemaker or Batch Jobs.</li><li>Ref: <a href="https://towardsdatascience.com/saving-95-on-infrastructure-costs-using-aws-lambda-for-scikit-learn-predictions-3ff260a6cd9d">https://towardsdatascience.com/saving-95-on-infrastructure-costs-using-aws-lambda-for-scikit-learn-predictions-3ff260a6cd9d</a></li></ul><p><strong>Fargate vs. Lambda</strong></p><ul><li>Lambda functions are stateless and short-lived. That means that a lot of container workloads in their current form may still suit the Fargate/ECS/EKS camp better. Fargate will remain useful for more traditional, longer-lived workloads that don’t have a need to scale quickly to 100’s or 1000’s of containers.</li><li>Container Image deployment to Lambda enables Lambda’s incredibly rapid and responsive scaling as well as Lambda’s integrations, error handling, destinations, DLQs, queueing, throttling and metrics.</li><li>Ref: <a href="https://dev.to/eoinsha/container-image-support-in-aws-lambda-deep-dive-2keh">https://dev.to/eoinsha/container-image-support-in-aws-lambda-deep-dive-2keh</a></li></ul><p><strong>AWS vs. Kubernetes</strong></p><ul><li>Deploying your container is entirely dependent on your tech stack. It could be as simple as a docker run -d -p 8080:80 myimage. Or you could use that container in a Kubernetes <a href="https://knative.dev/">KNative manifest</a> for full-on serverless machine learning deployments.</li></ul><p><strong>Cost &amp; Benchmarks</strong></p><p><a href="https://dev.to/eoinsha/container-image-support-in-aws-lambda-deep-dive-2keh">https://dev.to/eoinsha/container-image-support-in-aws-lambda-deep-dive-2keh</a></p><h3>Questions (from the workshop)</h3><p><em>How will this workflow be different from deploying a Deep learning model?</em></p><ul><li>This design can still be used for a Deep Learning model. However, if you have high computational needs for training, then you’ll need to think about <strong>where</strong> you are doing the training.</li><li><a href="https://datatalks.club/blog/ml-deployment-lambda.html#use-of-workers">Use Of Workers</a></li></ul><p><em>Is there any loading speed difference for large models from S3 vs them being part of the image in ECR?</em></p><ul><li>I’m unaware of the performance difference at the moment, but we can test it out.</li></ul><p><em>Why do we need SAM? What are the main benefits?</em></p><ul><li><a href="https://datatalks.club/blog/ml-deployment-lambda.html#advantages-of-sam">Advantages of SAM</a></li><li>Alternatives to SAM: CloudFormation/Terraform, serverless framework</li></ul><p><em>What are the benefits of using Docker compared to the old way of deploying lambda functions?</em></p><ul><li><a href="https://datatalks.club/blog/ml-deployment-lambda.html#benefits-new-aws">Benefits of new updates in AWS Lambda</a></li></ul><h3>Resources</h3><p><em>Workshops built on new AWS features:</em></p><ul><li><a href="https://aws.amazon.com/blogs/aws/new-for-aws-lambda-container-image-support/">AWS Lambda — Container Image tutorial</a></li><li><a href="https://github.com/alexeygrigorev/aws-lambda-docker">Deploying a DL Model on AWS</a> by <a href="https://datatalks.club/people/alexeygrigorev.html">Alexey Grigorev</a></li></ul><p><em>The Docker-based workflow (multi-stage build) is based on the following articles, and adapted to the new AWS features:</em></p><ul><li><a href="https://mlinproduction.com/docker-for-ml-part-1/">MLinProduction’s Docker for Machine Learning series</a> by <a href="https://mlinproduction.com/about/">Luigi Patruno</a></li><li><a href="https://winderresearch.com/a-simple-docker-based-workflow-for-deploying-a-machine-learning-model/">A Simple Docker-based Workflow For Deploying A Machine Learning Model</a> by <a href="https://winderresearch.com/team/phil-winder/">Phil Winder</a></li></ul><p><em>For more information on the use of Docker in Data Science &amp; ML, check out these excellent posts:</em></p><ul><li><a href="https://mlinproduction.com/docker-for-ml-part-1/">MLinProduction’s Docker for Machine Learning series</a> by <a href="https://mlinproduction.com/about/">Luigi Patruno</a></li><li><a href="https://towardsdatascience.com/how-docker-can-help-you-become-a-more-effective-data-scientist-7fc048ef91d5">Docker Can Help You Become A More Effective Data Scientist</a> by <a href="https://hamel.dev/">Hamel Husain</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=680540fb92f4" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[SQL in a Nutshell: Part 1 — Basic Real-World Scenarios]]></title>
            <link>https://medium.com/data-science/sql-in-a-nutshell-part-1-basic-real-world-scenarios-33a25ba8d220?source=rss-3402a1d6b9bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/33a25ba8d220</guid>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[analytics]]></category>
            <category><![CDATA[data-science]]></category>
            <dc:creator><![CDATA[Sejal Vaidya]]></dc:creator>
            <pubDate>Mon, 04 Dec 2017 15:06:57 GMT</pubDate>
            <atom:updated>2021-04-27T18:06:56.499Z</atom:updated>
            <content:encoded><![CDATA[<h4><em>This series focuses on the most frequent data science and analytical problems in the real-world, and aims at solving them with SQL.</em></h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*3KHwFLXghuHeG-sX" /><figcaption>Photo by <a href="https://unsplash.com/@kmuza?utm_source=medium&amp;utm_medium=referral">Carlos Muza</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p><strong>UPDATES (To the original post):</strong></p><ol><li>Links to queries on SQLFiddle have been removed. Future updates include links to Jupyter/IPython notebooks embedded with SQL.</li><li>Most of the queries here are written in ANSI SQL. For execution, PostgreSQL 9.6 has been included alongside MySQL 5.6, to demonstrate features such as Window Functions (which are not supported by MySQL). You may install either or both of them locally to test our queries.</li><li>The article has been expanded into a 3-part series, with the third part covering typical, non-trivial tasks in SQL.</li></ol><p>If you’re familiar with SQL’s syntax and with writing basic SELECT queries, you may find this series helpful to get a quick refresher or learn more tips and tricks to write cleaner and optimized queries to questions like <em>how to calculate a six-week rolling average of sales.</em></p><p>This is the <em>first post</em> of the series that will recap SQL concepts along with some basic problems, that will be answered with basic retrieval, joins, filters, aggregations, and transformations. <br>The <em>second post</em> will cover more complex scenarios and how to approach them with window functions, sub-queries, moving averages, common table expressions, and ranking.<br>The <em>third post </em>will cover typical, non-trivial tasks in SQL — such as pivoting, different ways to retrieve top results from each group, working with duplicates, and query tuning for performance.</p><h3>Introduction to SQL</h3><p>SQL (Structured Query Language) is a universal programming language designed for managing data in databases. Due to its standard syntax, it supports all relational databases as well as some <em>NoSQL </em>(Not-Only SQL) ones. SQL is popular because it is easy to understand with its usage designed to be like the English language, and because it can access the data directly where it’s stored instead of copying the data to other applications, which makes analytics relatively convenient.</p><p>With SQL, we can:</p><ul><li><em>Define </em>data (DDL), eg. normalize abbreviated values to full state names</li><li><em>Manipulate </em>data (DML), eg. replace a blank or null value with a word</li><li><em>Retrieve </em>data from databases (DQL), eg. get number of sales this month</li></ul><p>For the scope of this series, we will be dealing with queries only for retrieval (<em>SELECT</em>). We will also use ANSI SQL — a standard that’s compliant on most databases — of <a href="https://en.wikibooks.org/wiki/Structured_Query_Language">SQL:2011</a> version, for most of our queries. For demonstration, we’ll use MySQL 5.6 and/or PostgreSQL 9.6.</p><p><strong>NOTE</strong>: The latest standard (ISO/IEC or ANSI) released for SQL is <a href="https://en.wikipedia.org/wiki/SQL:2016">SQL:2016</a> that has additional support for JSON functionality.</p><h4>Create Database and Insert Data</h4><p>Before we begin to query our data, we will need to create a database and load data into it.</p><p>A <a href="http://en.wikipedia.org/wiki/Database"><em>database</em></a><em> </em>is an organized collection of data, and consists of structures such as tables among other elements. A <em>table </em>stores data in rows (tuples) and columns (attributes). A <em>schema </em>refers to a blueprint of how the database is constructed (or divided into database tables).</p><p>The following set of commands builds the schema for our database:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b7ca365672d0e5d8c5a0d536e2c6b32f/href">https://medium.com/media/b7ca365672d0e5d8c5a0d536e2c6b32f/href</a></iframe><p>Here, we create two tables — <em>Products</em>, <em>Sales — </em>that are related by <em>PRODUCT_ID </em>(Primary Key for <em>Products</em>, Foreign Key for <em>Sales</em>)</p><h3>Retrieve Data</h3><p>A <strong>SELECT</strong> command is used to retrieve data from the database.</p><p>Here’s a recap of some basic queries with different ways of getting <em>Sales </em>data:</p><ul><li><strong>All </strong>the details from a table</li><li><strong>LIMIT </strong>the results to first <em>N </em>rows (used for testing or sampling)</li><li>Specific columns from a table</li><li><strong>Aliases </strong>for column names (makes information human-readable)</li><li><strong>DISTINCT </strong>column values (unique values from one or more columns)</li></ul><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4fb81896c879b5927dab29cd62f64b38/href">https://medium.com/media/4fb81896c879b5927dab29cd62f64b38/href</a></iframe><h3>Join Tables</h3><p>Joining tables allows us to combine multiple datasets for a more complete set of results and comprehensive analysis.</p><p>In <a href="https://en.wikipedia.org/wiki/Relational_model">data modeling</a> (database design), <em>normalization </em>is a process used to organize data in separate well-structured tables to minimize redundancy and avoid insert, update and delete anomalies. The separation also tends to simplify the logic and enhance the flexibility of assembling precisely the items needed for a given purpose. This assembly is accomplished by means of the ‘JOIN’ operation.</p><p>Let’s say we had one single table <em>ProductSales</em> storing all our data:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/616/1*wbeqMTCq2DWim_CHcjMfdA.png" /><figcaption>Raw Data into ProductSales <em>table</em></figcaption></figure><p>And say, the business would like to see the products by the Product names, rather than only the Brand names. This means they could ask us to update each entry of ‘Nokia’ to ‘Nokia Lumia’, and ‘Samsung’ to ‘Samsung S5’. In this case, an operation for changing each and every row where <em>Product_Name </em>is ‘Nokia’ or ‘Samsung’, would be quite expensive computationally if there were, say, a million rows.</p><p>Instead, if we structure our data in such a way that, the <em>Products </em>data is taken out in a separate table, with a single row of information on each product then each sale in the <em>Sales </em>table can be related to a <em>Product_ID. </em>This is called a<em> one-to-many relationship. </em>Any change now to the <em>Product_Name </em>would be just a one-row operation, and all the <em>Sales </em>data related to the updated product could be easily accessed via a join on the <em>Product_ID</em>.</p><h4>Use-cases for Joins:</h4><ul><li><em>To combine related datasets</em>, for example — to find the name of a product that was sold, we would join the <em>Sales </em>table with the <em>Product </em>table</li><li><em>To perform comparisons</em>, for example — to perform year-over-year analysis, we could join partitions of similar data grouped over year values</li><li><em>To use tables as look-up resources</em>, for example — to get the status for a shipping delivery type, it would be more efficient to store enumerators for status codes to be mapped with their corresponding text.</li></ul><h4>Types of Joins</h4><ol><li>Inner Join: An <em>inner join</em> is a join that returns only those results from both tables where the <em>join condition</em> is True.</li></ol><p>For example — <em>Get the details of those Products that have made Sales:</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c483343cb897e6836d0b32a05785468b/href">https://medium.com/media/c483343cb897e6836d0b32a05785468b/href</a></iframe><p>In this example, it is observed from the result-set that there is no sale made by product ‘LG’, since there is no listing for product ‘LG’ (or no <em>Product_ID</em>)<em> </em>in the <em>Sales</em> table.</p><p>The <em>join condition</em> is specified via an <strong>ON</strong> clause. We can also add multiple conditions to it using the AND operator. For example, to <em>get the details of those Products that have made Sales in 2017:</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f2d4a3f9fb915fa456e5c52429194df8/href">https://medium.com/media/f2d4a3f9fb915fa456e5c52429194df8/href</a></iframe><p>2. Outer Join: An <em>outer join</em> is a join that returns all the results from one or both the tables (depending upon the <em>join type — left, right, or full</em>), and where the rows failing the <em>join condition </em>return NULL values.</p><p>To explain this better, we take our previous example, and modify the requirement slightly — <em>Get all the Products and their corresponding Sales information, if any:</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ebc7f969c59f82f59d4ac7a38e1a0232/href">https://medium.com/media/ebc7f969c59f82f59d4ac7a38e1a0232/href</a></iframe><p>Here we have every product listing along with each sale made. For ‘LG’, the result-set shows that the product exists in our database, but there is no sale made yet. This an example of a <strong>LEFT JOIN</strong>.</p><p>A RIGHT JOIN functions in a similar way, but in reverse, i.e. all the rows of the right table are returned, and the unmatched rows from left return as NULL values. It’s a standard practice to use a LEFT JOIN (over a RIGHT JOIN), wherever possible.</p><p><strong>NOTE: </strong>FULL JOIN<strong> </strong>is not supported in MySQL, but a similar query can be accomplished through UNION. The next update of this post will include topics on <em>Full Joins</em> as well as <em>Self Joins </em>as used in other databases like PostgreSQL. <em>Union </em>and <em>Union All </em>will be covered in the next post.</p><h3>Filter and Sort Data</h3><p>By filtering a dataset, we can restrict our results based on a search condition.</p><ul><li><em>To focus your analysis</em>, for example — narrow down results of a sales trend to a particular region</li><li><em>To limit the dataset for an aggregate condition, </em>for example — get products with an average rating of 4 or higher, whose total reviews are at least 10</li><li><em>To reduce the dataset size for computation</em>, for example — sampling a hypothesis</li></ul><p>Following are some operators that are commonly used in filtering:</p><p>The basic way to filter data is by using the WHERE clause along with one or more operators suited for your search condition.</p><p>Following are some operators that are commonly used:</p><ul><li><em>Comparison Operators</em>: = (Equal To) , &lt;&gt; or != (Not Equal To), and some other self-explanatory ones— &gt;, &lt;, &gt;=, &lt;=</li><li><em>Logical Operators: </em>AND, OR, NOT, IN (to look within a set of values), BETWEEN (with a start and end range), LIKE (to match similar values)</li></ul><h4>Sorting</h4><p>Sorting in SQL queries is optional. By default, the results are returned in the order of the Primary Key if it exists, or else in a random order. If we wish to sort the result in a particular manner, we must explicitly add an ORDER BY clause followed by a <em>list of column-names </em>to sort with, and finally the <em>order type (ASC — </em>Ascending<em>, DESC — </em>Descending<em>)</em>.</p><p>Let’s start with a simple example — <em>Get all Sales details for an ‘iPhone’:</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8d772eba5fc29b138a560f936252324b/href">https://medium.com/media/8d772eba5fc29b138a560f936252324b/href</a></iframe><p>Notice here, in the first query, we have filtered by Product_ID (if known). And in the second query, we have joined <em>Sales </em>with the <em>Products </em>table and filtered for a product that ends with “-Phone”. Note that the ‘%’ is a wildcard character which searches for any text in its place.</p><p>Let’s take another example — <em>Get Sales details between 1st November 2015 and 1st October 2017, sorted by Date:</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/19ed976801bc5bf284c70bd2e444adf8/href">https://medium.com/media/19ed976801bc5bf284c70bd2e444adf8/href</a></iframe><p>Here, in the first query, we use a set of comparison operators. In the second query, we use BETWEEN that searches for Date values within the same range.</p><p>Now, what if we needed to <em>get the Sales details for the products — ‘Nokia’, ‘Samsung’, and ‘LG’?</em> Or say it’s a list of about 20 products that we need to search for. Rather than including 20 AND conditions, a more feasible way to write this query would be to use the IN operator:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aecc9d82ce4ff30b1900d9de7bc514b8/href">https://medium.com/media/aecc9d82ce4ff30b1900d9de7bc514b8/href</a></iframe><h3>Aggregations</h3><p>Oftentimes we lose context while immersed in millions of details around us and wish to seek meaningful information from them. By aggregating data, we can observe insights like monthly totals, or averages across sales orders by category. This helps us gain a deeper understanding of our data, and allows us to ask deeper questions. At a high level, we can also identify patterns and trends in our data.</p><p>For this, we will use a collection of built-in functions in SQL, that summarize, or roll-up, our values. Some basic ones are min, max, average, sum and count, and the more analytical ones include standard deviation, variance, and rank.</p><p>Let’s start with using some basic functions, for example —</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3aab6544743d889ca601ed6c0ac467b7/href">https://medium.com/media/3aab6544743d889ca601ed6c0ac467b7/href</a></iframe><p>This is good for starters, but apart from telling us how many rows does the table contain or what are the minimum/maximum values for Price, it isn’t really much of a business insight. What if we wanted to see <em>how many Sales were made on a per-product basis</em>? Here comes <em>GROUP BY</em> to the rescue!</p><p><strong>GROUP BY</strong> allows you to separate data into groups, which can be aggregated independently of one another.</p><p>For example — <em>Get the total sales, average price and total quantity sold for each product in our database:</em></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c1154d5921a16d27f54488fb72fb77d3/href">https://medium.com/media/c1154d5921a16d27f54488fb72fb77d3/href</a></iframe><p>Now this gives a more comprehensive picture!</p><p><strong>NOTE</strong>: Using COALESCE, IFNULL (or equivalent) can replace the NULLs with specified values.</p><p>Let’s consider another example — <em>For every product sold from 1st October 2016 onward, get the total sales, average price and total quantity on a yearly basis, and return only those products that have sold at least 10 items, sorted by product name and year.</em></p><p>Now maybe that’s a lot to process, but here’s a break-down analysis to its solution:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d6d0757b1a2fae375a372814865d34fb/href">https://medium.com/media/d6d0757b1a2fae375a372814865d34fb/href</a></iframe><p><strong>NOTE</strong>: The HAVING clause, applies a filter after an aggregation is performed.</p><p>Aggregations are a core part of SQL Analysis and become necessary for any good data scientist/engineer to master. We’ll be learning about more such complex scenarios and how to approach them as we proceed with the series.</p><h3>Data Transformations with built-in functions</h3><p>Data Transformations are often required in situations where poorly collected data requires cleansing; such processes being called Data Wrangling (or munging). Transformations are also needed to present data reports in a more human-readable format.</p><p>Following are some examples for the quarterly and monthly Sales report, using built-in functions:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bed25eb01fc4e5c6a2a3879d810d31f0/href">https://medium.com/media/bed25eb01fc4e5c6a2a3879d810d31f0/href</a></iframe><p>The first query displays a quarterly Sales report. Here, the MONTH and YEAR functions transforms a Date value into its corresponding month and year values in Integer form, while CAST is used on <em>Price </em>to transform from <em>Integer </em>to <em>Decimal. </em>The CASE function (a variation of IF-ELSE) checks then checks the condition and maps each Date value into quarterly buckets, and thereafter the results are grouped by the quarter. Similarly, in order to display a monthly Sales report, we need to group over both <em>Year </em>and <em>Month</em>, as shown in the second query.</p><p>There are a lot more handy functions in SQL, which you can find in the documentation of the SQL version you’re using. More of those will be covered in the next posts of the series.</p><h3>References</h3><p>While you’re waiting for the next post, check out some interesting links on SQL for further knowledge:</p><ul><li><a href="https://en.wikibooks.org/wiki/Structured_Query_Language">Structured Query Language - Wikibooks, open books for an open world</a></li><li><a href="https://community.modeanalytics.com/sql/tutorial/introduction-to-sql/">The SQL Tutorial for Data Analysis</a></li><li><a href="https://bensullins.com/sql-tips-tricks-data-science/">SQL Tips and Tricks for Data Science - Ben Sullins | Data Geek</a></li></ul><p><em>Hope you found this post helpful! If you liked this article, do clap for it. Any thoughts? Put them in the comments below.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=33a25ba8d220" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/sql-in-a-nutshell-part-1-basic-real-world-scenarios-33a25ba8d220">SQL in a Nutshell: Part 1 — Basic Real-World Scenarios</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Private Messaging in the era of Open Data]]></title>
            <link>https://sejalv.medium.com/messaging-with-encryption-f498719ba99a?source=rss-3402a1d6b9bc------2</link>
            <guid isPermaLink="false">https://medium.com/p/f498719ba99a</guid>
            <category><![CDATA[social-media]]></category>
            <category><![CDATA[privacy]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[messaging]]></category>
            <dc:creator><![CDATA[Sejal Vaidya]]></dc:creator>
            <pubDate>Wed, 12 Oct 2016 12:25:46 GMT</pubDate>
            <atom:updated>2017-06-17T19:23:18.661Z</atom:updated>
            <content:encoded><![CDATA[<h4>Stumbling across “Secret Conversations” and critically eyeballing Encryption standards and User Data Surveillance methods</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Bc7E-3Kc7HxMcvRxSDPF5A.jpeg" /><figcaption><a href="https://www.caleupe.com/wp-content/uploads/2015/04/o-ONLINE-PRIVACY-facebook-1080x675.jpg">https://www.caleupe.com/wp-content/uploads/2015/04/o-ONLINE-PRIVACY-facebook-1080x675.jpg</a></figcaption></figure><p>The current age technology has taken such a turn, that you may just address your concerns over a chat with a dear pal and the chat itself may have solutions to provide already.</p><p>Sound scary?<br>It definitely was for me! When this happened-</p><p>A few weeks ago, I was discussing with a friend over a phone call, about an idea he was working on, for his FinTech startup. It was just a few minutes after the call ended, that I came across an article on FB NewsFeed, describing a product very similar to the one he spoke about.</p><p>I happened to share this link with that friend over Facebook Messenger, expressing a mild surprise over the coincidence of the appearance of that link on my NewsFeed, and proceeded to add a little joke to that conversation (doubtful if anyone was listening!):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HTx5_G0FVISJlRJcnsxJRA.jpeg" /><figcaption>Original copyright — not for use</figcaption></figure><p>What blew me away, was the occurrence of this FB notification, a few hours later:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jFqFbQCKHNFAwI9jvbtHMA.png" /></figure><p>This wasn’t my first ‘1984’ moment on Social Media, but my ears did keep ringing with “Big Brother is watching you!”</p><h3>So what are Secret Conversations, and how do you enable them?</h3><p>“Secret Conversations” is Facebook’s answer to the world of “End-to-End Encryption” -enabled messaging apps, and was rolled out to users in phases, beginning July this year.</p><p>The <a href="http://newsroom.fb.com/news/2016/07/messenger-starts-testing-end-to-end-encryption-with-secret-conversations/">Facebook newsroom announcement</a> reads, “We’ve heard from you that there are times when you want additional safeguards — perhaps when discussing private information like an illness or a health issue with trusted friends and family, or sending financial information to an accountant.”</p><p>Facebook’s secret conversations work only from a single device per user, and will allow you to set a timer. The feature also doesn’t support “rich content like GIFs and videos, making payments, or other popular Messenger features”, and cannot be applied to group messages.</p><p>To enable this feature, go to FB Messenger, switch to your profile tab and scroll through the list to find “Secret Conversations”, and tap on the option. To have a secret conversation, follow the steps as provided <a href="https://www.facebook.com/help/messenger-app/1084673321594605/?helpref=hc_fnav">here</a>.</p><h3>What’s the deal with End-to-End Encryption?</h3><p>End-to-End Encryption (E2EE) is a method of secure communication, where the key to the encryption is known only to the sender and the recipient, and your message can neither be decrypted by the company, nor can third parties like the government, law-enforcement agencies, Internet service providers, cyber-criminals or hackers, read it or tamper with it.</p><p>Many messaging apps, (including Telegram, Threema, Line and more), have by now deployed E2EE, and most of them use “The Signal Protocol”, designed by Open Whisper Systems.</p><p>Open Whisper Systems’ own messaging app “Signal” offers text, group chat, and calls on Android, all with end-to-end encryption, and is also <a href="https://twitter.com/Snowden/status/661313394906161152">recommended by Edward Snowden</a> — whistleblower, former NSA contractor, and one of the world’s best-known privacy experts.</p><p>While the 2013 NSA leaks by Edward Snowden may have turned the spotlight to privacy, it was the court battle between Apple and the FBI, that sparked the encryption war, in relation to unlocking an iPhone used by one of the San Bernardino shooters in 2015. Apple, which has used a form of end-to-end encryption in iMessage for years, <a href="http://www.apple.com/customer-letter/">refused to install a backdoor</a>, underlining the integral values many large communications companies hold when it comes to personal data, security and encryption.</p><p>Facebook-owned WhatsApp, thereafter, rolled out full end-to-end encryption to its billion-plus users in April, just under two months from the San Bernardino case, followed by Viber adding protection to its 700 million users’ messages just weeks after WhatsApp.</p><p>Yet, not all these apps have the same approach.</p><p>While WhatsApp and iMessage, automatically encrypt every message (default encryption), FB Messenger and Google’s Allo have an opt-in feature that users need to manually turn on, to chat in a separate window secretly or in “incognito” mode.</p><p>Also, messages in FB Messenger, Snapchat and Allo, have a feature to enable a self-destruct timer. But even when these messages disappear from your phone, they are still stored on the company servers, encrypted. Snapchat stores these messages for 30 days, while Allo stores it indefinitely. In case of Signal, however, no communications are sent to their servers, making it impossible for anyone to trace the call or text back to you.</p><h3>Why do all Messaging Apps not offer default encryption?</h3><p>Before answering that, let’s see what happens to the messages written in non-E2EE mode. For non-secret conversations, Facebook Messenger has been using a partial-path encryption method. In this case, the message that you send is encrypted and transmitted over a secure network to Facebook servers where it is decrypted and stored in plain text, along with its encryption keys. From here, it is re-encrypted and sent to your recipient, where it is decrypted again.</p><p>Such is the case for many other messaging apps like Allo, that provide the opt-in feature and are not completely end-to-end-encrypted. This also means the messages here, will be perfectly accessible for spying with lawful requests or for usage in generating targeted ads, similar to message data in Gmail and Hangouts, and location data collected by Android.</p><p>A few months back, in spite of being fully end-to-end encrypted, <a href="https://blog.whatsapp.com/10000627/Looking-ahead-for-WhatsApp">WhatsApp announced</a> its intent to share data with its parent company Facebook in order to draw in adverts to the platform. Third party companies will be able to send targeted messages directly to WhatsApp users should they accept the new terms and conditions.</p><p>In 2014, Facebook had announced that its users, “want to see ads that are more relevant to their interests.” <a href="https://www.facebook.com/full_data_use_policy">This is being done</a> by tracking their usage of websites and apps connected with FB accounts or through FB activity including messaging.</p><p>There’s no way to turn off this feature entirely unless you live in the US, Canada or Europe where you can register with the corresponding <a href="http://www.aboutads.info/choices/">Digital Advertising Alliance</a> to opt out of your device or browser.</p><p>Recently, there was a <a href="http://www.theverge.com/2016/9/21/12994362/allo-privacy-message-logs-google">controversy around Google</a> — “When Allo was announced at Google’s I/O conference earlier this year, the messaging app was presented as a step forward for privacy. Alongside the end-to-end-encrypted Incognito Mode, the Allo team talked about bold new message retention practices, storing messages only transiently rather than indefinitely. But with the release of the app today, Google is backing off on some of those features.”</p><p>Edward Snowden, criticized the app on Twitter saying:</p><h3>Edward Snowden on Twitter</h3><p>Google&#39;s decision to disable end-to-end encryption by default in its new #Allo chat app is dangerous, and makes it unsafe. Avoid it for now.</p><p>Google’s reason for the shift in Allo, was to improve the virtual assistant features. By not encrypting chats end-to-end, Google would be able to run its artificial intelligence on conversations to better suggest replies, relevant Web search results, info on any planned or dynamic events, (and witty jokes), just as it does with its other products.</p><h3>Privacy vs. AI vs. Surveillance</h3><p>A bitter truth known is- “<em>If You’re Not Paying For It, You Become The Product</em>”. Practically every site gathers data in some form, for analytics, for business or to help improve the user experience. A large amount of data is used in targeted advertising for monetization in businesses. Data is also used in tailoring each internet experience as per users’ preferences, pick up on trends and improve services at certain locations. All the major networks use cookies, search queries and our personal profiles to determine what it is that we like and dislike, and then build our future browsing experiences around such details. Breaking news, warnings, alerts and notifications are fed to interested users, depending on their tastes and geographical location.</p><p>But this does add a potential point of vulnerability to the design, no matter how secure the company servers may be. A concern that comes to mind is how governments or law agencies could potentially use this information. For example, could they use Facebook to find the people opposed to their views and even manipulate their moods?</p><p>For a responsible individual, encrypting only parts of messages like sharing account details or sensitive/embarrassing content only makes it more prone for hackers and spies to know where to find such info. Encrypting all your communications not only protects yourself but others as well. It is vital for targeted or marginalized user groups, including journalists, activists, and survivors of domestic abuse. If they are the only people who encrypt, then their communications stick out like sore end-to-end encrypted thumbs. But if everyone encrypts as their automatic default, potential surveillance targets are harder to spot.</p><p>Some law enforcement officials have warned that the spread of end-to-end encryption could help criminals and terrorists “go dark” because companies cannot access the content of the encrypted communications even when faced with a warrant.</p><p>Civil liberties advocates and technology experts, on the other hand, have generally praised the expansion of end-to-end encryption in consumer devices as a step forward for users’ privacy and cyber-security.</p><p>With the rapid growth in AI, Big Data, and IOT, how much are you willing to trade your privacy in the hope of better benefits?</p><p><em>Thoughts? Put them in the comments below. If you liked this article, click the 💚 below.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f498719ba99a" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>