<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Harshit Saklecha on Medium]]></title>
        <description><![CDATA[Stories by Harshit Saklecha on Medium]]></description>
        <link>https://medium.com/@harshitsaklecha?source=rss-833f4ae891d9------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*OMhnVlp7QCsY3V58.</url>
            <title>Stories by Harshit Saklecha on Medium</title>
            <link>https://medium.com/@harshitsaklecha?source=rss-833f4ae891d9------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 05:23:13 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@harshitsaklecha/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to: Configure and ec2 instance to run spark job on emr cluster.md]]></title>
            <link>https://harshitsaklecha.medium.com/how-to-configure-and-ec2-instance-to-run-spark-job-on-emr-cluster-md-18a27d564331?source=rss-833f4ae891d9------2</link>
            <guid isPermaLink="false">https://medium.com/p/18a27d564331</guid>
            <category><![CDATA[spark]]></category>
            <category><![CDATA[aws-ec2]]></category>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[aws-glue]]></category>
            <category><![CDATA[aws-elastic-beanstalk]]></category>
            <dc:creator><![CDATA[Harshit Saklecha]]></dc:creator>
            <pubDate>Thu, 28 Jan 2021 07:53:39 GMT</pubDate>
            <atom:updated>2021-02-02T10:15:30.729Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to: Configure an ec2 instance to run spark job on emr cluster</h3><p>This setup is for submitting Apache Spark jobs to the Amazon EMR cluster from a remote machine, such as the ec2 instance from the Elastic beanstalk instance. This might be useful for the services running in ebs which would also interact with the spark engine for computation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/349/1*Y4QZDRBH6_cuhzBdxXqjhQ.jpeg" /></figure><h3>Short Description</h3><p><strong>To submit Spark jobs to an EMR cluster from a remote machine, the following must be true:</strong></p><ol><li>Network traffic is allowed from the remote machine to all cluster nodes.</li><li>All Spark and Hadoop binaries are installed on the remote machine.</li><li>The configuration files on the remote machine point to the EMR cluster.</li></ol><p><strong>Resolution</strong><br>Confirm that network traffic is allowed from the remote machine to all cluster nodes</p><ul><li>If you are using an EC2 instance as a remote machine or edge node: Allow inbound traffic from that instance’s security group to the security groups for each cluster node.</li><li>If you are using your own machine: Allow inbound traffic from your machine’s IP address to the security groups for each cluster node.</li></ul><p>The above config will help you to submit the spark job in cluster mode to emr, but if you are looking to query the tables and hold data along in the remote machine make sure you have this remote machine too in the same security group as emr.</p><h3>Install the Spark and other dependent binaries on the remote machine</h3><p>To install the binaries, copy the files from the EMR cluster’s master node, as explained in the following steps. This is the easiest way to be sure that the same version is installed on both the EMR cluster and the remote machine.</p><ol><li>Run the following commands to create the folder structure on the remote machine:</li></ol><pre>sudo mkdir -p /var/aws/emr/ <br>sudo mkdir -p /etc/hadoop/conf <br>sudo mkdir -p /etc/spark/conf <br>sudo mkdir -p /var/log/spark/user/ <br>sudo chmod 777 -R /var/log/spark/</pre><p>2. Copy the following files from the EMR cluster’s master node to the remote machine. Don’t change the folder structure or file names.</p><pre>/etc/yum.repos.d/emr-apps.repo <br>/var/aws/emr/repoPublicKey.txt</pre><p>3. Run the following commands to install the Spark and Hadoop binaries:</p><pre>sudo yum install -y hadoop-client <br>sudo yum install -y hadoop-hdfs <br>sudo yum install -y spark-core <br>sudo yum install -y java-1.8.0-openjdk</pre><p>4. If you want to use the AWS Glue Data Catalog with Spark, run the following command on the remote machine to install the AWS Glue libraries:</p><pre>sudo yum install -y libgssglue</pre><p>if you are using Amazon Linux 2, the above lib is not available, you might also need to install</p><pre>sudo yum install aws-hm-client.noarch</pre><p>Additionally, we will have to configure an AWS account that will have the role <strong>AWSGlueServiceRole</strong> to access aws glue, you can configure this aws user via <strong>AWS configure</strong> command</p><p>5. Emr libs for accessing the tables data in s3:</p><pre>sudo yum install -y emrfs <br>sudo yum install -y emr-*</pre><p>6. Create working directories, create (mkdir -p) and allow write to (chmod 1777) the various scratch directories used by the EMR configuration (this list may change a bit with EMR versions)</p><pre>/mnt/s3 # fs.s3.buffer.dir<br>/mnt1/s3 # fs.s3.buffer.dir <br>/mnt/var/lib/hadoop/tmp # hadoop.tmp.dir <br>/mnt/tmp # java.io.tmpdir <br>/var/log/hive/user # hive.log.dir</pre><p><strong>Create the configuration files and point them to the EMR cluster</strong></p><p><strong>Note:</strong> You can also tools such as <strong>Rsync</strong> to copy the configuration files from the EMR master node to the remote instance.</p><ol><li>Run the following commands on the EMR cluster’s master node to copy the configuration files to Amazon Simple Storage Service (Amazon S3). Replace <strong>yours3bucket</strong> with the name of the bucket that you want to use.</li></ol><pre>aws s3 cp /etc/spark/conf s3://yours3bucket/emrhadoop-conf/sparkconf/ --recursive </pre><pre>aws s3 cp /etc/hadoop/conf s3://yours3bucket/emrhadoop-conf/hadoopconf/ --recursive</pre><p>2. Download the configuration files from the S3 bucket to the remote machine by running the following commands on the core and task nodes. Replace <strong>yours3bucket</strong> with the name of the bucket that you used in the previous step.</p><pre>sudo aws s3 cp s3://yours3bucket/emrhadoop-conf/hadoopconf/ /etc/hadoop/conf/ --recursive </pre><pre>sudo aws s3 cp s3://yours3bucket/emrhadoop-conf/sparkconf/ /etc/spark/conf/ --recursive</pre><p>3. Create the HDFS home directory for the user who will submit the Spark job to the EMR cluster. In the following commands, replace <strong>spark user</strong> with the name of your user.</p><pre>hdfs dfs –mkdir /user/sparkuser </pre><pre>hdfs dfs -chown sparkuser:sparkuser /user/sparkuser</pre><p>The remote machine is now ready for accessing Spark.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18a27d564331" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>