<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Benjamin R. Luginbuhl on Medium]]></title>
        <description><![CDATA[Stories by Benjamin R. Luginbuhl on Medium]]></description>
        <link>https://medium.com/@ben-luginbuhl?source=rss-8844619c2a37------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*zjRCVYLT5lZ58-flR9lA4Q.png</url>
            <title>Stories by Benjamin R. Luginbuhl on Medium</title>
            <link>https://medium.com/@ben-luginbuhl?source=rss-8844619c2a37------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 14:36:45 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ben-luginbuhl/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Writing a Python web scraper for academic papers with RapidAPI]]></title>
            <link>https://levelup.gitconnected.com/writing-a-python-reference-web-scraper-with-rapidapi-5e810e0e1d88?source=rss-8844619c2a37------2</link>
            <guid isPermaLink="false">https://medium.com/p/5e810e0e1d88</guid>
            <category><![CDATA[web-scraping]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[automation]]></category>
            <category><![CDATA[academia]]></category>
            <dc:creator><![CDATA[Benjamin R. Luginbuhl]]></dc:creator>
            <pubDate>Mon, 22 Feb 2021 01:36:02 GMT</pubDate>
            <atom:updated>2021-02-23T17:55:55.794Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Jumbotron — Python and RapidAPI web scraping" src="https://cdn-images-1.medium.com/max/450/1*ykQEd_yU9JOWpopT-8AU5w.png" /></figure><p>A common problem that crops up when working in academia is having a large list of references that you need to gather some metadata for (authors, journal, publication date, DOI, <em>etc</em>.). I recently encountered this problem when creating a database for my research, and I needed to get the URL links for about 300 separate references. I had a Word document that contained most of the information, including the primary author and the article title, but no easy way to get the links for them. Rather than going through each reference, copying the title, entering it in Google search, clicking the appropriate link, and then saving that URL, I decided to create a simple Python script to do this for me!</p><p>This tool uses Pandas and <a href="https://rapidapi.com/apigeek/api/google-search3">RapidAPI</a> to create a CSV file with links for all the references that are input to the script.</p><p>Here’s the story — a link to the Github repository for the code is included at the end!</p><h3>Environment Setup</h3><p>As with all coding projects, it’s good practice to work in a virtual environment. For my particular use case, I’ve been using the <a href="https://docs.microsoft.com/en-us/windows/wsl/install-win10">Windows Subsystem for Linux 2</a> (running Ubuntu 20.04), <a href="https://code.visualstudio.com/">Visual Studio Code</a>, and <a href="https://pipenv.pypa.io/en/latest/">Pipenv</a> (for Python package and environment management). You can read more about this setup <a href="https://medium.com/swlh/setting-up-a-secure-django-project-repository-with-docker-and-django-environ-4af72ce037f0">here</a>.</p><p>I did most of the prototyping using a Jupyter notebook, but the code is simple enough to be contained in a simple python file.</p><h4>Creating the Virtual Environment</h4><p>In a terminal window (I use <a href="https://www.microsoft.com/en-us/p/windows-terminal/9n0dx20hk701?activetab=pivot:overviewtab">Windows Terminal</a>, which is much prettier than cmd.exe or Powershell, in my opinion), run the following to create and activate a new environment to work in:</p><pre>$ mkdir refscraper &amp;&amp; cd refscraper<br>$ pipenv --python 3.9.1</pre><p>I’m using a relatively new version of Python for this project, but this project should work with older versions as well, just be aware of syntax differences for Python 2 <em>vs</em>. 3.</p><p>Next, we’ll install the only third-party package we will need, <a href="https://pandas.pydata.org/">Pandas</a>.</p><pre>(refscraper) $ pipenv install pandas</pre><p>Now we can create the other files that we will need:</p><pre>(refscraper) $ touch ref_scraper.py refs.txt</pre><p>This will create empty files that we will fill in a moment. Our directory should now look like this:</p><pre>/refscraper<br>  |--Pipfile<br>  |--Pipfile.lock<br>  |--ref_scraper.py<br>  |--refs.txt</pre><h4>The refs.txt file</h4><p>The format of our data file (we’ve named it refs.txt ) will be important for how we write our python code. For this project, I was given a Word document that had the references listed as shown in Figure 1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JjQvKX5J-MVovAsG3FtPEQ.png" /><figcaption>Figure 1. Example of reference document format</figcaption></figure><p>We could write a script to read these data directly from the document, since Word documents are essentially just bloated XML files. However, it was easy enough to copy the references straight into a plain text file, which is easier to parse using Python. After copying into refs.txt , the file looks like this:</p><pre>1 Sekiya, M. Sakaino, &amp; Toshiaki, S. T. Linear Logistic Regression<br>  for Estimation of Lower Limb Muscle Activations, IEEE Trans.<br>  Neural Syst. Rehabil Eng. 27, 523-532 (2019).<br>2 Malouf, R. A comparison of algorithms for maximum entropy<br>  parameter estimation, In proceedings of the 6th conference on<br>  Natural language learning, Association for Computational<br>  Linguistics, 20, 1-7 (2002).<br>3 Andrew, G. &amp; Gao, J. Scalable training of &lt;i&gt;L&lt;/i&gt;&lt;sup&gt;1&lt;/sup&gt;<br>  regularized log-linear models. In Proceedings of the 24th<br>  international conference on Machine learning, ACM: Corvalis,<br>  Oregon, USA, 33-40 (2007).<br>4 Elith, J. Leathwick, &amp; J. Hastie, R. T. A working guide to boosted<br>  regression trees. J. Anim. Ecol., 77 , 802-813 (2008).</pre><p>where the numbers in front of each reference are just the line numbers. Now, we can start working with Python to parse some links!</p><h3>The Python Script</h3><p>The first thing we need to do is read in our refs.txt file so that we can work with the data. For this, we will use Pandas DataFrames. Open the ref_scraper.py file in a text editor. To use VS Code, you can type code . in the directory with the file and it should open a new editor window.</p><pre># ref_scraper.py<br>import pandas as pd</pre><pre># declare a variable that points to the refs.txt file<br>REF_TEXT = &quot;./refs.txt&quot;</pre><pre># read the data into a DataFrame<br>df = pd.read_csv(<br>        REF_TEXT, <br>        delimiter=&#39;\n&#39;,<br>        header=None,<br>        names=[&#39;ref_full&#39;]<br>    )<br>df.index += 1</pre><p>This code has done two things: create a DataFrame with one column ref_full that contains the full text for each reference; shift the index to start from 1 instead of 0 (this is to make it easier to map the links to the actual reference numbers from the paper). Next, we need to parse the title from the full text of the reference so that we can use that to form our search queries later. Let’s look at an example from our refs.txt file to decide how to accomplish this.</p><pre>Sekiya, M. Sakaino, &amp; Toshiaki, S. T. Linear Logistic Regression for Estimation of Lower Limb Muscle Activations, IEEE Trans. Neural Syst. Rehabil Eng. 27, 523-532 (2019).</pre><p>It looks like we can split the full string on the . characters, which will give us a list. This won’t be perfect, but it will serve our purposes. Let’s fiddle around in a Python shell (in the terminal, run $ python ).</p><pre>&gt;&gt;&gt; import pandas as pd<br>&gt;&gt;&gt; df = pd.read_csv(<br>...    &#39;./refs.txt&#39;,<br>...    delimiter=&#39;\n&#39;,<br>...    header=None,<br>...    names=[&#39;ref_full&#39;])<br>&gt;&gt;&gt; df.index += 1<br>&gt;&gt;&gt; ref1 = df[&#39;ref_full&#39;][1]<br>&gt;&gt;&gt; print(ref1)<br>&#39;Sekiya, M. Sakaino, &amp; Toshiaki, S. T. Linear Logistic Reg...&#39;<br>&gt;&gt;&gt; ref1.split(&#39;.&#39;)<br>[&#39;Sekiya, M&#39;, &#39; Sakaino, &amp; Toshiaki, S&#39;, ..., ]</pre><p>I’ve truncated the string and the list so they fit on one line, but the important thing to note is that by calling .split(&#39;.&#39;) on the full string, one of the elements of the list that is returned contains the title (and possibly some other text, like the conference that the work was presented at). If we make the assumption that the element of the list that contains the title will be the longest one in the entire list, we can easily grab it and return it. We’ll also do a little bit more cleaning; back to the Python console!</p><pre>&gt;&gt;&gt; # removing quote characters<br>&gt;&gt;&gt; ref_1.replace(&#39;&quot;&#39;, &#39;&#39;).replace(&quot;&#39;&quot;, &#39;&#39;)<br>&gt;&gt;&gt; split_ref = ref1.split(&#39;.&#39;)<br>&gt;&gt;&gt; print(max(split_ref, key=len))<br>Linear Logistic Regression for Estimation of Lower Limb...</pre><p>Great! Now let’s combine our scratch work into a nice helper function in ref_scraper.py that we can apply to every row of our DataFrame.</p><pre># ref_scraper.py</pre><pre>def get_title_from_ref(ref):<br>    return(<br>        max(<br>            ref.split(&#39;.&#39;), key=len)<br>            .strip().replace(&#39;&quot;&#39;, &#39;&#39;).replace(&quot;&#39;&quot;, &#39;&#39;)<br>    )</pre><pre># create a new column and apply our new function<br>df[&#39;title&#39;] = df[&#39;ref_full&#39;].apply(lambda x: get_title_from_ref(x))</pre><p>Now we have a column of our DataFrame that hopefully contains only the title of the reference (again, plus some extras depending on the format, but it shouldn’t be a problem for our search query). Speaking of…</p><h4>Building the Search Query</h4><p>Before we can use our data to get things from the internet, we need to make some choices about the tools we will use to do this. For this project, I found that Python’s built-in requests library and a Google Search API from RapidAPI fit my needs (if anyone reading this has a better option, please let me know!).</p><p>To get started with this API, we need to <a href="https://rapidapi.com/signup">make a free account with RapidAPI</a>. Once you’ve made your account, navigate to the <a href="https://rapidapi.com/apigeek/api/google-search3">Google Search API from APIGeek</a>. In order to use this API, you need to subscribe to it by clicking the “Subscribe to Test” button. This will bring you to a page where you can choose which tier you want. I chose the free tier, which provides 600 free API calls per month. After choosing your subscription, you should be redirected back to the API page, where you should see a “Code Snippets” window on the right. Click the dropdown selector (which probably says “(Node.js) Unirest” currently), and select Python &gt; Requests. This will give us some hints as to how to structure our own code to use the API.</p><pre># Code snippet from RapidAPI.com<br>import requests</pre><pre>url=&quot;https://google-search3.p.rapidapi.com/api/v1/search/q=elon+musk&amp;num=100&quot;</pre><pre>headers={<br>  &#39;x-rapid-api-key&#39;: &quot;your-key-will-be-here&quot;,<br>  &#39;x-rapidapi-host&#39;: &quot;google-search3.p.rapidapi.com&quot;<br>}</pre><pre>response=requests.request(&quot;GET&quot;, url, headers=headers)</pre><pre>print(response.text)</pre><p>There are a few important things to note here:</p><ol><li>The url structure, which shows that our query string must have + signs instead of spaces.</li><li>The headers , which must include your personal x-rapid-api-key .</li><li>The response object that will be returned to us when making a GET request to the endpoint.</li></ol><p>Let’s go back to our code and create a function that will convert our title to a suitable query string and put the string into the appropriate url.</p><pre># ref_scraper.py</pre><pre>def create_request_url(https://medium.com/title):<br>    q_string = title.replace(&#39; &#39;, &#39;+&#39;)<br>    return f&quot;https://google-search3.p.rapidapi.com/api/v1/search/q={q_string}num=2&quot;</pre><p>This function will replace all the spaces in our title with + symbols, then insert that string into the url that we need to make the API call. I’ve also replaced the num value with 2 arbitrarily, because we don’t need 100 links for each reference.</p><p>While we’re here, we should also set a variable that contains our API key string:</p><pre># ref_scraper.py</pre><pre>RAPID_API_KEY = &quot;your-key-will-be-here&quot;</pre><p>Now let’s create a function that will take a title from our DataFrame and get the first link that is returned from the API endpoint:</p><pre># ref_scraper.py</pre><pre># import requests and json libraries<br>import requests<br>import json</pre><pre>def get_ref_link(i, title):<br>    <br>    # build the headers dictionary<br>    headers = {<br>        &#39;x-rapid-api-key&#39;: RAPID_API_KEY,<br>        &#39;x-rapid-api-host&#39;: &quot;google-search3.p.rapidapi.com&quot;<br>    }</pre><pre>    # get the query url for the given title<br>    query_url = create_request_url(https://medium.com/title)</pre><pre>    # create an empty string for the link<br>    link = &quot;&quot;</pre><pre>    # optional print statement to track progress<br>    print(f&quot;Getting link for ref[{i}]...&quot;)</pre><pre>    # get the response from the API<br>    try:<br>        r = requests.request(&quot;GET&quot;, query_url, headers=headers)<br>    except ConnectionError:<br>        pass</pre><pre>    # get the link from the response.text<br>    if r.status_code == 200:<br>        j = json.loads(r.text)<br>        try:<br>            link += j[&#39;results&#39;][0][&#39;link&#39;]<br>        except:<br>            link += &quot;no link found&quot;<br>    else:<br>        link += &quot;request failed&quot;<br>    <br>    # another optional print to track progress<br>    print(f&quot;Done. Link: [{link}]&quot;)</pre><pre>    return link</pre><p>This function will return the url of the first link that is found in the response from the API. It also has some basic error handling to deal with possible timeouts or cases where no link can be found. We’ve also included an argument i for the function; the reason for this will be clearer in the next step. The final step now is to use this function for every title in our DataFrame! Let’s add some functional code to our script:</p><pre># ref_scraper.py</pre><pre># import the time library so we can time the script<br>import time</pre><pre>start = time.time()</pre><pre># create an empty list for all the links<br>links = []</pre><pre># `iterrows()` returns the index `i` and the content `j` for each<br># row of the DataFrame<br>for i, j in df.iterrows():<br>    link = get_ref_link(i, j[&#39;title&#39;])<br>    links.append(link)</pre><pre>delta = time.time() - start</pre><p>If we run the script now, we will get a list links that contains all of the URLs that were found using the API. The final, <em>final </em>step is to add this list to our DataFrame and then save the data into a new file. We’ll also print the time that the script took to run, just for fun.</p><pre># ref_scraper.py</pre><pre>df[&#39;link&#39;] = links</pre><pre># set a variable for the output path of our new file<br>OUTPUT_FILE_PATH = &#39;./refs-links.csv&#39;</pre><pre>df.to_csv(OUTPUT_FILE_PATH)</pre><pre>print(f&quot;Finished in {delta:.2f seconds}&quot;)</pre><p>And we’re done! If we run the script in the terminal using $ python ref_scraper.py, we now have a .csv file in our directory with 3 columns: ref_full , title , and link. It’s now super easy to open this file in a spreadsheet app and copy the link column into whatever place you need.</p><p>Below is the full script:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4ced9f24921d7592bc1829418fc092cf/href">https://medium.com/media/4ced9f24921d7592bc1829418fc092cf/href</a></iframe><h3>Wrapping up</h3><p>The Github repository for this project can be found <a href="https://github.com/bluginbuhl/ref-scraper">here</a>. Please feel free to contribute!</p><h4>Future features</h4><p>Obviously this script has some limitations. Here’s a list of things that I’d like to add to this project in the future:</p><ol><li>Better handling of title parsing from reference list</li><li>Return other metadata, including DOIs</li><li>Add parsing for Word documents, Google Docs, and PDFs</li><li>Make use of an API that is faster and has more free uses</li><li>Alternatively, use Selenium and BeautifulSoup to scrape the web instead of an API</li></ol><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5e810e0e1d88" width="1" height="1" alt=""><hr><p><a href="https://levelup.gitconnected.com/writing-a-python-reference-web-scraper-with-rapidapi-5e810e0e1d88">Writing a Python web scraper for academic papers with RapidAPI</a> was originally published in <a href="https://levelup.gitconnected.com/">Level Up Coding</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Setting Up a Secure Django Project Repository With Docker and Django-Environ]]></title>
            <link>https://medium.com/swlh/setting-up-a-secure-django-project-repository-with-docker-and-django-environ-4af72ce037f0?source=rss-8844619c2a37------2</link>
            <guid isPermaLink="false">https://medium.com/p/4af72ce037f0</guid>
            <category><![CDATA[django]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[security]]></category>
            <category><![CDATA[docker-compose]]></category>
            <dc:creator><![CDATA[Benjamin R. Luginbuhl]]></dc:creator>
            <pubDate>Tue, 29 Sep 2020 23:50:48 GMT</pubDate>
            <atom:updated>2020-10-01T17:14:04.227Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fPEMj8rolS2OMEHD9sfC9w.png" /></figure><p>When following other tutorials for Django, I noticed that many don’t go into detail about how to make the project 12factor compliant, especially in regards to storing secrets like the SECRET_KEY and the database credentials. This tutorial will show how to start a new Django project and configure it to keep secrets safe. It will also use Docker to build containers for the python backend and a PostgreSQL database.</p><p>Code blocks show both typed commands (in bold) and some example output (not bold). If you’re following along, make sure to only type the parts listed in bold.</p><p>You can find the final repository for the project on my <a href="https://github.com/bluginbuhl/django-project-example">Github</a>.</p><h4>Initial setup</h4><p>Install Python and Docker Desktop. If you are a Windows 10 user, I highly recommend using the <a href="https://docs.microsoft.com/en-us/windows/wsl/install-win10">Windows Subsystem for Linux 2</a> (WSL2). <a href="https://docs.docker.com/docker-for-windows/wsl/">Docker plays very nicely with WSL2</a>, and if you also use <a href="https://code.visualstudio.com/">Visual Studio Code</a> as your IDE, there are some nice extensions that make development even easier.</p><p>VS Code Extensions:</p><ul><li><a href="https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.vscode-remote-extensionpack">Remote Development</a></li><li><a href="https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-wsl">Remote-WSL</a></li><li><a href="https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-containers">Remote-Containers</a></li><li><a href="https://marketplace.visualstudio.com/items?itemName=ms-azuretools.vscode-docker">Docker</a></li><li><a href="https://marketplace.visualstudio.com/items?itemName=ms-python.python">Python</a></li></ul><p>To confirm that Docker is working on WSL2, just type docker at the prompt. You should see some output listing possible docker commands. If you see</p><pre>$ <strong>docker</strong></pre><pre>The command &#39;docker&#39; could not be found in this WSL 2 distro.<br>We recommend to activate the WSL integration in Docker Desktop settings.</pre><pre>See https://docs.docker.com/docker-for-windows/wsl/ for details.</pre><p>then be sure to enable WSL2 integration in the Docker Desktop app and restart the terminal.</p><h3>Setting up the project environment</h3><p>Once you have the prerequisites installed, it’s time to setup our project environment.</p><p>First, we need to create an isolated environment for our project. I prefer to use <a href="https://pipenv.pypa.io/en/latest/">Pipenv</a> for python projects, as it makes both creating the environment and installing dependencies a breeze. Feel free to use virtualenv or venv if you are more familiar with those programs. To install Pipenv on WSL2, type:</p><pre>$ <strong>pip install pipenv</strong></pre><p>Then, create a folder for the project and initialize the environment:</p><pre>~ $ <strong>mkdir myproject &amp;&amp; cd myproject</strong><br>~/myproject $ <strong>pipenv shell</strong><br>Creating a virtualenv for this project…<br>Pipfile: /home/ben/myproject/Pipfile<br>Using /usr/bin/python3.8 (3.8.5) to create virtualenv…<br>Creating virtual environment...</pre><pre>✔ Successfully created virtual environment!<br>Virtualenv location: /home/ben/.local/share/virtualenv/myproject-2xqp8Bts<br>Creating a Pipfile for this project…<br>Launching subshell in virtual environment…</pre><p>The name of the environment, (myproject) should now be pre-pended to your prompt. Now install Django in the environment:</p><pre>(myproject) ~/myproject $ <strong>pipenv install django~=3.1.1</strong><br>Installing django~=3.1.1…<br>Adding django to Pipfile&#39;s [packages]…<br>✔ Installation Succeeded<br>Pipfile.lock not found, creating…<br>Locking [dev-packages] dependencies…<br>Locking [packages] dependencies…<br>Building requirements...<br>Resolving dependencies...<br>✔ Success!<br>Updated Pipfile.lock (7f443d)!<br>Installing dependencies from Pipfile.lock (7f443d)…<br>  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00</pre><p>We are just about ready to start the project, but I want to install two more packages before we begin. The first will be the <a href="https://pypi.org/project/psycopg2-binary/">psycopg2-binary</a> library, which is a database adapter for PostgreSQL and python. The second will be <a href="https://django-environ.readthedocs.io/en/latest/">django-environ</a> , which we will use to manage our project’s secrets (like the SECRET_KEY and the database credentials).</p><pre>(myproject) ~/myproject $ <strong>pipenv install psycopg2-binary~=2.8.6 django-environ~=0.4.5</strong><br>Installing psycopg2-binary~=2.8.6…<br>Adding psycopg2-binary to Pipfile&#39;s [packages]…<br>✔ Installation Succeeded<br>Installing django-environ~=0.4.5…<br>Adding django-environ to Pipfile&#39;s [packages]…<br>✔ Installation Succeeded<br>Pipfile.lock (7f443d) out of date, updating to (e551d6)…<br>Locking [dev-packages] dependencies…<br>Locking [packages] dependencies…<br>Building requirements...<br>Resolving dependencies...<br>✔ Success!<br>Updated Pipfile.lock (e551d6)!<br>Installing dependencies from Pipfile.lock (e551d6)…<br>  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 0/0 — 00:00:00</pre><p>Finally, we can start the Django project.</p><h4>Initial Django Project configuration</h4><p>First, run the django-admin startproject command:</p><pre>(myproject) ~/myproject $ <strong>django-admin startproject config .</strong></pre><p>Here, we are creating a Django project called config , which is where our main project settings will be stored. The directory should now look like this:</p><pre>myproject\<br>├── Pipfile<br>├── Pipfile.lock<br>├── config<br>│   ├── __init__.py<br>│   ├── asgi.py<br>│   ├── settings.py<br>│   ├── urls.py<br>│   └── wsgi.py<br>└── manage.py</pre><p>It is worth noting that as it stands, it is not really safe to commit this project to a git repository. Inside the settings.pyfile, the SECRET_KEY, DEBUG, and DATABASES settings are not secure. However, using django-environ, it’s simple to future-proof our project so that these settings can be easily kept secret and secure.</p><p>At this point, it also makes sense to open this directory in our text editor. If you’re using VS Code, just type:</p><pre>(myproject) ~/myproject $ <strong>code .</strong></pre><p>To start to secure the project, let’s create a .envfile within the config directory. This is where we will store our secrets as environment variables to be loaded using django-environ.</p><pre>(myproject) ~/myproject $ <strong>touch config/.env</strong></pre><p>Before we transfer our secrets to .env , let’s open settings.py and modify the crucial settings.</p><pre>&quot;&quot;&quot;<br>Django settings for config project.</pre><pre>Generated by &#39;django-admin startproject&#39; using Django 3.1.1.</pre><pre>For more information on this file, see<br><a href="https://docs.djangoproject.com/en/3.1/topics/settings/">https://docs.djangoproject.com/en/3.1/topics/settings/</a></pre><pre>For the full list of settings and their values, see<br><a href="https://docs.djangoproject.com/en/3.1/ref/settings/">https://docs.djangoproject.com/en/3.1/ref/settings/</a><br>&quot;&quot;&quot;</pre><pre>from pathlib import Path<br>from environ import Env               # new</pre><pre>env = Env()                           # new</pre><pre>env.read_env(env_file=&#39;config/.env&#39;)  # new</pre><pre><em>#</em> Build paths inside the project like this: BASE_DIR / &#39;subdir&#39;.<br>BASE_DIR = Path(__file__).resolve().parent.parent<br></pre><pre><em>#</em> Quick-start development settings - unsuitable for production<br><em>#</em> See https://docs.djangoproject.com/en/3.1/howto/deployment/checklist/</pre><pre><em>#</em> SECURITY WARNING: keep the secret key used in production secret!<br>SECRET_KEY = &#39;+qv2hh-c(#@e*#tf+$yehx8!t3hv+tc=38$y$cck-hwtcdhfmp&#39;</pre><pre><em>#</em> SECURITY WARNING: don&#39;t run with debug turned on in production!<br>DEBUG = True</pre><pre>...</pre><p>Note that the argument for env.read_env() is redundant, as this is the default for django-environ , but I prefer to make things explicit.</p><p>Next, let’s copy the SECRET_KEY string and put it into the .env file:</p><pre>DJANGO_SECRET_KEY=&#39;+qv2hh-c(#@e*#tf+$yehx8!t3hv+tc=38$y$cck-hwtcdhfmp&#39;</pre><p>While we’re here, let’s also add the DEBUG variable:</p><pre>DJANGO_SECRET_KEY=&#39;+qv2hh-c(#@e*#tf+$yehx8!t3hv+tc=38$y$cck-hwtcdhfmp&#39;<br>DJANGO_DEBUG=True</pre><p>Now change the SECRET_KEY and the DEBUG settings in settings.py to the following:</p><pre>...</pre><pre><em>#</em> SECURITY WARNING: keep the secret key used in production secret!<br>SECRET_KEY = env(&#39;DJANGO_SECRET_KEY&#39;)</pre><pre><em>#</em> SECURITY WARNING: don&#39;t run with debug turned on in production!<br>DEBUG = env(&#39;DJANGO_DEBUG&#39;, default=False)</pre><p>Save all the changes, and then do a test run of the server to make sure everything is working:</p><pre>(myproject) ~/myproject $ python manage.py runserver<br>Watching for file changes with StatReloader<br>Performing system checks...<br><br>System check identified no issues (0 silenced).<br><br>You have 18 unapplied migration(s). Your project may not work<br>properly until you apply the migrations for app(s): admin, auth, contenttypes, sessions.<br>Run &#39;python manage.py migrate&#39; to apply them.<br>September 29, 2020 - 22:25:39<br>Django version 3.1.1, using settings &#39;config.settings&#39;<br>Starting development server at http://127.0.0.1:8000/<br>Quit the server with CONTROL-C.</pre><p>In your browser, go to http://localhost:8000 and you should see the default Django start page.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0yzkUIgsS8rPiV7gT4JPMw.jpeg" /><figcaption>The default start page for a Django project</figcaption></figure><p>Good! Everything is working so far. Quit the local server by typing ⌘-C (MacOS) or CONTROL-C (Windows).</p><p>The next step is to change the DATABASES setting in settings.py to also use django-environ :</p><pre>...<br># Database<br># <a href="https://docs.djangoproject.com/en/3.1/ref/settings/#databases">https://docs.djangoproject.com/en/3.1/ref/settings/#databases</a></pre><pre># before<br>DATABASES = {<br>    &#39;default&#39;: {<br>        &#39;ENGINE&#39;: &#39;django.db.backends.sqlite3&#39;,<br>        &#39;NAME&#39;: BASE_DIR / &#39;db.sqlite3&#39;<br>    }<br>}</pre><pre># after<br>DATABASES = {<br>    &#39;default&#39;: env.db()<br>}</pre><p>And in config/.env :</p><pre>DJANGO_SECRET_KEY=&#39;+qv2hh-c(#@e*#tf+$yehx8!t3hv+tc=38$y$cck-hwtcdhfmp&#39;<br>DJANGO_DEBUG=True<br>DATABASE_URL=postgres://postgres:postgres@db:5432/postgres</pre><p>Note that the DATABASE_URL is in the format of</p><pre>postgres://{user}:{password}@{hostname}:{port}/{database-name}</pre><p>The user and password can be whatever you like, but the hostname of db will be important when we build our Docker files.</p><p>At this point, trying to use the runsever command again will not work. Attempting to do so will produce a long error stack that basically amounts to there is no database. So now, we need to build containers for our Django files and a database!</p><h3>Containerizing with Docker</h3><p>In the project’s root directory, create two files: Dockerfile and docker-compose.yml .</p><pre>(myproject) ~/myproject $ <strong>touch Dockerfile docker-compose.yml</strong></pre><p>Edit Dockerfile as follows:</p><pre><em>#</em> Pull a base image<br>FROM python:3.8-slim-buster</pre><pre><em>#</em> Set environment variables<br>ENV PYTHONDONTWRITEBYTECODE 1<br>ENV PYTHONUNBUFFERED 1</pre><pre><em>#</em> Create a working directory for the django project<br>WORKDIR /code</pre><pre><em>#</em> Copy requirements to the container<br>COPY Pipfile Pipfile.lock /code/</pre><pre><em>#</em> Install the requirements to the container<br>RUN pip install pipenv &amp;&amp; pipenv install --system</pre><pre><em>#</em> Copy the project files into the working directory<br>COPY . /code/</pre><pre><em>#</em> Open a port on the container<br>EXPOSE 8000</pre><p>And docker-compose.yml:</p><pre>version: &#39;3.8&#39;</pre><pre>services:<br>  web:<br>    build: .<br>    command: python /code/manage.py runserver 0.0.0.0:8000<br>    volumes:<br>      - .:/code<br>    ports:<br>      - 8000:8000<br>    depends_on:<br>      - db<br>  db:<br>    image: postgres:12<br>    volumes:<br>      - postgres_data:/var/lib/postgresql/data/<br>    environment:<br>      - &quot;POSTGRES_HOST_AUTH_METHOD=trust&quot;<br>      - &quot;POSTGRES_DB=postgres&quot;<br>      - &quot;POSTGRES_USER=postgres&quot;<br>      - &quot;POSTGRES_PASSWORD=postgres&quot;</pre><pre>volumes:<br>  postgres_data:</pre><p>When run using docker-compose, this file tells Docker to create two containers, one named web, the other named db. You should note that the environment variables specified in the db service are the default parameters for the PostgreSQL image. If you specifed a different user and password in the DATABASE_URL variable in the .env file above, you should include them here (although this will compromise the security of the database, as the docker-compose.yml file will be committed to the repository). When it comes time to deploy the project, create a separate docker-compose file that will not be committed.</p><p>Finally, build and run the containers:</p><pre>(myproject) ~/myproject $ <strong>docker-compose up --build<br></strong>Building web<br>Step 1/8 : FROM python:3.8-slim-buster<br> ---&gt; 62297c9f4e5c<br>Step 2/8 : ENV PYTHONDONTWRITEBYTECODE 1<br> ---&gt; Using cache<br> ---&gt; 4c65a79c32f1<br>Step 3/8 : ENV PYTHONUNBUFFERED 1<br> ---&gt; Using cache<br> ---&gt; d07c526fc592<br>Step 4/8 : WORKDIR /code<br> ---&gt; Using cache<br> ---&gt; c5b6fe0122e3<br>Step 5/8 : COPY Pipfile Pipfile.lock /code/<br> ---&gt; Using cache<br> ---&gt; 582d85ec55f8<br>Step 6/8 : RUN pip install pipenv &amp;&amp; pipenv install --system<br> ---&gt; Using cache<br> ---&gt; c18b4b211170<br>Step 7/8 : COPY . /code/<br> ---&gt; 086de67f57ce<br>Step 8/8 : EXPOSE 8000<br> ---&gt; Running in a1ac9496131a<br>Removing intermediate container a1ac9496131a<br> ---&gt; b836947a1d01<br><br>Successfully built b836947a1d01<br>Successfully tagged django-project-example_web:latest<br>Starting django-project-example_db_1 ... done<br>Recreating django-project-example_web_1 ... done<br>Attaching to django-project-example_db_1, django-project-example_web_1<br>db_1   |<br>db_1   | PostgreSQL Database directory appears to contain a database; Ski<br>pping initialization<br>db_1   |<br>db_1   | 2020-09-29 23:11:32.309 UTC [1] LOG:  starting PostgreSQL 12.4 (<br>Debian 12.4-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian <br>8.3.0-6) 8.3.0, 64-bit<br>db_1   | 2020-09-29 23:11:32.309 UTC [1] LOG:  listening on IPv4 address <br>&quot;0.0.0.0&quot;, port 5432<br>db_1   | 2020-09-29 23:11:32.309 UTC [1] LOG:  listening on IPv6 address <br>&quot;::&quot;, port 5432<br>db_1   | 2020-09-29 23:11:32.341 UTC [1] LOG:  listening on Unix socket &quot;<br>/var/run/postgresql/.s.PGSQL.5432&quot;<br>db_1   | 2020-09-29 23:11:32.393 UTC [25] LOG:  database system was shut <br>down at 2020-09-29 23:11:26 UTC<br>db_1   | 2020-09-29 23:11:32.415 UTC [1] LOG:  database system is ready t<br>o accept connections<br>web_1  | Watching for file changes with StatReloader<br>web_1  | Performing system checks...<br>web_1  |<br>web_1  | System check identified no issues (0 silenced).<br>web_1  |<br>web_1  | You have 18 unapplied migration(s). Your project may not work pr<br>operly until you apply the migrations for app(s): admin, auth, contenttyp<br>es, sessions.<br>web_1  | Run &#39;python manage.py migrate&#39; to apply them.<br>web_1  | September 29, 2020 - 23:11:34<br>web_1  | Django version 3.1.1, using settings &#39;config.settings&#39;<br>web_1  | Starting development server at http://0.0.0.0:8000/<br>web_1  | Quit the server with CONTROL-C.</pre><p>To veryify everything is working, go to http://localhost:8000/ again and you should see the default Django page. Close the server again by typing CONTROL-C or COMMAND-C.</p><p>The last step is to initialize a git repository and add a .gitignore file:</p><pre>(myproject) ~/myproject $ <strong>git init &amp;&amp; touch .gitignore</strong></pre><p>Make sure you add the .env file to .gitignore:</p><pre># local environments<br>.env</pre><p>At this point, you could also add the <a href="https://djangowaves.com/tips-tricks/gitignore-for-a-django-project/">other recommended files and folders</a> to .gitignore , including any temporary databases that were made when the containers were being built.</p><p>Now it’s safe to commit and push the project to Github.</p><pre>(myproject) ~/myproject $ <strong>git add . &amp;&amp; git commit -m &quot;Initial commit&quot;<br></strong>[master (root-commit) f7b1989] Initial commit<br> 11 files changed, 358 insertions(+)<br> create mode 100644 .gitignore<br> create mode 100644 Dockerfile<br> create mode 100644 Pipfile<br> create mode 100644 Pipfile.lock<br> create mode 100644 config/__init__.py<br> create mode 100644 config/asgi.py<br> create mode 100644 config/settings.py<br> create mode 100644 config/urls.py<br> create mode 100644 config/wsgi.py<br> create mode 100644 docker-compose.yml<br> create mode 100755 manage.py</pre><pre>(myproject) ~/myproject $ <strong>git branch -m &quot;main&quot;</strong></pre><p>You are now free to add new apps to the project and develop as normally.</p><p>The final repository for this project can be found on my <a href="https://github.com/bluginbuhl/django-project-example">Github</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4af72ce037f0" width="1" height="1" alt=""><hr><p><a href="https://medium.com/swlh/setting-up-a-secure-django-project-repository-with-docker-and-django-environ-4af72ce037f0">Setting Up a Secure Django Project Repository With Docker and Django-Environ</a> was originally published in <a href="https://medium.com/swlh">The Startup</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>