<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Lyle Okoth on Medium]]></title>
        <description><![CDATA[Stories by Lyle Okoth on Medium]]></description>
        <link>https://medium.com/@lyle-okoth?source=rss-2637c671f6af------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*IEW_TglRVfzDPDyc6eoABA.jpeg</url>
            <title>Stories by Lyle Okoth on Medium</title>
            <link>https://medium.com/@lyle-okoth?source=rss-2637c671f6af------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 16:02:06 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@lyle-okoth/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Creating A Documentation Generator for Python Projects using GPT3: Part Five]]></title>
            <link>https://medium.com/@lyle-okoth/creating-a-documentation-generator-for-python-projects-using-gpt3-part-five-af2588305633?source=rss-2637c671f6af------2</link>
            <guid isPermaLink="false">https://medium.com/p/af2588305633</guid>
            <category><![CDATA[queue]]></category>
            <category><![CDATA[threadpoolexecutor]]></category>
            <category><![CDATA[threads]]></category>
            <category><![CDATA[chain-of-responsibility]]></category>
            <category><![CDATA[python3]]></category>
            <dc:creator><![CDATA[Lyle Okoth]]></dc:creator>
            <pubDate>Sat, 10 Feb 2024 12:18:12 GMT</pubDate>
            <atom:updated>2024-02-10T12:18:12.797Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*jM2EjOaM-2MDG5o5rf6VDA.gif" /></figure><p>This is the fifth article in these series. We are building a python application for generating documentation for other python projects. In the <a href="https://medium.com/@lyle-okoth/creating-a-documentation-generator-for-python-projects-using-gpt3-part-four-4409f4f47bd5">previous article</a>, we finally assembled all the components in to a fully functional application. In this article, we will work on the following:</p><ul><li>The application command-line interface, to enable the user provide us with various inputs.</li><li>Speeding up the application using threads and queues.</li><li>Handling the output from chatgpt</li></ul><p>As usual, check out a new branch to track these new changes:</p><pre>git checkout -b speed-up-error-handling-ui</pre><h4>The User Interface</h4><p>This is a command-line app, that will take various parameters from the user including:</p><ul><li>The project directory or file to generate documentation for</li><li>Whether or not to overwrite existing documentation</li><li>The documentation style to use</li><li>Files and folders to ignore</li><li>The openai api key</li></ul><p>We will use <em>argparse</em> library to get these parameters when the application is executed and use our config object to store and supply them. First, update the <em>Config</em> object to account for these new parameters:</p><pre>from pydantic import BaseModel, Field<br>from typing import Optional<br><br><br>class Config(BaseModel):<br>    root_directory: list[str] = Field(description=&quot;The path to the source code directory&quot;)<br>    directories_ignore: set[str] = Field(<br>        description=&quot;Directories to ignore&quot;,<br>        default=set([&quot;venv&quot;, &quot;.venv&quot;, &quot;__pycache__&quot;, &quot;.git&quot;, &quot;build&quot;, &quot;dist&quot;, &quot;docs&quot;]),<br>    )<br>    files_ignore: set[str] = Field(<br>        description=&quot;Files to ignore&quot;,<br>        default_factory=set,<br>    )<br>    overwrite_function_docstring: Optional[bool] = Field(<br>        description=&quot;Whether or not to overwrite the existing function docstring&quot;,<br>        default=False,<br>    )<br>    documentation_style: Optional[str] = Field(<br>        description=&quot;The format of documentation to use&quot;,<br>        default=&quot;Numpy-Style&quot;,<br>        enum=[&quot;Numpy-Style&quot;, &quot;Google-Style&quot;, &quot;Sphinx-Style&quot;],<br>    )</pre><p>Then create a file called <em>ui.py</em> that will use <em>argparse</em> to get these parameters from the user:</p><pre>import os<br>from argparse import ArgumentParser, Namespace<br>from os import path<br><br>from .config import Config<br><br><br>def parse_arguments() -&gt; Namespace:<br>    parser = ArgumentParser(<br>        prog=&quot;docstring-generator&quot;,<br>        description=&quot;Generate docstrings for your python projects&quot;,<br>        epilog=&quot;Thanks for using %(prog)s! :)&quot;,<br>    )<br>    parser.add_argument(&quot;--path&quot;, nargs=&quot;*&quot;, default=[&quot;.&quot;], type=str)<br>    parser.add_argument(&quot;--config-file&quot;, nargs=&quot;?&quot;, default=&quot;&quot;, type=str)<br>    parser.add_argument(&quot;--OPENAI_API_KEY&quot;, nargs=&quot;?&quot;, default=&quot;&quot;, type=str)<br>    parser.add_argument(<br>        &quot;--overwrite-function-docstring&quot;, nargs=&quot;?&quot;, default=False, type=bool<br>    )<br>    parser.add_argument(&quot;--directories-ignore&quot;, nargs=&quot;*&quot;, default=[], type=str)<br>    parser.add_argument(&quot;--files-ignore&quot;, nargs=&quot;*&quot;, default=[], type=str)<br>    parser.add_argument(<br>        &quot;--documentation-style&quot;,<br>        nargs=&quot;?&quot;,<br>        default=&quot;Numpy-Style&quot;,<br>        choices=[&quot;Numpy-Style&quot;, &quot;Google-Style&quot;, &quot;Sphinx-Style&quot;],<br>        type=str,<br>    )<br>    args = parser.parse_args()<br>    paths: list[str] = args.path<br>    for entry in paths:<br>        if not path.exists(entry):<br>            print(f&quot;The target path &#39;{entry}&#39; doesn&#39;t exist&quot;)<br>            raise SystemExit(1)<br>    if args.OPENAI_API_KEY:<br>        os.environ[&quot;OPENAI_API_KEY&quot;] = args.OPENAI_API_KEY<br>    if not os.environ.get(&quot;OPENAI_API_KEY&quot;, None):<br>        print(&quot;You have not provided the open ai api key.&quot;)<br>        raise SystemExit(1)<br>    return args<br><br><br>def create_application_config(args: Namespace) -&gt; Config:<br>    config: Config = Config(<br>        root_directory=set(args.path),<br>        overwrite_function_docstring=args.overwrite_function_docstring,<br>        documentation_style=args.documentation_style,<br>    )<br>    config.directories_ignore.update(set(args.directories_ignore))<br>    config.files_ignore.update(set(args.files_ignore))<br>    return config</pre><ul><li>The <em>parse_argument</em> function simply parses the various arguments that the user provides, optionally giving us back default values. It also gets or sets the <em>openai</em> api key, and ensures that we give it an existing path. The path can either be a single file or folder path or multiple files and folder paths</li><li>The <em>create_application_config</em> function uses the arguments parsed to create the config object.</li></ul><p>Update the <em>__main__.py</em> file:</p><pre>from .ui import create_application_config, parse_arguments<br>from argparse import Namespace<br>from .config import Config<br>from .docstring_generator import generate_project_docstrings<br><br><br>def main():<br>    args: Namespace = parse_arguments()<br>    config: Config = create_application_config(args)<br>    # generate_project_docstrings(config=config)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>Let us test it out:</p><pre>python -m docstring_generator -h</pre><p>Here is the output:</p><pre>usage: docstring-generator [-h] [--path [PATH ...]] [--config-file [CONFIG_FILE]]<br>                           [--OPENAI_API_KEY [OPENAI_API_KEY]]<br>                           [--overwrite-function-docstring [OVERWRITE_FUNCTION_DOCSTRING]]<br>                           [--directories-ignore [DIRECTORIES_IGNORE ...]]<br>                           [--files-ignore [FILES_IGNORE ...]]<br>                           [--documentation-style [{Numpy-Style,Google-Style,Sphinx-Style}]]<br><br>Generate docstrings for your python projects<br><br>options:<br>  -h, --help            show this help message and exit<br>  --path [PATH ...]<br>  --config-file [CONFIG_FILE]<br>  --OPENAI_API_KEY [OPENAI_API_KEY]<br>  --overwrite-function-docstring [OVERWRITE_FUNCTION_DOCSTRING]<br>  --directories-ignore [DIRECTORIES_IGNORE ...]<br>  --files-ignore [FILES_IGNORE ...]<br>  --documentation-style [{Numpy-Style,Google-Style,Sphinx-Style}]<br><br>Thanks for using docstring-generator! :)</pre><p>To use it, you only need to set the OPENAI_API_KEY environment variable or supply. Comment all the lines in the <em>generate_docstring</em> function in the <em>helpers.py</em> module and return the dummy docstring to prevent the call to openai:</p><pre>def generate_doc_string(src_code: str, config: Config) -&gt; str:<br>    # documentation_style: str = &quot;Numpy-Style&quot;<br>    # prompt_formatted_str: str = function_prompt.format(<br>    #     function_code=src_code, documentation_style=config.documentation_style<br>    # )<br>    # function_and_docstring: str = chatgpt.invoke(prompt_formatted_str)<br>    # return function_and_docstring<br>    return function_dcstr</pre><p>Also notice the inclusion and use of the <em>Config</em> object to supply the documentation style. Update the <em>FunctionTransfromer</em> to take in the <em>Config</em> object and supply it to the <em>generate_doc_string</em> function:</p><pre>class FunctionTransformer(NodeTransformer):<br>    def __init__(self, config: Config) -&gt; None:<br>        super().__init__()<br>        self._config: Config = config<br><br>    def visit_FunctionDef(self, node: FunctionDef) -&gt; None:<br>        docstring: str = ast.get_docstring(node=node)<br>        if not docstring:<br>            src_code: str = ast.unparse(node)<br>            func_docstr: str = generate_doc_string(src_code=src_code, config=self._config)<br>            doc_str: str = parse_function_docstr(func_docstr.strip())<br>            dcstr_node: AST = make_docstring_node(doc_str)<br>            node.body.insert(0, dcstr_node)<br>        elif self._config.overwrite_function_docstring:<br>            src_code: str = ast.unparse(node)<br>            func_docstr: str = generate_doc_string(src_code=src_code, config=self._config)<br>            doc_str: str = parse_function_docstr(func_docstr.strip())<br>            dcstr_node: AST = make_docstring_node(doc_str)<br>            node.body[0] = dcstr_node<br>        return node</pre><p>Since we are supplying the path as a list of files and or folders, let us also update the mechanics of listing out the files. For the <em>DirectoryIterator</em>, remove the file and folder checks in the <em>__init__.py</em> method:</p><pre>def __init__(self, config: Config) -&gt; None:<br>        super().__init__()<br>        self._folders_ignore = set(config.directories_ignore)<br>        self._files_ignore = set(config.files_ignore)<br>        self._queue = deque(config.root_directory)</pre><p>Now let us create a Generator that will yield files for us. Within the <em>directory_iterators.py</em> file, create the <em>get_all_modules</em> function that will yield the files:</p><pre>def get_all_modules(config: Config) -&gt; None:<br>    &quot;&quot;&quot;Iterate throug all the directories from the root directory.&quot;&quot;&quot;<br>    for entry in config.root_directory:<br>        if path.isfile(entry):<br>            yield entry<br>        else:<br>            directory_iterator: DirectoryIterator = DirectoryIterator(config=config)<br>            for modules in directory_iterator:<br>                for module in modules:<br>                    yield module</pre><p>The method simply goes through all the entries in the path argument, in case it comes across a file, it is yielded immediately. If it’s a folder, it creates a <em>DirectoryIterator</em> that lists all the files in the directory and yields them one by one.</p><p>Update the <em>generate_project_docstrings</em> function in the <em>docstring_generator.py</em> file:</p><pre>def generate_project_docstrings(config: Config) -&gt; None:<br>    for file in get_all_modules(config=config):<br>        process_file(file_path=file, config=config)</pre><p>Remember that the <em>process_file</em> function now also takes in the <em>Config</em> object that will be supplied to the <em>NodeTransformer</em>.</p><p>And finally update the<em> __main__.py</em> file:</p><pre>from .ui import create_application_config, parse_arguments<br>from argparse import Namespace<br>from .config import Config<br>from .docstring_generator import generate_project_docstrings<br><br><br>def main():<br>    args: Namespace = parse_arguments()<br>    config: Config = create_application_config(args)<br>    generate_project_docstrings(config=config)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>We can now test out the application with the other parameters:</p><pre>python -m docstring_generator --path ./docstring_generator/utils/</pre><p>Next we need to speed up the application</p><h4>Application speed</h4><p>For this we will use threads and queues. We will separate out the process into two workflows, one for reading in the python files source codes, and another for processing the source codes, generating the docstrings and updating the docstrings. The first workflow will read in the python source code for each file and put that code into a queue. The second workflow will read in the source code from the queue, process it and update the source code before saving it.</p><p>In the <em>extensions.py</em> file, create a new queue called <em>source_code</em>.</p><pre>from queue import Queue<br><br><br>source_code_queue: Queue = Queue()</pre><p>Then let us update the <em>get_all_modules</em> function, that will add the file’s path and its source code into the source code queue:</p><pre>def get_all_modules(config: Config, source_code_queue: Queue) -&gt; None:<br>    &quot;&quot;&quot;Iterate throug all the directories from the root directory.&quot;&quot;&quot;<br>    for entry in config.root_directory:<br>        if path.isfile(entry):<br>            source_code_queue.put((entry, read_src(file_path=entry)))<br>        else:<br>            directory_iterator: DirectoryIterator = DirectoryIterator(config=config)<br>            for modules in directory_iterator:<br>                for module in modules:<br>                    source_code_queue.put((module, read_src(file_path=module)))</pre><p>Next we update the <em>process_file</em> function to take in each file path and corresponding source code from the queue:</p><pre>def process_file(config: Config, source_code_queue: Queue):<br>    while True:<br>        try:<br>            file_path, src = source_code_queue.get(timeout=1)<br>            src_tree: AST = parse_src(src)<br>            # print_src(src_tree)<br>            transformer: NodeTransformer = FunctionTransformer(config=config)<br>            new_tree = transformer.visit(src_tree)<br>            ast.fix_missing_locations(new_tree)<br>            new_module_code = ast.unparse(new_tree)<br>            print(new_module_code)<br>            save_src(file_path=file_path, new_src=new_module_code)<br>            format_file(file_path=file_path)<br>            source_code_queue.task_done()<br>        except Empty:<br>            print(&quot;Terminating the file processing..&quot;)<br>            break</pre><p>We then update the <em>generate_project_docstrings</em> function to use threads to execute the two workflows:</p><pre>def generate_project_docstrings(config: Config, source_code_queue: Queue) -&gt; None:<br>    with ThreadPoolExecutor(max_workers=2) as executor:<br>        executor.submit(get_all_modules, config, source_code_queue)<br>        executor.submit(process_file, config, source_code_queue)<br>    source_code_queue.join()</pre><p>And finally update the <em>__main__.py</em> file:</p><pre>from argparse import Namespace<br><br>from .config import Config<br>from .docstring_generator import generate_project_docstrings<br>from .extensions import source_code_queue<br>from .ui import create_application_config, parse_arguments<br><br><br>def main():<br>    args: Namespace = parse_arguments()<br>    config: Config = create_application_config(args)<br>    generate_project_docstrings(config=config, source_code_queue=source_code_queue)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>We can then run the application. Delete the .<em>env</em> file and set the openai api key at the command-line:</p><pre>export OPENAI_API_KEY=sk-xxxxxxxxxxxxxxx<br>python -m docstring_generator --path ./docstring_generator/utils/data --overwrite-function-docstring True</pre><p>To even make this faster we will create other workers for reading files and source codes and then process each function rather than whole files.</p><h4>Handling openai model output</h4><p>The docstring generated by chatgpt may not be what we expect. The output may:</p><ul><li>Contain the function code and docstring as we expect</li><li>Contain only the docstring</li><li>Contain function and docstring but part of the returned value maybe missing, if our function was very large</li><li>Contain badly formatted response</li></ul><p>To handle these various scenarios, we will use the chain of responsibility design pattern. At each stage, we will try out a different parser that can handle a particular scenario. These parsers will leverage chatgpt. For a start, we will only implement the default parser.</p><pre>from __future__ import annotations<br><br>import ast<br>from abc import ABC, abstractmethod<br>from ast import AST, FunctionDef<br><br>from .helpers import parse_src<br><br><br>class Parser(ABC):<br>    @abstractmethod<br>    def set_next(self, parser: Parser) -&gt; Parser:<br>        pass<br><br>    @abstractmethod<br>    def parse(self, docstring: str) -&gt; str:<br>        pass<br><br><br>class AbstractParser(Parser):<br>    _next_parser: Parser = None<br><br>    def set_next(self, parser: Parser) -&gt; Parser:<br>        self._next_parser = parser<br>        return parser<br><br>    @abstractmethod<br>    def parse(self, docstring: str) -&gt; str:<br>        if self._next_parser:<br>            return self._next_parser.parse(docstring)<br>        return None<br><br><br>class DefaultParser(AbstractParser):<br>    def parse(self, docstring: str) -&gt; str:<br>        try:<br>            src_tree: AST = parse_src(docstring)<br>            func_node: FunctionDef = src_tree.body[0]<br>            doc_str: str = ast.get_docstring(func_node)<br>        except Exception:<br>            return super().parse(docstring)<br>        else:<br>            return doc_str<br><br><br>model_parser: Parser = DefaultParser()<br><br><br>def parse_function_docstr(func_dcstr: str) -&gt; str:<br>    return model_parser.parse(docstring=func_dcstr)</pre><p>Later on we will chain different parsers to handle different scenarios.</p><p>So far, we now actually do have a working application. Try it out with your different projects. As usual, commit your changes, push them to github and merge them into the main branch:</p><pre>git add .<br>git commit -m &quot;feat: speeds up the application, adds ui and parses model output.&quot;<br>git push -u origin speed-up-error-handling-ui</pre><h4>Conclusion</h4><p>In this article:</p><ul><li>We created a simple commandline entry point for our application.</li><li>Implemented threads and queues for concurrent processing</li><li>Added model output parsing.</li></ul><p>In the next article we will add logging to our application, a bit of CI/CD and hopefully more internal and external documentation. Here is the <a href="https://github.com/twyle/docstring-generator">link to the project on GitHub</a>. If you found the article useful, like it and share it out. Also due connect with me on <a href="https://www.linkedin.com/in/lyle-okoth/">LinkedIn</a>, <a href="https://twitter.com/LyleOkoth">twitter</a> and <a href="https://github.com/twyle">GitHub</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=af2588305633" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating A Documentation Generator for Python Projects using GPT3: Part Four]]></title>
            <link>https://medium.com/@lyle-okoth/creating-a-documentation-generator-for-python-projects-using-gpt3-part-four-4409f4f47bd5?source=rss-2637c671f6af------2</link>
            <guid isPermaLink="false">https://medium.com/p/4409f4f47bd5</guid>
            <category><![CDATA[llm]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[prompt-engineering]]></category>
            <category><![CDATA[chatgpt3]]></category>
            <category><![CDATA[langchain]]></category>
            <dc:creator><![CDATA[Lyle Okoth]]></dc:creator>
            <pubDate>Fri, 09 Feb 2024 20:04:18 GMT</pubDate>
            <atom:updated>2024-02-09T20:04:18.623Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*jM2EjOaM-2MDG5o5rf6VDA.gif" /></figure><p>This is the fourth part in a series of articles that takes the user through the process of creating a python application for generating documentation for python projects. In the previous article we:</p><ul><li>Crafted a prompt for generating doc-strings for python functions by using lang-chain and openai’s gpt3.</li><li>Inserted the generated doc-strings into our functions</li><li>Built a fully working example demo of our app, that takes in a directory, then for each python file, parses out the function and inserts dummy doc-string.</li></ul><p>For this article, we will assemble all the components of the application into a fully functional python application.</p><p>Start by checking out a new branch:</p><pre>git checkout -b fully-functional-app</pre><h4>Refactoring</h4><p>First, let’s create a file called <em>directory_iterators.py</em> that will hold the <em>DirectoryIterator</em>. A changes made to this class include:</p><ul><li>Updating the initialization of the dequeue; the root directory is added as a list; previously it was added as a string, which ended up splitting it into characters and causing a path not found error.</li></ul><pre>from collections.abc import Iterator<br>from .config import Config<br>from os import path, listdir<br>from collections import deque<br><br><br>class DirectoryIterator(Iterator):<br>    def __init__(self, config: Config) -&gt; None:<br>        super().__init__()<br>        if not path.exists(config.root_directory):<br>            raise ValueError(<br>                f&#39;No such directory or file &quot;{config.root_directory}&quot; exists.&#39;<br>            )<br>        if path.isfile(config.root_directory):<br>            raise ValueError(&#39;The root directory should be a folder.&#39;)<br>        self._folders_ignore = set(config.folders_ignore)<br>        self._files_ignore = set(config.files_ignore)<br>        self._queue = deque([config.root_directory]) #adding the individual chars<br><br>    def __iter__(self) -&gt; Iterator:<br>        return super().__iter__()<br><br>    def __next__(self) -&gt; list[str]:<br>        if self._queue:<br>            files: list[str] = list()<br>            for _ in range(len(self._queue)):<br>                directory: str = self._queue.popleft()<br>                for entry in listdir(directory):<br>                    entry_path: str = path.join(directory, entry)<br>                    if (<br>                        path.isfile(entry_path)<br>                        and self._is_python_file(entry_path)<br>                        and entry not in self._files_ignore<br>                    ):<br>                        files.append(entry_path)<br>                    elif path.isdir(entry_path) and entry not in self._folders_ignore:<br>                        self._queue.append(entry_path)<br>            return files<br>        else:<br>            raise StopIteration()<br><br>    def _is_python_file(self, file_path: str) -&gt; bool:<br>        return file_path.split(&#39;.&#39;)[-1] == &#39;py&#39;</pre><h4>Transformers</h4><p>For updating the function docstring, we will need a <em>NodeTransformer</em>. Create a file called <em>transformers.py</em> and add to it the transformer code:</p><pre>from ast import AST, FunctionDef, NodeTransformer, Constant, Expr<br>import ast<br><br><br>class FunctionTransformer(NodeTransformer):<br>    def visit_FunctionDef(self, node: FunctionDef) -&gt; None:<br>        src_code: str = ast.unparse(node)<br>        func_docstr: str = generate_doc_string(src_code)<br>        doc_str: str = parse_function_docstr(func_docstr.strip())<br>        dcstr_node: AST = make_docstring_node(doc_str)<br>        node.body.insert(0, dcstr_node)<br>        return node</pre><p>Then add all the helper functions into the <em>helpers.py</em> file:</p><pre>import ast<br>from ast import AST, Constant, Expr, FunctionDef<br><br>from .llms import chatgpt<br>from .templates import function_prompt<br><br><br>def read_src(file_path: str) -&gt; str:<br>    with open(file_path, &quot;r&quot;, encoding=&quot;utf-8&quot;) as f:<br>        return f.read()<br><br><br>def save_src(file_path: str, new_src: str) -&gt; str:<br>    with open(file_path, &quot;w&quot;, encoding=&quot;utf-8&quot;) as f:<br>        return f.write(new_src)<br><br><br>def parse_src(file_src: str) -&gt; AST:<br>    return ast.parse(file_src)<br><br><br>def print_src(src_tree: AST) -&gt; None:<br>    print(ast.dump(src_tree, indent=4))<br><br><br>function_dcstr: str = &#39;&#39;&#39;<br>def subtract(a: int | float, b: int | float) -&gt; int | float:<br>    &quot;&quot;&quot;Subtracts two numbers<br><br>    Parameters<br>    ----------<br>    a : int or float<br>        The first number to subtract.<br>    b : int or float<br>        The second number to subtract.<br><br>    Returns<br>    -------<br>    int or float<br>        The result of subtracting b from a.<br>    &quot;&quot;&quot;<br>    return a - b<br>&#39;&#39;&#39;<br><br><br>def generate_doc_string(src_code: str) -&gt; str:<br>    # documentation_style: str = &#39;Numpy-Style&#39;<br>    # prompt_formatted_str: str = function_prompt.format(<br>    #     function_code=src_code, documentation_style=documentation_style<br>    # )<br>    # function_and_docstring: str = chatgpt.invoke(prompt_formatted_str)<br>    # return function_and_docstring<br>    return function_dcstr<br><br><br>def parse_function_docstr(func_dcstr: str) -&gt; str:<br>    src_tree: AST = parse_src(func_dcstr)<br>    func_node: FunctionDef = src_tree.body[0]<br>    doc_str: str = ast.get_docstring(func_node)<br>    return doc_str<br><br><br>def make_docstring_node(docstr: str):<br>    constant_str: Constant = Constant(docstr)<br>    return Expr(value=constant_str)</pre><p>This module contains the functions:</p><ul><li><em>read_src</em>; for reading each file source</li><li><em>save_src</em>: for saving the source code with updated docstrings</li><li><em>parse_src</em>; for parsing source code into <em>ast</em> tree</li><li><em>print_src</em>; for printing <em>ast</em> tree</li><li><em>generate_doc_string</em>; for generating the docstring using <em>chatgpt</em>; it currently just returns a hardcoded value to prevent calls to <em>openai</em> as we test out this new code layout</li><li><em>parse_function_docstr</em>; for parsing the docstring from the models output</li><li><em>make_docstring_node</em>; for creating a new docstring node</li></ul><p>All these function have been introduced earlier on and their usage explored.</p><p>Place the function docstring generator template into the <em>templates.py </em>file:</p><pre>from langchain.prompts import PromptTemplate<br><br>function_prompt_template: str = &quot;&quot;&quot;<br>Generate python docstring for the given python function using the provided documentation style.<br>Make sure to provide atleast two examples of the function usage only in the docstring as well<br>as the exceptions that may be raised when using the function. Make sure to return the<br>function and its docstring.<br>Function code: {function_code}<br>Documentation style: {documentation_style}<br>&quot;&quot;&quot;<br>function_prompt = PromptTemplate.from_template(template=function_prompt_template)</pre><p>Create a new file called <em>llms.py</em> and move the model creation code into that file. We rename the llm to <em>chatgpt</em> since that is what we are using. We could swap this for <em>bard</em> in future:</p><pre>import os<br><br>from langchain.llms.base import BaseLLM<br>from langchain_openai import OpenAI<br><br>api_key: str = os.environ[&quot;OPENAI_API_KEY&quot;]<br><br>chatgpt: BaseLLM = OpenAI(temperature=0, api_key=api_key)</pre><p>Move the process_file function into the <em>docstring_writer.py </em>file. This module will be responsible for actually processing each file:</p><pre>import ast<br>from ast import AST, NodeTransformer<br><br>from .helpers import parse_src, read_src, save_src<br>from .transformers import FunctionTransformer<br><br><br>def process_file(file_path: str) -&gt; str:<br>    src: str = read_src(file_path=file_path)<br>    src_tree: AST = parse_src(src)<br>    # print_src(src_tree)<br>    transformer: NodeTransformer = FunctionTransformer()<br>    new_tree = transformer.visit(src_tree)<br>    ast.fix_missing_locations(new_tree)<br>    new_module_code = ast.unparse(new_tree)<br>    print(new_module_code)<br>    # save_src(file_path=file_path, new_src=new_module_code)</pre><p>The <em>docstring_generator.py</em> will be the module that will put together all the components of our application. Add the following code:</p><pre>from .config import Config<br>from .directory_iterators import DirectoryIterator<br>from .docstring_writer import process_file<br><br><br>def generate_project_docstrings(config: Config) -&gt; None:<br>    directory_iterator: DirectoryIterator = DirectoryIterator(config=config)<br>    for files in directory_iterator:<br>        for file in files:<br>            process_file(file_path=file)</pre><p>And finally the entry point to our application will be __main__.py file. Update it to:</p><pre>from .config import Config<br>from .docstring_generator import generate_project_docstrings<br><br><br>def main():<br>    # root_dir: str = &#39;.&#39;<br>    root_dir: str = &quot;./docstring_generator/utils/data&quot;<br>    folders_ignore: list[str] = [&quot;venv&quot;, &quot;__pycache__&quot;, &quot;.git&quot;, &quot;.venv&quot;]<br>    config: Config = Config(<br>        root_directory=root_dir,<br>    )<br>    config.folders_ignore.extend(folders_ignore)<br>    generate_project_docstrings(config=config)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>Now, to test it out, create a new folder called <em>data</em> in the <em>utils</em> folder. Then within that folder create a <em>functions.py</em> file with the following four functions for testing:</p><pre>import operator<br><br><br>def add(a: int | float, b: int | float) -&gt; int | float:<br>    return operator.add(a, b)<br><br><br>def subtract(a: int | float, b: int | float) -&gt; int | float:<br>    return operator.sub(a, b)<br><br><br>def divide(a: int | float, b: int | float) -&gt; float:<br>    return operator.truediv(a, b)<br><br><br>def mutiply(a: int | float, b: int | float) -&gt; int | float:<br>    return operator.mul(a, b)</pre><p>We are now ready to test out our application.</p><h4>Initial test</h4><p>To test it out:</p><pre>python -m docstring_generator</pre><p>This should be the output:</p><pre>import operator<br><br>def add(a: int | float, b: int | float) -&gt; int | float:<br>    &quot;&quot;&quot;Subtracts two numbers<br><br>Parameters<br>----------<br>a : int or float<br>    The first number to subtract.<br>b : int or float<br>    The second number to subtract.<br><br>Returns<br>-------<br>int or float<br>    The result of subtracting b from a.&quot;&quot;&quot;<br>    return operator.add(a, b)<br><br>def subtract(a: int | float, b: int | float) -&gt; int | float:<br>    &quot;&quot;&quot;Subtracts two numbers<br><br>Parameters<br>----------<br>a : int or float<br>    The first number to subtract.<br>b : int or float<br>    The second number to subtract.<br><br>Returns<br>-------<br>int or float<br>    The result of subtracting b from a.&quot;&quot;&quot;<br>    return operator.sub(a, b)<br><br>def divide(a: int | float, b: int | float) -&gt; float:<br>    &quot;&quot;&quot;Subtracts two numbers<br><br>Parameters<br>----------<br>a : int or float<br>    The first number to subtract.<br>b : int or float<br>    The second number to subtract.<br><br>Returns<br>-------<br>int or float<br>    The result of subtracting b from a.&quot;&quot;&quot;<br>    return operator.truediv(a, b)<br><br>def mutiply(a: int | float, b: int | float) -&gt; int | float:<br>    &quot;&quot;&quot;Subtracts two numbers<br><br>Parameters<br>----------<br>a : int or float<br>    The first number to subtract.<br>b : int or float<br>    The second number to subtract.<br><br>Returns<br>-------<br>int or float<br>    The result of subtracting b from a.&quot;&quot;&quot;<br>    return operator.mul(a, b)</pre><p>The formatting is quite terrible and will have to be addressed.</p><h4>Using chatgpt</h4><p>Let us now test it with chat-gpt. Uncomment all the lines in the <em>generate_docstring</em> function in the <em>helpers.py</em> module:</p><pre>def generate_doc_string(src_code: str) -&gt; str:<br>    documentation_style: str = &#39;Numpy-Style&#39;<br>    prompt_formatted_str: str = function_prompt.format(<br>        function_code=src_code, documentation_style=documentation_style<br>    )<br>    function_and_docstring: str = chatgpt.invoke(prompt_formatted_str)<br>    return function_and_docstring</pre><p>And then run the application again. Here is the output:</p><pre>import operator<br><br>def add(a: int | float, b: int | float) -&gt; int | float:<br>    &quot;&quot;&quot;Returns the sum of two numbers.<br><br>Parameters<br>----------<br>a : int or float<br>    First number to be added.<br>b : int or float<br>    Second number to be added.<br><br>Returns<br>-------<br>int or float<br>    Sum of a and b.<br><br>Examples<br>--------<br>&gt;&gt;&gt; add(2, 3)<br>5<br>&gt;&gt;&gt; add(2.5, 3.5)<br>6.0<br><br>Raises<br>------<br>TypeError<br>    If either a or b is not an int or float.&quot;&quot;&quot;<br>    return operator.add(a, b)<br><br>def subtract(a: int | float, b: int | float) -&gt; int | float:<br>    &quot;&quot;&quot;Returns the difference between two numbers.<br><br>Parameters<br>----------<br>a : int or float<br>    The first number.<br>b : int or float<br>    The second number.<br><br>Returns<br>-------<br>int or float<br>    The difference between a and b.<br><br>Examples<br>--------<br>&gt;&gt;&gt; subtract(5, 3)<br>2<br>&gt;&gt;&gt; subtract(3.5, 2.5)<br>1.0<br><br>Raises<br>------<br>TypeError<br>    If either a or b is not an int or float.&quot;&quot;&quot;<br>    return operator.sub(a, b)<br><br>def divide(a: int | float, b: int | float) -&gt; float:<br>    &quot;&quot;&quot;Returns the quotient of two numbers.<br><br>Parameters<br>----------<br>a : int or float<br>    The first number.<br>b : int or float<br>    The second number.<br><br>Returns<br>-------<br>float<br>    The quotient of a divided by b.<br><br>Examples<br>--------<br>&gt;&gt;&gt; divide(10, 2)<br>5.0<br>&gt;&gt;&gt; divide(3.5, 0.5)<br>7.0<br><br>Raises<br>------<br>TypeError<br>    If either a or b is not an int or float.<br>ZeroDivisionError<br>    If b is 0.&quot;&quot;&quot;<br>    return operator.truediv(a, b)<br><br>def mutiply(a: int | float, b: int | float) -&gt; int | float:<br>    &quot;&quot;&quot;Returns the product of two numbers.<br><br>Parameters<br>----------<br>a : int or float<br>    First number to be multiplied.<br>b : int or float<br>    Second number to be multiplied.<br><br>Returns<br>-------<br>int or float<br>    Product of a and b.<br><br>Examples<br>--------<br>&gt;&gt;&gt; multiply(2, 3)<br>6<br>&gt;&gt;&gt; multiply(2.5, 4)<br>10.0<br><br>Raises<br>------<br>TypeError<br>    If either a or b is not an int or float.&quot;&quot;&quot;<br>    return operator.mul(a, b)</pre><p>Mine took quite a while to run, but it was successful. We will need to address the speed at which the application runs as well as other issues such as errors in working with <em>openai</em> in the upcoming articles. For now, we have a fully working application that given a project folder with python files, generates doc-strings for each function.</p><p>You can also un-comment the last line in the <em>process_file</em> function in the <em>docstring_writer.py</em> file to save the new source code with the doc-strings. As usual, commit your changes and push them to GitHub.</p><pre>git add .<br>git commit -m &quot;feat: assembles the whole application.&quot;<br>git push -u origin fully-functional-app</pre><h4>Conclusion</h4><p>In this article, we finally assembled our application and used it to generate doc-strings for four python functions. Here is the <a href="https://github.com/twyle/docstring-generator">link to the project on GitHub</a>. If you found the article useful, like it and share it out. Also due connect with me on <a href="https://www.linkedin.com/in/lyle-okoth/">LinkedIn</a>, <a href="https://twitter.com/LyleOkoth">twitter</a> and <a href="https://github.com/twyle">GitHub</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4409f4f47bd5" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating A Documentation Generator for Python Projects using GPT3: Part Three]]></title>
            <link>https://medium.com/@lyle-okoth/creating-a-documentation-generator-for-python-projects-using-gpt3-part-three-c46eb34d24e9?source=rss-2637c671f6af------2</link>
            <guid isPermaLink="false">https://medium.com/p/c46eb34d24e9</guid>
            <category><![CDATA[ast]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[python3]]></category>
            <dc:creator><![CDATA[Lyle Okoth]]></dc:creator>
            <pubDate>Thu, 08 Feb 2024 17:57:05 GMT</pubDate>
            <atom:updated>2024-02-08T17:57:05.776Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*jM2EjOaM-2MDG5o5rf6VDA.gif" /></figure><p>This is the third part in a series of articles that takes the reader through creating an application for generating documentation for python projects. Here is the progress so far:</p><ul><li>In the first part, we introduced the project, set up the initial layout and then created an iterator for obtaining python files from a given folder and its subfolders.</li><li>In the second part, we dived into the mechanics of parsing python source code. We used the ast module to parse python files into a tree structure that gave us access to the various components that make up a python program including the classes and methods. We went over the composite and visitor patterns and used these concepts to get, insert and replace a function’s docstrings with dummy docstrings.</li></ul><p>In this article, our focus will be on using openai’s chatgpt to generate the docstrings. We will parse each python file, get the source code for each function and then using langchain, send requests to chatgpt3 to generate docsstrings, then use those docstrings to update the function docstring.</p><p>As usual, let’s start by checking out a new branch:</p><pre>git checkout -b chatgpt-docstring</pre><p>And if you have not been following along, clone the code-parser branch and check out a new branch:</p><pre>git clone -b code-parser https://github.com/twyle/docstring-generator.git<br>git checkout -b chatgpt-docstring</pre><h4>Using chatgpt to generate docstrings</h4><p>To generate the docstring using chatgpt, we will need a model and a prompt. The prompt will simply tell the model what we want it to do. We will use gpt3.5. Start by installing langchain and its dependancies:</p><pre>pip install langchain, langchain-openai python-dotenv</pre><p>Then create a dotenv file at the project root and in it add your openai api key:</p><pre>touch .env</pre><p>The .env file should look like this:</p><pre>OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</pre><p>Replace the <em>sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx</em> with your own API key.</p><p>Here is sample code for generating docstring for a function that adds two numbers(<em>docstring_generator/utils/generate_docstrings.py</em>):</p><pre>from dotenv import load_dotenv<br><br>load_dotenv()<br>from langchain.prompts import PromptTemplate<br>from langchain_openai import OpenAI<br>from langchain.llms.base import BaseLLM<br>import os<br><br><br>api_key: str = os.environ[&#39;OPENAI_API_KEY&#39;]<br><br>llm: BaseLLM = OpenAI(<br>    temperature=0, api_key=api_key<br>)<br><br>function_prompt_template: str = &quot;&quot;&quot;<br>Generate python docstring for the given python function using the provided documentation style:<br>Function code: {function_code}<br>Documentation style: {documentation_style}<br>&quot;&quot;&quot;<br><br>function_code: str = &quot;&quot;&quot;<br>def add(a: int | float, b: int | float) -&gt; int | float:<br>    return operator.add(a, b)<br>&quot;&quot;&quot;<br>documentation_style: str = &#39;Numpy-Style&#39;<br><br>prompt = PromptTemplate.from_template(template=function_prompt_template)<br>prompt_formatted_str: str = prompt.format(<br>    function_code=function_code, documentation_style=documentation_style<br>)<br><br>function_and_docstring: str = llm.invoke(prompt_formatted_str)<br>print(function_and_docstring)</pre><ul><li>We start by importing the necessary libraries. <em>dotenv </em>enables us to load environment variables, in this case <strong>OPENAI_API_KEY</strong>.</li><li><em>langchain</em> provides us with a lot of utilities for working with large language models especially <em>openai</em>.</li><li><em>os</em> enbales us to access environment variables.</li><li>We start by creating an instance of <em>BaseLLM</em> that is a wrapper for openai’s <em>chat-completion</em> endpoint, with a <em>temperature of zero </em>to minimize creativity.</li><li>We then create a <em>prompt template</em> that simply commands the llm to generate function docstring for the given <em>add</em> function. We provide the style argument, in this case <em>Numpy-style</em>, but this could also be <em>Google-style</em> or <em>Sphinx-style</em>. These values together with the function code are provided using the <em>format</em> method. <em>langchain</em> uses the same syntax like python strings to provide template variables using a format method.</li><li>The formatted prompt string is then passed to the llm, using the invoke method. The llm then gives us back the function and its documentation.</li></ul><pre>def add(a, b):<br>    &quot;&quot;&quot;<br>    Add two numbers.<br><br>    Parameters<br>    ----------<br>    a : int or float<br>        First number to be added.<br>    b : int or float<br>        Second number to be added.<br><br>    Returns<br>    -------<br>    int or float<br>        Sum of a and b.<br>    &quot;&quot;&quot;<br>    return operator.add(a, b)</pre><p>As seen above, the input types as well as the output types are omitted by the llm. It is also not giving us examples, or showing us the exceptions that may be raised. Let us try a more complex prompt to address these issues.</p><h4>A more complex prompt</h4><p>With this prompt, we aim to:</p><ul><li>Ask the model to include a couple of examples as well as excpetions raised.</li></ul><pre>function_prompt_template: str = &quot;&quot;&quot;<br>Generate python docstring for the given python function using the provided documentation style. <br>Make sure to provide atleast two examples of the function usage as well as the exceptions that may be raised when using <br>the function may raise.<br>Function code: {function_code}<br>Documentation style: {documentation_style}<br>&quot;&quot;&quot;</pre><p>This is the part of llm development called prompt engineering. Try out different prompts until you find the one that works for you. Here is the output(<em>docstring_generator/utils/generate_docstrings_v1.py</em>):</p><pre>def add(a, b):<br>    &quot;&quot;&quot;<br>    Adds two numbers together.<br><br>    Parameters<br>    ----------<br>    a : int or float<br>        First number to be added.<br>    b : int or float<br>        Second number to be added.<br><br>    Returns<br>    -------<br>    int or float<br>        Sum of the two numbers.<br><br>    Raises<br>    ------<br>    TypeError<br>        If either a or b is not an int or float.<br><br>    Examples<br>    --------<br>    &gt;&gt;&gt; add(2, 3)<br>    5<br>    &gt;&gt;&gt; add(2.5, 3.5)<br>    6.0<br>    &gt;&gt;&gt; add(&quot;2&quot;, 3)<br>    TypeError: a must be an int or float<br>    &quot;&quot;&quot;<br><br>    if not isinstance(a, (int, float)) or not isinstance(b, (int, float)):<br>        raise TypeError(&quot;a and b must be an int or float&quot;)<br><br>    return a + b</pre><p>We did get what we asked for, as the docstring now contains examples as well as exceptions raised. However the llm also altered our function. This I do believe is what people refer to as hallucination.</p><pre>if not isinstance(a, (int, float)) or not isinstance(b, (int, float)):<br>        raise TypeError(&quot;a and b must be an int or float&quot;)</pre><p>At the moment, this is not bad, as we are interested in the docstring only.</p><h4>Parsing the docstring</h4><p>Now that we have the function and docstring, we need to parse the <em>llm</em> response and obtain only the docstring. This is where once again, we turn to the <em>ast</em> module. Form the <a href="https://medium.com/@lyle-okoth/creating-a-documentation-generator-for-python-projects-using-gpt3-part-two-0cb22fd01a2a">previous article</a>, we learnt that <em>ast</em> parses python code into a tree with various nodes, including function nodes(<em>FuncDef</em>), which have as the first entry an <em>Expr</em> node that contains the docstring. Let us see this in action (<em>docstring_generator/util/parse_llm_response.py</em>):</p><pre>from ast import AST<br><br>from utils import parse_src, print_src<br><br>src: str = &#39;&#39;&#39;<br>def add(a, b):<br>    &quot;&quot;&quot;<br>    Adds two numbers together.<br><br>    Parameters<br>    ----------<br>    a : int or float<br>        First number to be added.<br>    b : int or float<br>        Second number to be added.<br><br>    Returns<br>    -------<br>    int or float<br>        Sum of the two numbers.<br><br>    Raises<br>    ------<br>    TypeError<br>        If either a or b is not an int or float.<br><br>    Examples<br>    --------<br>    &gt;&gt;&gt; add(2, 3)<br>    5<br>    &gt;&gt;&gt; add(2.5, 3.5)<br>    6.0<br>    &gt;&gt;&gt; add(&quot;2&quot;, 3)<br>    TypeError: a must be an int or float<br>    &quot;&quot;&quot;<br><br>    if not isinstance(a, (int, float)) or not isinstance(b, (int, float)):<br>        raise TypeError(&quot;a and b must be an int or float&quot;)<br><br>    return a + b<br>&#39;&#39;&#39;<br><br><br>def main():<br>    src_tree: AST = parse_src(src)<br>    print_src(src_tree)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><ul><li>We use the output from the <em>llm</em> as the source to <em>ast</em>.</li></ul><p>Here is the output:</p><pre>Module(<br>    body=[<br>        FunctionDef(<br>            name=&#39;add&#39;,<br>            args=arguments(<br>                posonlyargs=[],<br>                args=[<br>                    arg(arg=&#39;a&#39;),<br>                    arg(arg=&#39;b&#39;)],<br>                kwonlyargs=[],<br>                kw_defaults=[],<br>                defaults=[]),<br>            body=[<br>                Expr(<br>                    value=Constant(value=&#39;\n    Adds two numbers together.\n\n    Parameters\n    ----------\n    a : int or float\n        First number to be added.\n    b : int or float\n        Second number to be added.\n\n    Returns\n    -------\n    int or float\n        Sum of the two numbers.\n\n    Raises\n    ------\n    TypeError\n        If either a or b is not an int or float.\n\n    Examples\n    --------\n    &gt;&gt;&gt; add(2, 3)\n    5\n    &gt;&gt;&gt; add(2.5, 3.5)\n    6.0\n    &gt;&gt;&gt; add(&quot;2&quot;, 3)\n    TypeError: a must be an int or float\n    &#39;)),<br>                If(<br>                    test=BoolOp(<br>                        op=Or(),<br>                        values=[<br>                            UnaryOp(<br>                                op=Not(),<br>                                operand=Call(<br>                                    func=Name(id=&#39;isinstance&#39;, ctx=Load()),<br>                                    args=[<br>                                        Name(id=&#39;a&#39;, ctx=Load()),<br>                                        Tuple(<br>                                            elts=[<br>                                                Name(id=&#39;int&#39;, ctx=Load()),<br>                                                Name(id=&#39;float&#39;, ctx=Load())],<br>                                            ctx=Load())],<br>                                    keywords=[])),<br>                            UnaryOp(<br>                                op=Not(),<br>                                operand=Call(<br>                                    func=Name(id=&#39;isinstance&#39;, ctx=Load()),<br>                                    args=[<br>                                        Name(id=&#39;b&#39;, ctx=Load()),<br>                                        Tuple(<br>                                            elts=[<br>                                                Name(id=&#39;int&#39;, ctx=Load()),<br>                                                Name(id=&#39;float&#39;, ctx=Load())],<br>                                            ctx=Load())],<br>                                    keywords=[]))]),<br>                    body=[<br>                        Raise(<br>                            exc=Call(<br>                                func=Name(id=&#39;TypeError&#39;, ctx=Load()),<br>                                args=[<br>                                    Constant(value=&#39;a and b must be an int or float&#39;)],<br>                                keywords=[]))],<br>                    orelse=[]),<br>                Return(<br>                    value=BinOp(<br>                        left=Name(id=&#39;a&#39;, ctx=Load()),<br>                        op=Add(),<br>                        right=Name(id=&#39;b&#39;, ctx=Load())))],<br>            decorator_list=[])],<br>    type_ignores=[])</pre><ul><li>As seen above, the docstring is parsed into the <em>Expr</em> node, which is the first node in the function body.</li></ul><h4>Getting the function source code</h4><p>So far, we can get chatgpt to generate docstrings for a given function. However, we are manually writing the function code then parsing it to the the prompt template. So how do we get the function code from the python file?</p><p>Well, once again we turn to the ast module. So so far, here’s what we’ve done with ast:</p><ul><li>Parse python file code into an ast tree with the various nodes</li><li>Walked through an ast tree to print out the function names and docstrings</li><li>Used the NodeTransformer to insert dummy doctsring</li><li>Converted ast tree back to source code</li></ul><p>That last part is what we need. Remember in the previous article, when after inserting the function docstring, we converted the ast tree back to source code and printed out the source code? Here is the code:</p><pre>src_tree: AST = parse_src(src)<br>transformer: NodeTransformer = MyTransformer()<br>new_tree = transformer.visit(src_tree)<br>ast.fix_missing_locations(new_tree)<br>new_module_code = ast.unparse(new_tree)<br>print(new_module_code)</pre><p>So <em>ast.unparse</em> takes in as <em>ast tree</em> and gives us back the code. How do we use that? Remember also the bit about composites? The tree is a composite object, so each node (<em>FuncDef</em>, <em>ClassDef</em>) is actually an <em>ast</em> tree. So we can just pass that node into <em>ast.unparse</em> to get the corresponding code.</p><p>What does all this mean? Well, given a python file, we read in the code. This code is passed to <em>ast</em> which creates the source code tree. We then subclass the <em>NodeVisitor</em> class and implement the <em>visit_Funcdef</em> method. This method will be called for each function in the source code and as one of its parameters, will be the function node itself. We can then use <em>ast</em> to get the source code for that function, then use that function source code to generate the docstring using the process we showed above. Let us see this in action.</p><h4>Getting the function source code</h4><p>We will subclass the NodeVisitor class and implement the visit_FunctionDef method. In that method, we will use ast.unparse to get the function source code(<em>docstring_generator/utils/get_function_src_code.py</em>):</p><pre>from ast import AST, FunctionDef, NodeVisitor<br>import ast<br><br>from utils import parse_src, print_src, read_src<br><br><br>class MyVisitor(NodeVisitor):<br>    def visit_FunctionDef(self, node: FunctionDef) -&gt; None:<br>        print(f&quot;Getting the source code for the function: {node.name}&quot;)<br>        src_code: str = ast.unparse(node)<br>        print(src_code)<br><br><br>def main():<br>    file_path: str = &quot;./docstring_generator/utils/functions.py&quot;<br>    src: str = read_src(file_path=file_path)<br>    src_tree: AST = parse_src(src)<br>    visitor = MyVisitor()<br>    visitor.visit(src_tree)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>Here is the output:</p><pre>Getting the source code for the function: add<br>def add(a: int | float, b: int | float) -&gt; int | float:<br>    return operator.add(a, b)<br>Getting the source code for the function: subtract<br>def subtract(a: int | float, b: int | float) -&gt; int | float:<br>    &quot;&quot;&quot;Subtracts two numbers<br><br>    Parameters<br>    ----------<br>    a : int or float<br>        The first number to subtract.<br>    b : int or float<br>        The second number to subtract.<br><br>    Returns<br>    -------<br>    int or float<br>        The result of subtracting b from a.<br>    &quot;&quot;&quot;<br>    return a - b</pre><h4>Recap</h4><p>So far:</p><ul><li>We can get all the python files in a given directory and its subdirectories</li><li>Parse the source code into an ast tree and get the source code for each function</li><li>Use the function code to generate docstrings</li><li>Parse the response from llm to get the function docstring</li><li>Modify the ast tree to insert or update function dosctring</li></ul><p>Can we now create a fully working example of what our application set out to do?</p><h4>A fully working example</h4><p>Let us start simple; just read in a python file, get the function source code, then use that source code to generate the docstring, parse the docstring and print it out to the terminal. We do this only for the <em>add</em> function (<em>docstring_generator/utils/simple_example.py</em>):</p><pre>from dotenv import load_dotenv<br><br>load_dotenv()<br>import os<br><br>from langchain.llms.base import BaseLLM<br>from langchain.prompts import PromptTemplate<br>from langchain_openai import OpenAI<br>from ast import AST, FunctionDef, NodeVisitor<br>import ast<br><br>from utils import parse_src, read_src<br><br>def generate_doc_string(src_code: str) -&gt; str:<br>    api_key: str = os.environ[&quot;OPENAI_API_KEY&quot;]<br><br>    llm: BaseLLM = OpenAI(temperature=0, api_key=api_key)<br><br>    function_prompt_template: str = &quot;&quot;&quot;<br>    Generate python docstring for the given python function using the provided documentation style. <br>    Make sure to provide atleast two examples of the function usage as well as the exceptions that may be raised when using <br>    the function may raise.<br>    Function code: {function_code}<br>    Documentation style: {documentation_style}<br>    &quot;&quot;&quot;<br>    <br>    documentation_style: str = &quot;Numpy-Style&quot;<br><br>    prompt = PromptTemplate.from_template(template=function_prompt_template)<br>    prompt_formatted_str: str = prompt.format(<br>        function_code=src_code, documentation_style=documentation_style<br>    )<br><br>    function_and_docstring: str = llm.invoke(prompt_formatted_str)<br>    return function_and_docstring<br><br><br>def parse_function_docstr(func_dcstr: str) -&gt; str:<br>    src_tree: AST = parse_src(func_dcstr)<br>    func_node: FunctionDef = src_tree.body[0]<br>    doc_str: str = ast.get_docstring(func_node)<br>    return doc_str<br><br><br>class MyVisitor(NodeVisitor):<br>    def visit_FunctionDef(self, node: FunctionDef) -&gt; None:<br>        if node.name == &#39;add&#39;:<br>            src_code: str = ast.unparse(node)<br>            func_docstr: str = generate_doc_string(src_code)<br>            doc_str: str = parse_function_docstr(func_docstr.strip())<br>            print(doc_str)<br><br><br>def main():<br>    file_path: str = &quot;./docstring_generator/utils/functions.py&quot;<br>    src: str = read_src(file_path=file_path)<br>    src_tree: AST = parse_src(src)<br>    visitor = MyVisitor()<br>    visitor.visit(src_tree)</pre><p>Here is the output:</p><pre>Add two numbers together.<br><br>Parameters<br>----------<br>a : int | float<br>    First number to be added.<br>b : int | float<br>    Second number to be added.<br><br>Returns<br>-------<br>int | float<br>    Sum of the two numbers.<br><br>Examples<br>--------<br>&gt;&gt;&gt; add(2, 3)<br>5<br>&gt;&gt;&gt; add(2.5, 3.5)<br>6.0<br><br>Raises<br>------<br>TypeError<br>    If either a or b is not an int or float.</pre><p>Just as a heads up, I encountered a lot of errors tyring o get this work. I had to constantly update the prompt, and here is what worked:</p><pre>function_prompt_template: str = &quot;&quot;&quot;<br>    Generate python docstring for the given python function using the provided documentation style. <br>    Make sure to provide atleast two examples of the function usage only in the docstring as well <br>    as the exceptions that may be raised when using the function. Make sure to return the <br>    function and its docstring.<br>    Function code: {function_code}<br>    Documentation style: {documentation_style}<br>    &quot;&quot;&quot;</pre><p>Now for the complete example, where we use the <em>NodeTransformer</em> class to actually insert the generated docstring:</p><pre>from dotenv import load_dotenv<br><br>load_dotenv()<br>import os<br><br>from langchain.llms.base import BaseLLM<br>from langchain.prompts import PromptTemplate<br>from langchain_openai import OpenAI<br>from ast import AST, FunctionDef, NodeTransformer, Constant, Expr<br>import ast<br><br>from utils import parse_src, read_src<br><br>def generate_doc_string(src_code: str) -&gt; str:<br>    api_key: str = os.environ[&quot;OPENAI_API_KEY&quot;]<br><br>    llm: BaseLLM = OpenAI(temperature=0, api_key=api_key)<br><br>    function_prompt_template: str = &quot;&quot;&quot;<br>    Generate python docstring for the given python function using the provided documentation style. <br>    Make sure to provide atleast two examples of the function usage only in the docstring as well <br>    as the exceptions that may be raised when using the function. Make sure to return the <br>    function and its docstring.<br>    Function code: {function_code}<br>    Documentation style: {documentation_style}<br>    &quot;&quot;&quot;<br>    <br>    documentation_style: str = &quot;Numpy-Style&quot;<br><br>    prompt = PromptTemplate.from_template(template=function_prompt_template)<br>    prompt_formatted_str: str = prompt.format(<br>        function_code=src_code, documentation_style=documentation_style<br>    )<br><br>    function_and_docstring: str = llm.invoke(prompt_formatted_str)<br>    return function_and_docstring<br><br><br>def parse_function_docstr(func_dcstr: str) -&gt; str:<br>    src_tree: AST = parse_src(func_dcstr)<br>    func_node: FunctionDef = src_tree.body[0]<br>    doc_str: str = ast.get_docstring(func_node)<br>    return doc_str<br><br><br>def make_docstring_node(docstr: str):<br>    constant_str: Constant = Constant(docstr)<br>    return Expr(value=constant_str)<br><br><br>class MyTransformer(NodeTransformer):<br>    def visit_FunctionDef(self, node: FunctionDef) -&gt; None:<br>        if node.name == &#39;add&#39;:<br>            src_code: str = ast.unparse(node)<br>            func_docstr: str = generate_doc_string(src_code)<br>            doc_str: str = parse_function_docstr(func_docstr.strip())<br>            dcstr_node: AST = make_docstring_node(doc_str)<br>            node.body.insert(0, dcstr_node)<br>        return node<br><br><br>def main():<br>    file_path: str = &quot;./docstring_generator/utils/functions.py&quot;<br>    src: str = read_src(file_path=file_path)<br>    src_tree: AST = parse_src(src)<br>    transformer: NodeTransformer = MyTransformer()<br>    new_tree = transformer.visit(src_tree)<br>    ast.fix_missing_locations(new_tree)<br>    new_module_code = ast.unparse(new_tree)<br>    print(new_module_code)<br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>And here is the output:</p><pre>import operator<br><br>def add(a: int | float, b: int | float) -&gt; int | float:<br>    &quot;&quot;&quot;Add two numbers together.<br><br>Parameters<br>----------<br>a : int | float<br>    First number to be added.<br>b : int | float<br>    Second number to be added.<br><br>Returns<br>-------<br>int | float<br>    Sum of the two numbers.<br><br>Examples<br>--------<br>&gt;&gt;&gt; add(2, 3)<br>5<br>&gt;&gt;&gt; add(2.5, 3.5)<br>6.0<br><br>Raises<br>------<br>TypeError<br>    If either a or b is not an int or float.&quot;&quot;&quot;<br>    return operator.add(a, b)<br><br>def subtract(a: int | float, b: int | float) -&gt; int | float:<br>    &quot;&quot;&quot;Subtracts two numbers<br><br>    Parameters<br>    ----------<br>    a : int or float<br>        The first number to subtract.<br>    b : int or float<br>        The second number to subtract.<br><br>    Returns<br>    -------<br>    int or float<br>        The result of subtracting b from a.<br>    &quot;&quot;&quot;<br>    return a - b</pre><p>And that works; as you can see, the add function gets its docstring generated, though its badly indented, an issue that we will have to address. We will also need to address functions with a lot of code, which might result in the model cutting off some of the output.</p><p>So far, we have a working prototype of the application we set out to build. We now need to put all the components together, something that we will address in the coming article. Save all the changes, push them to github and merge them into your main branch.</p><pre>git add .<br>git commit -m &quot;feat: uses chatgpt to generate the docstring.&quot;<br>git push -u origin chatgpt-docstring</pre><h4>Conclusion</h4><p>In this article:</p><ul><li>We used ast to parse a given file and obtain the source code for various functions.</li><li>Used the function source code to generate docstrings using openai’s chatgpt with the help of langchain</li><li>Parsed the response from openai to obtain the function docstring</li><li>Inserted the docstring into the function using the NodeTransformer class.</li></ul><p>In the next article, we will put all these components together into a fully functional application. Here is the <a href="https://github.com/twyle/docstring-generator">link to the project on github</a>. If you found the article useful, like it and share it out. Also due connect with me on <a href="https://www.linkedin.com/in/lyle-okoth/">linkedin</a>, <a href="https://twitter.com/LyleOkoth">twitter</a> and <a href="https://github.com/twyle">github</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c46eb34d24e9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating A Documentation Generator for Python Projects using GPT3: Part Two]]></title>
            <link>https://medium.com/@lyle-okoth/creating-a-documentation-generator-for-python-projects-using-gpt3-part-two-0cb22fd01a2a?source=rss-2637c671f6af------2</link>
            <guid isPermaLink="false">https://medium.com/p/0cb22fd01a2a</guid>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[visitor-pattern]]></category>
            <category><![CDATA[documentation]]></category>
            <category><![CDATA[composite-pattern]]></category>
            <dc:creator><![CDATA[Lyle Okoth]]></dc:creator>
            <pubDate>Thu, 08 Feb 2024 08:14:48 GMT</pubDate>
            <atom:updated>2024-02-08T08:14:48.015Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*jM2EjOaM-2MDG5o5rf6VDA.gif" /></figure><p>This is part two of a series of articles that take the reader through the process of creating a documentation generator for python projects. In the <a href="https://medium.com/@lyle-okoth/creating-a-documentation-generator-for-python-projects-using-gpt3-f78b63b835ec">previous article</a> we covered the following:</p><ul><li>Introduced the project</li><li>Created the initial layout based on a github template</li><li>Created a directory iterator that gives us access to all the python files in a given directory tree</li></ul><p>In this article, we will parse each one of those python files and generate dummy documentation for each python function, method and class. To keep these new changes separate from the previous changes, let us create a new branch called <em>code-parser:</em></p><pre>git checkout -b code-parser</pre><p>If you have not been following along, here is a link to the <a href="https://github.com/twyle/docstring-generator">github repo.</a> Check out the <em>directory-iterator </em>branch then create the<em> code-parser </em>branch<em>:</em></p><pre>git clone -b directory-iterator https://github.com/twyle/docstring-generator.git</pre><h3>How do we parse python code?</h3><p>For this, we will use the <em>ast</em> module provided by python. This module provides various utilities for parsing and modifying python source code. Of great interest to us is the <em>NodeTransformer</em> class. This class enables us to traverse python source code and update it. The python source code is parsed and represented as a tree with each node representing a different entity such as a function, class, variable e.t.c. What we are interested in is the function and class nodes. Before proceeding, let us look at an example source code representation by <em>ast. </em>Most of what I talk about will be a bit clearer after seeing this. Take the case of a file called <em>functions.py</em> with a single function called <em>add </em>that sums two integers or floats. These are just for demonstration purposes and should be kept separate from the project. But you can add them into the folder <em>docstring_generator/utils</em>:</p><pre>import operator<br><br>def add(a: int | float, b: int | float) -&gt; int | float:<br>  return operator.add(a,b)</pre><p>To highlight my points, there are a couple of functions in the docstring_generator/utils/utils.py that we will use. These include:</p><ul><li><em>read_src</em> for loading the source code for a given file</li><li><em>parse_src</em> for creating the source code tree with all the nodes</li><li><em>print_src</em> that prints out the parsed source code tree</li></ul><p>Here is the utils.py file with these various utilities:</p><pre>import ast<br>from ast import AST<br><br><br>def read_src(file_path: str) -&gt; str:<br>    with open(file_path, &#39;r&#39;, encoding=&#39;utf-8&#39;) as f:<br>        return f.read()<br>    <br>    <br>def parse_src(file_src: str) -&gt; AST:<br>    return ast.parse(file_src)<br><br>    <br>def print_src(src_tree: AST) -&gt; None:<br>    print(ast.dump(src_tree, indent=4))</pre><p>Then we will use the <em>main </em>function in the <em>main.py</em> file to print out the source code tree representation:</p><pre>from utils import parse_src, print_src, read_src<br>from ast import AST<br><br><br><br>def main(): <br>    file_path: str = &#39;./docstring_generator/utils/functions.py&#39;<br>    src: str = read_src(file_path=file_path)<br>    src_tree: AST = parse_src(src)<br>    print_src(src_tree)<br><br>if __name__ == &#39;__main__&#39;:<br>    main()</pre><p>Now, from the project root, execute the <em>main.py</em> file in the <em>docstring_generator/utils </em>folder:</p><pre>python docstring_generator/utils/main.py</pre><p>This is the output is:</p><pre>Module(<br>    body=[<br>        Import(<br>            names=[<br>                alias(name=&#39;operator&#39;)]),<br>        FunctionDef(<br>            name=&#39;add&#39;,<br>            args=arguments(<br>                posonlyargs=[],<br>                args=[<br>                    arg(<br>                        arg=&#39;a&#39;,<br>                        annotation=BinOp(<br>                            left=Name(id=&#39;int&#39;, ctx=Load()),<br>                            op=BitOr(),<br>                            right=Name(id=&#39;float&#39;, ctx=Load()))),<br>                    arg(<br>                        arg=&#39;b&#39;,<br>                        annotation=BinOp(<br>                            left=Name(id=&#39;int&#39;, ctx=Load()),<br>                            op=BitOr(),<br>                            right=Name(id=&#39;float&#39;, ctx=Load())))],<br>                kwonlyargs=[],<br>                kw_defaults=[],<br>                defaults=[]),<br>            body=[<br>                Return(<br>                    value=Call(<br>                        func=Attribute(<br>                            value=Name(id=&#39;operator&#39;, ctx=Load()),<br>                            attr=&#39;add&#39;,<br>                            ctx=Load()),<br>                        args=[<br>                            Name(id=&#39;a&#39;, ctx=Load()),<br>                            Name(id=&#39;b&#39;, ctx=Load())],<br>                        keywords=[]))],<br>            decorator_list=[],<br>            returns=BinOp(<br>                left=Name(id=&#39;int&#39;, ctx=Load()),<br>                op=BitOr(),<br>                right=Name(id=&#39;float&#39;, ctx=Load())))],<br>    type_ignores=[])</pre><h3>A brief overview of the source code tree</h3><p>As shown above, the source code is represented as a tree. The various nodes are found in the body and these nodes have thier own children which in turn have their own children. Some of the nodes shown include:</p><ul><li><em>Import</em>: This simple represnts an import statement</li><li><em>FunctionDef:</em> This represnts a function defination, with its children describing the functions inputs, outputs, documentation, operations e.t.c</li></ul><p>There are many many other nodes not shown here including <em>ClassDef</em>. Since the source code is represented as a tree, that tells us that we can traverse the tree in various ways including:</p><ul><li>Pre-order traversal (visit the parent, then the left subtree and finally the right subtree)</li><li>Post-order traversal (visit the left and right subtree, then visit the parent)</li><li>In-order traversal (visit the left subtree, then the parent, then the right subtree)</li></ul><p>Let us see the case for a <em>pre-order traversal</em>. For this, ast provides us with the <em>NodeVisitor</em> class which we shoul subclass and implement its various methods.</p><h3>The visitor pattern</h3><p>The <em>NodeVisitor</em> class uses the <em>visitor pattern</em> to traverse the source code tree. The visitor pattern separates objects from the algorithms that work on those objects. You pass it an object, which most of the times is a <em>composite</em>.</p><h4>The composite pattern</h4><p>A composite is an object composed of objects of the same types. The simplest example is a filesystem. A filesystem consists of <em>folders</em> and <em>files</em>. A folder can contain other <em>folders</em> as well as <em>files</em>. Here is a simple example of a composite object. The base object is the <em>Component</em>, which defines the composite interface. The <em>Leaf</em> object is the the lowest level of the composite object and can be found within the Folder object. This code is found in the <em>docstring_generator/utils/composite.py</em> file:</p><pre>from __future__ import annotations<br><br>from abc import ABC, abstractmethod<br><br><br>class Component(ABC):<br>    def __init__(self, name: str) -&gt; None:<br>        super().__init__()<br>        self._name = name<br><br>    @property<br>    def name(self) -&gt; Component:<br>        return self._name<br><br>    @name.setter<br>    def name(self, name: str) -&gt; None:<br>        self._name = name<br><br>    @property<br>    def parent(self) -&gt; Component:<br>        return self._parent<br><br>    @parent.setter<br>    def parent(self, parent: Component):<br>        self._parent = parent<br><br>    def add(self, component: Component) -&gt; None:<br>        pass<br><br>    def remove(self, component: Component) -&gt; None:<br>        pass<br><br>    def is_composite(self) -&gt; bool:<br>        return False<br><br>    @abstractmethod<br>    def operation(self) -&gt; str:<br>        pass<br><br><br>class File(Component):<br>    def __init__(self, name: str) -&gt; None:<br>        super().__init__(name)<br><br>    def operation(self) -&gt; str:<br>        return self.name<br><br><br>class Folder(Component):<br>    def __init__(self, name: str, children: list[Component] = []) -&gt; None:<br>        super().__init__(name=name)<br>        self._children: list[Component] = children<br><br>    def add(self, component: Component) -&gt; None:<br>        self._children.append(component)<br>        component.parent = self<br><br>    def remove(self, component: Component) -&gt; None:<br>        self._children.remove(component)<br>        component.parent = None<br><br>    def is_composite(self) -&gt; bool:<br>        return True<br><br>    def operation(self) -&gt; str:<br>        results = []<br>        for child in self._children:<br>            results.append(child.operation())<br>        return f&quot;{self.name}: ({&#39;+&#39;.join(results)})&quot;<br><br><br>def main():<br>    children: list[Component] = [<br>        Folder(<br>            name=&quot;utils&quot;, children=[File(name=&quot;classes.py&quot;), File(name=&quot;functions.py&quot;)]<br>        ),<br>        File(name=&quot;__main__.py&quot;),<br>        File(name=&quot;config.py&quot;),<br>        File(name=&quot;helpers.py&quot;),<br>        File(name=&quot;templates.py&quot;),<br>    ]<br>    root_folder: Folder = Folder(name=&quot;docstring_generator&quot;, children=children)<br>    print(root_folder.operation())<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>Execute it from the project root:</p><pre>python docstring_generator/utils/composite.py</pre><p>Simply prints out the folders and the files in each folder.</p><h4>Visitor pattern and Composite pattern</h4><p>Now say for each component of the composite tree (File, Folder), we wanted to do something with that component, how would we approach that situation? Well, that is what the visitor pattern is for. We implement a method called <em>visit_File</em> and <em>visit_Folder</em>, each which takes as a parametr the <em>File</em> or <em>Folder</em> and does something with it. Then we modify the <em>Component</em> class to declare a method called <em>accept</em> that accepts a <em>Visitor</em> and calls the appropriate method, for example <em>visit_File</em> will accept a <em>Visitor</em> and call the <em>visit_File</em> method of the <em>Visitor</em>. Here is the visitor code, found in <em>docstring_generator/utils/visitor.py.</em></p><pre>from __future__ import annotations<br><br>from abc import ABC, abstractmethod<br><br>from composite import File, Folder<br><br><br>class Visitor(ABC):<br>    def visit_File(self, file: File) -&gt; None:<br>        print(f&#39;visiting the file called &quot;{file.name}&quot;&#39;)<br><br>    def visit_Folder(self, folder: Folder) -&gt; None:<br>        print(f&#39;visiting the folder called &quot;{folder.name}&quot;&#39;)</pre><p>Each node of the composite object has an <em>accept</em> method that takes in a V<em>isitor</em> and calls the appropriate method. Here is the modified <em>Component</em>, found in the file <em>docstring_generator/utils/composite_visitor.py</em> file:</p><pre>from __future__ import annotations<br><br>from abc import ABC, abstractmethod<br><br>from visitor import Visitor<br><br><br>class Component(ABC):<br>    def __init__(self, name: str) -&gt; None:<br>        super().__init__()<br>        self._name = name<br><br>    @abstractmethod<br>    def accept(self, visitor: Visitor) -&gt; None:<br>        pass<br><br>    @property<br>    def name(self) -&gt; Component:<br>        return self._name<br><br>    @name.setter<br>    def name(self, name: str) -&gt; None:<br>        self._name = name<br><br>    @property<br>    def parent(self) -&gt; Component:<br>        return self._parent<br><br>    @parent.setter<br>    def parent(self, parent: Component):<br>        self._parent = parent<br><br>    def add(self, component: Component) -&gt; None:<br>        pass<br><br>    def remove(self, component: Component) -&gt; None:<br>        pass<br><br>    def is_composite(self) -&gt; bool:<br>        return False<br><br>    @abstractmethod<br>    def operation(self) -&gt; str:<br>        pass<br><br><br>class File(Component):<br>    def __init__(self, name: str) -&gt; None:<br>        super().__init__(name)<br><br>    def operation(self) -&gt; str:<br>        return self.name<br><br>    def accept(self, visitor: Visitor) -&gt; None:<br>        visitor.visit_File(self)<br><br><br>class Folder(Component):<br>    def __init__(self, name: str, children: list[Component] = []) -&gt; None:<br>        super().__init__(name=name)<br>        self._children: list[Component] = children<br><br>    def add(self, component: Component) -&gt; None:<br>        self._children.append(component)<br>        component.parent = self<br><br>    def remove(self, component: Component) -&gt; None:<br>        self._children.remove(component)<br>        component.parent = None<br><br>    def is_composite(self) -&gt; bool:<br>        return True<br><br>    def operation(self) -&gt; str:<br>        results = []<br>        for child in self._children:<br>            results.append(child.operation())<br>        return f&quot;{self.name}: ({&#39;+&#39;.join(results)})&quot;<br><br>    def accept(self, visitor: Visitor) -&gt; None:<br>        visitor.visit_Folder(self)<br>        for child in self._children:<br>            child.accept(visitor)<br><br><br>def main():<br>    children: list[Component] = [<br>        Folder(<br>            name=&quot;utils&quot;, children=[File(name=&quot;classes.py&quot;), File(name=&quot;functions.py&quot;)]<br>        ),<br>        File(name=&quot;__main__.py&quot;),<br>        File(name=&quot;config.py&quot;),<br>        File(name=&quot;helpers.py&quot;),<br>        File(name=&quot;templates.py&quot;),<br>    ]<br>    root_folder: Folder = Folder(name=&quot;docstring_generator&quot;, children=children)<br>    visitor: Visitor = Visitor()<br>    root_folder.accept(visitor)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>Here is the output:</p><pre>visiting the folder called &quot;docstring_generator&quot;<br>visiting the folder called &quot;utils&quot;<br>visiting the file called &quot;classes.py&quot;<br>visiting the file called &quot;functions.py&quot;<br>visiting the file called &quot;__main__.py&quot;<br>visiting the file called &quot;config.py&quot;<br>visiting the file called &quot;helpers.py&quot;<br>visiting the file called &quot;templates.py&quot;</pre><p>The <em>accept</em> method for a <em>Folder</em> first visits the folder, then its children.</p><h4>Generic visit for the ast tree</h4><p>Let us start by implementing the default method called <em>generic_visit. </em>In the<em> docstring_generator/util/generic_visit.py file:</em></p><pre>import ast<br>from ast import AST<br>from utils import parse_src, read_src<br><br>class MyVisitor(ast.NodeVisitor):<br>    def generic_visit(self, node: AST) -&gt; None:<br>        print(f&#39;entering {node.__class__.__name__}&#39;)<br>        super().generic_visit(node)<br>        <br>def main():<br>    file_path: str = &quot;./docstring_generator/utils/functions.py&quot;<br>    src: str = read_src(file_path=file_path)<br>    src_tree: AST = parse_src(src)<br>    visitor = MyVisitor()<br>    visitor.visit(src_tree)<br>    <br>if __name__ == &#39;__main__&#39;:<br>    main()</pre><p>Here is the output:</p><pre>entering Module<br>entering Import<br>entering alias<br>entering FunctionDef<br>entering arguments<br>entering arg<br>entering BinOp<br>entering Name<br>entering Load<br>entering BitOr<br>entering Name<br>entering Load<br>entering arg<br>entering BinOp<br>entering Name<br>entering Load<br>entering BitOr<br>entering Name<br>entering Load<br>entering Return<br>entering Call<br>entering Attribute<br>entering Name<br>entering Load<br>entering Load<br>entering Name<br>entering Load<br>entering Name<br>entering Load<br>entering BinOp<br>entering Name<br>entering Load<br>entering BitOr<br>entering Name<br>entering Load</pre><p>It simply uses <em>pre-order</em> traversal to to walk through the source code tree.</p><h4>Visiting Function nodes</h4><p>Let us now create a vistor that only visits the function nodes(<em>docstring_generator/utils/visit_functions.py</em>):</p><pre>import ast<br>from _ast import FunctionDef<br>from ast import AST<br>from typing import Any<br><br>from utils import parse_src, read_src<br><br><br>class MyVisitor(ast.NodeVisitor):<br>    def visit_FunctionDef(self, node: FunctionDef) -&gt; Any:<br>        print(f&quot;entering {node.name}&quot;)<br><br><br>def main():<br>    file_path: str = &quot;./docstring_generator/utils/functions.py&quot;<br>    src: str = read_src(file_path=file_path)<br>    src_tree: AST = parse_src(src)<br>    visitor = MyVisitor()<br>    visitor.visit(src_tree)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>This one simply prints the name of the function. Let us now get the functions docstrings. Add the function called subtract with its docstrings to the <em>functions.py</em> file:</p><pre>def subtract(a: int | float, b: int | float) -&gt; int | float:<br>    &quot;&quot;&quot;Subtracts two numbers<br><br>    Parameters<br>    ----------<br>    a : int or float<br>        The first number to subtract.<br>    b : int or float<br>        The second number to subtract.<br><br>    Returns<br>    -------<br>    int or float<br>        The result of subtracting b from a.<br>    &quot;&quot;&quot;<br>    return a - b</pre><p>Then create a new visitor that will print out the function docstring. ast provides us with a utility function for getting the function docstring(<em>get_function_docstring.py</em>):</p><pre>import ast<br>from _ast import FunctionDef<br>from ast import AST<br>from typing import Any<br><br>from utils import parse_src, read_src<br><br><br>class MyVisitor(ast.NodeVisitor):<br>    def visit_FunctionDef(self, node: FunctionDef) -&gt; Any:<br>        print(f&quot;entering {node.name}&quot;)<br>        docstring: str = ast.get_docstring(node=node)<br>        if docstring:<br>            print(f&#39;The function &quot;{node.name}&quot; has docstring.&#39;)<br>            print(docstring)<br>        else:<br>            print(f&#39;The function &quot;{node.name}&quot; has no docstring.&#39;)<br><br><br>def main():<br>    file_path: str = &quot;./docstring_generator/utils/functions.py&quot;<br>    src: str = read_src(file_path=file_path)<br>    src_tree: AST = parse_src(src)<br>    visitor = MyVisitor()<br>    visitor.visit(src_tree)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>Here is the output:</p><pre>entering add<br>The function &quot;add&quot; has no docstring.<br>entering subtract<br>The function &quot;subtract&quot; has docstring.<br>Subtracts two numbers<br><br>Parameters<br>----------<br>a : int or float<br>    The first number to subtract.<br>b : int or float<br>    The second number to subtract.<br><br>Returns<br>-------<br>int or float<br>    The result of subtracting b from a.</pre><h4>Transforming Nodes</h4><p>In the previous sections, we’ve simply traversed the various nodes. However our application requires us to alter those nodes, specifically add docstrings. For this, we will use another utility class provided by <em>ast</em> called the <em>NodeTransformer</em> that allows us to alter nodes in place. It is similar to the <em>NodeVisitor</em> class but returns the transformed node.</p><p>First, let us look at how the docstring node looks like:</p><pre>python docstring_generator/utils/main.py</pre><pre>Module(<br>    body=[<br>        Import(<br>            names=[<br>                alias(name=&#39;operator&#39;)]),<br>        FunctionDef(<br>            name=&#39;add&#39;,<br>            args=arguments(<br>                posonlyargs=[],<br>                args=[<br>                    arg(<br>                        arg=&#39;a&#39;,<br>                        annotation=BinOp(<br>                            left=Name(id=&#39;int&#39;, ctx=Load()),<br>                            op=BitOr(),<br>                            right=Name(id=&#39;float&#39;, ctx=Load()))),<br>                    arg(<br>                        arg=&#39;b&#39;,<br>                        annotation=BinOp(<br>                            left=Name(id=&#39;int&#39;, ctx=Load()),<br>                            op=BitOr(),<br>                            right=Name(id=&#39;float&#39;, ctx=Load())))],<br>                kwonlyargs=[],<br>                kw_defaults=[],<br>                defaults=[]),<br>            body=[<br>                Return(<br>                    value=Call(<br>                        func=Attribute(<br>                            value=Name(id=&#39;operator&#39;, ctx=Load()),<br>                            attr=&#39;add&#39;,<br>                            ctx=Load()),<br>                        args=[<br>                            Name(id=&#39;a&#39;, ctx=Load()),<br>                            Name(id=&#39;b&#39;, ctx=Load())],<br>                        keywords=[]))],<br>            decorator_list=[],<br>            returns=BinOp(<br>                left=Name(id=&#39;int&#39;, ctx=Load()),<br>                op=BitOr(),<br>                right=Name(id=&#39;float&#39;, ctx=Load()))),<br>        FunctionDef(<br>            name=&#39;subtract&#39;,<br>            args=arguments(<br>                posonlyargs=[],<br>                args=[<br>                    arg(<br>                        arg=&#39;a&#39;,<br>                        annotation=BinOp(<br>                            left=Name(id=&#39;int&#39;, ctx=Load()),<br>                            op=BitOr(),<br>                            right=Name(id=&#39;float&#39;, ctx=Load()))),<br>                    arg(<br>                        arg=&#39;b&#39;,<br>                        annotation=BinOp(<br>                            left=Name(id=&#39;int&#39;, ctx=Load()),<br>                            op=BitOr(),<br>                            right=Name(id=&#39;float&#39;, ctx=Load())))],<br>                kwonlyargs=[],<br>                kw_defaults=[],<br>                defaults=[]),<br>            body=[<br>                Expr(<br>                    value=Constant(value=&#39;Subtracts two numbers\n\n    Parameters\n    ----------\n    a : int or float\n        The first number to subtract.\n    b : int or float\n        The second number to subtract.\n\n    Returns\n    -------\n    int or float\n        The result of subtracting b from a.\n    &#39;)),<br>                Return(<br>                    value=BinOp(<br>                        left=Name(id=&#39;a&#39;, ctx=Load()),<br>                        op=Sub(),<br>                        right=Name(id=&#39;b&#39;, ctx=Load())))],<br>            decorator_list=[],<br>            returns=BinOp(<br>                left=Name(id=&#39;int&#39;, ctx=Load()),<br>                op=BitOr(),<br>                right=Name(id=&#39;float&#39;, ctx=Load())))],<br>    type_ignores=[])</pre><p>Here is the docstring node:</p><pre>Expr(<br>                    value=Constant(value=&#39;Subtracts two numbers\n\n    Parameters\n    ----------\n    a : int or float\n        The first number to subtract.\n    b : int or float\n        The second number to subtract.\n\n    Returns\n    -------\n    int or float\n        The result of subtracting b from a.\n    &#39;))</pre><p>It’s the very first node in a <em>FunctionDef</em>, and is absent for functions without docstrings. It is an <em>Expr</em> node with a <em>Constant</em> node whose value is a string.</p><h4>Creating and adding a Docstring Node</h4><p>Let us use the <em>NodeTransformer</em> class to add docstrings to the <em>add</em> function. We create a function for generating a dummy docstring for the <em>add</em> function, then another function for creating a docstring node given the docstring then use the <em>NodeTransfromer</em> subclass to add the docstring node, only for the add function. We then print out the new source code. (generate_docstring/utils/add_dummy_docstring.py):</p><pre>import ast<br>from _ast import FunctionDef<br>from ast import AST, Constant, Expr, NodeTransformer<br>from typing import Any<br><br>from utils import parse_src, read_src<br><br>function_dcstr: str = &quot;&quot;&quot;<br>    Add two numbers.<br><br>    Parameters<br>    ----------<br>    a : int or float<br>        First number to be added.<br>    b : int or float<br>        Second number to be added.<br><br>    Returns<br>    -------<br>    int or float<br>        Sum of a and b.<br>    &quot;&quot;&quot;<br><br><br>def get_function_docstring() -&gt; str:<br>    return function_dcstr<br><br><br>def make_docstring_node(docstr: str):<br>    constant_str: Constant = Constant(docstr)<br>    return Expr(value=constant_str)<br><br><br>class MyTransformer(NodeTransformer):<br>    def visit_FunctionDef(self, node: FunctionDef) -&gt; FunctionDef:<br>        if node.name == &quot;add&quot;:<br>            add_dcstr: str = get_function_docstring()<br>            dcstr_node: AST = make_docstring_node(add_dcstr)<br>            node.body.insert(0, dcstr_node)<br>        return node<br><br><br>def main():<br>    file_path: str = &quot;./docstring_generator/utils/functions.py&quot;<br>    src: str = read_src(file_path=file_path)<br>    src_tree: AST = parse_src(src)<br>    transformer: NodeTransformer = MyTransformer()<br>    new_tree = transformer.visit(src_tree)<br>    ast.fix_missing_locations(new_tree)<br>    new_module_code = ast.unparse(new_tree)<br>    print(new_module_code)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>Here is the output:</p><pre>import operator<br><br>def add(a: int | float, b: int | float) -&gt; int | float:<br>    &quot;&quot;&quot;<br>    Add two numbers.<br><br>    Parameters<br>    ----------<br>    a : int or float<br>        First number to be added.<br>    b : int or float<br>        Second number to be added.<br><br>    Returns<br>    -------<br>    int or float<br>        Sum of a and b.<br>    &quot;&quot;&quot;<br>    return operator.add(a, b)<br><br>def subtract(a: int | float, b: int | float) -&gt; int | float:<br>    &quot;&quot;&quot;Subtracts two numbers<br><br>    Parameters<br>    ----------<br>    a : int or float<br>        The first number to subtract.<br>    b : int or float<br>        The second number to subtract.<br><br>    Returns<br>    -------<br>    int or float<br>        The result of subtracting b from a.<br>    &quot;&quot;&quot;<br>    return a - b</pre><h4>Handling existing docstrings</h4><p>Existing docstrings can either be skipped or replaced. Let us replace existing docstring(<em>replace_docstring.py</em>):</p><pre>import ast<br>from _ast import FunctionDef<br>from ast import AST, Constant, Expr, NodeTransformer<br>from typing import Any<br><br>from utils import parse_src, read_src<br><br>function_dcstr: str = &quot;&quot;&quot;<br>    Add two numbers.<br><br>    Parameters<br>    ----------<br>    a : int or float<br>        First number to be added.<br>    b : int or float<br>        Second number to be added.<br><br>    Returns<br>    -------<br>    int or float<br>        Sum of a and b.<br>    &quot;&quot;&quot;<br><br><br>def get_function_docstring() -&gt; str:<br>    return function_dcstr<br><br><br>def make_docstring_node(docstr: str):<br>    constant_str: Constant = Constant(docstr)<br>    return Expr(value=constant_str)<br><br><br>class MyTransformer(NodeTransformer):<br>    def visit_FunctionDef(self, node: FunctionDef) -&gt; FunctionDef:<br>        docstr: str = ast.get_docstring(node=node)<br>        add_dcstr: str = get_function_docstring()<br>        dcstr_node: AST = make_docstring_node(add_dcstr)<br>        if docstr:<br>            node.body[0] = dcstr_node<br>        else:<br>            node.body.insert(0, dcstr_node)<br>        return node<br><br><br>def main():<br>    file_path: str = &quot;./docstring_generator/utils/functions.py&quot;<br>    src: str = read_src(file_path=file_path)<br>    src_tree: AST = parse_src(src)<br>    transformer: NodeTransformer = MyTransformer()<br>    new_tree = transformer.visit(src_tree)<br>    ast.fix_missing_locations(new_tree)<br>    new_module_code = ast.unparse(new_tree)<br>    print(new_module_code)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>Here is the output:</p><pre>import operator<br><br>def add(a: int | float, b: int | float) -&gt; int | float:<br>    &quot;&quot;&quot;<br>    Add two numbers.<br><br>    Parameters<br>    ----------<br>    a : int or float<br>        First number to be added.<br>    b : int or float<br>        Second number to be added.<br><br>    Returns<br>    -------<br>    int or float<br>        Sum of a and b.<br>    &quot;&quot;&quot;<br>    return operator.add(a, b)<br><br>def subtract(a: int | float, b: int | float) -&gt; int | float:<br>    &quot;&quot;&quot;<br>    Add two numbers.<br><br>    Parameters<br>    ----------<br>    a : int or float<br>        First number to be added.<br>    b : int or float<br>        Second number to be added.<br><br>    Returns<br>    -------<br>    int or float<br>        Sum of a and b.<br>    &quot;&quot;&quot;<br>    return a - b</pre><p>Both functions now have the add docstring.</p><p>Finally, add all the new changes and commit them to github.</p><pre>git add .<br>git commit -m &quot;feat: parses te source code and adds dummy docstring.&quot;<br># you may have to run the two commands a couple of times<br>git push -u origin code-parser<br></pre><h3>Conclusion</h3><p>Since this article has gotten very long, I will end it here. We have covered the following today:</p><ul><li>Parsing python source code into a tree structure</li><li>Learnt how to traverse the parsed source code tree structure</li><li>Gone through the composite design pattern and how to create one</li><li>Gone through the visitor pattern and how to create one</li><li>Used the visitor pattern to operate on a composite object</li><li>Traversed a function defination to obtain the docstring</li><li>Inserted and replaced the docstring for a given function.</li></ul><p>In the next article, we will use langchain to access openai and generate the docstrings.Here is the <a href="https://github.com/twyle/docstring-generator">link to the project on github</a>. If you found the article useful, like it and share it out. Also due connect with me on <a href="https://www.linkedin.com/in/lyle-okoth/">linkedin</a>, <a href="https://twitter.com/LyleOkoth">twitter</a> and <a href="https://github.com/twyle">github</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=0cb22fd01a2a" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Creating A Documentation Generator for Python Projects using GPT3]]></title>
            <link>https://blog.gopenai.com/creating-a-documentation-generator-for-python-projects-using-gpt3-f78b63b835ec?source=rss-2637c671f6af------2</link>
            <guid isPermaLink="false">https://medium.com/p/f78b63b835ec</guid>
            <category><![CDATA[gpt-3]]></category>
            <category><![CDATA[langchain]]></category>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[large-language-models]]></category>
            <dc:creator><![CDATA[Lyle Okoth]]></dc:creator>
            <pubDate>Wed, 07 Feb 2024 18:44:41 GMT</pubDate>
            <atom:updated>2024-02-09T11:03:08.273Z</atom:updated>
            <content:encoded><![CDATA[<p>In these series of articles, I will take you through the process of creating a python package that generates docstrings for a python project. In simpler terms, using the package, you will be able to generate docstrings for python functions, classes and class methods.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*jM2EjOaM-2MDG5o5rf6VDA.gif" /></figure><h3>What you will learn.</h3><p>Along the way, you will learn how to:</p><ul><li>Structure, build and upload a python package to pypi.</li><li>Create and use custom iterators to traverse the file system.</li><li>Parse and alter python source code</li><li>Use langchain to generate documentation using openai’s gpt3</li></ul><h3>What are Docstrings?</h3><p>So, how is the dosctsing generator going to work? And how are we going to build it? For a start, docstrings are these pieces of text that appear after a function, methods and class declaration that explain to the user what the class, method or function does as well as the inputs, outputs and errors that that may be raised when using them. They can also include example uses.</p><h3>Function Dosctrings</h3><p>Take the case of a function that adds two numbers:</p><pre>def add(a: int, b: int) -&gt; int:<br>  return a + b</pre><p>The functions docstring would look like this (numpy-style):</p><pre>def add(a, b):<br>  &quot;&quot;&quot;<br>  Add two numbers.<br>  <br>  Parameters<br>  ----------<br>  a : int or float<br>      First number to be added.<br>  b : int or float<br>      Second number to be added.<br>  <br>  Returns<br>  -------<br>  int or float<br>      Sum of a and b.<br>  &quot;&quot;&quot;<br>  return a + b</pre><h3>Class Docstrings</h3><p>For a class like:</p><pre>import math<br><br>class TestClass:<br>  def __init__(self, radius: int | float) -&gt; None:<br>    self._radius = radius<br><br>  def calculate_area(self) -&gt; float:<br>    return math.pi * self._radius * self._radius<br><br>  def calculate_perimeter(self) -&gt; float:<br>    return 2 * math.pi * self._radius</pre><p>The docstring would be:</p><pre>import math<br><br><br>class MyTestClass:<br>    &quot;&quot;&quot;Class representing a circle with a given radius.<br><br>    Parameters<br>    ----------<br>    radius : int or float<br>        The radius of the circle.<br><br>    Attributes<br>    ----------<br>    radius : int or float<br>        The radius of the circle.<br><br>    Methods<br>    -------<br>    calculate_area()<br>        Calculates the area of the circle.<br>    calculate_perimeter()<br>        Calculates the perimeter of the circle.&quot;&quot;&quot;<br><br>    def __init__(self, radius: int | float) -&gt; None:<br>        &quot;&quot;&quot;Initializes a new instance of MyTestClass.<br><br>        Parameters<br>        ----------<br>        radius : int or float<br>            The radius of the circle.&quot;&quot;&quot;<br>        self.radius = radius<br><br>    def calculate_area(self) -&gt; float:<br>        &quot;&quot;&quot;Calculates the area of the circle.<br><br>        Returns<br>        -------<br>        float<br>            The area of the circle.&quot;&quot;&quot;<br>        return math.pi * self.radius * self.radius<br><br>    def calculate_perimeter(self) -&gt; float:<br>        &quot;&quot;&quot;Calculates the perimeter of the circle.<br><br>        Returns<br>        -------<br>        float<br>            The perimeter of the circle.&quot;&quot;&quot;<br>        return 2 * math.pi * self.radius</pre><h3>What are we going to build?</h3><p>That is what we aim to build. A python package that when run from the commandline, will iterate through a folder and its subfolders and generate docstrings for each python file in each subfolder. To use it we will provide it a couple of arguments such as:</p><ul><li>The project folder</li><li>The subfolders to avoid</li><li>The files to avoid</li><li>Whether or not to override existing docstrings</li><li>The documentation style, which could be Google-style, Numpy style or even Sphinx style.</li></ul><h3>How are we going to use it?</h3><p>We will package the python application and upload it to pypi. To use it, you will need to install it from pypi, then run it as a commandline application and providing the parameters mentioned above. i.e</p><pre>pip install awesome-docstring-package # install the package<br>python -m awesome_docstring_package --path /home/lyle/tutorial/test-docstrings-generator/test/ --directories-ignore venv<br>--files-ignore test.py --overwrite-function-docstring True --overwrite-class-docstring True --overwrite-class-methods-docstring True --documentation-style Numpy-Style</pre><h3>How does it work?</h3><p>Given a directory, the application starts by finding all the python files in the directory and its subdirectories. Each files’ source code is parsed using the ast module to find the functions and classes. For each function and class, a prompt is generated and sent to openai that requests it to generate the docstring. The generated docstring is then used to fill in the dosctring, and finally the file is saved.</p><h3>Let us build it</h3><p>In this first part, we will set up the project as well as build an iterator for traversing the file system. The iterator will give us back the python files for the project directory. To follow along, start by cloning the starter template. We will modify this along the way. Here is the <a href="https://github.com/twyle/docstring-generator">starter template on github</a>, clone the starter-template branch.</p><pre>git clone -b starter-template https://github.com/twyle/docstring-generator.git</pre><p>Here is the initial project layout:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*pJN8_3T2VexA2aWYelSkJQ.png" /></figure><p>Set up the development environment, by running the Makefile, which will install the following packages:</p><ul><li>black</li><li>commitizen</li><li>flake8</li><li>isort</li><li>pre-commit</li><li>pytest</li><li>recommonmark</li><li>setuptools</li><li>sphinx</li><li>sphinx_rtd_theme</li><li>twine</li><li>wheel</li></ul><p>Then create a virtual environment:</p><pre>python3 -m venv venv<br>source venv/bin/activate</pre><p>Black, commitizen, flake8, isort and pre-commit will help us format our files. We will discuss the rest later. For commitizen, follow the steps shown by the gif shown below:</p><pre>make install-dev-reqs # install the development requirements<br>make pre-commit # install pre-commit<br>make initial-tag # create the initial tag v0.0.1<br>make init-cz # initialize commitizen</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*5ZqWMvBrOCNlWt7wrmd0XQ.gif" /></figure><h3>Creating the directory iterator</h3><p>An iterator enables us to traverse or move through a given data structure without knowing about how the underlying data structure works. Think of using a for loop to move through a list. In our case, we need to traverse the filesystem for a given directory. We could use os.walk, but for our case, we want more control, so we will build an iterator from scratch. Our iterator is relatively simple:</p><ul><li>It will take in a directory as the input as well as a set of directories and files to avoid.</li><li>It will loop through each subdirectory and at each step, yield the files for the current directory</li><li>It will use a queue to iterate through the file system, and this will be done top down, from the root directory to the very last subdirectory</li><li>At each directory, we will list the contents but only yield a list of files and add the directories to the queue. This will go on until we run out of subdirectories.</li><li>For the iterator, we will inherit from <em>collections.abc.Iterator</em> and implement the __next__ and __iter__ methods.</li></ul><p>Check out a new branch called <em>directory-iterator</em> to save these new features.</p><pre>git checkout -b directory-iterator</pre><p>Within the <em>dosctring/helpers.py</em> file add the following code:</p><pre>from collections import deque<br>from collections.abc import Iterator<br>from os import listdir, path<br><br><br>class DirectoryIterator(Iterator):<br>    def __init__(<br>        self,<br>        root_directory: str,<br>        folders_ignore: list[str] = [],<br>        files_ignore: list[str] = [],<br>    ) -&gt; None:<br>        super().__init__()<br>        if not path.exists(root_directory):<br>            raise ValueError(f&#39;No such directory or file &quot;{root_directory}&quot; exists.&#39;)<br>        if path.isfile(root_directory):<br>            raise ValueError(&quot;The root directory should be a file.&quot;)<br>        self._folders_ignore = set(folders_ignore) <br>        self._files_ignore = set(files_ignore)<br>        self._queue = deque(root_directory)<br><br>    def __iter__(self) -&gt; Iterator:<br>        return super().__iter__()<br><br>    def __next__(self) -&gt; list[str]:<br>        if self._queue:<br>            files: list[str] = list()<br>            for _ in range(len(self._queue)):<br>                directory: str = self._queue.popleft()<br>                for entry in listdir(directory):<br>                    entry_path: str = path.join(directory, entry)<br>                    if (<br>                        path.isfile(entry_path)<br>                        and self._is_python_file(entry_path)<br>                        and entry not in self._files_ignore<br>                    ):<br>                        files.append(entry_path)<br>                    elif (<br>                        path.isdir(entry_path)<br>                        and entry not in self._folders_ignore<br>                    ):<br>                        self._queue.append(entry_path)<br>            return files<br>        else:<br>            raise StopIteration()<br><br>    def _is_python_file(self, file_path: str) -&gt; bool:<br>        return file_path.split(&#39;.&#39;)[-1] == &quot;py&quot;</pre><p>To test it out, add the following code to the <em>docstring_generator/__main__.py file</em>:</p><pre>from .helpers import DirectoryIterator<br><br><br>def main():<br>    root_dir: str = &quot;.&quot;<br>    folders_ignore: list[str] = [&quot;venv&quot;, &quot;__pycache__&quot;, &#39;.git&#39;, &#39;.venv&#39;]<br>    directory_iterator: DirectoryIterator = DirectoryIterator(<br>        root_directory=root_dir, folders_ignore=folders_ignore<br>    )<br>    for files in directory_iterator:<br>        print(files)<br>        <br>if __name__ == &#39;__main__&#39;:<br>    main()</pre><p>Then run the file:</p><pre>python -m docstring_generator</pre><p>With this, we have an iterator that can get us all the python files in a given file system while ignoring the files that we do not want. To tidy up this code, let us create a configuration object that can keep track of the various options that we can pass to this iterator. We will need pydantic for this. So install it and add it to the requirements.txt file:</p><pre>pip install pydantic</pre><p>In the <em>doctring_generator/config.py</em> file add the following:</p><pre>from pydantic import BaseModel, Field<br><br><br>class Config(BaseModel):<br>    root_directory: str = Field(description=&#39;The path to the source code directory&#39;)<br>    folders_ignore: list[str] = Field(<br>        description=&#39;Directories to ignore&#39;,<br>        default=[&#39;venv&#39;, &#39;.venv&#39;, &#39;__pycache__&#39;, &#39;.git&#39;, &#39;build&#39;, &#39;dist&#39;, &#39;docs&#39;],<br>    )<br>    files_ignore: list[str] = Field(<br>        description=&#39;Files to ignore&#39;,<br>        default_factory=list,<br>    )</pre><p>The new directory iterator looks like this:</p><pre>from collections import deque<br>from collections.abc import Iterator<br>from os import listdir, path<br><br>from .config import Config<br><br><br>class DirectoryIterator(Iterator):<br>    def __init__(self, config: Config) -&gt; None:<br>        super().__init__()<br>        if not path.exists(config.root_directory):<br>            raise ValueError(<br>                f&#39;No such directory or file &quot;{config.root_directory}&quot; exists.&#39;<br>            )<br>        if path.isfile(config.root_directory):<br>            raise ValueError(&quot;The root directory should be a file.&quot;)<br>        self._folders_ignore = set(config.folders_ignore)<br>        self._files_ignore = set(config.files_ignore)<br>        self._queue = deque(config.root_directory)<br><br>    def __iter__(self) -&gt; Iterator:<br>        return super().__iter__()<br><br>    def __next__(self) -&gt; list[str]:<br>        if self._queue:<br>            files: list[str] = list()<br>            for _ in range(len(self._queue)):<br>                directory: str = self._queue.popleft()<br>                for entry in listdir(directory):<br>                    entry_path: str = path.join(directory, entry)<br>                    if (<br>                        path.isfile(entry_path)<br>                        and self._is_python_file(entry_path)<br>                        and entry not in self._files_ignore<br>                    ):<br>                        files.append(entry_path)<br>                    elif path.isdir(entry_path) and entry not in self._folders_ignore:<br>                        self._queue.append(entry_path)<br>            return files<br>        else:<br>            raise StopIteration()<br><br>    def _is_python_file(self, file_path: str) -&gt; bool:<br>        return file_path.split(&quot;.&quot;)[-1] == &quot;py&quot;</pre><p>Here is the update <em>docstring_generator/__main__.py file</em>:</p><pre>from .config import Config<br>from .helpers import DirectoryIterator<br><br><br>def main():<br>    root_dir: str = &quot;.&quot;<br>    folders_ignore: list[str] = [&quot;venv&quot;, &quot;__pycache__&quot;, &quot;.git&quot;, &quot;.venv&quot;]<br>    config: Config = Config(<br>        root_directory=root_dir,<br>    )<br>    config.folders_ignore.extend(folders_ignore)<br>    directory_iterator: DirectoryIterator = DirectoryIterator(config=config)<br>    for files in directory_iterator:<br>        print(files)<br><br><br>if __name__ == &quot;__main__&quot;:<br>    main()</pre><p>Commit these new changes in the directory-iterator branch and push them them to github:</p><pre>git add .<br>git commit -m &quot;feat: creates the directory iterator.&quot; # You may have to run these two commands a couple of times as commitizen may reformat some files<br>git push --set-upstream origin directory-iterator</pre><p>Do not forget to merge these changes into your main branch.</p><h3>Conclusion</h3><p>In this article we achieved the following:</p><ul><li>Introduced the project</li><li>Created the basic layout</li><li>Created an iterator for traversing the file system.</li></ul><p>In the <a href="https://medium.com/@lyle-okoth/creating-a-documentation-generator-for-python-projects-using-gpt3-part-two-0cb22fd01a2a">next article</a>, we will go through the process on parsing the files and generating docstrings for python functions, classeses and methods. Here is the <a href="https://github.com/twyle/docstring-generator">link to the project on github</a>. If you found the article useful, like it and share it out. Also due connect with me on <a href="https://www.linkedin.com/in/lyle-okoth/">linkedin</a>, <a href="https://twitter.com/LyleOkoth">twitter</a> and <a href="https://github.com/twyle">github</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=f78b63b835ec" width="1" height="1" alt=""><hr><p><a href="https://blog.gopenai.com/creating-a-documentation-generator-for-python-projects-using-gpt3-f78b63b835ec">Creating A Documentation Generator for Python Projects using GPT3</a> was originally published in <a href="https://blog.gopenai.com/">GoPenAI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How To Get A Google API Key]]></title>
            <link>https://medium.com/@lyle-okoth/how-to-get-a-google-api-key-d3c38649eaae?source=rss-2637c671f6af------2</link>
            <guid isPermaLink="false">https://medium.com/p/d3c38649eaae</guid>
            <dc:creator><![CDATA[Lyle Okoth]]></dc:creator>
            <pubDate>Thu, 14 Dec 2023 09:42:10 GMT</pubDate>
            <atom:updated>2023-12-14T11:35:18.741Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using Google Services programatically.</h4><p>In this post, I will take you through the process of obtaining a Google API key and provide an example use case of such a key. But first, what is this API key and what is it useful for?</p><p>Well, Google provides their users with various services such as:</p><ul><li>Google Search</li><li>Google Calendar</li><li>YouTube</li><li>Gmail</li><li>Google Drive</li><li>Google Sheets</li></ul><p>These services are mostly used by non-developers through the web such as searching for and watching videos on youtube or sending an email through gmail. However for developers, their is the possiblity of using these services by writing applications that use on their behalf. For example, an application that schedules meetings weekly thorugh google calendar. to make such applications, google provides developers with API keys, that enables google to identify the particular application and user that is using their service.</p><p>For this tutorial, I will walk you through obtaining an API key for working with youtube. That is we will search through youtube by writing a python application. To get started, you need a <strong><em>Google</em></strong> account. Creating that is pretty simple, just <a href="https://support.google.com/mail/answer/56256?hl=en">sign up for a gmail account</a>. Once you have a Gmail account, head over to the <a href="https://console.cloud.google.com/cloud-resource-manager?pli=1">Google Developer Console</a>.</p><p>Start off by creating a new project. The button is close to the top left part of the screen. Give it a good name, I called mine <em>YouTube Search</em>. Next select the <em>humburger</em> menu (the three horizontal lines at the top left of the page) to access the various options. From that menu, select the <em>APIs &amp; Services</em> option then finally select <em>Credentials</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*z3ioYw6HoETLt5iZjI5REQ.png" /></figure><p>This gets you to the credentials page. On this page, just select the <em>Create Credentials</em> button, then select the <em>OAuth Client ID</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1EI0sn2X6zU0OtzW1qQqqg.png" /></figure><p>If this is the first time you are doing this, you will need to set up consent. That is, the accounts (emails) that can use the credentials that you will create. So head over to the Oauth consent page. Just click on the <em>Configure Consent Screen</em> button. Select the <em>external</em> option and then click on <em>create</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pOoGA_wDqhNF2FD4ZzTfCw.png" /></figure><p>This brings you to the app information screen. Fill in the fields appropriately, then select <em>save and continue</em>. Here is mine:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*A6qcIUP3_xQ5YHHnMwZrQg.gif" /></figure><p>In the next screen, for selecting scoipes, just skip it. Select <em>Save and continue</em>. You then need to add test users, these are email adresses that are authorized or you can use to test consent. Add atleast one. Just click on <em>add user</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*HLMpro4N9JFKTd4gIIfa7A.gif" /></figure><p>Once done, go back to the dashboard, then select Credentials once again, then Client Oauth ID. This time it brings you to the application type screen, and your selection should be <em>desktop</em>. Give it an appropriate name and then selecet <em>create</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*We36azUJ6u8e_PBSyrv7rQ.png" /></figure><p>This finally gets you an oauth client, a json file, which you should download.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*R47-EtdHbXhVu0pG1JfCOg.png" /></figure><p>Now for the fun part, making use of the client secret file that you just downloaded. First, you will need to enable the particular service that you want to use with this oauth id. Select the Enabled services and APIs button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YIstAFkLHSnneJ3fnQlKBw.png" /></figure><p>Then select the <em>Enable APIs and Services</em> button.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6C5F81f_VV9bwoUF3wqFfQ.png" /></figure><p>For our use case, we will enable <em>YouTube</em>. You can enable any other service that you want.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*nMjgDv-_XpNSRlTjGfMvtg.gif" /></figure><p>Now for the fun part, searching through YouTube. We will use the <em>oryks-youtube</em> python library. Go ahead and install it.</p><pre>pip install oryks-youtube</pre><p>Then search through youtube (replace the <em>clients_secrets_file</em>) with where you downloaded your secrets file:</p><pre>from youtube import YouTube<br>from youtube.schemas import (<br>    SearchFilter, SearchOptionalParameters, SearchPart, YouTubeResponse, YouTubeRequest<br>)<br>from typing import Iterator<br>from youtube.models import Video<br><br>client_secrets_file = &#39;/home/lyle/Downloads/secrets.json&#39;<br>youtube = YouTube(client_secrets_file)<br>youtube.authenticate()<br><br>query: str = &#39;Python programming videos&#39;<br>max_results: int = 10<br>part: SearchPart = SearchPart()<br>optional_parameters: SearchOptionalParameters = SearchOptionalParameters(<br>    q=query,<br>    maxResults=max_results,<br>    type=[&#39;video&#39;, &#39;playlist&#39;, &#39;channel&#39;]<br>)<br>search_request: YouTubeRequest = YouTubeRequest(<br>    part=part, <br>    optional_parameters=optional_parameters<br>)<br>video_iterator: Iterator = youtube.get_search_iterator(search_request)<br>videos: list[Video] = next(video_iterator)</pre><p>When you run the above code, you will be taken through an authentication process as shown below:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/600/1*LxvQU7j-SJLkI1ET_E56xA.gif" /></figure><p>And with that, you can now obtain Google Authentication for use with any of their services.</p><p>My name is Lyle, a python developer who enjoys building applications that leverage large language models for various purposes. Find me on <a href="https://www.linkedin.com/in/lyle-okoth/">linkedin</a>, <a href="https://twitter.com/LyleOkoth">twitter</a> and <a href="https://github.com/twyle">github</a>. Read more about the <a href="https://youtube-wrapper.readthedocs.io/en/latest/">youtube python library</a> for working with the youtube v3 api.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=d3c38649eaae" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Manage Application Secrets with Hashicorp’s Vault.]]></title>
            <link>https://medium.com/@lyle-okoth/how-to-manage-application-secrets-with-hashicorps-vault-865216896a39?source=rss-2637c671f6af------2</link>
            <guid isPermaLink="false">https://medium.com/p/865216896a39</guid>
            <category><![CDATA[hashicorp-vault]]></category>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[hashicorp]]></category>
            <dc:creator><![CDATA[Lyle Okoth]]></dc:creator>
            <pubDate>Thu, 23 Jun 2022 12:21:08 GMT</pubDate>
            <atom:updated>2022-06-23T12:21:08.893Z</atom:updated>
            <content:encoded><![CDATA[<h4>Using vault with a file backend to manage key value pairs.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*qqTRI6u8MjLofxbq501ZiQ.png" /></figure><p>In this article, I want to show you how to manage application secrets with Hashicorp Vault. As a software developer, especially a backend developer, there are so many values that your application uses, some of which you may not want the general public to see. These secrets, could include:</p><ul><li>Database usernames, passwords, ports as well as the database name and the server’s address, where the database is hosted.</li><li>Secret keys for encryption and decryption.</li><li>Default admin passwords and emails</li><li>Server addresses, where an application is deployed as well as the usernames and passwords for logging in.</li><li>And many more…</li></ul><p>Keeping track of these secrets become difficult, especially if you deal with numerous applications or work in large teams where different developers need access to these secrets with different levels of authorization i.e a database administrator vs the tech lead vs a junior developer. This is where Hashicorp’s vault comes in.</p><h4>Introduction</h4><p>To manage these secrets, I currently store them in a .env file both in development and production, something that I quite don’t like. I bet you do too. And then I have to make them simple, to avoid mistyping them into the .env file. But with vault, i can overcome both. So what is Vault?</p><p><a href="https://www.vaultproject.io/">Vault</a> is an open-source tool used for securely storing and managing secrets.</p><blockquote><strong>What is a secret?</strong> Secrets, in the context of this tutorial, are securely-sensitive or personally identifiable info like database credentials, SSH keys, usernames and passwords, AWS IAM credentials, API tokens, Social Security Numbers, credit card numbers, just to name a few.</blockquote><p>There are numerous components that make up Vault. In this tutorial, we will concern ourselves with only two concepts:</p><ul><li><strong>Storage</strong>; this describes where the secrets are stored by the Vault. It could be a file system backed ona our computter, Consul offered by Hashicorp (needs to be configured), a database such as PostreSQL or even AWS S3. In this tutorial, we will use the filesystem backend to store our secrets.</li><li><strong>Secrets</strong>: this describes the type of secrets that are to be stored. This could be static (do not change over time) or dynamic (generated on the fly). These secrets vary from AWS credentials, database credentials as well as key value pairs. In this tutorial, we will only look at key value pairs.</li></ul><h4>Application Secrets</h4><p>In this article, we will assume that a Flask application is to be deployed with a postgres database. If you are familiar with flask, then you know that we would need to give out at least:</p><ul><li>The flask environment i.e development as FLASK_ENV</li><li>The flask app, as FLASK_APP</li><li>The database credentials, as POSTGRES_USER, POSTGRES_PASSWORD, POSTGRES_HOST, POSTGRES_PORT and the POSTGRES_DB.</li><li>The app secret as SECRET_KEY.</li></ul><p>These values could be provided by exporting them before running the application:</p><p>export FLASK_APP=api/__init__.py</p><p>or using a .env file. In this tutorial, we will store the secrets using Vault and then when needed, read them from vault and then populate the .env file with the values.</p><h4>Vault Configuration</h4><p>To configure a vault server instance, we will use docker compose to spin up a vault server, then store our secrets in the created server. To get started at the comandline:</p><pre>mkdir vault-consul-docker &amp;&amp; cd vault-consul-data</pre><pre>mkdir vault &amp;&amp; cd vault</pre><pre>mkdir config data logs policies # you should have the folder structure below within the vault-consul-docker folder</pre><pre>└── vault<br>    ├── config<br>    ├── data<br>    ├── logs<br>    └── policies</pre><ul><li>The config folder will hold the vault server configuration, in this case, where to store the data as well as the logs.</li><li>The data folder will contain the data that we add to vault</li><li>The logs folder as the name implies will contain the logs</li><li>The policies folder will contain our policies. Remember when I hinted to limiting access to secrets by different team members in a project? Policies help with that</li></ul><p>Let us create the configuration file that vault will use. Within the config folder, create a file called vault-config.json:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b38b898f713eeb6febf41cea3eea53f7/href">https://medium.com/media/b38b898f713eeb6febf41cea3eea53f7/href</a></iframe><ul><li>Here we use a file system as our back-end and the data is stored within the vault/data directory.</li><li>The server listens for connections on port 8200</li></ul><p>Next, we create the Dockerfile that creates the vault server container. In the vault directory, create a file called Dockerfile and add the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d01b715640ecf9d52f6d19a1ba731114/href">https://medium.com/media/d01b715640ecf9d52f6d19a1ba731114/href</a></iframe><ul><li>We use an alpine base image</li><li>The vault version is then set, then the vault directory created.</li><li>We then add bash, that we will use later and wget for downloading the vault instance.</li><li>We then download and install vault, set the path to the vault directory and then copy the config file.</li><li>The vault port 8200 is then exposed for access to the container.</li></ul><p>Let us then create the docker compose file for bringing up the container. Create a file called docker-compose.yml at the project root (the vault-consul-docker folder) and add the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ab16db3649d92e93d4e3d14b043587e/href">https://medium.com/media/1ab16db3649d92e93d4e3d14b043587e/href</a></iframe><ul><li>We build the vault service using the above Dockerfile.</li><li>We then map port 8200 on our local computer to port 8200 of the container that is built.</li><li>We then map the vault volumes in the container to the directories on our computer.</li><li>We then set the command to execute when the container starts up.</li></ul><p>To start the container:</p><p>sudo docker-compose up --build</p><p>If you followed the instructions as given out, you should now have a vault server running on port 8200.</p><h4>Vault setup</h4><p>Now that we have the vault server up and running, we need to set it up before we can store and access secrets. This involves:</p><ul><li>Initializing vault</li><li>Unsealing vault</li><li>Logging in</li></ul><p>First, let us initialize vault. This will give us a root token(for logging in) and 5 keys for unsealing vault. Let us access the vault command-line; in a different command-line tab:</p><p>sudo docker-compose exec vault bash # get a bash session on the running container</p><p>Then within the bash session, initialize vault:</p><p>vault operator init #initialize vault</p><p>You should get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DnXYuQiSqpK0E5fTGiSS7Q.png" /></figure><blockquote>Keep those 5 unseal keys secret at a safe location. Without them you cannot unseal vault, and if vault is unsealed you cannot read its secrets or write to it.</blockquote><p>Next, let us unseal vault. We will need three keys. On the bash session:</p><p>vault operator unseal</p><p>Then paste one of the unseal keys at the prompt. Due this three times with a different key each time. Once done, the vault should be initialized and unsealed:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Es74_NnTrSJDrC-VUR-R5g.png" /></figure><p>Then let us log in:</p><p>vault login</p><p>Use the root token that you obtained earlier on.</p><h4>Create and Read a secret at the Vault Commandline</h4><p>With us logged in, it’s time to create and read a secret. We will create a key value pair, so let us us the kv engine offered by vault. We start by enabling it. Remember this is a static secret (does not expire). At the bash session:</p><p>vault secrets enable kv</p><p>Then let us create a key value pair, in this case, the flask environment:</p><p>vault kv put kv/vault-secret-management/local/flask_environment FLASK_ENV=development</p><p>In this case we create a new key value pair at the path kv/vault-secret-management/local/flask_environment . The path is composed of:</p><ul><li>vault-secret-management, which is the current project name.</li><li>local, which tells me that this value is used during development on my local computer</li><li>flask_environment which is what the secret is keeping</li></ul><p>I use this format for storing secrets since I create tons of flask applications, and host them on AWS, Linode and Heroku. Knowing what a secret is and which application and what environment it is for becomes easier with that format.</p><p>To read it:</p><p>vault kv get kv/vault-secret-management/local/flask_environment</p><p>You get back:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*6uB_LHj9gec0VnAdse6mZA.png" /></figure><p>Well, that is how to set and read secrets using the kv engine. You can also set and read secrets using the vault UI or via HTTPS.</p><h4>Read Secrets over HTTPS</h4><p>I will not go through creating a secret using the UI or HTTPS. We will only read the secrets over HTTPS, since that is what we are concerned with. At a new command-line tab, use curl to access the vault server and read the secret, but first, you need to provide the root access token:</p><p>export VAULT_TOKEN=s.GMRc7gPtdE0rBNbmyt1AKecn # the root access token</p><p>Then read the secret:</p><pre>curl \<br> -H “X-Vault-Token: $VAULT_TOKEN” \<br> -X GET \<br> <a href="http://127.0.0.1:8200/v1/kv/vault-secret-management/local/flask_environment">http://127.0.0.1:8200/v1/kv/vault-secret-management/local/flask_environment</a></pre><p>You get back:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LtpAtJM9g5_0z4YXo-otPA.png" /></figure><p>And with that, we have set up and then read a secret using vault. Now try to set up and read the other secrets that we mentioned earlier on. Here is an example for FLASK_APP:</p><p>vault kv put kv/vault-secret-management/local/flask_app FLASK_APP=api/__init.py</p><p>And to read it:</p><pre>curl \<br> -H “X-Vault-Token: $VAULT_TOKEN” \<br> -X GET \<br> <a href="http://127.0.0.1:8200/v1/kv/vault-secret-management/local/flask_environment">http://127.0.0.1:8200/v1/kv/vault-secret-management/local/flask_</a>app</pre><h4>Automate the reading of secrets</h4><p>Finally, let us automate the reading of these secrets. Remember that at the start, we wanted our flask applications to read these values from a dotenv file using the dotenv python library. Create a file called run.sh in the project root and add the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/82589edb33f0bcded1d481f285cdefa1/href">https://medium.com/media/82589edb33f0bcded1d481f285cdefa1/href</a></iframe><ul><li>When executed, the script first deletes the existing .env file</li><li>It then makes a call to the vault server to get the particular secret. The vault server returns a JSON response, which is passed using the jq utility. The jq utility is a command-line tool that is used to parse JSON. Once the JSON is parsed and the desired key obtained, this is written to the .env file.</li></ul><p>To run the script, first install the jq utility:</p><pre>sudo apt install jq -y # install jq</pre><pre>chmod +x run.sh # make the script executable</pre><pre>./run.sh</pre><p>A file called .env is created in the project root. Anytime you use this .env file make sure that you add it to the .gitignore file. You get a file called .env with the following values (if you do not add the rest of the secrets like me):</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/0011321bea07ee822a62dbd96e5139b1/href">https://medium.com/media/0011321bea07ee822a62dbd96e5139b1/href</a></iframe><p>This can then be picked up by your application.</p><p>With vault, you can now set secrets that are long and complicated and reuse them as much as you wan. You can even version them and read the versions that you want as well as set policies that govern access to these secrets.</p><p>Once done, exit the bash session:</p><p>exit</p><p>Then stop the running container.</p><p>To read them, you need to restart the container, then unseal vault.</p><h4>Conclusion</h4><p>In this tutorial:</p><ul><li>We briefly went over secrets, and the current not so secure or scalable way of managing them</li><li>We then briefly went over Hashicorp’s vault</li><li>We then set up a very simple vault and used the key value engine to set up key value pairs</li><li>We read the secrets from the command-line then over HTTPS</li><li>We finally created a script that automatically loads the secrets into a .env file for our application.</li></ul><p>In the next tutorial, we will deploy vault to a server on AWS or Linode and access it from a GitHub workflow and from a deployed application. We will also use a docker swarm with a consul back-end. To see the vault server secret management in use check out my tutorial series on <a href="https://medium.com/@lyle-okoth/how-to-deploy-a-production-grade-flask-application-to-an-aws-ec2-instance-using-github-actions-6241886b197">deploying a production grade flask application to AWS</a>.</p><p>I hope you enjoyed it and learnt something. Give it a clap or share it out and do not hesitate to reach out to me in-case of an issue. The code for this application is here <a href="https://github.com/twyle/vault-secret-management">vault-secret-management</a>. I am Lyle, a junior software engineer with a passion for developing, testing and deploying scalable services. You can find me on <a href="https://twitter.com/lylethedesigner"><em>twitter</em></a>,<a href="https://www.linkedin.com/in/lyle-okoth/"><em> linkedin,</em></a> <a href="https://github.com/twyle"><em>github</em></a> and here’s my <a href="https://twyle.github.io/portfolio-simple/"><em>portfolio</em></a>. See you next time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=865216896a39" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Deploy a Production Grade Flask application to an AWS EC2 Instance using GitHub Actions…]]></title>
            <link>https://medium.com/@lyle-okoth/how-to-deploy-a-production-grade-flask-application-to-an-aws-ec2-instance-using-github-actions-9bfd82698000?source=rss-2637c671f6af------2</link>
            <guid isPermaLink="false">https://medium.com/p/9bfd82698000</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[flask-framework]]></category>
            <category><![CDATA[postgresql]]></category>
            <category><![CDATA[flask-sqlalchemy]]></category>
            <category><![CDATA[docker-compose]]></category>
            <dc:creator><![CDATA[Lyle Okoth]]></dc:creator>
            <pubDate>Sat, 18 Jun 2022 09:30:10 GMT</pubDate>
            <atom:updated>2022-06-18T12:09:58.042Z</atom:updated>
            <content:encoded><![CDATA[<h3>How to Deploy a Production Grade Flask application to an AWS EC2 Instance using GitHub Actions: Part Four.</h3><h4>Logging and Database Integration.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*f2Stg6DNIalcdVlv.png" /></figure><p>This is the fourth article in a series that takes the reader through the process of deploying a production ready flask API to an AWS EC2 instance. In the <a href="https://medium.com/@lyle-okoth/how-to-deploy-a-production-grade-flask-application-to-an-aws-ec2-instance-using-github-actions-163be1d5fbd5">previous article</a> we:</p><ul><li>We set up nginx to route HTTPS traffic to our server.</li><li>We then added a domain tour application for easier access from the internet.</li><li>We then used certbot to generate an SSL certificate for our site.</li></ul><p>In this article, we will accomplish the following:</p><ul><li>Set up databases locally for testing and development using docker compose.</li><li>Set up our application to connect to and make use of the created database.</li><li>Update our workflow by setting up a PostgreSQL service for testing our application using GitHub Actions.</li><li>Set up a database on AWS and connect our application to it.</li><li>Set up logging.</li></ul><p>By the end of this tutorial, you will have an application that uses a PostgreSQL database hosted on AWS. Let’s get started.</p><h4>Step 1: Test our application.</h4><p>At the end of the previous tutorial, we had set up a domain name for our application and in an earlier tutorial, we had created a service that automatically starts up our application when the server is started up. To test it out, first boot up the AWS EC2 server that hosts our application, then navigate to <a href="https://dev.twyle.xyz/">https://dev.twyle.xyz</a> (replace dev.twyle.xyz with your own domain name) and you will get:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ntRs8pcXkoFvxgUUeRu09Q.png" /></figure><h4>Step 2: Test our application locally</h4><p>Let us ensure that our local code works. If you don’t have tho code, just follow the previous tutorial to get up to speed, then on the command-line:</p><pre>source venv/bin/activate # activate the virtual environment</pre><pre>make update # update the package manager</pre><pre>make install &amp;&amp; make install-dev # install the project requirements (just to be safe)</pre><pre>make run # start the application </pre><pre>curl localhost:5000/</pre><p>You will get:</p><pre>{“hello”:”from template api auto-deployed with GitHub actions and being tested!”}</pre><p>Terminate the application.</p><h4>Step 3: Set up the databases</h4><p>We then need to set up the database for local use. We could install and use a PostgreSQL database, but I think it would be easier and more convenient to spin up a docker container for the development and testing. We will use docker compose, since we need two databases, one for development and one for testing. However in this article, we will not use the test database. In the project root, create a new file called docker-compose.yml and enter the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/397302a10016193f342d8cc8d1b14b13/href">https://medium.com/media/397302a10016193f342d8cc8d1b14b13/href</a></iframe><ul><li>We create two services one called test_db and the other called dev_db, both of which spin up a PostgreSQL database running on the local-host, with a user called postgres with password lyle and database name set to lyle. The test database is accessible on port 5435 and the development database is available on port 5436. I chose those ports since I have PostgreSQL installed locally and running on port 5432.</li><li>We also included a volume, for the development database since we do not want to lose our application data.</li></ul><p>Spin up the two databases:</p><p>sudo docker-compose up -d --build</p><p>To test the databases, let us connect to them:</p><p>sudo docker-compose exec dev_db psql --username=postgres --dbname=lyle --port=5432</p><p>You should get a prompt for the development database, called lyle.</p><p>\q # at the prompt to exit.</p><p>Stop the running containers:</p><p>sudo docker-compose down -v</p><h4>Step 4: Set up our application to access the database</h4><p>The database that we created will enable our application to store users and admins. The admins will have the authority to create new users by supplying a JSON web token that will be issued out to them after they have logged into their accounts. So, with a database, how would our application look like?</p><ul><li>We will have two database models, Admin and User, built with flask-sqlalchemy. The User model, as it name suggests will represent a user in our application, with an id, a name and whether or not they are active. The Admin model, will represent an admin in our application, with an id, name, email and password.</li><li>We will add another blueprint, called auth, that will hold all the functionality that is associated with the admin. This includes registration, and logging in. The default blueprint will be updated to allow the creation, reading, updating and deletion of a user.</li><li>We will create a configuration, that tells our application how to connect to the database, then add extensions that enable the application to interact with the database.</li><li>We will also add helpers, that would make the overall application structure better.</li></ul><p>Let us start with the application configuration. For that, we will first need to supply the database connection information, including it’s host address, the user , the password and the database. Open up the .env file at the project root and update it with the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/707a0544962284ed9cb2bb14b8f10d83/href">https://medium.com/media/707a0544962284ed9cb2bb14b8f10d83/href</a></iframe><p>Replace the POSTGRES_HOST (192.168.xxx.x) with your computer’s IP address.</p><p>For our application to make use of this information, we need a way to read them. We will create a config object that later will be used by the application. Open up the api/config.py file and update it with the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77822d3790c99371dbd476ffd91fee14/href">https://medium.com/media/77822d3790c99371dbd476ffd91fee14/href</a></iframe><p>We create configurations for development, testing, staging and production all of which inherit from the base config. Then to use the configuration, we need to pas it to the flask app instance.Open up api/__init__.py and update it with the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a4244448ed5e748b830069ebe9a3310/href">https://medium.com/media/4a4244448ed5e748b830069ebe9a3310/href</a></iframe><ul><li>This module first imports all the necessary modules, as well as the blueprints, extensions and helpers.</li><li>It starts by loading the environment variables, line 16, then checks if the all the environment variables are set in line 19 to 22. The function are_environment_variables_set is declared in the helpers module. Find it on the project repo and add it to your project.</li><li>It then creates the app instance on line 25, then registers the blueprints. At line 33, the environment is set using the set_flask_environment function also from the helpers module by passing in the app instance. This function uses the config class that we created earlier on. Check it out in the project repo.</li><li>We then initiate the database, the migration script as well as the flask JSON web tokes extension, then finally register an error handler for HTTP 400 error. The migrate instance is declared in the extensions module, while the error handler is declared in the error_handlers module.</li></ul><p>Let us briefly look at the two database models. The user model is declared in the default package within the models module. Open up api/blueprints/default/models.py and add the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/31d75be60c4412dd6373d18d0e43a0b2/href">https://medium.com/media/31d75be60c4412dd6373d18d0e43a0b2/href</a></iframe><p>This is a very simple model that represents a user with only an email address and whether or not they are active, which defaults to True. The id is automatically generated by the database.</p><p>Then create a new folder called auth in the blueprints folder. Within the auth folder, create a __init__.py file to mark it as a package, the create the models.py module to hold our models. The auth package declares all the functionality associated with creating and authorizing our admins. Open up the models.py file and enter the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4883cbfb764b638bf6966489fb350831/href">https://medium.com/media/4883cbfb764b638bf6966489fb350831/href</a></iframe><ul><li>It first imports the dataclass, some constants as well as the database instance. The constants are declared in the constants module within the blueprint package while the database instance db is declared in the extensions module in the blueprints package. The extensions module also contains the declaration of the flask JSON web token extensions declaration as well as the creation of the application logger.</li><li>The admin is a bit more complex, with an email, name and password.</li></ul><p>Some notes about the models:</p><ul><li>The user’s id and email cannot be null, and the id has to be an integer while the email has to be a string. Both have length limits and the email has a specific format that has to be adhered to. The email address also has to be unique. When creating the user, these are checked for, and if not met, an exception is raised. Check the helpers module in the default package, where these checks are performed. The exceptions module in the blueprints package declares the exceptions that are raised when the user attributes fall outside of the specified values.</li><li>The admin’s email, name and password cannot be null. They also have length limits. The name and password have to be unique and alphanumeric, while the email has to adhere to a specific format. If any of these rules are not met, an exception is raised. Check the helpers module in the auth package for details.</li></ul><p>Finally let us look at the views that allow our application user to interact with our database. First, open up the api/blueprints/default/views.py file and update it with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/032d27e715060d4f16a163fd6ef9904f/href">https://medium.com/media/032d27e715060d4f16a163fd6ef9904f/href</a></iframe><ul><li>There’s the default route that simply informs you that this is an API.</li><li>Then we have routes for creating, updating, reading and deleting a user all which require authorization, the @jwt_required tag. There is also a route for getting all the users. These routes make use of methods declared in the <a href="https://github.com/twyle/flask-heroku-docker/blob/development/api/blueprints/default/helpers.py">helpers module</a> to handle their respective requests.</li></ul><p>Then open up the api/blueprints/auth/views.py file and update it with the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eb30427d2180b174ecc1f2a984d35ca4/href">https://medium.com/media/eb30427d2180b174ecc1f2a984d35ca4/href</a></iframe><ul><li>In this module, we are mostly interested in the register and login functionality. The other routes are still under development. The register route allows one to create an admin user. Once created, the login route allows the admin to log in and obtain an access token which they can use to create a new user.</li></ul><h3>Step 5: Test the application.</h3><p>To test the application, we first need to update the requirements. Open up the requirements.txt file and add the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e38f24924adca70e7cfb5ccec686789f/href">https://medium.com/media/e38f24924adca70e7cfb5ccec686789f/href</a></iframe><p>Then create a script for creating the database and a few users. Create a file called manage.py in the project root and add the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f03a767f9da5ceb1f3e61d34d1fb5f5e/href">https://medium.com/media/f03a767f9da5ceb1f3e61d34d1fb5f5e/href</a></iframe><ul><li>The create_db function simply creates all the database tables.</li><li>The seed_db function creates two dummy users.</li></ul><p>Then on the command-line:</p><pre>source venv/bin/activate # activate the virtual environment</pre><pre>make install # install the project requirements</pre><pre>sudo docker-compose up -d — build # start the databases</pre><pre>python manage.py create_db # create the database</pre><pre>python manage.py seed_db # create two default records in the database</pre><pre>make run # start the application</pre><p>To test the application:</p><p>curl <a href="http://localhost:5000/">http://localhost:5000/</a></p><p>You get:</p><p>{<br> “hello”: “from template api”<br>}</p><p>curl <a href="http://localhost:5000/users">http://localhost:5000/users</a></p><p>You get:</p><p>[<br> {<br> “active”: true, <br> “email”: “<a href="/service/mailto:test@example.com">test@example.com</a>”, <br> “id”: 1<br> }, <br> {<br> “active”: true, <br> “email”: “<a href="/service/mailto:test1@example.com">test1@example.com</a>”, <br> “id”: 2<br> }<br>]</p><p>Then to try the authorization, open up postman. Create a post request to the create route to create a new admin. This route will be localhost:5000/auth/register:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*dOVUh1zwFu3fQlmq.png" /></figure><p>Then create a post request to the login endpoint, localhost:5000/auth/login to get a refresh and access tokens:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*9DsKxq7db9hZ6j7e.png" /></figure><p>Create a new post request to localhost:5000/user:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*aQmyKk59Yl5DGw72.png" /></figure><p>Then to add authorization, copy the access token that you got when you registered a new admin, then under the <strong>Authorization </strong>tab, in the <strong>Type</strong> drop-down select Bearer Token:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*n19sATqTiLPDdw-a.png" /></figure><p>Then paste it into the <strong>Token</strong> field and press send:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*JdtCFVEMJBNOxs-g.png" /></figure><p>You should have created a new user:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*j2SEyhz2WTCUWNZi.png" /></figure><p>With that, we have created a flask app that enables an admin to register and login in, then authorizes them to create new users using JSON web tokens. Stop the flask server as well as the running containers.</p><p>sudo docker-compose down -v # stop the running containers</p><p>Since this tutorial is already long enough, we will set up the database in the next article.</p><h4>Conclusion</h4><p>In this article, we did the following:</p><ul><li>Tested that our application works, both on AWS and on our local computer.</li><li>Set up two databases by creating two postgres containers, one for development and one for testing, using docker-compose.</li><li>Connected to the created databases using docker compose.</li><li>Set up our application to connect to the created databases as well as log the interactions.</li><li>Created authorization for admin users using the flask JSON web extension</li><li>Tested that our application worked by connecting to the database and creating new users</li></ul><p>In the next tutorial, we will install a database on AWS and connect our application to the created database. We will also update the GitHub workflow to create postgres database during the workflow run.</p><p>I hope you enjoyed it and learnt something. Give it a clap or share it out and do not hesitate to reach out to me in-case of an issue. The code for this application is here <a href="https://github.com/twyle/flask-ec2-deployment/tree/features">flask-ec2-deployment</a>. I am Lyle, a junior software engineer with a passion for developing, testing and deploying scalable services. You can find me on <a href="https://twitter.com/lylethedesigner"><em>twitter</em></a>,<a href="https://www.linkedin.com/in/lyle-okoth/"><em> linkedin,</em></a> <a href="https://github.com/twyle"><em>github</em></a> and here’s my <a href="https://twyle.github.io/portfolio-simple/"><em>portfolio</em></a>. See you next time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9bfd82698000" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Deploy a Containerized Flask Application to Heroku using GitHub Actions: Part Two.]]></title>
            <link>https://medium.com/@lyle-okoth/how-to-deploy-a-containerized-flask-application-to-heroku-using-github-actions-part-two-4b209c7a65c6?source=rss-2637c671f6af------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b209c7a65c6</guid>
            <category><![CDATA[docker-compose]]></category>
            <category><![CDATA[flask-sqlalchemy]]></category>
            <category><![CDATA[flask-api]]></category>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[heroku]]></category>
            <dc:creator><![CDATA[Lyle Okoth]]></dc:creator>
            <pubDate>Wed, 15 Jun 2022 07:04:48 GMT</pubDate>
            <atom:updated>2022-06-15T07:04:48.721Z</atom:updated>
            <content:encoded><![CDATA[<h4>Logging and Database Integration.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*MysfmAKrrjgUEhaL.png" /></figure><p>This is the second article in a series of articles that takes the reader through the process of creating and deploying a containerized application to Heroku. In the <a href="https://medium.com/@lyle-okoth/how-to-deploy-a-containerized-flask-application-to-heroku-using-github-actions-b0353880afc6">previous article</a>, we did the following:</p><ul><li>Used a template to create and deploy a Flask application to Heroku</li><li>Went through the application layout.</li><li>Created a Heroku account and obtained the API keys.</li><li>Created a GitHub environment.</li><li>Deployed our containerized application to Heroku using GitHub Actions.</li></ul><p>In this article, we will:</p><ul><li>Set up a database for development in our local environment using docker.</li><li>Use JSON web tokens to authorize a user.</li><li>Set up our application to access and the created database.</li><li>Set up logging for our application.</li><li>Provision a database using GitHub actions in our workflow.</li><li>Provision database on Heroku.</li></ul><p>By the end of this tutorial, you will have an application that uses a PostgreSQL database hosted on Heroku. Let’s get started.</p><h4>Step 1: Test Our Application</h4><p>If you followed the previous tutorial and have the application code, at the command-line:</p><pre>source venv/bin/activate # activate the virtual environment</pre><pre>make update # update the package manager</pre><pre>make install # install the requirements (just to be safe)</pre><pre>make install-dev</pre><pre>make test # execute the tests</pre><pre>make run # execute the application</pre><p>In a different terminal:</p><p>curl <a href="http://127.0.0.1/">http://127.0.0.1</a>:5000/ # to get the application output</p><p>You will get: {“hello”:”from template api”}</p><p>Let us test the container. Stop the running server, then:</p><pre>make build imagename=flask-heroku-docker </pre><pre>make run-docker imagename=flask-heroku-docker</pre><p>curl <a href="http://127.0.0.1/">http://127.0.0.1</a>:5000/ # to get the application output</p><p>You will get: {“hello”:”from template api”}</p><p>Terminate the docker application.</p><h4>Step 2: Set up the databases</h4><p>In this step, we will set up two databases, one for testing and the other for development. We will use docker compose to spin up two docker images. At the project root, create a new file called docker-compose.yml and enter the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac3893a7d2128f003ff77e2a5e155087/href">https://medium.com/media/ac3893a7d2128f003ff77e2a5e155087/href</a></iframe><ul><li>We create two services one called test-db and the other called dev-db, both of which spin up a PostgreSQL database running on the local-host, with a user called postgres with password lyle and database name set to lyle. The test database is accessible on port 5433 and the development database is available on port 5434. I chose those ports since I have PostgreSQL installed locally and running on port 5432.</li><li>We also included a volume, for the development database since we do not want to lose our application data.</li></ul><p>Spin up the two databases:</p><p>sudo docker-compose up -d --build</p><p>To test the databases, let us connect to them:</p><p>sudo docker-compose exec dev-db psql --username=postgres --dbname=lyle --port=5432</p><p>You should get a prompt for the development database, called lyle.</p><p>\q # at the prompt to exit.</p><p>Stop the running containers:</p><p>sudo docker-compose down -v</p><h4>Step 3: Update the container.</h4><p>We need to use a newer container version, since the previous container generated some errors during the installation of pycopg2 that we need in order to connect to the database. Open up the Dockerfile and update it with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/165fe52df03648426accd6f19442db9a/href">https://medium.com/media/165fe52df03648426accd6f19442db9a/href</a></iframe><ul><li>This one is pretty straight forward; we use python3.9.5 as the base image, set the working directory, install the project dependencies, copy the project into the container and set the command to run at start up. Copying the requirements at the beginning gives us some caching since the installation of the requirements now only occurs when the requirements change.</li></ul><p>Try the image:</p><pre>make build imagename=flask-heroku-docker</pre><pre>make run-docker imagename=flask-heroku-docker</pre><h4>Step 4: Set up the Database Access in the application.</h4><p>The database that we will add will give us the ability to create and store user records. To create users, we will need administrators who have authority to create users. So we will have two tables,one for the users and the other for the administrators. For authorization, we will use JSON web tokens.</p><p>For database access to work, we will need to do the following:</p><ul><li>Create environment secrets</li><li>Create the application configuration</li><li>Create the database models</li><li>Create the routes to interact with the database</li><li>Create the database tables</li></ul><p>Let us create the environment variables first. Create a file called .env and enter the following values:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/81b6325e9438007650d3680e5a4f6df3/href">https://medium.com/media/81b6325e9438007650d3680e5a4f6df3/href</a></iframe><p>These are mostly configurations for the application as well as the database. Next, let us create the application configuration. Open up the api/config.py file and add the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/77822d3790c99371dbd476ffd91fee14/href">https://medium.com/media/77822d3790c99371dbd476ffd91fee14/href</a></iframe><p>This also just sets up the different configurations for development, staging, testing and production. Then to set the configuration, api/__init__.py and update it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4a4244448ed5e748b830069ebe9a3310/href">https://medium.com/media/4a4244448ed5e748b830069ebe9a3310/href</a></iframe><ul><li>This module first imports all the necessary modules, as well as the blueprints, extensions and helpers.</li><li>It starts by loading the environment variables, line 16, then checks if the all the environment variables are set in line 19 to 22. The function are_environment_variables_set is declared in the helpers module. Find it on the <a href="https://github.com/twyle/flask-heroku-docker/blob/development/api/helpers.py">project repo</a> and add it to your project.</li><li>It then creates the app instance on line 25, then registers the blueprints. At line 33, the environment is set using the set_flask_environment function also from the helpers module by passing in the app instance. This function uses the config class that we created earlier on. Check it out in the <a href="https://github.com/twyle/flask-heroku-docker/blob/development/api/helpers.py">project repo</a>.</li><li>We then initiate the database, the migration script as well as the flask JSON web tokes extension, then finally register an error handler for HTTP 400 error. The migrate instance is declared in the <a href="https://github.com/twyle/flask-heroku-docker/blob/development/api/extensions.py">extensions module</a>, while the error handler is declared in the <a href="https://github.com/twyle/flask-heroku-docker/blob/development/api/error_handlers.py">error_handlers</a> module.</li></ul><p>Let us briefly look at the two database models. The user model is declared in the default package within the models module. Open up api/blueprints/default/models.py and add the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/31d75be60c4412dd6373d18d0e43a0b2/href">https://medium.com/media/31d75be60c4412dd6373d18d0e43a0b2/href</a></iframe><p>This is a very simple model that represents a user with only an email address and whether or not they are active, which defaults to True. The id is automatically generated by the database.</p><p>Then create a new folder called auth in the blueprints folder. Within the auth folder, create a __init__.py file to mark it as a package, the create the models.py module to hold our models. The auth package declares all the functionality associated with creating and authorizing our admins. Open up the models.py file and enter the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4883cbfb764b638bf6966489fb350831/href">https://medium.com/media/4883cbfb764b638bf6966489fb350831/href</a></iframe><ul><li>It first imports the dataclass, some constants as well as the database instance. The constants are declared in the <a href="https://github.com/twyle/flask-heroku-docker/blob/development/api/blueprints/constants.py">constants module</a> within the blueprint package while the database instance db is declared in the <a href="https://github.com/twyle/flask-heroku-docker/blob/development/api/blueprints/extensions.py">extensions module</a> in the blueprints package. The <a href="https://github.com/twyle/flask-heroku-docker/blob/development/api/blueprints/extensions.py">extensions module</a> also contains the declaration of the flask JSON web token extensions declaration as well as the creation of the application logger.</li><li>The admin is a bit more complex, with an email, name and password.</li></ul><p>Some notes about the models:</p><ul><li>The user’s id and email cannot be null, and the id has to be an integer while the email has to be a string. Both have length limits and the email has a specific format that has to be adhered to. The email address also has to be unique. When creating the user, these are checked for, and if not met, an exception is raised. Check the <a href="https://github.com/twyle/flask-heroku-docker/blob/development/api/blueprints/default/helpers.py">helpers module</a> in the default package, where these checks are performed. The <a href="https://github.com/twyle/flask-heroku-docker/blob/development/api/blueprints/exceptions.py">exceptions module</a> in the blueprints package declares the exceptions that are raised when the user attributes fall outside of the specified values.</li><li>The admin’s email, name and password cannot be null. They also have length limits. The name and password have to be unique and alphanumeric, while the email has to adhere to a specific format. If any of these rules are not met, an exception is raised. Check the <a href="https://github.com/twyle/flask-heroku-docker/blob/development/api/blueprints/auth/helpers.py">helpers module</a> in the auth package for details.</li></ul><p>Finally let us look at the views that allow our application user to interact with our database. First, open up the api/blueprints/default/views.py file and update it with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/032d27e715060d4f16a163fd6ef9904f/href">https://medium.com/media/032d27e715060d4f16a163fd6ef9904f/href</a></iframe><ul><li>There’s the default route that simply informs you that this is an API.</li><li>Then we have routes for creating, updating, reading and deleting a user all which require authorization, the @jwt_required tag. There is also a route for getting all the users. These routes make use of methods declared in the <a href="https://github.com/twyle/flask-heroku-docker/blob/development/api/blueprints/default/helpers.py">helpers module</a> to handle their respective requests.</li></ul><p>Then open up the api/blueprints/auth/views.py file and update it with the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/eb30427d2180b174ecc1f2a984d35ca4/href">https://medium.com/media/eb30427d2180b174ecc1f2a984d35ca4/href</a></iframe><ul><li>In this module, we are mostly interested in the register and login functionality. The other routes are still under development. The register route allows one to create an admin user. Once created, the login route allows the admin to log in and obtain an access token which they can use to create a new user.</li></ul><h4>Step 5: Test the application.</h4><p>To test the application, we first need to update the requirements. Open up the requirements.txt file and add the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e38f24924adca70e7cfb5ccec686789f/href">https://medium.com/media/e38f24924adca70e7cfb5ccec686789f/href</a></iframe><p>Then create a script for creating the database and a few users. Create a file called manage.py in the project root and add the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f03a767f9da5ceb1f3e61d34d1fb5f5e/href">https://medium.com/media/f03a767f9da5ceb1f3e61d34d1fb5f5e/href</a></iframe><ul><li>The create_db function simply creates all the database tables.</li><li>The seed_db function creates two dummy users.</li></ul><p>Then on the command-line:</p><pre>make install # install the requirements</pre><pre>sudo docker-compose up -d — build # start the database</pre><pre>python manage.py create_db # create the databases</pre><pre>python manage.py seed_db # creates two dummy users</pre><pre>make run # start the application</pre><p>To test the application:</p><p>curl <a href="http://localhost:5000/">http://localhost:5000/</a></p><p>You get:</p><p>{<br> “hello”: “from template api”<br>}</p><p>curl <a href="http://localhost:5000/users">http://localhost:5000/users</a></p><p>You get:</p><p>[<br> {<br> “active”: true, <br> “email”: “<a href="/service/mailto:test@example.com">test@example.com</a>”, <br> “id”: 1<br> }, <br> {<br> “active”: true, <br> “email”: “<a href="/service/mailto:test1@example.com">test1@example.com</a>”, <br> “id”: 2<br> }<br>]</p><p>Then to try the authorization, open up postman. Create a post request to the create route to create a new admin. This route will be localhost:5000/auth/register:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FBBymQki3Ktn1B3Ppe5uPQ.png" /></figure><p>Then create a post request to the login endpoint, localhost:5000/auth/login to get a refresh and access tokens:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*g5Im7fqp0EjPpSvde6tVDg.png" /></figure><p>Create a new post request to localhost:5000/user:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4sV1qcSZ2Y53AzRhUYgIUg.png" /></figure><p>Then to add authorization, copy the access token that you got when you registered a new admin, then under the <strong>Authorization </strong>tab, in the <strong>Type</strong> drop-down select Bearer Token:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*hVJ-sGGIICQqqhzRPAalmg.png" /></figure><p>Then paste it into the <strong>Token</strong> field and press send:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gaemZbN-YggJT_pyuDOBMw.png" /></figure><p>You should have created a new user:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*08WTuHKwY2bQ-yo4RBkJEA.png" /></figure><p>Now let us try the container. Stop the running flask server, then startup the docker container:</p><pre>source .env # Make the env variables available for the docker build</pre><pre>make build imagename=flask-heroku-docker</pre><pre>make run-docker imagename=flask-heroku-docker</pre><p>curl localhost:5000/users # to see the newly created user (we left the database containers running):</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*TVcew-Z0pr_sDDj88f4HzQ.png" /></figure><p>And with that we have successfully created and containerized a flask app that connects to a database and allows authorized users to create other users.</p><h4>Step 6: Provision a database on Heroku.</h4><p>Head on over to Heroku and create a new account or sign into your existing account. To create a database, you will need to authenticate your account, by providing a credit card. Then create a dummy app, since we will need one to create a database.</p><p>On the dashboard page, select the create drop-down and select new app:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5R0JPfGd_es2_GhnLLVQvQ.png" /></figure><p>Give it a unique name and press the <strong>Create app</strong> button:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rw2lWJMW6MJZwSl4lpBB3Q.png" /></figure><p>Head on over to <a href="https://elements.heroku.com/addons">heroku addons</a>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jGrxcpeErEulyAOpqOJl1g.png" /></figure><p>Scroll down and select <strong>Heroku Postgres</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xr69JUE2gispG39HJ2Fk0Q.png" /></figure><p>Scroll down to the <strong>Plans and Pricing</strong> section and select <strong>Hobby Dev </strong>then click on <strong>Install Heroku Postgres</strong> button:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*FykIdDxaO2z_6l6YpgwCjQ.png" /></figure><p>Select the app that we created and click on the <strong>Submit Order Form</strong>:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Oxsf8O6ehtwyzYgiC3w6iA.png" /></figure><p>And with that, you will have created a PostgreSQL database on Heroku. Stop the docker container running our flask server as well the containers running the databases.</p><p>Conclusion</p><p>In this article, we did the following:</p><ul><li>Tested that our application works, both on our computer and in a container.</li><li>Set up two databases by creating two postgres containers, one for development and one for testing, using docker-compose.</li><li>Connected to the created databases using docker compose.</li><li>Updated the docker image that our application uses to python 3.9.5</li><li>Set up our application to connect to the created databases as well as log the interactions.</li><li>Created authorization for users using the flask JSON web extension</li><li>Tested that our application worked by connecting to the database and creating new users</li><li>Created a database on Heroku</li></ul><p>In the next tutorial, we will connect to the database created on Heroku from our computer to test out the connection. We then will provide databases using github actions on GitHub, to test our application as well as create secrets on GitHub and Heroku for our application. We then will test our application with the created Heroku database.</p><p>That’s it for this article. I hope you enjoyed it and learnt something. Give it a clap or share it out and do not hesitate to reach out to me in-case of an issue. The code for this application is here <a href="https://github.com/twyle/flask-heroku-docker">flask-heroku-docker</a>. I am Lyle, a junior software engineer with a passion for developing, testing and deploying scalable services. You can find me on <a href="https://twitter.com/lylethedesigner"><em>twitter</em></a>,<a href="https://www.linkedin.com/in/lyle-okoth/"><em> linkedin,</em></a> <a href="https://github.com/twyle"><em>github</em></a> and here’s my <a href="https://twyle.github.io/portfolio-simple/"><em>portfolio</em></a>. See you next time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b209c7a65c6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How To Create a GitHub Action that Shows Code Coverage using Pytest: Part Two.]]></title>
            <link>https://medium.com/@lyle-okoth/how-to-create-a-github-action-that-shows-code-coverage-using-pytest-part-two-57d38a54064b?source=rss-2637c671f6af------2</link>
            <guid isPermaLink="false">https://medium.com/p/57d38a54064b</guid>
            <category><![CDATA[github-api]]></category>
            <category><![CDATA[code-coverage]]></category>
            <category><![CDATA[github-actions]]></category>
            <dc:creator><![CDATA[Lyle Okoth]]></dc:creator>
            <pubDate>Mon, 13 Jun 2022 09:18:42 GMT</pubDate>
            <atom:updated>2022-06-13T09:18:42.144Z</atom:updated>
            <content:encoded><![CDATA[<h4>Create a GitHub Issue.</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*alJeETCm-Eoxv07G.png" /></figure><p>This is the third article in a series that takes the reader through the process of creating a GitHub action that shows code coverage for their python projects hosted on GitHub. In the <a href="https://medium.com/@lyle-okoth/how-to-create-a-pytest-github-action-that-shows-code-coverage-part-two-43d4e4fb20a8">previous tutorial,</a> the following were accomplished:</p><ul><li>The Action code was created that runs pycov on a given project, then the results were displayed at the action output.</li><li>The GitHub Action was deployed to the Market place.</li><li>The Action was tested on a repository.</li></ul><p>In this article we will do the following:</p><ul><li>Replace the python docker image with a newer version.</li><li>Update the action such that it creates a GitHub issue when run.</li></ul><h4>Step 1: Update the docker image</h4><p>Open up the Dockerfile and update it with the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/b54cafa09e15353d04011658d169d2c3/href">https://medium.com/media/b54cafa09e15353d04011658d169d2c3/href</a></iframe><ul><li>We replaced the python3 image with python3.9.5 image since the previous image could not properly install numpy that we will need.</li><li>We set the working directory as /usr/src/app</li><li>The ENV PYTHONDONTWRITEBYTECODE 1 command prevents Python from writing pyc files to disc (equivalent to python -B <a href="https://docs.python.org/3/using/cmdline.html#id1">option</a>)</li><li>The ENV PYTHONUNBUFFERED 1 command prevents Python from buffering stdout and stderr (equivalent to python -u <a href="https://docs.python.org/3/using/cmdline.html#cmdoption-u">option</a>)</li><li>We then upgrade pip, copy the requirements file into the application directory and then install them.</li><li>We then copy the project files into the working directory and set the entry point.</li></ul><h4>Step 2: Update the Action Code</h4><p>Open the main.py file and update it with the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cd196fe4473b43acc758578abf81b393/href">https://medium.com/media/cd196fe4473b43acc758578abf81b393/href</a></iframe><ul><li>We have removed the code for generating the coverage report. In this updated version, we will simply use the output of pycov to generate the report.</li><li>The action still takes in the four inputs, code_directory, test_directory, pycov_config_file and the pytest_config_file as well as a few more that we will see later.</li><li>These are then used together with subprocess to run pytest with coverage. The output from the command is then stored for later use.</li><li>We then use the process output to generate the GitHub issue. If the action is able to generate the GitHub issue, we set the output, TESTCOVERAGE, to true, otherwise it defaults to false.</li></ul><p>Let us then create the create_issue function. Create a folder called helpers and within the folder create an __init__.py file as well as create_issue.py file. Open the create_issue.py file and add the following:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1968045c39bab2b806449de3dd2e388e/href">https://medium.com/media/1968045c39bab2b806449de3dd2e388e/href</a></iframe><ul><li>We first get the token as an environment variable. We need a github token in order to be able to create an issue on GitHub. This token is supplied by GitHub and later on we will see how to supply this token in a workflow.</li><li>We then get the repository details which are also supplied as environment variables, but this time by GitHub. The supplied value is in the form username/repo-name. We split this value to get the username and the repository name.</li><li>In the create_df function, we use the output from subprocess to generate the data frame for the tests.</li><li>Then in the create_markdown function, we use the data frame created with the above function to generate markdown that would be rendered by GitHub.</li><li>Then using the markdown string created with the above function, we create a GitHub issue. We supply the title, the body, the assignee as well as the label. Then using the username, repository name and the access token, we make a request to GitHub to create a new issue. If the request is successful, we return true otherwise false is returned.</li></ul><h4>Step 3: Update the Metadata file</h4><p>Open up the action.yml file and update it with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a3dca794351532b6771a04ea07cb9c09/href">https://medium.com/media/a3dca794351532b6771a04ea07cb9c09/href</a></iframe><ul><li>We simply added github_token to the inputs.</li><li>We also changed the output to show whether or not the action run was successful.</li></ul><h4>Step 4: Update the Workflow file</h4><p>Then to use our updated workflow, we need to update the workflow file. Open up the .github/workflows/main.yml and update it with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/61924d5bca76672ebca7fe6ddfa3156f/href">https://medium.com/media/61924d5bca76672ebca7fe6ddfa3156f/href</a></iframe><ul><li>We are using version 0.15.0 in my case, but for you this value will depend on how you tag the updated action.</li><li>For the inputs, we supply the code directory and test directory as before then add a new entry in line 20. This is how we supply the token to our action. The token is supplied by github.</li><li>Then we display the action output.</li></ul><h4>Step 5: Update the requirements file</h4><p>We then need to update the project requirements to include the requests library as well as pandas. Open the requirements.txt file and update it with:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cefb25aa6eb774430fae722eeb39cf9b/href">https://medium.com/media/cefb25aa6eb774430fae722eeb39cf9b/href</a></iframe><h4>Step 6: Update the project description</h4><p>You can also update the project description to show your users how to use the action. Open up README.md and update it with:</p><pre><strong>## Inputs</strong></pre><pre><strong>## `codedirectory`</strong></pre><pre><strong>**Required**</strong> The directory containing the source code. Default `”.”`.</pre><pre><strong>## `testdirectory`</strong></pre><pre><strong>**Required**</strong> The directory containing the tests. Default `”tests\”`.</pre><pre><strong>## `pycovconfigfile`</strong></pre><pre><strong>**Optional**</strong> The pycov configuration file Default `”.coveragerc”`.</pre><pre><strong>## `pytestconfigfile`</strong></pre><pre><strong>**Optional**</strong> The pytest configuration file. Default `”setup.cfg”`.</pre><pre><strong>## Outputs</strong></pre><pre><strong>## `testcoverage`</strong></pre><pre>The Test coverage</pre><pre><strong>## Example usage</strong></pre><pre>```</pre><pre>- name: oryks code coverage action</pre><pre>    id: selftest</pre><pre>    uses: twyle/oryks-code-coverage@v1.0.0</pre><pre>    with:</pre><pre>      codedirectory: src/</pre><pre>      testdirectory: tests/</pre><pre>      github_token: ${{ github.token }}</pre><pre>    - name: action output</pre><pre>      run: |</pre><pre>      echo “${{ steps.selftest.outputs.testcoverage }}</pre><pre>```</pre><pre>[bandit-image]: <a href="https://img.shields.io/badge/security-bandit-yellow.svg">https://img.shields.io/badge/security-bandit-yellow.svg</a></pre><pre>[bandit-url]: <a href="https://github.com/PyCQA/bandit">https://github.com/PyCQA/bandit</a></pre><pre>[isort-image]: <a href="https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&amp;labelColor=ef8336">https://img.shields.io/badge/%20imports-isort-%231674b1?style=flat&amp;labelColor=ef8336</a></pre><pre>[isort-url]: <a href="https://pycqa.github.io/isort/">https://pycqa.github.io/isort/</a></pre><h4>Step 7: Tag and Push your Code to GitHub</h4><p>Next, just commit your code, tag it and push it to GitHub (make sure you are in a feature branch since their is a pre-commit hook that we set up that prevents commit to the development branch):</p><pre>source venv/bin/activate</pre><pre>git add .</pre><pre>git commit -m “feat: the action creates an issue.”</pre><pre>git push </pre><pre>cz bump</pre><p>The output of cz bump should be a git tag. Use this to update the workflow file. Then once again, commit your code:</p><pre>git add .</pre><pre>git commit -m “fix: using a working version.”</pre><pre>git push origin vx.x.x # use the tag generated above</pre><h4>Step 8: The Action output</h4><p>When successfully ran, the action will generate an issue:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QuH5IYJGAsSWzzbmzgO2zQ.png" /></figure><p>And with that, we have an action that when successfully run, generates coverage for a given project then creates an issue.</p><h4>Conclusion</h4><p>In this article, we did the following:</p><ul><li>Created a new Dockerfile based on python 3.9.5</li><li>Updated the action code, such that on successful runs, our action generates an issue that indicates the coverage</li><li>Updated the metadata such that the action gets a github token that allows access to the github api.</li><li>Tested the action.</li></ul><p>In the next article, we will handle failing tests, as well as allow the user to specify threshold for failing or passing coverage.</p><p>The complete code can be found on the <a href="https://github.com/twyle/oryks-code-coverage">project github page</a>. I’m Lyle, a junior software engineer with a passion for developing, testing and deploying scalable services. You can find me on <a href="https://twitter.com/lylethedesigner"><em>twitter</em></a>,<a href="https://www.linkedin.com/in/lyle-okoth/"><em> linkedin,</em></a> <a href="https://github.com/twyle"><em>github</em></a> and here’s my <a href="https://twyle.github.io/portfolio-simple/"><em>portfolio</em></a>. See you next time.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=57d38a54064b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>