<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Dane Bulat on Medium]]></title>
        <description><![CDATA[Stories by Dane Bulat on Medium]]></description>
        <link>https://medium.com/@dane-bulat?source=rss-24e03e26bd42------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/proxy/1*TGH72Nnw24QL3iV9IOm4VA.png</url>
            <title>Stories by Dane Bulat on Medium</title>
            <link>https://medium.com/@dane-bulat?source=rss-24e03e26bd42------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 11:24:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@dane-bulat/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[An Introduction to Emacs Lisp: Working with Files and Their Attributes]]></title>
            <link>https://dane-bulat.medium.com/an-introduction-to-emacs-lisp-working-with-files-and-their-attributes-4c0777601e76?source=rss-24e03e26bd42------2</link>
            <guid isPermaLink="false">https://medium.com/p/4c0777601e76</guid>
            <category><![CDATA[emacs]]></category>
            <category><![CDATA[programming-languages]]></category>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[linux-tutorial]]></category>
            <category><![CDATA[emacs-lisp]]></category>
            <dc:creator><![CDATA[Dane Bulat]]></dc:creator>
            <pubDate>Sun, 31 Jan 2021 21:34:34 GMT</pubDate>
            <atom:updated>2021-02-01T04:41:03.003Z</atom:updated>
            <content:encoded><![CDATA[<h4>A hands-on introduction to the Emacs Lisp scripting language and working with files in Emacs</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N0jHVILNezy4kcN9fAAsLA.png" /></figure><h3>Introduction</h3><p>Emacs Lisp is a scripting language built into Emacs. Most of the editing functionality Emacs provides is implemented in Emacs Lisp in the form of callable functions. As well as being able to call any built-in Emacs function, it is also possible to define functions ourselves using the Emacs Lisp scripting language.</p><p>This article provides an introduction to working with the Emacs Lisp language as a whole. We go through how to develop several functions that deal with accessing and outputting items on the file system. Essential features of the Emacs Lisp language will be discussed along the way, such as defining functions, creating variables, looping, and list manipulation to name a few. With this in mind, each section of the article can serve as either an introduction or refresher on using certain features of Emacs Lisp. In addition to fundamental language features, we will also take advantage of built-in Emacs functions that deal with the file system.</p><p>Going into more detail, we will implement some functions that deal with retrieving and outputting the attributes of a requested file, directory, or symbolic link on the file system. These attributes convey important information such as the item’s owner, the group it belongs to, its byte size, permissions, and so on — very similar to what is output after running the ls -l command in a terminal. At a high level, we will use Emacs Lisp to develop functions that do the following tasks:</p><ul><li>Retrieve the name of a requested file along with its attributes.</li><li>Format output correctly such that information is easy to read.</li><li>Output file and directory information to the *Messages* buffer.</li></ul><p>Our functions will also be interactive, which means that they can be invoked just like any other command in Emacs.</p><h4>Source Code</h4><p>A <a href="https://github.com/danebulat/elisp-samples">Git repository</a> has been set up on Github to accompany this article. You can find all the discussed code in a single source file called file-attributes.el, which can be downloaded in a couple of ways.</p><p>One option is to invoke cURL on the command line to download the file without its parent repository:</p><pre>$ curl -L <a href="https://raw.githubusercontent.com/danebulat/elisp-samples/master/file-attributes/file-attributes.el">https://raw.githubusercontent.com/danebulat/elisp-samples/master/file-attributes/file-attributes.el</a> &gt; file-attributes.el</pre><p>Alternatively, invoke git if you would like to download the entire elisp-samples repository. Keep in mind that this repository may contain additional directories and source code for other published Emacs Lisp articles:</p><pre>$ git clone <a href="https://github.com/danebulat/elisp-samples">https://github.com/danebulat/elisp-samples</a></pre><p>Feel free to refer to the source code whilst reading through this article.</p><h3>Emacs Refresher</h3><p>This section will serve as a quick refresher on how to accomplish certain tasks within Emacs and <a href="https://github.com/hlissner/doom-emacs">Doom Emacs</a> — the Emacs distribution I currently use for development. if you are completely new to Emacs, I certainly recommend installing Doom Emacs, especially if you plan on following this tutorial.</p><p><em>Doom Emacs includes many fantastic development modules and sensible default key bindings out-of-the-box. If you are a developer who wants an open source, extensible, and configurable IDE, Doom Emacs will be my number one recommendation. The </em><a href="https://github.com/hlissner/doom-emacs/blob/develop/docs/getting_started.org"><em>getting started guide</em></a><em> on Github is a great resource for setting it up on your system.</em></p><p>If you are using another distribution of Emacs or your own configuration, I trust you will be able to find the correct key bindings for certain commands mentioned in this article.</p><h4>Evaluating Emacs Lisp Code</h4><p>Functions and variables written in Emacs Lisp must be installed from within an Emacs session before they can be accessed. In other words, functions and variables must be <em>evaluated</em> by the Emacs Lisp interpreter so they can be utilized in either source code or via a regular Emacs command.</p><p>We will firstly take a look at evaluating a variable. Variables in Emacs Lisp are usually created by making a call to the setq function. A variable name along with its initial value are passed as arguments to this function. Its syntax can be demonstrated with some pseudo code:</p><pre>(setq variable-name initial-value)</pre><p>Go ahead and create an Emacs Lisp file called something like intro.el so we can evaluate some code. Doom Emacs will recognize that files with an .el extension contain Emacs Lisp source code. Consequently, a major mode called Emacs Lisp mode will be enabled for the buffer, which will add functionality such as syntax highlighting, correct tabbing behavior, and a completion menu that will pop up as we type Emacs Lisp code.</p><p>Let’s make a call to the setq function to define a variable called greeting and initialize it to the string &quot;Hello, world!&quot;. Notice that strings in Emacs Lisp are surrounded by double quotes:</p><pre>(setq greeting &quot;Hello, world!&quot;)</pre><p>You will quickly realize that parenthesis are used extensively in Emacs Lisp. This is because everything we define in Emacs Lisp is in fact a list, and a list is always enclosed in opening and closing parenthesis. In addition to variables, other language facilities such function definitions, if statements, and loop constructs are also defined using list syntax. The Emacs Lisp interpreter (the program which evaluates and runs our code) has the job of deducing what each list in our source code represents.</p><p>Whitespace also separates a function’s name and its subsequent arguments. It is therefore important to remember that specifying a comma in between arguments will cause the interpreter to throw an error.</p><p>Another interesting feature within Emacs Lisp mode is the ability to evaluate source code directly from within a source file. Individual lines, blocks, or functions may be evaluated by issuing the appropriate command. For example, to evaluate our greeting variable in Doom Emacs, place your cursor on the line where it is defined — the convention is to navigate your cursor to the last closing parenthesis on the line. Then press the key combination gr followed by RETURN to invoke a command called +eval:region. A floating line of output will display the return value of the expression that was just evaluated.</p><p><em>If you have evil mode enabled, press </em><em>S-% (hold the </em><em>Shift key and press the key that is mapped to the percent sign, which is usually </em><em>5) to move your cursor to the opening parenthesis of a list. Repeat this combination to move the cursor to the closing parenthesis of a list.</em></p><p>Now our greeting variable has been evaluated, it can be referenced in any other part of the source file. For example, we can pass it to the message function to perform some additional formatting:</p><pre>(message &quot;The value of greeting variable: %s&quot; greeting)</pre><p>This line of code calls the message function and outputs our greeting string along with some additional text before it. Notice that we include the %s format specifier in the first argument to act as a string placeholder. Our variable is passed as the second argument to fill in this placeholder. Moving your cursor to the closing parenthesis and invoking gr will output our new message.</p><p>We know how to evaluate single lines of code, but what if we have a function that is comprised of many lines of code? Let’s define a simple function and take a look at how we can evaluate it in our source file. The syntax for defining functions in Emacs Lisp is displayed below:</p><pre>(defun function-name (arg1, arg2)<br>   &quot;Document string explaining ARG1 and ARG2.&quot;<br>   ;; body<br>   )</pre><p>As with every other language item in Emacs Lisp, functions are enclosed in parenthesis. The defun macro is specified immediately after the opening parenthesis. This keyword lets the interpreter know that it should evaluate the list definition as a function.</p><p>The function’s name is specified next, followed by its arguments in another set of nested parenthesis. If a function doesn’t require any arguments, an empty list () is specified after the function name. A documentation string follows the argument list, which should explain what the function does as a whole, along with the type of argument(s) it is expecting. Code specific to the function follows a documentation string, which defines that function&#39;s behavior.</p><p>We also use two semi-colons in the code snippet above to demonstrate Emacs Lisp comments. An arbitrary sequence of semi-colons denote the start of a comment in Emacs Lisp. However, it is not uncommon to see comments that start with only a single semi-colon.</p><p>With this knowledge of functions under our belt, let’s define a simple function which outputs a string we pass as its only argument:</p><pre>(defun output-message (string-to-output)<br>   &quot;A simple function that outputs STRING-TO-OUTPUT via the<br>`message&#39; function.&quot;<br>   (message &quot;Output message: %s&quot; string-to-output))</pre><p>Notice that function arguments are specified in uppercase within a document string. This convention makes it easy for developers to find where arguments are mentioned within documentation strings— particularly long ones.</p><p>When it comes to evaluating a function, the gr command is no longer sufficient, because we need to take into account multiple commands on multiple lines. In these cases, we can use the C-M-x binding to evaluate a block of code. This means pressing and holding the Control and Alt keys, followed by pressing the x key. If done successfully, a floating prompt displaying our function&#39;s name will be output after its final closing parenthesis in the form of =&gt; output-message.</p><p>Now our output-message function is installed, it can be invoked from within Emacs in a number of ways. One way is to make a call in source code, very similar to the way we called message earlier:</p><pre>(output-message &quot;Hello, world!&quot;)</pre><p>Now repeat the usual process of pressing gr RETURN after placing your cursor on the closing parenthesis of the function call. This will call our function and output its return value. The return value of a function is whatever value is returned by its last expression.</p><p>Another way to invoke functions in Emacs is via the M-: key binding, which means to hold down ALT and SHIFT, followed by pressing the semi-colon (;) key - or whatever key is mapped to the colon character. An Eval: prompt will be displayed at the bottom of Emacs and will be waiting for input. At this point, type (output-message &quot;This is cool!&quot;) followed by pressing RETURN to invoke our function. Its output will now be shown in the status line. We can use M-: to invoke any installed command in our Emacs session.</p><p>The last key binding we will mention in this section is one that comes configured with Doom Emacs, namely SPC c e. After using this key combination, Emacs will run the +eval/buffer-or-region command. As a result, the interpreter will evaluate all Emacs Lisp code in the current buffer. This is a good way to check that no errors exist within a source file. If an error does exist, the Emacs Lisp debugger will let us know.</p><h3>Working With File Attributes</h3><p>This section will take a look at how to retrieve and output attributes for any given item on the file system. It would also be nice to output this data in a consistent and readable way. To meet this goal, we will develop three functions that each perform a specific task that when combined together, will enable us to output properly formatted file attribute data to the built-in *Messages* buffer.</p><p>To retrieve attributes of a given file, we will make use of a built-in Emacs function called file-attributes. The only argument we must provide this function is a string representing a valid filename on the file system. For completeness, let&#39;s take a look at its full signature:</p><pre>(file-attributes FILENAME &amp;optional ID-FORMAT)</pre><p>As its name suggests, the &amp;optional keyword tells the Emacs Lisp interpreter that subsequent arguments are completely optional, and do not need to be passed a value at invocation time. One optional argument called ID-FORMAT is given for the file-attributes function, which simply dictates whether to return a file&#39;s user ID and group ID attributes as a string or integer value. A string value is returned If nothing is given to this argument, which is what we want.</p><p>Moving forward, a populated list of data is returned after evaluating a call to file-attributes. Each item in the list corresponds to a particular attribute of the file we requested. In total, twelve items will make up the list of attributes:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6a045186c9e09f96c54cacf5ccfc0c18/href">https://medium.com/media/6a045186c9e09f96c54cacf5ccfc0c18/href</a></iframe><p>Of particular interest is the list’s first element, which tells us whether the requested item is an actual file, directory, or symbolic link. For example, it will contain a true value if we requested a regular file — true is specified with the t character in Emacs Lisp.</p><p>Moreover, items that contain a time stamp will store the number of seconds since 1970-01-01 00:00:00. Emacs documentation points out that the current-time function returns its value in the same way. We will therefore need to format this value into a readable string before writing it to a target buffer.</p><p>At this point, we can test out file-attributes by making a call to it and examining its return value within Emacs. For example, evaluate the following code to output attributes of your .bashrc file:</p><pre>(file-attributes &quot;~/.bashrc&quot;)<br>    -| (nil 1 1000 1000<br>           (24588 61261 769551 700000)<br>           (24574 56146 828467 818000)<br>           (24574 56146 828467 818000)<br>           1693 &quot;-rw-r--r--&quot; t 14686928 66310)</pre><p>When observing the output, we realize that it would be beneficial to write a function that formats this data into a more readable form. If we don’t do any formatting, it will be very difficult to determine what the data in each item conveys.</p><p>Let’s now switch our focus on accessing items within the list returned by file-attributes. One way to do this would be to create a variable that points to the list, and then passing it to list functions, such as car, cdr and nthcdr to output its individual items:</p><pre><strong>;; Variable that points to the file attributes list</strong><br>(setq attributes (file-attributes &quot;~/.bashrc&quot;))</pre><pre><strong>;; Output first item</strong><br>(car attributes)</pre><pre><strong>;; Output fifth item (last access time)</strong><br>(car (nthcdr 4 attributes))</pre><pre><strong>;; Another way to output the fifth item</strong><br>(car (cdr (cdr (cdr (cdr attributes)))))</pre><pre><strong>;; Output tenth item (file modes)</strong><br>(car (nthcdr 9 attributes))</pre><p>The car function returns the first item in a list, while the cdr function returns the list starting at its second item. Multiple nested calls to cdr along with car allow us to traverse a list and access individual items.</p><p>An easier method to access list items is also demonstrated above, which is to use the nthcdr function. All we need to do here is pass an index number along with a list to work on. For example, (nthcdr 4 attributes) returns the fifth item in the attributes list, which is the &quot;date last accessed&quot; time stamp. Remember that a list&#39;s first item is always at index zero, meaning the nth item&#39;s index number is always n - 1.</p><p>We could certainly utilize these functions to retrieve individual items from a list. However, Emacs also provides some functions that specifically deal with accessing items from a list returned by file-attributes. These functions are described in the next table:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/05447026d16d2ce44f769c7d94b45a51/href">https://medium.com/media/05447026d16d2ce44f769c7d94b45a51/href</a></iframe><p>Let’s test these functions out by evaluating some commands. We will start things off by creating a variable that points to a list of file attributes, and then pass it to a couple of the file-attribute-* functions detailed in the table above:</p><pre><strong>;; Create a variable that points to the file attributes list</strong><br>(setq attributes (file-attributes &quot;~/.bashrc&quot;))</pre><pre><strong>;; Pass variable to file attribute functions</strong><br>(file-attribute-user-id attributes)<br>(file-attribute-size attributes)</pre><p>That’s all very well and good, but we are not limited to only passing predefined variables to functions — we can also call functions directly within other functions. We know this because every expression in Emacs Lisp must return a value. As such, when a function is called within the parenthesis of another function, its return value is passed as an argument. For example, instead of passing our attributes variable to a function, we can just as easily call file-attributes directly:</p><pre>(file-attribute-device-number (file-attributes &quot;~/.bashrc&quot;))<br>(file-attribute-inode-number (file-attributes &quot;~/.bashrc&quot;))</pre><p>Taking our discussion a step further, functions can be invoked within calls to other functions at any level. In these cases, the Emacs Lisp interpreter will evaluate the inner-most invocation first, and work its way outward, gradually moving up the nested function hierarchy. With that said, consider what happens in the following examples:</p><pre>(message &quot;Last access time: %s&quot;<br>         (file-attribute-access-time<br>          (file-attributes &quot;~/.bashrc&quot;)))</pre><pre>(message &quot;Last modification time: %s&quot;<br>         (file-attribute-modification-time<br>          (file-attributes &quot;~/.bashrc&quot;)))</pre><p>The first call to message contains an invocation of file-attribute-access-time, which in turn contains an invocation to file-attributes. The Emacs Lisp interpreter will firstly evaluate (file-attributes &quot;~/.bashrc&quot;), followed by (file-attributes-access-time (...)) , and finally (message ...). The same situation occurs in the second message invocation, where the inner most function is evaluated first.</p><h3>Developing Functions to Handle File Attributes</h3><h4>Appending a Pair of Strings to a List</h4><p>The first function we will implement will have the responsibility of appending two string items to a list. This list will be passed as an argument to the function and subsequently returned once the two string items are appended to it. Because the appended string items are to represent exactly one line of output, this function will be called add-line. Let&#39;s firstly take a look at its signature and describe what particular processing will take place:</p><pre>(add-line (description attribute list))</pre><p>Our add-line function must receive three arguments:</p><ul><li>description : A string describing the passed attribute argument.</li><li>attribute : An item from a list returned by the file-attributes function.</li><li>list : A list containing description and attribute string items.</li></ul><p>We could construct a list by invoking add-line in the following way:</p><pre>(add-line &quot;File modes&quot; <br>    (file-attribute-modes attributes) output-list)<br>(add-line &quot;File size (in bytes)&quot; <br>    (file-attribute-size attributes) output-list)</pre><p>We will discuss how add-line is used within another function we will develop very shortly. Without getting too far ahead of ourselves, we should also talk about the fact that our list should only contain string data; the reason being is that we want to eventually write our list data to a buffer. Because some attributes in a list returned by file-attributes are not of a string type, we will need to convert these values to a string before appending them to our output list.</p><p>Now we understand what the add-line function should do along with some technical considerations, let&#39;s discuss its actual implementation next. If you are implementing the functions discussed in this article yourself, go ahead and copy the following code into an .el file in Emacs:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1ea413ddd2a30c863ad456fd12902279/href">https://medium.com/media/1ea413ddd2a30c863ad456fd12902279/href</a></iframe><p>The function starts by checking if its attribute argument is a nil value. This will occur if we retrieve attributes from a physical file on the file system. We invoke the equal function within a call to the when macro to compare the value of attribute to a nil value. As a result, attribute is set to a string value of &quot;nil&quot; if it originally contained a nil value:</p><pre>(when (equal attribute nil)<br>   (setq attribute &quot;nil&quot;))</pre><p>There will also be times when attribute contains a number, such as when it contains a file size or time stamp. To handle these situations, we invoke the stringp predicate function which returns t if its argument is indeed a string. Because we want to act when attribute does not point to a string, we wrap stringp inside a unless macro. In other words, unless attribute is a string, we execute some code to convert it to a string. This is accomplished by calling the number-to-string function and using its return value to overwrite attribute. In order for number-to-string to not throw an error, its argument must be an integer type:</p><pre>(unless (stringp attribute)<br>    (setq attribute (number-to-string attribute)))</pre><p>At this point, we can be sure both description and attribute are string values. The cons function is then called to prepend these variables to the passed list. The first argument to cons is always a single data item that is to be added to a list - if a list is provided for this argument, it will be appended as a single item (a list within a list). The second argument is the list we wish to add this item to:</p><pre>(setq list (cons description list))<br>(setq list (cons attribute list)))</pre><p>Lastly, setq returns list with our appended description and attribute items, which will cause add-line to also return list to its calling code. Feel free to evaluate add-line using C-M-x and make some test invocations to check its output:</p><pre><strong>;; Temporary code for testing the add-line function</strong><br>(setq attributes (file-attributes &quot;~/.bashrc&quot;))<br>(setq output-list ())</pre><pre>(setq output-list (add-line &quot;File modes&quot; <br>                  (file-attribute-modes attributes) output-list))</pre><pre>(setq output-list (add-line &quot;File size (in bytes)&quot; <br>                  (file-attribute-size attributes) output-list))</pre><pre>(message &quot;%s&quot; output-list)</pre><h4>Constructing a Formatted List</h4><p>The second function we develop will have the job of utilizing our add-line function to construct a list of &quot;attribute&quot; and &quot;description&quot; string pairs. We also want this function to return the constructed list so other functions can receive it and use its data in certain ways. Let&#39;s firstly take a look at the signature of our next function, which we shall name formatted-file-attributes:</p><pre>(defun formatted-file-attributes (filename))</pre><p>The function contains one argument where we must pass a valid file name from which we will receive its corresponding attributes. If a valid filename is not provided, the function will be certain to throw an error.</p><p>Before we walk through how formatted-file-attributes works, its full implementation is displayed in the next gist. Go ahead and input the code into your Emacs Lisp file if you are typing out each function yourself:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f282274a1e8bdd2a3fc61b8fe2ddddfb/href">https://medium.com/media/f282274a1e8bdd2a3fc61b8fe2ddddfb/href</a></iframe><p>Following the documentation string, a new function is called that we have yet to discuss called interactive. In our case, it enables us to pass a string to the function&#39;s filename argument when it is called interactively. A function in Emacs is called interactively when we invoke it using either SPC : in Doom Emacs, or M-x, which is available in both Doom Emacs and vanilla Emacs. These key bindings invoke the &quot;Meta-x&quot; command, which calls a function called execute-extended-command behind the scenes.</p><p>After invoking Meta-x and typing formatted-file-attributes followed by hitting Return, Emacs will prompt us to also input a filename that will be given to the function as its first argument. We set up this functionality by passing a string defined as &quot;fFile name: &quot; to the interactive function. The lower-case f signifies that we want to pass the name of an existing file. The following File name: text will be the prompt that is displayed when we call our function interactively:</p><pre>(interactive &quot;fFile name: &quot;)</pre><p>Our function continues to set up a couple of local variables. A list of local variables can be defined and provided an initial value inside an Emacs Lisp let statement. What makes local variables special is that they can only be accessed from within the function in which they are defined, and within the let statements outer-most parenthesis. If a variable defined outside a function has an identical name as a local variable defined inside a function, that outer variable is masked by the local variable. So be aware that you will not be able to access masked variables. In addition, a function&#39;s arguments are also set up as local variables.</p><p>Moving forward, we define two local variables called attributes and out-list. The attributes variable is initialized to point to the list returned by file-attributes, that is in turn passed the function&#39;s filename argument. As a result, we are able to access the attributes of our requested file through the attributes variable in subsequent code:</p><pre>(let ((attributes (file-attributes filename))<br>        (out-list ()))</pre><p>Secondly, the out-list variable is initialized to an empty list, which can be done by specifying an opening and closing parenthesis. This variable will be passed to add-line to construct a formatted list. It will be returned at the end of formatted-file-attributes so other functions can utilize its data for further processing, such as writing it to other buffers.</p><p><em>Notice how nested parenthesis are specified within a </em><em>let statement — an outer set of parenthesis contain each variable definition, which are also wrapped with parenthesis. Be careful to specify the order of parenthesis correctly if you are implementing this function yourself.</em></p><p>Our function then appends string items to the out-list variable by making use of our add-line function Notice that we use the setq function to continuously set out-list to the list returned by add-line. As a result, the string items appended in add-line are not lost when the program returns to the outer formatted-file-attributes function. For each attribute, we pass a hard-coded string as the description argument for add-line . When it comes to passing an item from the attributes list, we opt to call the built-in file-attribute-* functions:</p><pre>(setq out-list (add-line &quot;File type&quot;<br>        (file-attribute-type attributes) out-list))<br>(setq out-list (add-line &quot;Links&quot;<br>        (file-attribute-link-number attributes) out-list))</pre><pre><strong>;; And so on</strong></pre><p>One extra step is taken when a time stamp is passed to add-line. The built-in format-time-string function is called to format a time stamp into a readable string understandable to humans. A %- sequence is passed as the first argument to format-string, which defines how a time stamp will be formatted. We specify the %D sequence, which is a synonym for %m/%d/%y. As a result, a string is returned that shows the current month, day, and year. The second argument that format-string expects is the actual time stamp to work on:</p><pre>(setq out-list <br> (add-line &quot;Last Access Time&quot;<br>            (format-time-string &quot;%D&quot; <br>             (file-attribute-access-time attributes))<br>             out-list))</pre><p><em>Many </em><em>%- sequences are available in Emacs Lisp to format a date and time to your exact specifications. Refer to the </em><a href="https://www.gnu.org/software/emacs/manual/html_node/elisp/Time-Parsing.html"><em>Emacs official documentation</em></a><em> to discover the full set of sequences available.</em></p><p>The last line in formatted-file-attributes reverses the order of items in out-list such that its order mimics that of the list returned by file-attributes. We call a built-in function called nreverse and pass it out-list to do the reverse. This is wrapped inside a setq command, which will cause the function to return the final state of out-list:</p><pre>(setq out-list (nreverse out-list))))</pre><p>Some test calls to formatted-file-attributes can be made to make sure its working correctly. Remember to firstly move your cursor within the function itself and evaluate it with the C-M-x. From there, you can make a few calls directly in your source file and evaluate them with gr RETURN. The constructed list will be returned and displayed as output:</p><pre>(formatted-file-attributes &quot;~/.bashrc&quot;)</pre><h4>Outputting to the Messages Buffer</h4><p>The last functions we develop will be a wrapper around our formatted-file-attributes function. It will utilize the returned list to write its data somewhere. This new function will be called file-attributes-to-messages, and will be tasked to output file attribute information to the built-in Emacs *Messages* buffer:</p><pre>(defun file-attributes-to-messages (filename))</pre><p>Similar to our previous function, file-attributes-to-messages exposes a single argument called filename, where we supply a string representing the file whose attributes we wish to retrieve. The gist below lists the complete file-attributes-to-messages function. We will then discuss its contents:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2116d747cd63b5eb244c87f4d67fbbf0/href">https://medium.com/media/2116d747cd63b5eb244c87f4d67fbbf0/href</a></iframe><p>Notice that file-attributes-to-messages is also an interactive function, therefore allowing us to invoke it using the Meta-x (SPC : or M-x) key binding. When the function is invoked in this way, a prompt will again be displayed, instructing us to provide a file name.</p><p>One local variable called out-list is then defined in a let statement which is set up to point to the list returned by our format-file-attributes function. This means we can reference out-list at any time to access the list data we wish to output:</p><pre>(let ((out-list (format-file-attributes filename)))</pre><p>Within the let statement&#39;s outer-most parenthesis, we invoke message to write the requested file name to the *Messages* buffer. Notice that we specify an initial format specifier sequence as %20s. As a result, the &quot;description&quot; part of output will be rendered as a column of twenty characters. In other words, %Ns will render a string of N characters. A blank space character is prepended to our passed string until it reaches N characters:</p><pre>(message &quot;%20s: %s\n\n&quot; &quot;Filename&quot; filename)</pre><p>Our function then enters a while loop which will continue to run as long as out-list is pointing to items in the list. A single iteration firstly calls message to output the next &quot;description&quot; and &quot;attribute&quot; pair from the list. The last command results in the out-list pointer being shifted two items up the list. When the while condition is evaluated next, out-list will either be pointing to a &quot;description&quot; item, or nil. When it points to nil, we know that the entire list has has been written to the *Messages* buffer. At this point, the while loop will break:</p><pre>(while out-list<br>      (message &quot;%20s %s\n&quot; <br>               (car out-list) (car (nthcdr 1 out-list)))<br>      (setq out-list (cdr (cdr out-list))))))</pre><p>Notice that we make a recursive call to cdr to make out-list shift two items up the list.</p><p>At this point, feel free to test file-attributes-to-messages by calling it interactively or directly in source code. After an invocation, file attribute information will be written to the *Messages* buffer:</p><pre>(file-attributes-to-messages &quot;~/.bashrc&quot;)</pre><p>The *Messages* buffer can be opened in Doom Emacs by using the SPC-b B key binding. From there, select *Messages* from the list of buffers to open it in the current window. On vanilla Emacs, use the C-x b key binding, type *Messages* and press RETURN.</p><h3>Conclusion</h3><p>This article has provided a hands-on introduction to Emacs Lisp by walking through the development of three functions that deal with interacting with files. From here, you can add the functions we developed to your <a href="https://www.gnu.org/software/emacs/manual/html_node/emacs/Init-File.html">Emacs initialization file</a>, which will cause Emacs to automatically evaluate them when you launch a new session. For example, Doom Emacs will automatically evaluate our functions if they exist in the ~/.doom.d/config.el file.</p><p>To build upon the functions even further, I would recommend playing with the <a href="https://www.gnu.org/software/emacs/manual/html_node/elisp/Time-Parsing.html">sequence options</a> available to the format-time-string function to output timestamps in various formats. You can also read up on the <a href="https://www.gnu.org/software/emacs/manual/html_node/elisp/Formatting-Strings.html">many formatting options</a> that can be passed to the message function to further customize output to the *Messages* buffer.</p><p>Also keep an eye out for future Emacs Lisp and Doom Emacs related articles from myself. We will build upon the content presented in this article when we look at developing more custom functionality within Emacs.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4c0777601e76" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python: Building an Image Manipulation Tool (Part 2)]]></title>
            <link>https://dane-bulat.medium.com/python-building-an-image-manipulation-tool-part-2-21531e3c2609?source=rss-24e03e26bd42------2</link>
            <guid isPermaLink="false">https://medium.com/p/21531e3c2609</guid>
            <category><![CDATA[linux-tutorial]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[image-processing]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[application-development]]></category>
            <dc:creator><![CDATA[Dane Bulat]]></dc:creator>
            <pubDate>Wed, 13 Jan 2021 04:36:12 GMT</pubDate>
            <atom:updated>2021-01-22T06:05:54.100Z</atom:updated>
            <content:encoded><![CDATA[<h4>A tutorial and case study on developing an image processing application with Python</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*czoBVcxhb_bnSNC2Lu65XA.png" /></figure><h3>Introduction</h3><p>This is the second part of a tutorial and case study that details how to develop an image processing application with Python. The first article focused on setting up a virtual environment with <a href="https://pipenv.pypa.io/en/latest/">Pipenv</a> and installing our application’s package dependencies. Subsequent sections also discussed our tool’s command line arguments in addition to some example invocations to demonstrate its usage.</p><p>Refer to the content presented in <a href="https://dane-bulat.medium.com/python-building-an-image-manipulation-tool-part-1-8299dd97bfe8">part one</a> to get up to speed on the project. It is comprised of three sections that discuss the following topics:</p><ol><li><strong>Project Overview<br></strong>The project’s directories and source files are discussed. From there, we go through the process of creating a virtual environment with Pipenv.</li><li><strong>How to Use The Image Manipulation Tool<br></strong>Our tool’s usage and command line options are detailed. We also discuss the difference between positional arguments and optional arguments.</li><li><strong>Example Commands with Screenshots<br></strong>A few example invocations of our program are demonstrated to show how it can be used in real-world scenarios. Before and after screenshots accompany each example.</li></ol><p><a href="https://dane-bulat.medium.com/python-building-an-image-manipulation-tool-part-1-8299dd97bfe8">Python: Building an Image Manipulation Tool (Part 1)</a></p><p>This part will walk through our project’s source code. We will take a look at using the standard <a href="https://docs.python.org/3/library/argparse.html">argparse</a> Python module, as well as how to process images in different ways with the <a href="https://pillow.readthedocs.io/en/stable/">Pillow API</a>. Topics of discussion are organized into three sections:</p><ol><li><strong>Passing Command Line Arguments<br></strong>This section details how the standard argparse Python module is used to parse command line arguments and convert them into Python objects.</li><li><strong>Handling Parsed Arguments<br></strong>We then talk more about the program’s structure and adopted coding conventions in order to improve readability, prevent duplication, and maintain performance.</li><li><strong>Image Processing with Pillow<br></strong>Lastly, we discuss how our application integrates the Pillow API and inspect the code used for performing image processing operations.</li></ol><h4>Source Files</h4><p>A <a href="https://github.com/danebulat/py-image-manip-demo">GitHub repository</a> has been set up for readers wishing to download the application discussed in this article. Navigate to an appropriate place on your file system and invoke Git to clone the repository:</p><pre>$ git clone <a href="https://github.com/danebulat/py-image-manip-demo.git">https://github.com/danebulat/py-image-manip-demo.git</a></pre><p>Refer to<a href="https://dane-bulat.medium.com/python-building-an-image-manipulation-tool-part-1-8299dd97bfe8"> part one</a> of this tutorial if you need help installing a virtual environment, or would like to find out more about how to use the program.</p><h3>1. Parsing Command Line Arguments</h3><p>The first major task our program does is parse command line arguments. This entire process is handled in the parser.py module. Within it, the standard <a href="https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser">argparse</a> module is utilized to parse arguments out of sys.argv. A lot of useful functionality is provided in the argparse module and its corresponding <a href="https://docs.python.org/3/library/argparse.html#argumentparser-objects">ArgumentParser</a> object. This includes:</p><ul><li>The automatic generation of configurable help and usage messages.</li><li>Appropriate error messages are issued when a user provides invalid arguments.</li><li>A flexible API for defining both positional and optional arguments.</li><li>The ability to apply rules to arguments to govern their type and the number of values they can receive.</li></ul><p>Subsections that follow describe how the argparse API is used to define a usable command line interface. The official Python website also provides <a href="https://docs.python.org/3/library/argparse.html#dest">extensive documentation</a> that covers the entire module in a linear fashion. It is therefore a good resource to find out more information about its usage.</p><h4>Creating an Argument Parser</h4><p>The first line in our application makes a call to a method we define ourselves called parse_arguments():</p><pre>namespace = parser.parse_arguments()</pre><p>This line demonstrates that parse_arguments() is defined in the parser module and returns an object that is referenced by a variable called namespace. Providing a user has specified valid arguments on the command line, these arguments are converted into usable Python objects and attached to the object referenced by namespace.</p><p>Command line arguments are parsed before any image processing is performed within our program. In order to do this, we firstly instantiate a class defined in the argparse module of type ArgumentParser. This object provides methods for adding command line arguments, parsing input, and getting them back in the form of Python objects.</p><p>Let’s open up parser.py and inspect the parse_arguments() function to see how an ArgumentParser is created:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/649c799611955a23d5d42ff52e91ad26/href">https://medium.com/media/649c799611955a23d5d42ff52e91ad26/href</a></iframe><p>Our configured ArgumentParser object is created and referenced by a variable called parser. Notice that parameters are specified as keyword arguments in its constructor. The values we assign to each argument govern how the program&#39;s help text is displayed when issuing the -h and --help options. Let&#39;s take a look at what each parameter does:</p><ul><li>prog=&#39;imgmanip&#39;<br>A string representing the name of our program.</li><li>usage=&#39;%(prog)s FILES [-e ENHANCERS] [-f FILTERS] [--thumbnail] ...&#39; <br>A string describing the program’s usage. Displayed on the first line of the help text.</li><li>description=&#39;\Perform a range of manipulations on images files...&#39;<br>A description of what the program does. Displayed immediately after the usage string.</li><li>epilog=&#39;Part of the PIL tutorial at: https://dane-bulat.medium.com&#39;<br>Text that is displayed after the argument list.</li><li>formatter_class=argparse.RawDescriptionHelpFormatter<br>A class to format specific text output. We use a formatter class called RawDescriptionHelpFormatter to give us more control of how textual descriptions are displayed.</li><li>add_help=False <br>A boolean indicating whether to add a -h / --help option by default. Because we want to define a custom help message, this parameter is assigned a False value.</li><li>allow_abbrev=False<br> A boolean indication whether a long optional argument can be abbreviated if the abbreviation is unambiguous. For example, if assigned True, we could specify --res instead of --resize. Our program opts to turn this feature off and assigns False.</li><li>exit_on_error=True<br>Determines whether ArgumentParser exits and provides error information when an invalid argument is encountered. We opt to enable this functionality by assigning a True value.</li></ul><p>We take the opportunity to describe our program’s available filters and enhancers in the description text. This section usually consists of only a few sentences and is really meant to be brief. Developers wishing to stick to this convention could create an additional option, such as --usage, to output our very long description.</p><h4>Positional Arguments</h4><p>Once a configured ArgumentParser object is created, we call its <a href="https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.add_argument">add_argument()</a> method to define both positional and optional command line arguments. We start off by defining our program’s only positional argument that receives one or more image filenames:</p><pre>parser.add_argument(&#39;inputs&#39;, action=&#39;store&#39;, nargs=&#39;+&#39;,<br>                    help=&#39;A list of image files to process.&#39;)</pre><p>The first value given to add_argument() is always a Python string representing the name of an argument - inputs in our case. This name is displayed in the <em>usage</em> and <em>argument list</em> sections of the help text. It is therefore important to provide a name that is both short and easy to understand.</p><p>A keyword parameter called action describes the type of action that is taken when a particular argument is encountered on the command line. We instruct the parser to simply store the value given to our argument in a variable by providing a string defined as &#39;store&#39;. This means argparse will create an internal variable called inputs to store whatever is provided on the command line.</p><p>The <strong>type</strong> of variable a parser generates to store an argument’s values is governed by what is assigned to other parameters in add_argument(). One of them is a parameter called nargs, which we assign &#39;+&#39; above. The plus character means that an argument can receive one or more values. Our parser therefore determines that the inputs object is in fact a Python list. This also means that an error will be issued if no values are given to the inputs argument.</p><p>Lastly, a help string should always be provided for each argument added to a parser. As you would expect, this string is displayed in the program’s help text. It should provide a brief description of what the argument does.</p><h4>Special Optional Arguments</h4><p>Let’s switch our focus to talking about “special” optional arguments that are provided by the argparse module. These are the standard --help and --version options. The add_argument() method is called again to generate these options:</p><pre>parser.add_argument(&#39;-h&#39;, &#39;--help&#39;, action=&#39;help&#39;,<br>                    default=argparse.SUPPRESS,<br>                    help=&quot;Show this help message and exit.&quot;)<br>  <br>parser.add_argument(&#39;--version&#39;, action=&#39;version&#39;,<br>                    version=&#39;%(prog)s &#39; + get_version_string(),<br>                    help=&quot;Show program&#39;s version number and exit.&quot;)</pre><p>Just like positional arguments, a name must firstly be specified when defining optional arguments. A short name is given before a long name, and they are prefixed with one and two dash characters respectively.</p><p>When it comes to handling the --help argument, a special &#39;help&#39; string is assigned to the action parameter. This instructs our parser object to print its associated help text and exit the program.</p><p>Similarly, the --version argument&#39;s action parameter is assigned a special &#39;version&#39; string. As a result, a keyword parameter called version is exposed that should be assigned the program&#39;s version string. We assign it a string containing a special format specifier denoted by %(prog)s - which returns the string we assigned to prog in the ArgumentParser constructor.</p><h4>Boolean Optional Arguments</h4><p>Arguments that are parsed into boolean objects must be assigned either &#39;store_true&#39; or &#39;store_false&#39; actions. We use this technique for detecting whether the --info or --thumbnail options were specified on the command line:</p><pre>parser.add_argument(&#39;-i&#39;, &#39;--info&#39;, action=&#39;store_true&#39;,<br>                    help=&#39;Display image properties.&#39;)<br>​<br>parser.add_argument(&#39;--thumbnail&#39;, action=&#39;store_true&#39;,<br>                    help=&#39;A thumbnail is generated.&#39;)</pre><p>We assign &#39;store_true&#39; action strings to both of our boolean arguments. As a result, our parser object will generate corresponding variables called info and thumbnail which will either contain True - if the argument was given, or False - if the argument was left out.</p><h4>Single Value Optional Arguments</h4><p>The --factor and --thumbnail-width optional arguments are configured to accept a single value:</p><pre>parser.add_argument(<br>    &#39;--factor&#39;, action=&#39;store&#39;, default=1.2, type=float,<br>    help=&#39;Float to specify the strength of enhancers.&#39;)<br>​<br>parser.add_argument(<br>    &#39;--thumbnail-width&#39;, action=&#39;store&#39;, type=int, default=128,<br>    help=&#39;Specify a width for the generated thumbnail.&#39;)</pre><p>Notice that we specify a default value that our parser will use to initialize the argument&#39;s internal variable if one if not provided on the command line. The factor argument is tied to a default value of 1.2, whilst the default width for a generated thumbnail is 128 pixels.</p><p>We also provide a type that the parser will use for generating an argument&#39;s internal variable. In this case, a factor is parsed into a float, while a thumbnail width is passed into an int.</p><h4>Optional Arguments with Pre-Determined Choices</h4><p>The choices keyword parameter allows us to provide a list of allowed values for a given argument. We take advantage of this functionality when defining the --flip and --rotate optional arguments:</p><pre>parser.add_argument(<br>    &#39;--flip&#39;, action=&#39;store&#39;, nargs=&#39;+&#39;, choices=[&#39;horz&#39;, &#39;vert&#39;],<br>    help=&#39;Flip an image vertically or horizontally.&#39;)<br>​<br>parser.add_argument(<br>    &#39;--rotate&#39;, action=&#39;store&#39;, choices=[90, 180, 270], type=int,<br>    help=&quot;Rotate an image in a 90 degree increment.&quot;)</pre><p>As demonstrated above, the --flip argument only accepts &#39;horz&#39; and &#39;vert&#39; strings. In addition, the --rotate argument can only be given three values — either 90, 180 or 270.</p><h4>Multiple Value Optional Arguments</h4><p>Similar to the inputs positional argument, we set the nargs parameter to a plus character (&#39;+&#39;) when defining the --filter and --enhancer optional arguments. It is also possible to control the name of an argument&#39;s internal variable by assigning a string to the <a href="https://docs.python.org/3/library/argparse.html#dest">dest</a> parameter.</p><pre>parser.add_argument(<br>    &#39;-f&#39;, &#39;--filter&#39;, action=&#39;store&#39;, nargs=&#39;+&#39;,<br>    help=&#39;Apply filters to an images.&#39;, dest=&#39;filters&#39;)<br>​<br>parser.add_argument(<br>    &#39;-e&#39;, &#39;--enhance&#39;, action=&#39;store&#39;, nargs=&#39;+&#39;,<br>    help=&#39;Apply enhancers to an images.&#39;, dest=&#39;enhancers&#39;)</pre><p>When we later call parse_args() to return an object containing our parsed values, the list of filter strings will be assigned to an attribute called filters. Likewise, the list of enhancer strings will be assigned to an attribute called enhancers.</p><p>In addition, we opt to validate filter and enhancer strings ourselves rather than utilizing the choices parameter for a couple of reasons:</p><ul><li>The help text will become very hard to read if every possible value is listed.</li><li>We want to do some additional processing to make filter and enhancer names case-insensitive.</li></ul><p>We will take a look at the corresponding validation functions shortly.</p><h4>Mutually Exclusive Optional Arguments</h4><p>Lastly, we create a mutually exclusive argument group that is used to define our resize options. As a result, only one argument defined in this group can be given on the command line at any given time. It makes sense to do this because each of the resize options modify the same properties of an image, but in different ways. For example, it would not be logically correct to specify both --resize-width and --resize-width-proportional on the command line.</p><p>A mutually exclusive group is created by calling a parser object’s <a href="https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.add_mutually_exclusive_group">add_mutually_exclusive_group()</a> method. We call add_argument() on the returned group object itself to append mutually exclusive arguments to it:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2b95163e5f36d91046e7ee1d9f373b8d/href">https://medium.com/media/2b95163e5f36d91046e7ee1d9f373b8d/href</a></iframe><h4>Returning Arguments to the Application</h4><p>An ArgumentParser object contains a method called <a href="https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.parse_args">parse_args()</a> which we call to generate actual Python object’s that are derived from arguments specified on the command line:</p><pre>args = parser.parse_args()<br>return args</pre><p>At this point, our application can check the attributes attached to this object and run appropriate operations requested by the user.</p><h3>2. Handling Parsed Arguments</h3><p>When the parsing phase is complete, our application receives a collection of Python objects that correspond to each argument specified on the command line. These objects are attached to a variable we call namespace. We pass this object around the application in order to check its attributes and perform appropriate operations.</p><p>When the namespace object is returned by parse_arguments() in imgmanip.py, the application immediately checks its inputs attribute to iterate through image filenames that were specified on the command line:</p><pre>for index, filename in enumerate(<strong>namespace.inputs</strong>):<br>    try:<br>        with Image.open(filename) as opened_image:<br>            handle_arguments(namespace, opened_image, index)</pre><pre>    except FileNotFoundError as err:<br>        print(f&quot;WARNING: {err.filename} not found. Skipping...&quot;)</pre><ul><li>This is the first time we make use of the Pillow API. A call to <a href="https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.open">Image.open()</a> loads an image file into memory and a variable called opened_image points to the loaded image data. In other words, opened_image is our handle to the loaded image.</li><li>A call to handle_arguments() is made upon loading an image and is passed both the namespace object and opened_image handle. This function is implemented in handler.py — a module located in our project&#39;s imgmanip/ directory.</li><li>An exception of type FileNotFoundError is thrown if a particular filename is not found. If this occurs, we output the filename in question and continue to process the next filename in namespace.inputs.</li></ul><p>This is a good time to talk about how the program handles image loading. An image is loaded into memory by calling Image.open() and is pointed at by a variable called opened_image. From here, any image processing that is invoked will work on this single source of data - no temporary copies of image data is made.</p><p>In addition, a save operation will only ever be performed once for a loaded image. This will occur after all command line arguments have been handled and image processing is finished. At this time, a new image is saved as a physical file on the file system.</p><h4>Handling Arguments</h4><p>Let’s now switch our focus to the handler.py module, which is responsible for checking objects in namespace and performing the correct operations. The job of this file is to simply detect which image manipulation operations to perform, and then offload these operations to <em>another object</em>. It doesn&#39;t contain any code that modifies the internal representation of the loaded image.</p><p>This <em>other object</em> is an instantiation of a class called ImageManipulator that we implement ourselves in the manipulator.py module. An ImageManipulor object contains functions that actually modify loaded image data. We issue an import statement at the top of handler.py to gain access to this object:</p><pre>from manipulator import ImageManipulator</pre><p>Looking inside the handle_arguments() function, the first thing it does is instantiate an ImageManipulator object:</p><pre>def handle_arguments(namespace, image):<br>    <strong>&quot;&quot;&quot;Function to handle command line arguments.&quot;&quot;&quot;</strong></pre><pre>    manip = ImageManipulator(image, index)</pre><p>Notice that we pass our image handle to the ImageManipulator constructor. The constructor proceeds to cache the handle in one of its own class attributes called self.image. This means that functions defined in ImageManipulator will always have access to the image it is working on.</p><p>From here, handle_arguments() proceeds to check attributes attached to the namespace object through simple if statements. If the namespace object is found to have a particular attribute, the image manipulator object (manip) or a corresponding handler function is called to fulfil the operation:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/75ada1b32c4e3069077bff14dc6f5024/href">https://medium.com/media/75ada1b32c4e3069077bff14dc6f5024/href</a></iframe><p>A new image is saved to disk once all arguments have been handled. The is accomplished by calling ImageManipulator.save_image():</p><pre><strong># Save the new file if necessary<br></strong>manip.save_image()</pre><h4>Handler Functions</h4><p>Individual handler functions are defined in handler.py to fulfil particular operations. They also keep our Python code readable, and prevent duplication. A handler function is defined for operations that need some preliminary setup code before the ImageManipulator object can be used.</p><p>For example, the _handle_filters() function must firstly validate filter names received from the command line before it can give the go-ahead for the ImageManipulator to do any processing. Because namespace.filters is a Python list, it is also necessary to iterate through each filter name and apply them individually. This same approach is taken to apply enhancers.</p><p>There are three handler functions defined in handler.py. Because their implementations are easy to understand, we won&#39;t spend too much time talking about their code. Instead, a summary of each function is provided here:</p><ul><li>_handle_filters(namespace, manip)<br>Validates filter names received from the command line and instructs the image manipulator object to apply each valid filter to the loaded image. Each filter string is converted to uppercase before it is passed to _is_valid_filter() to perform validation. If a filter name fails validation, a message is displayed and the next filter string in the list is processed.</li><li>_handle_enhancers(namespace, manip)<br>Validates enhancer names received from the command line and instructs the image manipulator object to apply each valid enhancer to the loaded image. Each enhancer string is converted to uppercase before it is passed to _is_valid_enhancer() to perform validation. If an enhancer name fails validation, a message is displayed and the next enhancer string in the list is processed.</li><li>_handle_resizing(namespace, manip) <br>Checks which resize argument (if any) was specified on the command line and calculates a new width or height value if necessary, before offloading image processing to the image manipulator object.</li></ul><p>It is also important to be aware of the number of scopes (or indentation) that exist in any particular function. For example, many nested if statements and looping constructs could lead to code that is hard to follow or comprehend. To mitigate this issue, our application makes an effort to only use two or three scopes in any given function.</p><h4>Validation Functions</h4><p>Let’s wrap up our discussion on the handler.py module by discussing its validation features. Separate functions are defined to validate both filter and enhancer names received from the command line.</p><p>Parsed filter names are contained in a list of strings that correspond to values that were specified on the command line (after the -f or --filter flags). Our _handle_filters() function validates these strings by firstly converting them to upper case and calling _is_valid_filter():</p><pre>def _is_valid_filter(filter_string):<br>    <strong>&quot;&quot;&quot;Validates a passed filter string.&quot;&quot;&quot;</strong><br>    <br>    if filter_string in [<br>            &#39;BLUR&#39;, &#39;CONTOUR&#39;, &#39;DETAIL&#39;, &#39;EDGE_ENHANCE&#39;,  <br>            &#39;EDGE_ENHANCE_MORE&#39;,&#39;EMBOSS&#39;, &#39;FIND_EDGES&#39;, <br>            &#39;SHARPEN&#39;, &#39;SMOOTH&#39;, &#39;SMOOTH_MORE&#39;]:<br>        return True<br>    return False</pre><p>If the filter_string argument matches any string in the python list following the in keyword, True is returned; otherwise, False is returned.</p><p>The same approach is taken to validate enhancer names:</p><pre>def _is_valid_enhancer(enhancer_string):<br>    <strong>&quot;&quot;&quot;Validates a passed enhancer string.&quot;&quot;&quot;</strong></pre><pre>    if enhancer_string in [<br>            &#39;BRIGHTNESS&#39;, &#39;COLOR&#39;, &#39;COLOUR&#39;, &#39;CONTRAST&#39;, <br>            &#39;SHARPNESS&#39;, &#39;GREYSCALE&#39;, &#39;GAUSSIANBLUR&#39;, &#39;BOXBLUR&#39;]:<br>        return True<br>    return False</pre><h3>3. Image Processing with PIL</h3><p>The manipulator.py module contains a class called ImageManipulator which implements methods that process and modify image data. This section will go through each of these methods and detail how the Pillow API is used to manipulate image data.</p><p>We start by importing necessary Pillow modules at the top of manipulator.py:</p><pre>from PIL import Image, ImageEnhance, ImageFilter</pre><p>We import the Image class to access an image’s internal data and perform a range of operations on it. PIL also provides <a href="https://pillow.readthedocs.io/en/stable/reference/ImageEnhance.html#module-PIL.ImageEnhance">ImageEnhance</a> and <a href="https://pillow.readthedocs.io/en/stable/reference/ImageFilter.html">ImageFilter</a> modules for applying specific filters and enhancements to image data.</p><p>Now is a good time to mention the ImageManipulator constructor, which initializes four important attributes which are referenced in corresponding class methods:</p><pre>def __init__(self, image):<br>    self.image = image<br>    self.original_filename = image.filename<br>    self.save_flag = False<br>    self.index = index</pre><ul><li>self.image = image<br>A handle to the current loaded image is cached, making it readily accessible to every method defined in the class.</li><li>self.original_filename = image.filename<br>The filename of the current loaded image is cached for convenience and is used in formatting operations.</li><li>self.save_flag = False<br>This flag determines whether a new image file should be saved to disk after all command line arguments have been handled. It will be assigned True if the loaded image data is modified in any way.</li><li>self.index = index<br>If multiple image files are specified on the command line, this attribute will keep track of which image is currently being processed. It is used to generate uniquely named image files.</li></ul><h4>Displaying Image Properties</h4><p>Moving forward, a method called output_information() is responsible for outputting properties of the loaded image. Attributes defined in the Image class are used to output its corresponding filename, image size (in pixels), format and band information (such as RGB, RGBA, L, etc.):</p><pre>def output_information(self):<br>    <strong>&quot;&quot;&quot;Sends image properties to standard output.&quot;&quot;&quot;</strong></pre><pre>    print(&quot;{0:&lt;9}: {1}&quot;.format(&quot;Filename&quot;, self.image.filename))<br>    print(&quot;{0:&lt;9}: {1}&quot;.format(&quot;Size&quot;, self.image.size))<br>    print(&quot;{0:&lt;9}: {1}&quot;.format(&quot;Format&quot;, self.image.format))<br>    print(&quot;{0:&lt;9}: {1}\n&quot;.format(&quot;Bands&quot;, self.image.getbands()))</pre><p>Format specifiers are passed to the print() function to output information cleanly. A column width of nine characters is output by specifying the &lt;9 character combination.</p><h4>Applying Image Filters</h4><p>Two methods are defined in ImageManipulator that handle filter processing. The apply_filter_and_save() method firstly sets the save flag to True and calls the _apply_filter() &quot;private&quot; method to apply an actual filter to the loaded image data:</p><pre>def apply_filter_and_save(self, filter_string):<br>    <strong>&quot;&quot;&quot;Apply a filter to a loaded image and save the new version.&quot;&quot;&quot;</strong><br>​<br>    self.save_flag = True<br>    self.image = self._apply_filter(filter_string)<br>    print(f&quot;{filter_string} filter applied to \      <br>            {self.original_filename}&quot;)</pre><p>Within _apply_filter(), we call <a href="https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.filter">Image.filter()</a> to apply a particular filter. This method is passed <a href="https://pillow.readthedocs.io/en/stable/reference/ImageFilter.html#filters">a filter name</a> defined in the Pillow ImageFilter module that indicates which filter to use. The correct filter is applied based on the filter_string argument:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c16d6c8863724c73273f28fbc16a77f4/href">https://medium.com/media/c16d6c8863724c73273f28fbc16a77f4/href</a></iframe><h4>Applying Image Enhancements</h4><p>The apply_enhancer_and_save() method is responsible for determining which enhancer to apply to the loaded pixel data. Its enhancer_string argument is firstly compared with hard coded enhancer strings such as &#39;BRIGHTNESS&#39;, &#39;CONTRAST&#39;, &#39;SHARPNESS&#39;, and so on. We also allow the color enhancer to be specified using either &quot;color&quot; or &quot;colour&quot; spellings.</p><p>When the enhancer_string argument matches one of these hard-coded strings, a corresponding enhancer object defined in the ImageEnhance module is created. This object is referenced by a variable we create ourselves called enh — which is then pass to another class method called _apply_enhancer() along with the factor value:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/95e7f00595bf6b3572321eb6a8d72f59/href">https://medium.com/media/95e7f00595bf6b3572321eb6a8d72f59/href</a></iframe><p>A call to the enhancer object&#39;s enhance() method is made inside _apply_enhancer(), which returns the resulting image data to our self.image handle:</p><pre>def _apply_enhancer(self, enhancer, enhancer_string, factor):<br>    <strong>&quot;&quot;&quot;Enhances an image and saves a new file.&quot;&quot;&quot;</strong><br>​<br>    self.image = enhancer.enhance(factor)<br>    print(&quot;{0} enhancer applied to {1} [factor = {2}]&quot;<br>        .format(enhancer_string, self.original_filename, factor))</pre><h4>Generating Greyscale Images</h4><p>We take advantage of the <a href="https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.convert">Image.convert()</a> method to generate a greyscale image. A mode value of &#39;L&#39; is passed to this method to convert the loaded image’s pixel format to 8-bit pixels (black and white):</p><pre>elif enhancer_string == &#39;GREYSCALE&#39;:<br>    self.image = self.image.convert(&#39;L&#39;)<br>    print(&quot;{0} enhancer applied to {1}&quot;<br>          .format(enhancer_string, self.original_filename))</pre><h4>Gaussian Blur and Box Blur Filters</h4><p>The Gaussian blur and box blur effects are handled a little bit differently compared to the other enhancers because they are defined as filters in Pillow. We process them independently by calling our image handle’s <a href="https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.filter">Image.filter()</a> method and passing either <a href="https://pillow.readthedocs.io/en/stable/reference/ImageFilter.html#PIL.ImageFilter.GaussianBlur">ImageFilter.GaussianBlur()</a> or <a href="https://pillow.readthedocs.io/en/stable/reference/ImageFilter.html#PIL.ImageFilter.BoxBlur">ImageFilter.BoxBlur()</a> to fulfil these operations:</p><pre>elif enhancer_string in [&#39;GAUSSIANBLUR&#39;, &#39;BOXBLUR&#39;]:<br>    <br>    if enhancer_string == &#39;GAUSSIANBLUR&#39;:<br>        self.image = <br>            self.image.filter(ImageFilter.GaussianBlur(factor))<br>    else:<br>        self.image = self.image.filter(ImageFilter.BoxBlur(factor))<br>​<br>    print(&quot;{0} filter applied to {1} [radius = {2}]&quot;.format(<br>        enhancer_string, self.original_filename, factor))</pre><h4>Generating a Thumbnail</h4><p>The generate_thumbnail() method uses pixel data from the loaded image to create a separate thumbnail. A width is passed to this method to control the thumbnail output size:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/48e1763daab31d111af17a6efc703aa3/href">https://medium.com/media/48e1763daab31d111af17a6efc703aa3/href</a></iframe><p>This is the only time that a copy is made of the loaded image data to fulfil an operation. We purposefully make this copy to ensure that thumbnails are generated independently of other image manipulation functions.</p><p>We create a variable called thumb_image to point to the copy. It subsequently calls <a href="https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.thumbnail">Image.thumbnail()</a> to generate a thumbnail, which is then saved to the file system.</p><h4>Flipping Images</h4><p>Our application allows images to be flipped both vertically and horizontally. The apply_flips() method handles this functionality:</p><pre>def apply_flips(self, flips):<br>    <strong>&quot;&quot;&quot;Flips the image horizontally or vertically.&quot;&quot;&quot;</strong><br>​<br>    for flip in flips:<br>        if flip.upper() == &#39;HORZ&#39;:<br>            self.image = self.image.transpose(Image.FLIP_LEFT_RIGHT)<br>            print(f&quot;{self.original_filename} flipped horizontally.&quot;)<br>        <br>        elif flip.upper() == &#39;VERT&#39;:<br>            self.image = self.image.transpose(Image.FLIP_TOP_BOTTOM)<br>            print(f&quot;{self.original_filename} flipped vertically.&quot;)<br>​<br>    self.save_flag = True</pre><p>The <a href="https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.transpose">Image.transpose()</a> method is called to flip image data. An attribute is passed to this method to specify a flipping direction:</p><ul><li>Image.FLIP_LEFT_RIGHT: An image will be flipped horizontally.</li><li>Image.FLIP_TOP_BOTTOM: An image will be flipped vertically.</li></ul><h4>Resizing Images</h4><p>The resize_image() method is responsible for resizing a loaded image. It receives a tuple containing two integer values that represent new width and height values. This tuple is subsequently passed to an invocation of our Image object&#39;s <a href="https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.resize">resize()</a> method. After a resize operation completes, our image handle is updated to point to the modified image data:</p><pre>def resize_image(self, size):<br>    <strong>&quot;&quot;&quot;Resizes cached image to the X and Y values in the <br>    passed tuple.&quot;&quot;&quot;</strong></pre><pre>    self.image = self.image.resize(size)<br>    print(f&quot;{self.original_filename} resized to: {size}&quot;)<br>​<br>    self.save_flag = True</pre><h4>Rotating Images</h4><p>A method called rotate_image() handles Image rotation. It receives an integer value denoted by a parameter called increment which specifies how many degrees our image will rotate in a counter-clockwise direction.</p><p><a href="https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.transpose">Image.transpose()</a> is called to rotate its internal image data. Either theImage.ROTATE_90, Image.ROTATE_180 or Image.ROTATE_270 attribute is passed to this method depending on the value of increment. As usual, the resulting image data is returned to self.image:</p><pre>def rotate_image(self, increment):<br>    <strong>&quot;&quot;&quot;Rotates the cached image counter-clockwise in a 90 degree<br>    increment.<br>    &quot;&quot;&quot;</strong><br>​<br>    if increment == 90:<br>        self.image = self.image.transpose(Image.ROTATE_90)<br>    elif increment == 180:<br>        self.image = self.image.transpose(Image.ROTATE_180)<br>    else:<br>        self.image = self.image.transpose(Image.ROTATE_270)<br>    <br>    print(f&quot;{self.original_filename} rotated {increment} &quot;<br>          f&quot;degrees CCW.&quot;)<br>    self.save_flag = True</pre><h4>Saving the Image</h4><p>A new image file is written to the file system if any image processing was performed. This functionality is handled by a method called save_image(). It is called after all command line arguments have been handled, and saves a new image to the file system if self.save_flag is set to True:</p><pre>def save_image(self):<br>    <strong>&quot;&quot;&quot;Saves the image to a new file if save flag is set.&quot;&quot;&quot;</strong></pre><pre>    if self.save_flag is True:<br>        name, ext = os.path.splitext(self.original_filename)<br>        out_filename = name + &#39;-manip-&#39; + str(self.index) + ext<br>        self.image.save(out_filename)<br>        print(f&quot;New file saved as: {out_filename}&quot;)</pre><ul><li>A new file name is calculated based on the self.original_filename attribute: A &quot;-manip&quot; string is inserted between the original file identifier and its extension to derive a new file name.</li><li>The <a href="https://pillow.readthedocs.io/en/stable/reference/Image.html#PIL.Image.Image.save">Image.save()</a> method is called to actually write its data to disk.</li></ul><h3>In Conclusion — Part 2</h3><p>A lot of content has been covered in this tutorial where we have explored various stages that take place when developing Python applications. We started off by creating a virtual environment in which to install our project. This involved installing a particular version of Python along with specific packages required by the project. Documentation on how to use our program along with some example commands were also presented. It is important to be aware of such information because learning about a tool is necessary if you intend to contribute to its source code.</p><p>We then examined the project’s source code to discover how particular functionality is handled with Python. The application takes advantage of the standard <a href="https://docs.python.org/3/library/argparse.html">argparse</a> module to define a CLI and parse command line arguments. Image processing is also handled by a fork of the PIL (Python Imaging Library) called <a href="https://pillow.readthedocs.io/en/stable/">Pillow</a>.</p><p>Moving forward, you now know enough about the application to fork the project on GitHub and continuously develop it according to your own specifications. For example, you may wish to expand on the program’s command line interface and experiment with more features that Pillow exposes with its API. You may also want to create a standalone executable using a tool such as <a href="https://cx-freeze.readthedocs.io/en/latest/">cx_Freeze</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=21531e3c2609" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Python: Building an Image Manipulation Tool (Part 1)]]></title>
            <link>https://dane-bulat.medium.com/python-building-an-image-manipulation-tool-part-1-8299dd97bfe8?source=rss-24e03e26bd42------2</link>
            <guid isPermaLink="false">https://medium.com/p/8299dd97bfe8</guid>
            <category><![CDATA[python3]]></category>
            <category><![CDATA[application-development]]></category>
            <category><![CDATA[python-programming]]></category>
            <category><![CDATA[linux-tutorial]]></category>
            <category><![CDATA[image-processing]]></category>
            <dc:creator><![CDATA[Dane Bulat]]></dc:creator>
            <pubDate>Mon, 11 Jan 2021 05:23:21 GMT</pubDate>
            <atom:updated>2021-01-22T06:05:08.282Z</atom:updated>
            <content:encoded><![CDATA[<h4>A tutorial and case study on developing an image processing application with Python</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-Rmk2my3OdvSi2O32gvILg.png" /></figure><h3>Introduction</h3><p>Python allows development projects to be approached in a goal-orientated way. Programs written in Python are interpreted rather than compiled, which enables features such as dynamic typing and automatic memory management behind the scenes. This means more time can be invested in things like adding functionality and improving an application’s behavior, instead of tedious tasks like hardening code infrastructure and making sure memory is handled efficiently. With this in mind, Python is a great tool for productivity.</p><p>This tutorial and case study will detail how to use Python to create an image manipulation tool. This tool is invoked on the command line and is able to process image files in different ways, such as:</p><ul><li>Applying filters to images such as blur, sharpen and smooth functions.</li><li>Enhancing an image by adjusting properties such as its brightness, contrast and color.</li><li>Convert an image into a greyscale version.</li><li>Applying Gaussian blur and box blur filters to an image.</li><li>Flip an image horizontally and vertically, as well as rotating an image.</li><li>Resizing images to specific widths and heights.</li><li>Generating thumbnails of image files.</li></ul><p>Various stages of development will be talked about, including setting up a virtual environment, defining an interface for command line arguments, and performing actions based on arguments the program receives.</p><p>Our tool also utilizes a fork of the Python Imaging Library (PIL) called <a href="https://pillow.readthedocs.io/en/stable/index.html">Pillow</a> for performing image processing tasks. At a high level, the Pillow API is used to load image files into memory, process its pixel data in various ways, and finally save a new image to the file system.</p><p>Two articles will make up this tutorial. This first article will detail how to set up a virtual environment in which to install our project, followed by going over how the image manipulation tool is used:</p><ol><li><strong>Project Overview<br></strong>The project’s directories and source files are discussed. We then go through the process of creating a virtual environment with <a href="https://pipenv.pypa.io/en/latest/">Pipenv</a>.</li><li><strong>How to Use the Image Manipulation Tool<br></strong>Our tool’s usage and command line options are detailed. We also discuss the difference between positional arguments and optional arguments, and how they are used.</li><li><strong>Example Commands with Screenshots<br></strong>A few example invocations of our program are demonstrated to show how it can be used in real-world scenarios. Before and after screenshots accompany each example.</li></ol><p><a href="https://dane-bulat.medium.com/python-building-an-image-manipulation-tool-part-2-21531e3c2609">Part two</a> walks through the actual Python code that makes up the project. We take a look at the standard <a href="https://docs.python.org/3/library/argparse.html">argparse</a> Python module, as well as how to process images in different ways with the Pillow API:</p><ol><li><strong>Passing Command Line Arguments<br></strong>This section details how the standard <a href="https://docs.python.org/3/library/argparse.html">argparse</a> Python module is used to parse command line arguments and convert them into Python objects, ready to be used by the application.</li><li><strong>Handling Parsed Arguments</strong><br>We then talk more about the program’s structure and adopted coding conventions in order to improve readability, prevent duplication, and maintain performance.</li><li><strong>Image Processing with Pillow<br></strong>Lastly, we discuss how our application integrates the Pillow API and the code used for performing image processing operations.</li></ol><p><a href="https://dane-bulat.medium.com/python-building-an-image-manipulation-tool-part-2-21531e3c2609">Python: Building an Image Manipulation Tool (Part 2)</a></p><h4>Source Files</h4><p>A <a href="https://github.com/danebulat/py-image-manip-demo">GitHub repository</a> has been set up for readers wishing to download the application and follow along with instructions presented in this article. Navigate to an appropriate place on your file system and invoke Git to clone the repository:</p><pre>$ git clone <a href="https://github.com/danebulat/py-image-manip-demo.git">https://github.com/danebulat/py-image-manip-demo.git</a></pre><p>We will go over the project’s structure, it’s source files, and how to set up a virtual environment in the next section.</p><h3>1. Project Overview</h3><p>This section provides an overview of our application’s project structure where each directory and source file is described. We also take a look at how to create a Python virtual environment using the pipenv package. Let&#39;s start by taking a look at what the project is made up of:</p><pre>&gt; py-image-manip-demo/<br>      --&gt; images/<br>             --&gt; 001.jpg<br>             --&gt; 002.jpg<br>             --&gt; 003.jpg<br>      --&gt; imgmanip/<br>             --&gt; imgmanip.py<br>             --&gt; parser.py<br>             --&gt; handler.py<br>             --&gt; manipulator.py<br>      --&gt; Pipfile<br>      --&gt; Pipfile.lock</pre><ul><li>images/ <br>Three JPEG image files are provided in this directory. Feel free to specify these files as inputs to our program.</li><li>imgmanip/<br>This directory is named after our application and contains its source code.</li><li>imgmanip.py<br>Our application’s entry point. The code in this file loads image files into memory and instructs the appropriate modules to parse command line arguments and handle image processing.</li><li>parser.py<br>A Python module that implements functions to parse command-line arguments into Python objects.</li><li>handler.py<br>A Python module that runs appropriate portions of code based on what was specified on the command line. It also calls image processing functions defined in manipulator.py.</li><li>manipulator.py<br>Implements a class called ImageManipulator that performs image processing. Items from the Pillow API are imported and used in this module.</li><li>Pipfile<br>This file is read by pipenv to create and configure a virtual environment. Important information such as the project&#39;s Python version and package dependencies are specified here.</li><li>Pipfile.lock<br>Another important file to configure the virtual environment correctly. This file defines the exact package versions required by a project.</li></ul><p>The project is structured where related functionality is defined in its own module. For example, we have individual modules for parsing command line arguments, handling generated objects derived from those arguments, and performing image manipulation with Pillow. Part two of this case study details the code in each of these modules.</p><h4>Setting Up a Virtual Environment</h4><p>A virtual environment is essentially a container that houses a Python and pip installation. A few solutions exist for creating Python virtual environments. One of them is a built-in module called <a href="https://docs.python.org/3/tutorial/venv.html">venv</a> which enables you to create and manage system-wide virtual environments.</p><p>We want to be able to create virtual environments on the project level. The <a href="https://pipenv.pypa.io/en/latest/">pipenv</a> package provides such functionality and allows us to:</p><ul><li>Specify packages along with their version number in a Pipfile to determine a project&#39;s dependencies.</li><li>Generate a file called Pipfile.lock for producing deterministic builds.</li><li>Install a specific version of Python that will run when the environment is active.</li><li>Run shell commands inside the environment just as though it was the actual system-wide shell environment.</li></ul><p>As a matter of fact, you can determine which Python version our project requires along with its dependencies by inspecting Pipfile:</p><pre>[packages]<br>pillow = &quot;*&quot;</pre><pre>[requires]<br>python_version = &quot;3.9&quot;</pre><p>The [packages] section specifies a package called pillow which pip will download to our virtual environment. Its corresponding version is defined in Pipfile.lock:</p><pre>&quot;pillow&quot;: {<br>    &quot;hashes&quot;: [<br>        ...<br>    ],<br>    &quot;index&quot;: &quot;pypi&quot;,<br>    &quot;version&quot;: &quot;==8.1.0&quot;<br>}</pre><p>As a result of what is specified in Pipfile and Pipfile.lock, an exact environment is set up whenever the project is installed with Pipenv on any given system - even if a newer version of Python or Pillow is available. This ensures that our project&#39;s source code remains compatible with its dependencies. For example, if a new API is introduced in a future version of Pillow, we can be assured that our application will still run, as it is tied to an older version of that particular package.</p><p>The following steps will detail how to create a virtual environment with <a href="https://pipenv.pypa.io/en/latest/">Pipenv</a> to run our project. Let’s start by installing the pipenv package with pip. We will also output its version string to confirm the package is found by the system:</p><pre>$ pip install pipenv<br>$ pipenv --version<br>    -| pipenv, version 2020.11.15</pre><p>From here we can enter the root directory of our project and create a virtual environment. Pipenv will read Pipfile and Pipfile.lock to install the correct version of Python and package dependencies that are specified in these files:</p><pre>$ cd py-image-manip-demo<br>$ pipenv install</pre><p>If you do not have an installation of Python 3.9 on your system, a prompt will ask if you would like to install it. In this case, input y and press Enter to continue the installation. A directory containing your new virtual environment is created before you regain control of the terminal.</p><p>A useful feature of Pipenv is the graph option, which will list packages that have been installed to the virtual environment:</p><pre>$ pipenv graph<br>    -| Pillow==8.1.0</pre><p>Moving forward, there are a couple of ways to run commands using the virtual environment’s own python and pip installations. Firstly, pipenv run is used to invoke a single command relative to the virtual environment. For example, the following commands output its Python and Pip installations:</p><pre>$ pipenv run which python pip<br>    -| /home/dane/.local/share/virtualenvs/py-image-manip-demo-<br>       lnYNAppa/bin/python<br>       /home/dane/.local/share/virtualenvs/py-image-manip-demo-<br>       lnYNAppa/bin/pip<br>​<br>$ pipenv run ls -l $(which python pip)</pre><p>An easier way to work in an environment is to spawn a shell by running the pipenv shell command. Any subsequent commands will use the virtual environment&#39;s corresponding Python and Pip installations by default:</p><pre>$ pipenv shell<br>​<br><strong># From here you can work inside the virtual environment</strong><br>$ python --version &amp;&amp; which python &amp;&amp; pip --version<br>$ pip list &amp;&amp; pip show pillow</pre><p>Run the exit command when you are finished working on a project and wish to leave a virtual environment:</p><pre>$ exit</pre><p>It is also a good idea to read through the pipenv help menu, which provides a full list of available commands along with brief descriptions:</p><pre>$ pipenv --help</pre><p>Of particular importance is how to install packages to a virtual environment with Pipenv. This is done with the following syntax:</p><pre>pipenv install &lt;package_name&gt;</pre><p>When a package is installed in this way, a project’s Pipfile is updated to include the newly installed package. If new packages have been installed and used within a project&#39;s source code, a new Pipfile.lock file should also be generated before committing your code and making a push remotely:</p><pre>$ pipenv lock</pre><p>I encourage you to try out each Pipenv command in our project if this is your first time working with virtual environments.</p><h3>2. How to Use the Image Manipulation Tool</h3><p>The image manipulation tool is invoked on the command-line and will process images based on the flags it receives. This section will detail how to invoke the tool and discuss its flags in detail. An understanding of its usage will put us in good stead when we look at the corresponding source code in part two.</p><h4>Invocation</h4><p>Let’s start off by updating the shebang character sequence at the top of imgmanip.py to point to the virtual environment&#39;s Python executable. Doing this will enable us to invoke our program using a shorthand syntax, which will be discussed shortly. We can find out the location of an environment&#39;s python executable in the following way:</p><pre>$ pipenv shell<br>$ which python<br>    -| /Users/dane/.local/share/virtualenvs/py-image-manip-demo-<br>       e67HmAh0/bin/python</pre><p>Go ahead and replace the first line in imgmanip.py with this output and make sure to follow up with a save:</p><pre><strong>&gt;&gt;</strong> #!/Users/dane/.local/share/virtualenvs/py-image-manip-demo-<br>     e67HmAh0/bin/python<br>​<br>   &quot;&quot;&quot;Module for performing a range of image manipulations.</pre><p>The imgmanip.py file is our application&#39;s entry point. It receives one or more inputs (image file names) and calls code defined in other modules to perform appropriate operations. We therefore invoke the application by parsing this file to the Python interpreter:</p><pre>python imgmanip.py &lt;image_files&gt; &lt;flags&gt;</pre><p>Because we updated the shebang character sequence to point to a valid Python binary, the application can also be invoked using a shorthand syntax by specifying ./ instead of python. In order for this to work, imgmanip.py must have executable permissions:</p><pre>$ chmod u+x imgmanip.py</pre><p>It is now possible to invoke our application using the following syntax:</p><pre>./imgmanip.py &lt;image_files&gt; &lt;flags&gt;</pre><p>Let’s test this out to make sure everything is working correctly. To prevent having to specify relative paths to image files when invoking our program, we can firstly create some symbolic links in the same directory as imgmanip.py:</p><pre>$ ln -s ../images/001.jpg 001.jpg<br>$ ln -s ../images/002.jpg 002.jpg<br>$ ln -s ../images/003.jpg 003.jpg</pre><p>Let’s now use these image files as inputs to our program and find out their corresponding properties by specifying the -i flag:</p><pre><strong># One input file</strong><br>$ ./imgmanip.py 001.jpg -i<br>    -| Filename : 001.jpg<br>       Size     : (2560, 1440)<br>       Format   : JPEG<br>       Bands    : (&#39;R&#39;, &#39;G&#39;, &#39;B&#39;)<br>​<br><strong># Multiple input files</strong><br>$ ./imgmanip.py 001.jpg 002.jpg 003.jpg -i<br>​<br><strong># Multiple input files with wildcard character</strong><br>$ ./imgmanip.py *.jpg -i</pre><p>If properties are output for each image — their filename, size in pixels, format, and bands — the image manipulation tool is working correctly on your system.</p><h4>Positional Arguments and Optional Arguments</h4><p>Let’s move on to discuss the command line arguments that our tool can receive. Command line arguments come in two flavors — <strong>positional arguments</strong> and <strong>optional arguments</strong>:</p><ul><li>Position arguments must be specified after the program’s name in a certain order. For example, two positional arguments are typically passed to the cp command: A source file to copy followed by a target file. In this case, the order in which files are specified are very important because positional arguments for cp have been set up in a certain way.</li><li>Optional arguments are denoted with one or two dash characters and can be specified in any order. They typically govern how an application behaves. For example, the cp command defines an optional argument called -n, which, when specified, ensures an existing file is not overwritten when performing a copy operation.</li></ul><p>An optional argument can have both a shorthand and longhand version. A single dash followed by a single character (I.e. -h) defines a shorthand argument. Two dashes followed by a word (I.e. --help) usually define longhand arguments.</p><p>Our program has only one positional argument called inputs that represents a list of one or more image filenames. Quite a few optional arguments are also defined, which are described in the following table:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/bd6963e97dc59c366216d8405f8a1bc4/href">https://medium.com/media/bd6963e97dc59c366216d8405f8a1bc4/href</a></iframe><p>You will notice that the standard -h, --help and --version optional arguments are defined for our program. The remaining arguments process input images in some way, and have a specific usage pattern as detailed below:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/816bea6ccfdbcc42518a657d18fa5ebd/href">https://medium.com/media/816bea6ccfdbcc42518a657d18fa5ebd/href</a></iframe><p>The usage of most arguments is easy to understand. However, the -f, --filter and -e, --enhance arguments warrant further explanation. In short, they allow us to specify functions to modify an image&#39;s pixel data in some way. As demonstrated in the table above, a particular filter or enhancer can be specified more than once, in an arbitrary order, for any given command.</p><p>Altogether, our program exposes ten filters which are listed here:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/208d96557560be60e11643d4026782a9/href">https://medium.com/media/208d96557560be60e11643d4026782a9/href</a></iframe><p>Enhancers are more powerful than filters because their strength can be customized by parsing a float value to the --factor argument. The application uses a default factor of 1.2 if a value is not specified on the command line. In total, seven enhancers are available:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/60d1cfad9044919b1a06959e174c62ce/href">https://medium.com/media/60d1cfad9044919b1a06959e174c62ce/href</a></iframe><p>The application is quite flexible when it comes to specifying filter and enhancer names. Firstly, names are case-insensitive because command line input is converted into uppercase before being processed further. Therefore, commands like this are completely legal:</p><pre>$ ./imgmanip.py 001.jpg -f detAIL BLUR SmooTH -e color GREYSCALE</pre><pre>    -| DETAIL filter applied to 001.jpg<br>       BLUR filter applied to 001.jpg<br>       SMOOTH filter applied to 001.jpg<br>       COLOR enhancer applied to 001.jpg [factor = 1.2]<br>       GREYSCALE enhancer applied to 001.jpg<br>       New file saved as: 001-manip-0.jpg</pre><p>Furthermore, invalid filter and enhancer names are skipped if any exist:</p><pre>./imgmanip.py 001.jpg -f DETAIL banana SMOOTH \<br>                      -e BRIGHTNESS apple --factor 1.5</pre><pre>    -| DETAIL filter applied to 001.jpg<br>       <strong>&#39;BANANA&#39; not a valid filter name. Skipping...</strong><br>       SMOOTH filter applied to 001.jpg<br>       BRIGHTNESS enhancer applied to 001.jpg [factor = 1.5]<br>       <strong>&#39;APPLE&#39; not a valid enhancer name. Skipping...</strong><br>       New file saved as: 001-manip-0.jpg</pre><h3>3. Example Commands with Screenshots</h3><p>The application provides quite a large command line interface without being overly complex. At this point, feel free to experiment with the tool by combining options and seeing what interesting images you can generate. Keep in mind that arguments can be specified in an arbitrary order, and the help text is always available if you need a refresher on the tool’s arguments and usage patterns:</p><pre>$ ./imgmanip.py --help</pre><p>Our first example will demonstrate how to apply a greyscale and brightness enhancer to an image with a custom factor (strength) value:</p><pre>$ ./imgmanip.py 001.jpg -e greyscale brightness --factor 1.25</pre><pre>   -| GREYSCALE enhancer applied to 001.jpg<br>      BRIGHTNESS enhancer applied to 001.jpg [factor = 1.25]<br>      New file saved as: 001-manip-0.jpg</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*T94hFI3_h9wFN1Fhp2T0Pw.png" /><figcaption>Converting an image into a brightened greyscale version.</figcaption></figure><p>Filters can be applied in the same manner as enhancers. The next example applies four filters and one enhancer to an image:</p><pre>$ ./imgmanip.py 002.jpg -f contour detail detail smooth -e greyscale</pre><pre>    -| CONTOUR filter applied to 002.jpg<br>       DETAIL filter applied to 002.jpg<br>       DETAIL filter applied to 002.jpg<br>       SMOOTH filter applied to 002.jpg<br>       GREYSCALE enhancer applied to 002.jpg</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3ed50gJ1_B_TtHYywyFIUA.png" /><figcaption>Combining the contour filter with other effects.</figcaption></figure><p>If you need a larger or smaller version of an image but wish to maintain its proportionality — meaning no pixel stretching or distortion — one of the proportional resize options will do the trick:</p><pre>$ ./imgmanip.py 002.jpg --resize-width-proportional 512</pre><pre>    -| 002.jpg resized to: (512, 320)<br>       New file saved as: 002-manip-0.jpg</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Sx4vCWg0va7z83_s1buRBA.png" /><figcaption>Resizing an image proportionally.</figcaption></figure><p>Images can be flipped both vertically and horizontally. It is also possible to rotate an image by a 90 degree increment in a counter-clockwise direction. The next example demonstrates this functionality:</p><pre>$ ./imgmanip.py 001.jpg --flip horz --rotate 180</pre><pre>    -| 003.jpg flipped vertically.<br>       003.jpg flipped horizontally.<br>       003.jpg rotated 180 degrees CCW.<br>       New file saved as: 003-manip-0.jpg</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*opA84XlXf427TY8w1ZrY5w.png" /><figcaption>Flipping an image horizontally and rotating it 180 degrees CCW.</figcaption></figure><p>A thumbnail will inherit a width of 128 pixels if a custom width is not specified on the command line. The next example generates a thumbnail of 256 pixels for three images:</p><pre>$ ./imgmanip.py 001.jpg 002.jpg 003.jpg \<br>        --thumbnail --thumbnail-width 256</pre><pre>    -| Thumbnail of 001.jpg generated: 001-256x144-0.thumb<br>       Thumbnail of 002.jpg generated: 002-256x160-1.thumb<br>       Thumbnail of 003.jpg generated: 003-256x173-2.thumb</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*5LUKzZuH7-rCRoS6w6EyNQ.png" /><figcaption>Generating a thumbnail for three image files.</figcaption></figure><h3>In Conclusion — Part 1</h3><p>At this point you should have a virtual environment setup that is able to run our image manipulation program.</p><p>In addition to detailing how to create project-level virtual environments with Python, this article walked through the usage of our image manipulation tool. The difference between positional arguments and optional arguments was also discussed.</p><p>We will dive into our project’s source code and take a look at what makes our program tick in the next part. When you are ready, click the following link to start reading <a href="https://dane-bulat.medium.com/python-building-an-image-manipulation-tool-part-2-21531e3c2609">part two</a>:</p><p><a href="https://dane-bulat.medium.com/python-building-an-image-manipulation-tool-part-2-21531e3c2609">Python: Building an Image Manipulation Tool (Part 2)</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8299dd97bfe8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CMake: Building SFML and Game Projects on Linux]]></title>
            <link>https://dane-bulat.medium.com/cmake-building-sfml-and-game-projects-on-linux-3947b3ba6e8?source=rss-24e03e26bd42------2</link>
            <guid isPermaLink="false">https://medium.com/p/3947b3ba6e8</guid>
            <category><![CDATA[cmake]]></category>
            <category><![CDATA[game-development]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[linux-tutorial]]></category>
            <category><![CDATA[open-source]]></category>
            <dc:creator><![CDATA[Dane Bulat]]></dc:creator>
            <pubDate>Fri, 01 Jan 2021 10:35:37 GMT</pubDate>
            <atom:updated>2021-01-03T09:58:33.111Z</atom:updated>
            <content:encoded><![CDATA[<h4>A tutorial on how to build and install game projects using CMake and SFML</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*K0YhuqDLutA1pBdKk1GYkA.png" /></figure><h3>Introduction</h3><p>The <a href="https://www.sfml-dev.org/">Simple and Fast Multimedia Library</a> (SFML) is a cross-platform development library that provides simple APIs for creating games and multimedia applications. SFML can compile and run on Linux, Mac OS and Windows. The project is also working towards being able to compile Android and iOS applications out of the box. On top of this, SFML is written in C++ and provides an easy-to-use C++ interface. Several <a href="https://www.sfml-dev.org/download/bindings.php">language bindings</a> also exist for many popular programming languages.</p><p>SFML can facilitate the development of large-scale commercial game projects. It is also very accessible for smaller teams and hobbyists who are new to the realm of developing games and multimedia applications with C++.</p><h4>Tutorial Overview</h4><p>The first part of this article details how to install SFML on Linux using CMake. This process involves downloading the official <a href="https://github.com/SFML/SFML">SFML GitHub repository</a> and using CMake to configure a build environment for generating artifacts such as the project’s shared libraries, header files and example projects. We then copy these artifacts to standard locations on the file system to complete an installation.</p><p>Once SFML is installed, we will switch our focus to developing applications that use its libraries. The traditional approach for developing SFML applications is to use an IDE that provides graphical tools to configure a project. Good examples of such IDEs are Codeblocks, Xcode and Visual Studio. This approach usually involves opening a dialog box to set search paths for libraries and header files, as well as specifying the actual library names an application will link to. The process of configuring a project will therefore be different depending on the IDE you use.</p><p>On the other hand, this article demonstrates how to configure SFML projects using CMake. Because the CMake program is a cross-platform build system generator, it is possible to configure how a project is built in a platform agnostic way. This means we can use an identical configuration script to compile an SFML project on Linux, Mac OS and Windows. With that said, we will focus on building SFML projects with CMake on Linux in this article to keep things simple.</p><p>More specifically, we will look at a minimal boilerplate project in addition to a pong game. Both projects link to SFML libraries and include a CMake configuration file for handling the build and install phases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/804/1*aTpDntiMdfh5dLC_NbN1lg.png" /><figcaption>Running the Pong game.</figcaption></figure><h4>Source Files</h4><p>A <a href="https://github.com/danebulat/cmake-sfml-demo">GitHub repository</a> has been setup to accompany this tutorial. Run the following Git command to download it to your system:</p><pre>$ git clone <a href="https://github.com/danebulat/cmake-sfml-demo.git">https://github.com/danebulat/cmake-sfml-demo.git</a></pre><p>The repository contains a couple of applications that are discussed in subsequent sections of this article. Feel free to check out the source code and try out the applications after you have SFML installed on your system.</p><h3>1. Building and Installing SFML</h3><p>This section details how to download, build and install the SFML project to your Linux system. It is worth pointing out that your distribution’s package manager probably provides an SFML package which will automatically install the SFML libraries for you. While convenient, this installation approach strips away your ability to customize an SFML build that matches your exact specifications. For example, a pre-configured package won’t allow you to do things like:</p><ul><li>Install SFML modules as only static libraries or only shared libraries.</li><li>Select specific example projects to generate.</li><li>Specify custom install locations for generated artifacts.</li><li>Install a local copy of the documentation files.</li></ul><p>With these points in mind, I recommend installing SFML with CMake whilst knowing that a pre-configured package is available if a system issue prevents you from completing the manual approach.</p><p>We will, however, use the package manager as an inspection tool to discover SFML’s dependencies. It is therefore a good idea to find out the name of the SFML package that your package manager points to. For example, pacman identifies the sfml package on Arch Linux:</p><pre>$ pacman -Ss sfml<br>    -| community/sfml 2.5.1-2<br>           A simple, fast, cross-platform, and object-oriented<br>           multimedia API</pre><h4>Installing SFML Dependencies</h4><p>The first step for getting SFML on a Linux system is to install a few libraries it depends on along with their development headers. The <a href="https://www.sfml-dev.org/tutorials/2.5/compile-with-cmake.php">official SFML website</a> provides a general list of project names whose libraries are required to install SFML. While it is not necessary to memorize each individual dependency, it is certainly useful to be aware of some back-end libraries SFML uses in its APIs:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f83e24cac09f98644f30667849d2a316/href">https://medium.com/media/f83e24cac09f98644f30667849d2a316/href</a></iframe><p>The exact name of the package you will need to install for each dependency will vary depending on the package manager you are using. We can find out these package names by querying the SFML package that the package manager points to. On Arch Linux this is achieved by invoking pacman with the information flag:</p><pre><strong># Use your package manager to identify SFML dependencies</strong><br>$ pacman -Si sfml<br>    -| Depends On : libsndfile  libxrandr  openal  glew  freetype2<br>       libx11</pre><p>You will also discover that SFML depends on an OpenGL back-end to perform graphics rendering if you read some of the online documentation. It is therefore a good idea to append the mesa package (an open-source implementation of the OpenGL specification) to your installation command to ensure this dependency is satisfied. So, on Arch Linux your installation command will look like this:</p><pre>$ sudo pacman -S mesa libsndfile libxrandr openal glew freetype2 <br>                 libx11</pre><p>Lastly, the Doxygen program is required for building SFML documentation. Go ahead and install it if you plan on generating a copy of the SFML help files on your system:</p><pre>$ sudo pacman -S doxygen graphviz</pre><p>At this point you should have all of the dependencies installed. Keep in mind that CMake will detect any missing libraries when we build SFML in the next section, so don’t worry about being left in the dark if you are missing a particular dependency.</p><h4>Downloading SFML</h4><p>In order to build SFML we must first download its source code which is currently <a href="https://github.com/SFML/SFML">hosted on GitHub</a>. Download the project to an appropriate place on your filesystem by running a Git clone command:</p><pre>$ git clone <a href="https://github.com/SFML/SFML.git">https://github.com/SFML/SFML.git</a><br>$ du -hs SFML/<br>    -| 159M    SFML/</pre><p>The du command reveals that the SFML repository will consume 159MB of storage space on your hard drive. Inspecting the root directory of the repository gives you an idea of what it provides:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/46b41a11532e7948ea3c3060d71dc4da/href">https://medium.com/media/46b41a11532e7948ea3c3060d71dc4da/href</a></iframe><p>It is important to note that the cmake/ directory and CMakeLists.txt file have different roles. Scripts contained in cmake/ enable external projects to locate necessary SFML artifacts on the filesystem - such as its shared libraries and header files. In order to read these files, an external project must be using CMake to generate its native build environment.</p><p>On the other hand, CMakeLists.txt is read by CMake to generate a native build environment for installing SFML on the system. We will take a look at what this entails next.</p><h4>Building SFML</h4><p>A common practice that takes place when building a project is to separate two directory hierarchies called the <strong><em>source tree</em></strong> and <strong><em>build tree</em></strong>. A source tree refers to a project’s directory tree that houses source code. On the other hand, a build tree houses native build files that are compiled into a project’s artifacts.</p><p>The root directory of the SFML repository is also the source tree root. This is because CMakeLists.txt references source files in nested sub-directories. It is therefore a good idea to create a new directory, also at the project root to contain our build tree. Let&#39;s call this directory build:</p><pre><strong># Enter SFML repository</strong><br>$ cd SFML/</pre><pre><strong># Create and enter a build directory</strong><br>$ mkdir build &amp;&amp; cd build</pre><p>We proceed by invoking the CMake program to read CMakeLists.txt in the parent directory:</p><pre>$ cmake ..</pre><p>This command will generate native build files and a CMake cache containing an initial build configuration. The type of build files CMake will generate depends on what generator it has selected before CMakeLists.txt is read. On a UNIX based operating system such as Linux, CMake selects the Unix Makefiles generator by default. You can confirm which generator is selected by passing --help to the cmake program and inspecting a subsection called Generators at the bottom:</p><pre>$ cmake --help | less<br>    -| Generators<br>       ...<br>       * Unix Makefiles = Generates standard UNIX makefiles.</pre><p>Notice that the selected generator is marked with an asterisk symbol. A short description is also output to let us know the type of files it will produce. In our case, standard UNIX makefiles are generated which will allow us to invoke the make utility to generate SFML&#39;s artifacts such as its shared libraries, header files, and so on.</p><p>Let’s now turn our focus to the cache file that was generated by CMake after it read CMakeLists.txt. The cache is a file called CMakeCache.txt that is saved at the root of our build tree. We launch either the ccmake or cmake-gui programs to inspect and change particular configuration settings:</p><ul><li>ccmake: A CLI that launches from within a terminal.</li><li>cmake-gui: A GUI that provides a more traditional application experience.</li></ul><p>I would personally recommend ccmake because it is fast, easy-to-use and launches directly within the terminal. With that said, let&#39;s open CMakeCache.txt and inspect its build options:</p><pre>$ ccmake .</pre><p>Pressing t will toggle advance mode and display a complete list of available build options. In addition to ON and OFF switches, you will notice other options for setting things like search paths, installation paths, and compiler flags. Cycle through options by pressing the up and down arrow keys. The Enter key will also allow you to change a value tied to an option:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*LCDIiucB7Ca-kWgDt9Gzeg.gif" /><figcaption>Inspecting and setting options in the SFML cache.</figcaption></figure><p>Settings within the cache dictate which SFML artifacts are generated at build time. For example, boolean options are set to either ON or OFF to control whether shared libraries, documentation, and example projects are generated. Paths to SFML dependencies are also saved in the cache. If a certain dependency cannot be found on the system, the corresponding field will contain a message to let you know. Important SFML related options to be aware of are listed here:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c558d12560d4c8afa958442a59202899/href">https://medium.com/media/c558d12560d4c8afa958442a59202899/href</a></iframe><p>Let’s go ahead and turn on the following options to generate documentation files, example projects, and the test suit:</p><ul><li>SFML_BUILD_DOC: <strong>ON</strong></li><li>SFML_BUILD_EXAMPLES: <strong>ON</strong></li><li>SFML_BUILD_TEST_SUIT: <strong>ON</strong></li></ul><p>Press c to update CMakeCache.txt followed by g to generate the corresponding build files. At this point our build tree contains doc/, lib/, examples/, test/ and src/ directories to house the corresponding items at build time.</p><p>From here we can build SFML according to our configuration:</p><pre>$ make</pre><p>It may take a couple of minutes for the build process to complete. When it is finished, the SFML shared libraries and development header files will be stored in appropriate locations within our build tree.</p><p>The last step is to copy, or <strong><em>install</em></strong>, the generated components to appropriate places on the filesystem. Doing so will enable development projects to import the SFML libraries and header files automatically. Let’s invoke make install with administrative privileges to complete our installation:</p><pre>$ sudo make install</pre><p>SFML components will be copied to install locations specified in the CMakeCache.txt file. If no installation paths were modified in the cache, SFML components will be installed to the following locations:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/934070e7fbb04ff11087c800804b0143/href">https://medium.com/media/934070e7fbb04ff11087c800804b0143/href</a></iframe><h3>2. Creating a Boilerplate SFML Project</h3><p>This section will walk through a boilerplate project that can be used as a starting point for games and multimedia applications built with SFML and CMake. It is included in this article’s <a href="https://github.com/danebulat/cmake-sfml-demo">companion repository</a> within a directory called Boilerplate:</p><pre>&gt; Boilerplate/<br>    ---&gt; CMakelists.txt<br>    ---&gt; config.h.in<br>    ---&gt; main.cpp</pre><p>CMake is our tool of choice for compiling main.cpp into an executable and linking it to the SFML shared libraries. The program itself simply renders a blank window via the SFML graphics API, and can be closed by pressing the escape key. It also sends the project&#39;s version number to standard output. Let&#39;s go through each command in CMakeLists.txt and talk about how the SFML library components are imported into the project.</p><h4>CMake Configuration File</h4><p>As with most configurations, a minimum CMake version is specified at the top of the file followed by a project name and version number:</p><pre>cmake_minimum_required(VERSION 3.12 FATAL_ERROR)<br>project(SFMLBoilerplate VERSION 0.1)</pre><p>We then request to use features of the C++ 11 standard to build the project. The CMAKE_CXX_EXTENSIONS variable is also set to OFF to ensure the standard -std=c++11 flag is passed to the compiler. Feel free to replace 11 with a more recent C++ standard such as 14 or 17 if you would like to take advantage of newer language features:</p><pre>set(CMAKE_CXX_STANDARD 11)<br>set(CMAKE_CXX_EXTENSIONS OFF)<br>set(CMAKE_CXX_STANDARD_REQUIRED TRUE)</pre><p>A default build type is set in the next part of our configuration. The term &quot;build type&quot; in CMake refers to a certain build configuration such as debug and release. We give our configuration a debug build type by default. A debug build often results in fast compile times with debugging symbols baked into executables - enabling debugging with tools as such as GDB and LLDB:</p><pre>if(NOT CMAKE_BUILD_TYPE)<br>  set(CMAKE_BUILD_TYPE Debug<br>    CACHE STRING &quot;Choose the type of build (Debug or Release)&quot; FORCE)<br>endif()</pre><p>The next command generates a header file called config.h which includes preprocessor definitions that specify our project&#39;s major and minor version number. CMake reads config.h.in to produce this header file:</p><pre>configure_file(config.h.in config.h)</pre><p>At this point we call <a href="https://cmake.org/cmake/help/latest/command/find_package.html">find_package()</a> to locate the SFML shared libraries and their corresponding header files. CMake is smart enough to detect the host operating system and deduce where standard locations are on its filesystem. With this in mind, all we need to do is pass the version of SFML along with the specific components (libraries) we wish to locate:</p><pre>find_package(SFML 2.5 <br>  COMPONENTS <br>    system window graphics network audio REQUIRED)</pre><p>We opt to specify both a major and minor version (2.5) to ensure CMake locates the most recent version of SFML. It is also possible to not specify any version at all - in this case CMake will return the first version of SFML it finds on the system.</p><p>SFML contains <a href="https://www.sfml-dev.org/documentation/2.5.1/modules.php">six modules</a> called Audio, Graphics, Network, System and Window, and are specified as components in the <a href="https://cmake.org/cmake/help/latest/command/find_package.html">find_package()</a> command. Components can be left out if you don&#39;t plan on linking to the corresponding library. For example, if you know a project will only ever require the system, window and graphics components, network and audio do not need to be specified.</p><p>Our configuration proceeds by instructing CMake to compile main.cpp into an executable file called SFMLBoilerplate via the <a href="https://cmake.org/cmake/help/latest/command/add_executable.html">add_executable()</a> command. We call <a href="https://cmake.org/cmake/help/latest/command/target_include_directories.html">target_include_directories()</a> immediately after to tell the build system where to find config.h:</p><pre>add_executable(SFMLBoilerplate main.cpp)<br>target_include_directories(SFMLBoilerplate <br>  PRIVATE &quot;${PROJECT_BINARY_DIR}&quot;)</pre><p>We also link our SFMLBoilerplate target to the sfml-graphics shared library. Because our application creates an sf::Window object whose implementation is defined in SFML/Graphics.hpp, the graphics module must be loaded at runtime:</p><pre>target_link_libraries(SFMLBoilerplate sfml-graphics)</pre><p>A single installation command wraps up our configuration. It specifies that the SFMLBoilerplate executable is copied to /usr/local/bin after invoking a sudo make install command:</p><pre>install(TARGETS SFMLBoilerplate DESTINATION bin)</pre><p>Keep in mind that additional installation commands are necessary if an application loads resource files such as images, audio and fonts at runtime. We will take a look at how to handle this in a following section.</p><h4>Building and Running the Boilerplate Project</h4><p>We will use the same commands to build our boilerplate project as we did for building SFML. Let’s start by creating a build directory and invoking cmake to read CMakeLists.txt in the parent directory:</p><pre>$ cd Boilerplate<br>$ mkdir build &amp;&amp; cd build<br>$ cmake ..</pre><p>Now is a good time to inspect the cache before committing to the build process. For example, you may want to select another build type or check which compiler flags are passed to the compiler:</p><pre>$ ccmake .</pre><p>Finally, initiate a build with make and run the generated executable:</p><pre>$ make<br>$ ./SFMLBoilerplate</pre><p>If this is your first time building an SFML application with CMake, congratulations! An empty window is rendered and the project’s current version is sent to standard output. You can also press the escape key to exit the application:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2lElDvVOGXvztIGzgjjlVw.gif" /><figcaption>Building and running the boilerplate application.</figcaption></figure><h3>3. Building and Installing a Pong Game</h3><p>The boilerplate project is a nice foundation for developing multimedia applications completely from scratch. However, it still leaves some important questions unanswered such as:</p><ul><li>How do you load external resources into an application such as fonts, images and audio files?</li><li>How can the source tree be amended to logically separate a project’s implementation and header files?</li><li>How can a game be installed globally on a Linux system so any user can play it?</li></ul><p>We will walk through the configuration of a pong game in order to answer these questions. The pong project is part of this article’s <a href="https://github.com/danebulat/cmake-sfml-demo">accompanying repository</a> and is located in the top-level Pong/ directory. Subsequent sections will detail how to build pong locally as well as doing a system-wide installation.</p><p>Let’s firstly take a look at the source tree and its corresponding files that make up the project:</p><pre>&gt; Pong/<br>    ---&gt; CMakeLists.txt<br>    ---&gt; content/<br>           ---&gt; background.png<br>           ---&gt; bip.wav<br>           ---&gt; sansation.ttf<br>    ---&gt; include/<br>           ---&gt; Ball.h<br>           ---&gt; Bat.h<br>           ---&gt; config.h.in<br>    ---&gt; src/<br>           ---&gt; Ball.cpp<br>           ---&gt; Bat.cpp<br>           ---&gt; main.cpp</pre><p>Notice that src/ and include/ contain the application&#39;s source files:</p><ul><li>Ball.h, Ball.cpp: Implements a Ball class that handles rendering and moving the pong ball around the window.</li><li>Bat.h, Bat.cpp: Implements a Bat class that renders a rectangular platform which will moves left or right based on input from the keyboard.</li><li>config.h.in: An input file we pass to CMake&#39;s <a href="https://cmake.org/cmake/help/latest/command/configure_file.html">configure_file()</a> command to generate a config.h header file.</li><li>main.cpp: Defines the applications entry point and handles user input, the game loop, and rendering.</li></ul><p>It is not necessary to understand all of the code to complete this tutorial. You can get an idea of how the application works by carefully reading the comments in each source file and looking at how SFML objects are used throughout — all SFML classes are defined in a namespace called sf. I encourage you to check out the <a href="https://www.sfml-dev.org/learn.php">many books and tutorials</a> online If you are interested in learning more about the SFML API.</p><p>Lastly, the content/ directory contains resources that our application loads at runtime. It includes a background image, sound effect and font for text rendering. The sansation.ttf font is provided by SFML and is usually included as part of the standard <a href="https://www.sfml-dev.org/tutorials/2.5/start-osx.php">Xcode project template</a>.</p><p>Resources such as images, fonts, audio, and configuration files should be put into the content/ directory. Not only does this keep our project organized, it also simplifies our installation script in CMakeLists.txt.</p><h4>CMake Configuration File</h4><p>One of the main challenges we must solve in our build configuration is how the executable is going to find resource files it loads at runtime. To understand this problem more clearly, consider the following points:</p><ul><li>When the executable is generated within the build tree, it will load resources locally from the project’s Pong/content/ directory.</li><li>When the executable is installed to a standard system location such as /usr/local/bin/, it needs to load resources from another standard location such as /usr/local/share/Pong/content/ to ensure the resources will always be available.</li></ul><p>Looking at the problem another way, if we install an executable globally that looks for resources in an arbitrary project saved somewhere in a user’s home directory, there is no guarantee that those files, or even the project will exist at any given point in time. The executable will break if the user decides to delete, rename, or move the project to a different location. This problem becomes even more apparent if the executable is installed on another system altogether.</p><p>Our solution firstly involves defining an option in our build configuration called USE_INSTALL_RESOURCE_PATH along with a variable called RESOURCE_PATH such that:</p><ul><li>When USE_INSTALL_RESOURCE_PATH is set to ON, RESOURCE_PATH will point to the resource installation location.</li><li>When USE_INSTALL_RESOURCE_PATH is set to OFF, RESOURCE_PATH will point to our local project’s resource directory.</li></ul><p>Let’s see how this logic is expressed in CMakeLists.txt:</p><pre>option(USE_INSTALL_RESOURCE_PATH <br>  &quot;Set resource path to install location&quot; OFF)</pre><pre>if(NOT USE_INSTALL_RESOURCE_PATH)<br>  set(RESOURCE_PATH &quot;${PROJECT_SOURCE_DIR}/content/&quot;)<br>else()<br>  set(RESOURCE_PATH <br>    &quot;${CMAKE_INSTALL_PREFIX}/share/${CMAKE_PROJECT_NAME}/content/&quot;)<br>endif()</pre><p>The path contained in RESOURCE_PATH is passed to our application through the config.h header file. The corresponding preprocessor definition is then referenced whenever a resource needs to be loaded at runtime. This happens in main.cpp:</p><pre>std::string source_dir = <strong>RESOURCE_PATH</strong>; </pre><pre><strong>// ... Load font</strong>​<br>sf::Font font;<br>if (!font.loadFromFile(source_dir + &quot;sansation.TTF&quot;))<br>    return EXIT_FAILURE;</pre><pre><strong>// ... Load texture</strong><br>sf::Texture texture;<br>if (!texture.loadFromFile(source_dir + &quot;background.png&quot;))<br>    return EXIT_FAILURE;</pre><pre><strong>// ... Load sound buffer</strong><br>sf::SoundBuffer buffer;<br>if (!buffer.loadFromFile(source_dir + &quot;bip.wav&quot;))<br>   return EXIT_FAILURE;</pre><p>Moving forward, we instruct CMake to find SFML version 2.5 somewhere on the system in addition to its libraries:</p><pre>find_package(SFML 2.5 <br>  COMPONENTS system window graphics audio REQUIRED)</pre><p>Our configuration subsequently instructs CMake to compile the source files in src/ into an executable called Pong:</p><pre>add_executable(Pong src/main.cpp src/Ball.cpp src/Bat.cpp)</pre><p>We also specify the build/ and include/ directories as include search paths by calling <a href="https://cmake.org/cmake/help/latest/command/target_include_directories.html">target_include_directories()</a>. Built-in CMake variables are referenced here so our paths remain valid even if our project is moved to a different place on the filesystem:</p><pre>target_include_directories(Pong <br>  PRIVATE<br>    &quot;${PROJECT_BINARY_DIR}&quot;<br>    &quot;${CMAKE_CURRENT_SOURCE_DIR}/include&quot;)</pre><p>We also set the INSTALL_RPATH_USE_LINK_PATH target property to TRUE for the Pong target near the bottom of our configuration. As a result, an installed executable is able to locate the SFML shared libraries and subsequently load them at runtime. Alternatively, the INSTALL_RPATH property can be set to a specific search path if the SFML libraries are installed in a non-standard location:</p><pre><strong># Using INSTALL_RPATH_USE_LINK_PATH</strong><br>set_target_properties(Pong<br>  PROPERTIES<br>    INSTALL_RPATH_USE_LINK_PATH TRUE)<br>    <br><strong># Using INSTALL_RPATH</strong><br>set_target_properties(Pong<br>  PROPERTIES<br>    INSTALL_RPATH &quot;${CMAKE_INSTALL_PREFIX}/lib&quot;)</pre><p>Also keep in mind that the sfml-system and sfml-window libraries are linked implicitly if you explicitly link a target to the sfml-graphics library. We also pass sfml-audio to enable our game to play sound effects:</p><pre>target_link_libraries(Pong sfml-graphics sfml-audio)</pre><p>Lastly, we make a couple of calls to the <a href="https://cmake.org/cmake/help/latest/command/install.html">install()</a> command for setting up correct install locations for the Pong executable and its resource files:</p><pre><strong># Copy Pong to /usr/local/bin</strong><br>install(TARGETS Pong DESTINATION bin)<br>​<br><strong># Copy content/ to /usr/local/share/Pong</strong><br>install(DIRECTORY content<br>  DESTINATION &quot;${CMAKE_INSTALL_PREFIX}/share/${CMAKE_PROJECT_NAME}&quot;)</pre><p>Refer to <a href="https://github.com/danebulat/cmake-sfml-demo">this article’s repository</a> to see the full listing of CMakeLists.txt with comments.</p><h4>Building and Installing Pong</h4><p>We have another chance to go through the build process with CMake! Let’s start by building a debug version of our pong game that can be run directly in the build tree:</p><pre>$ cd Pong <br>$ mkdir build &amp;&amp; cd build<br>$ cmake ..<br>$ cmake --build .</pre><p>Remember that our USE_INSTALL_RESOURCE_PATH option is set to OFF by default. As a result, the generated Pong executable will point to the content/ directory within our local project to resources. For debugging and testing purposes, this is absolutely fine. Go ahead and run Pong to try out the game before we commit to an install:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Jaw2Nrbk2b1-tnImtKoxnA.png" /><figcaption>Running Pong locally in the project build tree.</figcaption></figure><p>Pong is working pretty well — we can now proceed to build a release version and install it to the system. Let’s firstly remove our debug version by cleaning the build tree. This is necessary as we don’t want to install an executable that attempts to load resources from an arbitrary place on the filesystem by mistake:</p><pre>$ cmake --build . --target clean</pre><p>From here, open the cache with ccmake and modify the following settings:</p><ul><li>CMAKE_BUILD_TYPE: Release<br>A release build type will optimize our application&#39;s internal assembly code for improved performance and strip out any debugging symbols.</li><li>USE_INSTALL_RESOURCE_PATH: ON <br>Ensure that the executable points to a standard system location to load its resources.</li></ul><p>Press c to update CMakeCache.txt followed by g to generate corresponding native build files. Then build the project from the command line:</p><pre>$ make</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*nNU6uVsgICXNdGbZ6-WOFg.gif" /><figcaption>Configuring a release version of Pong and then building it.</figcaption></figure><p>Finally, let’s install our application and its resources:</p><pre>$ sudo make install<br>  [100%] Built target Pong<br>  Install the project...<br>  -- Install configuration: &quot;Release&quot;<br>  -- Installing: /usr/local/bin/Pong<br>  -- Set runtime path of &quot;/usr/local/bin/Pong&quot; to &quot;/usr/local/lib&quot;<br>  -- Installing: /usr/local/share/Pong/content<br>  -- Installing: /usr/local/share/Pong/content/bip.wav<br>  -- Installing: /usr/local/share/Pong/content/background.png<br>  -- Installing: /usr/local/share/Pong/content/sansation.ttf</pre><p>Pong can now be launched just like a regular Linux application. For example, one method to launch it is to simply run Pong in a terminal.</p><h3>In Conclusion</h3><p>In addition to building the SFML libraries from source, this article has detailed how to build and install SFML-based applications without using a traditional IDE. Instead, we used the CMake DSL (domain-specific language) to write build and install scripts for our projects. These scripts can be further developed to support other operating systems and take advantage of other CMake tools such as CPack and CTest.</p><h4>Related Articles</h4><p>Other articles I have written that focus on developing with CMake:</p><ul><li><a href="https://dane-bulat.medium.com/cmake-how-to-inspect-and-configure-the-compiler-877e6cb0317f">CMake: How to Inspect and Configure the Compiler</a></li><li><a href="https://dane-bulat.medium.com/vim-setting-up-a-build-system-and-code-completion-for-c-and-c-eb263c0a19a1">Vim: Setting up a Build System and Code Completion for C and C++</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3947b3ba6e8" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[CMake: How to Inspect and Configure the Compiler]]></title>
            <link>https://dane-bulat.medium.com/cmake-how-to-inspect-and-configure-the-compiler-877e6cb0317f?source=rss-24e03e26bd42------2</link>
            <guid isPermaLink="false">https://medium.com/p/877e6cb0317f</guid>
            <category><![CDATA[cmake]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[linux-tutorial]]></category>
            <category><![CDATA[compilers]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[Dane Bulat]]></dc:creator>
            <pubDate>Thu, 24 Dec 2020 17:18:46 GMT</pubDate>
            <atom:updated>2021-01-01T19:33:45.751Z</atom:updated>
            <content:encoded><![CDATA[<h4>A tutorial on using CMake to configure compilers and their flags for development projects</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*P1-aJ2rcyziXgq_W2iJlgQ.png" /></figure><h3>Introduction</h3><p>The CMake program makes it possible to write simple configuration files to control a project’s compilation process and generate native build files across platforms. It is also possible to configure the compilation process based on the host system’s environment in a granular way.</p><p>This tutorial details how to inspect, configure and output a project’s compiler settings using CMake. We will progress in incremental steps as described below:</p><ol><li><strong>Inspecting the Default Compiler</strong><br>This section details how to find out which compiler and compiler flags CMake uses on your system by default. These settings are then used to compile the sample project included with this tutorial.</li><li><strong>Selecting a Compiler and Inspecting its Properties</strong><br>We then look at how to select another C++ compiler on the system to build our project, as well as outputting properties such as its path, ID and version.</li><li><strong>Inspecting the Default Build Types<br></strong>The default compiler flags CMake uses for its build configurations are identified and discussed.</li><li><strong>Conditionally Adding Compiler Flags</strong><br>Our configuration is modified to conditionally set compiler flags depending on the selected compiler and build type.</li></ol><p>Upon completing this tutorial, you will have built up enough knowledge to be able to configure the compiler to your exact specifications for building a project with CMake.</p><h4>Source Files</h4><p>A small C++ <a href="https://github.com/danebulat/cmake-compiler-flags">sample project</a> has been setup on GitHub to accompany this tutorial. Run the following git command to download it to your system:</p><pre>$ git clone <a href="https://github.com/danebulat/cmake-compiler-flags.git">https://github.com/danebulat/cmake-compiler-flags.git</a></pre><p>The project contains an initial CMake configuration file along with some source files in a single directory:</p><pre>&gt; root<br>   ---&gt; CMakeLists.txt      <strong># Initial CMake configuration</strong><br>   ---&gt; interpolate.cpp     <strong># Source file for &#39;interp&#39; library</strong><br>   ---&gt; interpolate.h       <strong># Header file for &#39;interp&#39; library</strong><br>   ---&gt; main.cpp            <strong># Source file for the executable</strong><br>   ---&gt; steps/</pre><ul><li>interpolate.cpp and interpolate.h<br>These files implement some standard easing functions that are commonly used to produce tween animations. It&#39;s not necessary to understand how the easing functions work to complete this tutorial. CMake simply compiles these files into a shared library called interp.</li><li>main.cpp<strong><br></strong>A simple program that calls some of the easing functions and outputs the results. CMake compiles this file into an executable and links it to the interp shared library.</li><li>CMakeLists.txt<strong><br></strong>The CMake configuration file which describes how the project is built.</li><li>steps/<br>A directory containing snapshots of project files that correspond to each stage of the tutorial.</li></ul><p>Our initial configuration file sets some project properties and compiles the source files into an executable and shared library. Let’s go through some of the configuration to get acquainted with our initial setup.</p><pre>set(CMAKE_CXX_STANDARD 14)<br>set(CMAKE_CXX_EXTENSIONS OFF)<br>set(CMAKE_CXX_STANDARD_REQUIRED ON)</pre><p>Features of the C++ 14 standard are requested to compile the project’s targets. We also opt to not request compiler specific extensions.</p><pre>set(BUILD_SHARED_LIBS ON)</pre><p>The BUILD_SHARED_LIBS variable is turned on, which means a shared library is always produced if we do not pass a library type to the add_library command.</p><pre>add_library(interp interpolate.h interpolate.cpp)</pre><p>The interpolate.h and interpolate.cpp source files are compiled into a shared library called interp. CMake will use the host operating system&#39;s preferred naming scheme for shared libraries. For example, a shared library called libinterp.so will be generated on a Linux operating system.</p><pre>add_executable(main main.cpp)<br>target_link_libraries(main interp)</pre><p>The main.cpp file is compiled into an executable file called main, which subsequently links to the interp shared library</p><p>Take some time to read the comments in each source file to get a better understanding of our initial setup. We will take a look at how to build and run the program next.</p><h3>1. Inspecting the Default Compiler</h3><p>When CMake is invoked to read a configuration file, temporary files are generated, including a cache and a CMakeFiles directory containing information specific to the environment. CMake uses this information to select the most appropriate compiler for the project.</p><p>Let’s produce these files by creating a build directory within our sample project and invoking CMake to read the configuration file:</p><pre>$ cd cmake-compiler-flags<br>$ mkdir build &amp;&amp; cd build<br>$ cmake ..<br>    -| The CXX compiler identification is GNU 10.2.0<br>   ... Check for working CXX compiler: /usr/bin/c++ - skipped</pre><p>Information about the selected compiler is output when the configuration file is read. The compiler ID and version number complete the first line of output. The compiler’s location on the file system is also output a few lines down. In my case, GCC version 10.2.0 is selected, and is located at /usr/bin/c++.</p><p>These values may be different on your own system. For example, the Clang compiler will probably be selected on Mac OS with a corresponding binary named clang++. We will take a look at ways to switch the compiler in a following section.</p><p>It’s worth mentioning that your file system may have multiple copies of the same compiler mapped to different names. For example, common binary names for the GCC compiler include c++, gcc and g++. This may happen when you download a Linux distribution&#39;s development group which contains various packages to facilitate development.</p><p>If you are unsure about which compilers you have on your system, I recommend passing the --version and --help flags to the binaries in question:</p><pre><strong># A path is returned if the compiler is installed on your system</strong><br>$ which c++ g++ gcc</pre><pre><strong># Check compiler name and version</strong><br>$ c++ --version<br>$ gcc --version<br>$ g++ --version</pre><pre><strong># Review the help listing</strong><br>$ g++ --help | less</pre><pre><strong># Compare file size</strong><br>$ ls -l $(which c++ g++ gcc)</pre><p>Moving forward, it is also important to know which flags are passed to the compiler when targets are built. Compiler flags will also vary depending on the selected <strong>build type</strong>.</p><p>For example, a debug build will likely generate artifacts containing debugging symbols and less-optimized assembly code. Conversely, a release build usually strips out all debugging information and generates optimized assembly code to improve a program’s performance. The compiler must receive appropriate flags in order to satisfy a particular build type.</p><p>For now, we need to know that all build types use compiler flags defined in the CMAKE_CXX_FLAGS variable. Projects that haven&#39;t selected a build type, such as our own, will pass the flags defined in this variable to the compiler. It is defined as an empty string by default, and flags are appended either through the cache interface or by modifying CMakeLists.txt. Another important variable to be aware of is CMAKE_CXX_COMPILER, which stores the path to the selected compiler.</p><p>Before we compile our sample project, let’s inspect its cache file to confirm the presence of CMAKE_CXX_FLAGS in addition to which compiler is selected:</p><pre><strong># Open cache in build directory<br></strong>$ ccmake .</pre><p>The ccmake program presents a CLI which enables us to interact with our project&#39;s cache file. The very top variable called CMAKE_BUILD_TYPE is an empty string, meaning our project has not selected a build type yet. The variables we are interested in are revealed after pressing the t key to toggle advance mode — CMAKE_CXX_COMPILER and CMAKE_CXX_FLAGS will now be displayed. When you are finished inspecting the cache, press q to exit:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c8wunhStRcsm58ww4cnbDA.gif" /><figcaption>Inspecting variables in the project’s cache.</figcaption></figure><p>Now we know which compiler is selected and the (lack of) flags it will receive, let’s build the project and run our program:</p><pre><strong># Build project</strong><br>$ cmake --build .</pre><pre><strong># Run program</strong><br>$ ./main<br>    -| Begin...<br>       Result 0: 1 ...</pre><h3>2. Selecting a Compiler and Inspecting its Properties</h3><p>This section will cover a few useful techniques for finding information about the compiler and outputting its properties. We will firstly take a look at the CMake --system-information flag, which dumps detailed information about your system either to the screen or a file. Let&#39;s try it out in the sample project&#39;s root directory:</p><pre>$ cmake --system-information information.txt</pre><p>From here we can open information.txt in a text editor such as Vim and search for specific information about our environment. Go ahead and search for the following variables to check which values are assigned by default:</p><ul><li>CMAKE_CXX_COMPILER_LOADED: True if C++ was enabled for the project.</li><li>CMAKE_CXX_COMPILER_ID: The unique compiler identification string.</li><li>CMAKE_COMPILER_IS_GNUCXX: True if the C++ compiler is part of GCC.</li><li>CMAKE_CXX_COMPILER_VERSION: A string of the C++ compiler version.</li><li>CMAKE_CXX_COMPILER: Path to the selected C++ compiler.</li><li>CMAKE_C_COMPILER: Path to the selected C compiler.</li></ul><p>Values given to each variable in information.txt will vary depending on the host operating system. Also note that identical variables are defined for other CMake supported languages. For example, C language variable names specify C instead of CXX.</p><p>All variables included in this file can be referenced inside CMakeLists.txt and even sent to your application to facilitate conditional compiling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*jEQd0GvQ-llTFkVujn1O7Q.gif" /><figcaption>Inspecting system information generated by CMake.</figcaption></figure><h4>Selecting Another Compiler</h4><p>Let’s discuss how to select another C++ compiler to build a project. Perhaps you are a fan of the Clang/LLVM compiler and would prefer to use it instead of GCC. As you may have discovered, CMake stores the path of the selected compiler inside a variable called CMAKE_CXX_COMPILER. This variable can be set in two ways:</p><ul><li>Using a switch on the command line:<br>cmake -DCMAKE_CXX_COMPILER=&lt;compiler name&gt; ..</li><li>Using an environment variable:<br>export CXX=&lt;compiler name&gt;</li></ul><p>If you would like the --system-information flag to detect another compiler, you will need to export an environment variable called CXX:</p><pre><strong># Create CXX environment variable</strong><br>$ export CXX=clang++<br>$ echo $CXX<br>    -| clang++</pre><pre><strong># Overwrite information.txt with updated system information</strong><br>$ cmake --system-information information.txt</pre><p>Clang will now be selected if CMake is re-invoked to read CMakeLists.txt. Alternatively, a compiler can be specified on the command line by explicitly setting CMAKE_CXX_COMPILER. Doing so would take precedence over any value stored in the CXX environment variable:</p><pre><strong># Enter build directory and remove all files (clean)</strong><br>$ cd build &amp;&amp; rm -fr *</pre><pre><strong># Read configuration and set g++ compiler</strong><br>$ cmake -DCMAKE_CXX_COMPILER=clang++ ..<br>    -| The CXX compiler identification is Clang 11.0.0<br>   ... Check for working CXX compiler: /usr/bin/clang++ - skipped</pre><p>At this point, the sample project can be built and run in the same way as before. However, the compilation and linking phases will be handled by Clang instead of GCC:</p><pre>$ cmake --build .<br>$ ./main<br>    -| Begin...<br>       Result 0: 1 ...</pre><h4>Outputting Compiler Information at Configuration Time</h4><p>Additional information can be sent to standard output when CMake is invoked to read the configuration file. For example, an application may have a configuration option to control whether it will link to shared libraries or static libraries at compile time. It would be nice to see a visual confirmation of this setting at configuration time before we commit to the build process.</p><p>With this in mind, we will update our configuration file to output information about the selected compiler. As a matter of fact, we will reference the same variables that we searched for in the information.txt file earlier on. Add the following code to CMakeLists.txt to see this in action:</p><pre>   set(CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS ON)</pre><pre><strong>&gt;&gt;</strong> if(CMAKE_CXX_COMPILER_LOADED)<br>     message(STATUS &quot;Compiler path: ${CMAKE_CXX_COMPILER}&quot;)<br>     message(STATUS &quot;Compiler ID: ${CMAKE_CXX_COMPILER_ID}&quot;)<br>     message(STATUS &quot;Compiler version:    <br>             ${CMAKE_CXX_COMPILER_VERSION}&quot;)<br>     message(STATUS &quot;Compiler is part of GCC: <br>             ${CMAKE_COMPILER_IS_GNUCXX}&quot;)<br>   endif()</pre><p>Our configuration firstly checks if a C++ compiler is enabled via the CMAKE_CXX_COMPILER_LOADED variable. If a compiler is enabled, the message command is called for each variable we wish to log. Notice that CMake variables are referenced inside a string by specifying a dollar sign followed by curly braces around the variable name.</p><p>Now our messages are output whenever CMake is invoked to read CMakeLists.txt:</p><pre>$ cmake ..<br>    -| Compiler path: /usr/bin/clang++<br>       Compiler ID: Clang<br>       Compiler version: 11.0.0<br>       Compiler is part of GCC:  ...</pre><h4>Outputting Compiler Information at Runtime</h4><p>It is also possible to access CMake variables in an application’s source code. This is achieved by providing CMake a configuration file that contains preprocessor definitions corresponding to variables we’d like to expose to the application. CMake will process this file and output a regular C++ header file that our application can include.</p><p>Having an executable output compiler information is a good feature for a debug build. However, we won’t want to do this in a release build for obvious reasons. To solve this issue, we will define an <strong>option</strong> in CMakeLists.txt that will instruct the application to either compile the output code or not:</p><pre>   set(CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS ON)<br>   <br><strong>&gt;&gt;</strong> option(OUTPUT_COMPILER_INFO<br>     &quot;Output compiler information when launching the main<br>      executable.&quot; ON)</pre><p>Another issue we face is the behavior of the CMAKE_COMPILER_IS_GNUCXX variable. Because CMAKE_COMPILER_IS_GNUCXX either contains the value 1 or no value at all, mapping it to a preprocessor definition is error prone. Instead, we will set up a new variable called CXX_COMPILER_IS_GNU and assign it either 1 or 0 depending on whether CMAKE_COMPILER_IS_GNUCXX contains a value:</p><pre>   option(OUTPUT_COMPILER_INFO &quot;...&quot; ON)</pre><pre><strong>&gt;&gt;</strong> if(CMAKE_COMPILER_IS_GNUCXX MATCHES 1)<br>     set(CXX_COMPILER_IS_GNU 1)<br>   else()<br>     set(CXX_COMPILER_IS_GNU 0)<br>   endif()</pre><p>Let’s now turn our focus to generating a header file that contains definitions to corresponding variables in our configuration. This is done by calling the configure_file command using the following syntax:</p><pre>configure_file(&lt;input&gt; &lt;output&gt;)</pre><p>Similar to a template, the input file contains placeholders for variables defined in CMakeLists.txt. The syntax for declaring a variable placeholder is @VARIABLE_NAME@. An output file will copy the input file and set the placeholders to actual values defined in our build configuration. Let&#39;s firstly create an input file in the sample project&#39;s root directory called config.h.in and add some definitions within it:</p><pre>#pragma once<br>#define COMPILER_LOADED @CMAKE_CXX_COMPILER_LOADED@<br>#define COMPILER_IS_GNU @CXX_COMPILER_IS_GNU@<br>#define COMPILER_NAME &quot;@CMAKE_CXX_COMPILER@&quot;<br>#define COMPILER_ID &quot;@CMAKE_CXX_COMPILER_ID@&quot;<br>#define COMPILER_VERSION &quot;@CMAKE_CXX_COMPILER_VERSION@&quot;<br>#cmakedefine OUTPUT_COMPILER_INFO</pre><p>We proceed by making a call to configure_file in our configuration file. Immediately after, a call to target_include_directories specifies the location of where config.h will be generated such that main can find it on the file system:</p><pre>   target_link_libraries(main interp)<br> <br><strong>&gt;&gt;</strong> <strong># generate configuration file</strong><br>   configure_file(config.h.in config.h @ONLY)</pre><pre>   <strong># include binary directory to find config.h</strong><br>   target_include_directories(main PRIVATE ${PROJECT_BINARY_DIR})</pre><ul><li>@ONLY specifies that placeholders can be declared only with the @ symbol.</li><li>The PRIVATE keyword means that only the main target will look in the binary directory for include files. Any target that links to main will not search directories listed in this call.</li></ul><p>Lastly, update main.cpp to include config.h. The preprocessor definitions defined in config.h can now be referenced within the application to do things like conditional compiling and debugging. In our case, we define a function that outputs properties of the selected compiler, and subsequently call it in the main function. Additionally, OUTPUT_COMPILER_INFO is used to perform conditional compiling - if it contains a false value, our new code is not compiled:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/6c1d1a434aab9eff8d5301fdc27944ff/href">https://medium.com/media/6c1d1a434aab9eff8d5301fdc27944ff/href</a></iframe><p>With our new build configuration complete, let’s see what happens when we select another compiler:</p><pre>$ cd build &amp;&amp; rm -fr *<br>$ cmake -DCMAKE_CXX_COMPILER=g++ ..<br>$ cmake --build .<br>$ ./main<br>   -| Compiler name: /usr/bin/g++<br>      Compiler ID: GNU<br>      Compiler version: 10.2.0<br>      GCC compiler: 1</pre><p>Furthermore, if you set OUTPUT_COMPILER_INFO to OFF in the cache and rebuild, our output function will not compile.</p><h3>3. Inspecting the Default Build Types</h3><p>This section will focus on inspecting build types and their corresponding compiler flags. The CMake BUILD_TYPE variable specifies which build type configuration is selected at build time, and is empty by default. When a build type is not selected for a project, the compiler will only receive flags defined in the CMAKE_CXX_FLAGS variable.</p><p>It is therefore a good idea to select a build type before compiling a project. CMake includes four build types out-of-the-box which are:</p><ul><li>Debug: Fast compilation times, no optimization, debugging symbols are maintained.</li><li>Release: Compiles with a high level of optimization. Prioritizes runtime performance over compilation time.</li><li>RelWithDebugInfo: Compiles with a good level of optimization, debugging symbols are maintained.</li><li>MinSizeRel: Compiles with optimization that prioritizes executable file size and speed over compilation time.</li></ul><p>For each build type mentioned above, CMake creates a corresponding variable to store its compiler flags. The variable names along with their flags are detailed below:</p><ul><li>CMAKE_CXX_FLAGS_DEBUG: -g</li><li>CMAKE_CXX_FLAGS_MINSIZEREL: -Os -DNDEBUG</li><li>CMAKE_CXX_FLAGS_RELEASE: -O3 -DNDEBUG</li><li>CMAKE_CXX_FLAGS_RELWITHDEBUGINFO: -O2 -g -DNDEBUG</li></ul><p>Optimization flags increases both compilation time and performance of the generated code. The following table describes each compiler flag mentioned above:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/c0ed0bfccbbb30b0f9d4657611151604/href">https://medium.com/media/c0ed0bfccbbb30b0f9d4657611151604/href</a></iframe><p>You should now have a good idea of which build type to adopt at certain stages of development. A debug build would be selected If you are implementing new features or analyzing runtime behavior for bottlenecks. However, a release build would be adopted if your product is on the verge of being feature-complete and you need to measure the application’s performance in a real-world environment.</p><p>Build type variables and their corresponding compiler flags can be accessed from within the cache. Toggle advanced mode after launching either ccmake or cmake-gui to check them out.</p><h4>Selecting a Build Type</h4><p>Selecting a build type can be done in a number of ways. One method is to explicitly set the CMAKE_BUILD_TYPE variable in your configuration:</p><pre>   set(CMAKE_WINDOWS_EXPORT_ALL_SYMBOLS ON)</pre><pre><strong>&gt;&gt;</strong> if(NOT CMAKE_BUILD_TYPE)<br>     set(CMAKE_BUILD_TYPE Release CACHE STRING &quot;Build type&quot; FORCE)<br>   endif()</pre><p>This code sets the build type to Release providing a value for CMAKE_BUILD_TYPE is not passed on the command line. This means the project will always inherit a built type when it is configured. Keep in mind that a new value can always be set via the cache interface before starting the build process.</p><p>Another method usually adopted to set the CMAKE_BUILD_TYPE variable is via the command line at configuration time:</p><pre>$ cmake -D CMAKE_BUILD_TYPE=Debug ..</pre><p>In this case, the Debug configuration is selected instead of a Release.</p><p>To demonstrate build types in action, some code can be added to main.cpp to conditionally compile some code. We will output a message that will change depending on whether the debug build type is selected or not. Remember the -NDEBUG symbol that the release build types define? We will use that to determine which build type is selected.</p><p>Let’s proceed by adding some code to main.cpp:</p><pre><strong>&gt;&gt;</strong> #ifdef NDEBUG<br>       std::cout &lt;&lt; &quot;NDEBUG Defined (Release)&quot; <br>        &lt;&lt; std::endl &lt;&lt; std::endl;<br>   #else<br>    std::cout &lt;&lt; &quot;NDEBUG Not Defined (Debug)&quot; <br>                 &lt;&lt; std::endl &lt;&lt; std::endl;<br>   #endif</pre><pre>   std::cout &lt;&lt; &quot;Begin...&quot; &lt;&lt; std::endl;</pre><p>Test out this new functionality by doing a few more builds of the sample project, making sure to select a different build type and compiler combination each time:</p><pre><strong># Clean generated files</strong><br>$ cmake --build . --target clean</pre><pre><strong># Configure a debug build</strong><br>$ cmake -DCMAKE_BUILD_TYPE=Release -DCMAKE_CXX_COMPILER=clang++ ..<br>$ cmake --build .</pre><pre><strong># Run exectuable</strong><br>$ ./main<br>   -| Compiler name: /usr/bin/clang++<br>      ...<br>      NDEBUG Defined (Release)</pre><h3>4. Conditionally Adding Compiler Flags</h3><p>Even though many compiler flags work with both GCC and Clang, there could be times were you want to use a flag that is only available for a single compiler. For example, a developer who is compiling with GCC on a Linux system may want to use additional GCC specific flags to optimize the program even more. The same scenario may happen when the Clang compiler is adopted for a project.</p><p>It would also be nice to accumulate a set of flags for each build type our project supports. To facilitate this functionality, our solution firstly creates separate variables to store global (all build types), debug, and release flags:</p><pre>     message(STATUS &quot;Compiler is part of GCC: ${ ... }&quot;)<br>   endif()</pre><pre><strong>&gt;&gt;</strong> set(CXX_FLAGS)<br>   set(CXX_FLAGS_DEBUG)<br>   set(CXX_FLAGS_RELEASE)</pre><p>A couple of flags are immediately appended to CXX_FLAGS in order to compile all targets with position independent code, and to enable the recommended compiler warnings:</p><pre>   set(CXX_FLAGS_RELEASE)</pre><pre><strong>&gt;&gt;</strong> list(APPEND CXX_FLAGS &quot;-fPIC&quot; &quot;-Wall&quot;)</pre><p>From here we construct a couple of if statements that check which compiler is selected at build time. Our configuration checks for a GNU compiler as well as the Clang compiler. Depending on the selected compiler, appropriate flags are appended to our variables:</p><pre>   list(APPEND CXX_FLAGS &quot;-fPIC&quot; &quot;-Wall&quot;)<br>   <br><strong>&gt;&gt;</strong> if(CMAKE_CXX_COMPILER_ID MATCHES GNU)<br>     list(APPEND CXX_FLAGS &quot;-fno-rtti&quot; &quot;-fno-exceptions&quot;)<br>     list(APPEND CXX_FLAGS_DEBUG &quot;-Wsuggest-final-types&quot; <br>          &quot;-Wsuggest-final-methods&quot; &quot;-Wsuggest-override&quot;)<br>     list(APPEND CXX_FLAGS_RELEASE &quot;-O3&quot; &quot;-Wno-unused&quot;)<br>   endif()</pre><pre>   if(CMAKE_CXX_COMPILER_ID MATCHES Clang)<br>     list(APPEND CXX_FLAGS &quot;-fno-rtti&quot; &quot;-fno-exceptions&quot; <br>          &quot;-Qunused-arguments&quot; &quot;-fcolor-diagnostics&quot;)<br>     list(APPEND CXX_FLAGS_DEBUG &quot;-Wdocumentation&quot;)<br>     list(APPEND CXX_FLAGS_RELEASE &quot;-O3&quot; &quot;-Wno-unused&quot;)<br>   endif()</pre><p>Note that additional if statements can be added to check for other compilers your project wishes to support.</p><p>Moving forward, a call to target_compile_options will let us assign compiler flags to a given target. We firstly instruct CMake to compile the interp shared library using compiler flags defined in CXX_FLAGS:</p><pre><strong>&gt;&gt; </strong>target_compile_options(interp PRIVATE ${CXX_FLAGS})</pre><pre>   add_executable(main main.cpp)</pre><p>On top of the global compiler flags, generator expressions are used to assign either the debug or release compiler flags to the main target:</p><pre><strong>&gt;&gt;</strong> target_compile_options(main PRIVATE ${CXX_FLAGS}<br>     &quot;$&lt;$&lt;CONFIG:Debug&gt;:${CXX_FLAGS_DEBUG}&gt;&quot;<br>     &quot;$&lt;$&lt;CONFIG:Release&gt;:${CXX_FLAGS_RELEASE}&gt;&quot;)</pre><pre>   target_link_libraries(main interp)</pre><p>The CONFIG generator expression returns the selected build type. We use it here to assign correct compiler flags based on the selected build type. For example, assuming CONFIG returns Debug, $&lt;CONFIG:Debug&gt; will expand to $&lt;Debug:Debug&gt;, resulting in a true value. In this case, compiler flags defined in CXX_FLAGS_DEBUG are used. On the other hand, CMake will select the flags defined in CXX_FLAGS_RELEASED if CONFIG returns Release.</p><p>It is also important to know that compile options can be added with three levels of visibility:</p><ul><li>PRIVATE: Options will only be applied to the given target and not to other targets consuming it.</li><li>INTERFACE: Options on a given target will only be applied to targets consuming it.</li><li>PUBLIC: Options will be applied to the given target and all other targets consuming it.</li></ul><p>Our configuration specifies the PRIVATE attribute, which means consuming targets will not inherit the consumed target&#39;s compiler flags.</p><p>Build your sample project a few times again, but this time append --<strong> </strong>VERBOSE=1 at the end of the build command. This option will enable us to confirm which compiler flags were received by the compiler:</p><pre>$ cd build &amp;&amp; rm -fr *<br>$ cmake -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_BUILD_TYPE=Debug ..<br>$ cmake --build . <strong>-- VERBOSE=1</strong></pre><p>You will notice that CMake compiles the shared library and executable using different compiler flags. The selected build type also influences which flags are passed to the compiler.</p><h4>Outputting Selected Compiler Flags</h4><p>Confirming which compiler flags a build receives via the verbose option on the command line is not very efficient. A better option would be to output the flags at configuration time. With this in mind, we will use the CMakePrintHelpers module to output our variables.</p><p>Update your configuration file to output compiler flags just before the shared library is built:</p><pre><strong>&gt;&gt;</strong> include(CMakePrintHelpers)<br>   cmake_print_variables(CXX_FLAGS)<br>   cmake_print_variables(CXX_FLAGS_DEBUG)<br>   cmake_print_variables(CXX_FLAGS_RELEASE)<br>   <br>   add_library(interp interpolate.h interpolate.cpp)</pre><p>We include the CMakePrintHelpers module and immediately call cmake_print_variables to output the contents of each variable on a single line. It is also possible to specify multiple variables in a single call.</p><p>The manual page for any CMake module can be accessed on the command line via the --help-module option. For example, the following command opens a manual page for the CMakePrintHelpers module:</p><pre>$ cmake --help-module CMakePrintHelpers</pre><p>Selected flags are now output for a particular build at configuration time:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*1-EjKfvT5yeeyhN6dIpKLg.gif" /><figcaption>Building a release configuration of our project.</figcaption></figure><h3>In Conclusion</h3><p>I hope this tutorial has provided enough information for you to configure your own development projects with confidence when it comes to the compiler. I encourage you to experiment further with the sample project and perform some more conditional compilation based on the build type and compiler selected for a particular build.</p><h4>Related Articles</h4><p>Other articles I have written that focus on developing with CMake:</p><ul><li><a href="https://dane-bulat.medium.com/cmake-building-sfml-and-game-projects-on-linux-3947b3ba6e8">CMake: Building SFML and Game Projects on Linux</a></li><li><a href="https://dane-bulat.medium.com/vim-setting-up-a-build-system-and-code-completion-for-c-and-c-eb263c0a19a1">Vim: Setting up a Build System and Code Completion for C and C++</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=877e6cb0317f" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Vim: Setting up a Build System and Code Completion for C and C++]]></title>
            <link>https://dane-bulat.medium.com/vim-setting-up-a-build-system-and-code-completion-for-c-and-c-eb263c0a19a1?source=rss-24e03e26bd42------2</link>
            <guid isPermaLink="false">https://medium.com/p/eb263c0a19a1</guid>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[linux-tutorial]]></category>
            <category><![CDATA[vim]]></category>
            <category><![CDATA[cmake]]></category>
            <dc:creator><![CDATA[Dane Bulat]]></dc:creator>
            <pubDate>Tue, 15 Dec 2020 15:43:43 GMT</pubDate>
            <atom:updated>2021-01-01T19:35:00.630Z</atom:updated>
            <content:encoded><![CDATA[<h4>A guide on integrating CMake and the YouCompleteMe engine with Vim</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eedXnEDHoJ7ZDPBFckSWcw.png" /></figure><h3>Introduction</h3><p>This guide details how to turn Vim into a powerful C and C++ development environment. After working through each incremental step you will finish with a Vim configuration and boilerplate C++ project that you can use as a starting point for future applications. The steps presented in this guide will also work with Neovim.</p><p>CMake will be used to manage our application build process in a platform-independent way. We will also take a look at how to install and configure the <a href="https://github.com/ycm-core/YouCompleteMe">YouCompleteMe</a> Vim plug-in (from here referred to as YCM) to enable automatic code completion and error detection when working on source files.</p><p>Here’s a preview of how Vim will behave after completing this guide:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VTn3KktcYCNT6emqrCD10A.gif" /><figcaption>Preview of Vim running CMake and YCM.</figcaption></figure><p>Implementation details are broken down into six incremental sections where each one completes a specific task:</p><ol><li><strong>Project Structure and Initial CMake Script</strong><br>We will install CMake and set up a project directory structure containing a simple C++ application. An initial CMake script will then be presented for compiling the application.</li><li><strong>YouCompleteMe Installation</strong> <br>Detailed instructions are provided for installing the YCM code-completion engine on your Linux distribution. A configuration is also provided to improve the overall experience of running YCM with Vim.</li><li><strong>Setting Up Debug and Release Configurations</strong><br>The project’s directory structure and CMake script are modified to support debug and release builds of the boilerplate application. This section will also detail how to configure compiler flags with CMake.</li><li><strong>Vim Mappings for Automatic Building and Running</strong><br>This section will detail setting up some key mappings for automatically building and running the application. The <a href="https://github.com/tpope/vim-dispatch">vim-dispatch</a> plug-in is utilized to run commands tied to the mappings asynchronously.</li><li><strong>Compiling and Linking a Dynamic Library<br></strong>A source file containing some mathematical functions is added to the project and compiled into a shared library. We use CMake to link to the shared library from our application.</li><li><strong>Final Tweaks<br></strong>The Vim configuration will be modified for setting a good color scheme.</li></ol><h4>Starting Vim Configuration</h4><p>This article recommends having a Vim configuration with some plug-ins already installed that accomplish some common editor tasks — such as having language packs for syntax highlighting, correct tabbing behavior, and a file system explorer. Having these features set up in advance will complement the functionality you will be integrating in this guide.</p><p>If you have a vanilla Vim installation without any plug-ins, I recommend going through my previous article which covers how to turn Vim into a lightweight IDE. As well as installing the Vundle package manager, it details how to configure a range of plug-ins to facilitate programming with Vim:</p><p><a href="https://dane-bulat.medium.com/how-to-turn-vim-into-a-lightweight-ide-6185e0f47b79">How to Turn Vim Into a Lightweight IDE</a></p><h4>Project Source Files</h4><p>I have set up a <a href="https://github.com/danebulat/vim-cmake-boilerplate">GitHub repository</a> for readers wishing to check out the final project detailed in this guide. You can download the repository by running the following git command:</p><pre>$ git clone <a href="https://github.com/danebulat/vim-cpp-completion">https://github.com/danebulat/vim-cmake-boilerplate</a></pre><p>In addition to a boilerplate C++ application and build scripts, the repository also includes the corresponding Vim configuration file at config/vimrc. Keep in mind that the configuration does rely on some system packages for everything to work. Instructions for Installing YCM dependencies are provided in subsequent sections.</p><h3>Project Structure and Initial CMake Script</h3><p>The boilerplate project will have the following directory structure:</p><pre>- root<br>   --&gt; bin        <strong># Executables</strong><br>   --&gt; build      <strong># Build system files</strong><br>   --&gt; config     <strong># Vim configuration</strong><br>   --&gt; include    <strong># Application header files</strong><br>   --&gt; lib        <strong># Library source files</strong><br>   --&gt; src        <strong># Application source files</strong></pre><p>Common directory names used in C++ projects are adopted. Our solution also opts for out-of-source builds, meaning executables and shared libraries generated by the build system are placed outside of directories containing the actual source code.</p><p>The build directory will contain native build files generated by CMake after it reads the top-level CMakeLists.txt script that we write ourselves. Native Makefiles, executables and shared libraries will be placed in this directory when the project is built.</p><p>The config directory is a good location to store your corresponding Vim configuration file. This means a working configuration will always be available if you modify your live .vimrc file to a point where it breaks some functionality.</p><p>For readers implementing the project from scratch, navigate to an appropriate place on your file system and create the directory structure mentioned above:</p><pre>$ mkdir vim-cmake-boilerplate &amp;&amp; cd vim-cmake-boilerplate<br>$ mkdir bin build config include lib src</pre><p>Let’s also add a simple main.cpp file in the src directory that we can build with CMake shortly:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/911cc1d8bc9213e919d60ee8d3e73de3/href">https://medium.com/media/911cc1d8bc9213e919d60ee8d3e73de3/href</a></iframe><h4>CMake Installation</h4><p>Moving forward, go ahead and install CMake using your distribution’s package manager if its not on your system already:</p><pre>$ sudo pacman -S cmake      <strong># Arch</strong><br>$ sudo apt install cmake    <strong># Debian and Ubuntu</strong><br>$ sudo dnf install cmake    <strong># Fedora and CentOS</strong></pre><p>I also recommend making a note of your CMake version and reading the short DESCRIPTION paragraph on its manual page:</p><pre>$ cmake --version<br>    -| cmake version 3.19.1 ...<br>$ man cmake</pre><p>The manual describes CMake as a cross-platform build system generator — meaning we can write a single CMake script that will generate native build files on platforms such as Linux, Windows and Mac OS.</p><p>Reading further also reveals that the operating system’s native build tool can be invoked to build or install a project that uses CMake. This means that we could invoke the make utility on Linux to build our application after generating the necessary native build files with CMake. Alternatively, the cmake --build command can be used to do the same thing.</p><p>Lastly, CMake provides a GUI program to modify a project’s build configuration which can be invoked by running the cmake-gui command. Although I personally prefer using the ccmake command in a terminal, a GUI equivalent is also available.</p><h4>Initial Build Script</h4><p>With CMake installed we can write a short script to compile the main.cpp file and output an executable. We refer to this script as the top-level CMakeLists.txt file, and it is read by CMake to generate native build files for the system. Let&#39;s create a CMakeLists.txt file in the project&#39;s root directory containing some initial commands:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/851de36322b9e1bdea6f6fcf43693f34/href">https://medium.com/media/851de36322b9e1bdea6f6fcf43693f34/href</a></iframe><p>Most of the commands are self-explanatory. We give the executable a name of Boilerplate and request C++ 11 standard features to build the target. The last command sets the RUNTIME_OUTPUT_DIRECTORY variable to the location of our project&#39;s bin directory. As a result, the generated executable will be placed in this location.</p><p>At this point we have done enough to generate an executable with the cmake utility. Let&#39;s test this out by running some commands in a terminal:</p><pre><strong># Enter build directory</strong><br>$ cd build</pre><pre><strong># Read CMakeLists.txt and generate native build files</strong><br>$ cmake ..</pre><pre><strong># Build project and generate executable in bin/ directory</strong><br>$ cmake --build .</pre><pre><strong># Run the executable</strong><br>$ ../bin/Boilerplate<br>    -| Hello, CMake!</pre><h4>Configuration Header</h4><p>The next thing we’ll do is generate a header file containing definitions that correspond to variables in CMakeLists.txt. Our application can include this header file and reference its definitions to do things like output build-specific information and compile code conditionally.</p><p>In our case, we will generate a header file called Config.h that contains definitions of the project&#39;s version number. This is done by calling the configure_file command in CMakeLists.txt using the syntax :</p><pre>configure_file(&lt;input&gt; &lt;output&gt;)</pre><p>The input file is similar to a template. It contains placeholders for variables that exist in CMakeLists.txt with the syntax @VARIABLE_NAME@. An output file will copy the input file and set the placeholders to actual values defined in our build configuration. Let&#39;s firstly create the input file at include/Config.h.in and define a couple of placeholders:</p><pre>#pragma once<br>#define VERSION_MAJOR @Boilerplate_VERSION_MAJOR@<br>#define VERSION_MINOR @Boilerplate_VERSION_MINOR@</pre><p>Then call the configure_file command in CMakeLists.txt:</p><pre>   set(CMAKE_CXX_STANDARD_REQUIRED True)</pre><pre><strong>&gt;&gt; </strong>configure_file(include/Config.h.in <br>      &quot;${CMAKE_CURRENT_SOURCE_DIR}/include/Config.h&quot;)</pre><p>In order for CMake to find Config.h when the target is compiled, we must tell it to search the include directory:</p><pre>   add_executable(Boilerplate <br>       &quot;${CMAKE_CURRENT_SOURCE_DIR}/src/main.cpp&quot;)<br>  <br><strong>&gt;&gt;</strong> target_include_directories(Boilerplate PUBLIC<br>       &quot;${CMAKE_CURRENT_SOURCE_DIR}/include&quot;)</pre><p>Lastly, include Config.h in src/main.cpp and output the application&#39;s version:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/71db8e7a995f3308ce7af12ee154c7c6/href">https://medium.com/media/71db8e7a995f3308ce7af12ee154c7c6/href</a></iframe><p>If you now update the version number in CMakeLists.txt and rebuild your project, the application will output the correct version number — there is no need to manually update any source code.</p><h3>YouCompleteMe Installation</h3><p>This section will go through installing the YCM plug-in for C and C++ development. YCM is a fast and powerful code completion engine that supports many languages. Its dependencies will vary depending on what language you want code completion for.</p><p>We will install and configure YCM to perform code completion for the C-family languages. The engine will provide IDE-like features, including:</p><ul><li>A completion menu giving suggestions for anything we type in source files.</li><li>Function and method signature help and argument hints.</li><li>Detection of warning and errors that are presented in various ways.</li><li>Commands for refactoring and finding symbols.</li><li>Many configuration options to personalize the behavior of the plug-in.</li></ul><p>This guide will provide enough information to get started with the YCM engine. With that said, multiple articles could be written going over all of its features and possible configurations. I certainly recommend browsing through the official <a href="https://github.com/ycm-core/YouCompleteMe">YCM GitHub page</a> to learn more about the plug-in.</p><h4>Installation</h4><p>The current build of YCM requires the minimum version of Vim to be v.8.1.2269. It is therefore a good idea to check that you are running a recent version of Vim on your system:</p><pre>$ vim --version | awk &#39;NR==1 {print; exit}&#39;<br>    -| VIM - Vi IMproved <strong>8.2</strong> (2019 Dec 12, compiled ... 16:46:37)</pre><p>The YCM documentation also recommends using the <a href="https://github.com/VundleVim/Vundle.vim">Vundle</a> plug-in manager to install it — feel free to check out my <a href="https://dane-bulat.medium.com/how-to-turn-vim-into-a-lightweight-ide-6185e0f47b79">previous article</a> if you need help setting up Vundle. Although not mentioned in the YCM documentation, I’ve found that <a href="https://github.com/junegunn/vim-plug">vim-plug</a> also installs YCM without any problems.</p><p>Additionally, Python and a couple of packages that provide front-end compiler technologies are required by the YCM server. These packages can be installed with your distribution’s package manager:</p><pre><strong># Arch</strong><br>$ sudo pacman -S base-devel python clang llvm</pre><pre><strong># Debian and Ubuntu</strong><br>$ sudo apt install build-essential python3-dev llvm-defaults</pre><pre><strong># Fedora</strong><br>$ sudo dnf install gcc-c++ make python3-devel clang <br>  clang-tools-extra</pre><p><em>Please leave a note or comment If you are having trouble installing YCM on your distribution. I’ll investigate and update this guide with instructions accordingly!</em></p><p>With all dependencies taken care of we can now download the YCM plug-in with Vundle. Go ahead and add the ycm-core/YouCompleteMe repository to your plug-in list in .vimrc. Then refresh your configuration file before calling the :PluginInstall command:</p><pre><strong>&quot; Add YCM repository to plugin list</strong><br>Plugin &#39;ycm-core/YouCompleteMe&#39;</pre><pre><strong>&quot; Refresh configuration</strong><br>:source ~/.vimrc</pre><pre><strong>&quot; Download YCM</strong><br>:PluginInstall</pre><p>After :PlugInstall completes, exit Vim and navigate to the downloaded YCM repository on your file system. In addition to the main repository, several nested Git repositories called submodules are configured to work with YCM. In other words, these submodules need to be downloaded in order for the completion engine to function. You can check which submodules are configured for the YouCompleteMe repository by opening the .gitmodules file in the root directory:</p><pre>$ cd ~/.vim/plugged/YouCompleteMe<br>$ cat .gitmodules<br>    -| [submodule &quot;third_party/requests-futures&quot;]<br>            path = third_party/requests-futures<br>            url = <a href="https://github.com/ross/requests-futures">https://github.com/ross/requests-futures</a><br>            ...</pre><p>The path value specifies where the submodule will be cloned in the repository directory structure, and the url value identifies the remote location of the submodule. Run the following commands to download the submodules and complete the YCM installation:</p><pre><strong># Clone submodules</strong><br>$ git submodule update --init --recursive</pre><pre><strong># Install YCM</strong><br>$ python3 install.py --clangd-completer</pre><p>Notice that we specify the --clangd-completer option when invoking the install.py script. This means that YCM will provide semantic support (code completion) for C-family languages. You can provide additional flags to enable support for other languages, or specify the -all flag to enable everything. Keep in mind that doing this will require additional dependencies.</p><p>The YCM server will attempt to start the next time Vim is launched. However, it still needs to know how to compile our source files behind the scenes to analyze the code and give feedback. We solve this issue in the next section.</p><h4>Generating a Compiler Flags Database with CMake</h4><p>The YCM server will be able to compile source code and provide correct feedback when it detects a database of compiler flags in our project. We turn on the CMAKE_EXPORT_COMPILE_COMMANDS variable in CMakeLists.txt to generate a compiler flags database whenever the project is built:</p><pre>   set(CMAKE_CXX_STANDARD_REQUIRED True)<br><strong>&gt;&gt;</strong> set(CMAKE_EXPORT_COMPILE_COMMANDS ON)</pre><p>Now a file called compile_commands.json is generated and put in the build directory when we initiate a build - this file is the database we need:</p><pre>$ cd build<br>$ cmake .. &amp;&amp; cmake --build .<br>$ ls compile_commands.json<br>      -| compile_commands.json</pre><p>Lastly, this file needs to be located in the project’s root directory. The YCM documentation tells us to either make a copy, or create a symbolic link to the generated database. We will opt for the second option by creating a soft symbolic link to the actual build/compile_commands.json file:</p><pre>$ cd .. &amp;&amp; pwd<br>    -| /home/dane/Github/vim-cpp-completion<br>$ ln -s build/compile_commands.json compile_commands.json</pre><p>If Vim was open whilst you were generating the compiler flags database, restart the YCM server within the Vim editor so it detects the new file:</p><pre>:YcmRestartServer</pre><p>Now start typing some code to see the completion menu in action. Completion suggestions are based on a sub-sequence of what you type. Press the TAB key to cycle through items in the completion menu and Shift-TAB to cycle back. The default mapping to close the completion menu is &lt;C-y&gt;. Warnings and errors will also be presented in various ways:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d3fpeH8lOQog9KDWxzrtVg.gif" /><figcaption>Testing the default behavior of the YCM plug-in.</figcaption></figure><h4>Configuring YCM</h4><p>What YCM provides out of the box is great, but it would be nice to modify some settings. For example, making the help window (documentation panel) close automatically when we leave a highlighted item in the completion menu will make development more comfortable. You may also wish to modify the maximum amount of items displayed in the completion menu.</p><p>The following gist lists my YCM configuration. Feel free to copy these settings into your own .vimrc file while reading the comments to get an idea of what the setting does:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2b880871a59237594434e60831f584d0/href">https://medium.com/media/2b880871a59237594434e60831f584d0/href</a></iframe><p>The editor will also “snap” a couple of characters to the right when YCM detects an error or warning in your source file. When this happens, the &gt;&gt; characters are rendered in the sign column. It is a good idea to always show the sign column in Vim to prevent this snapping behavior:</p><pre><strong>&quot; Make Vim always render the sign column:</strong><br>set signcolumn=yes</pre><h3>Setting Up Debug and Release Configurations</h3><p>This section will discuss how to set up debug and release configurations for our project. The CMake BUILD_TYPE variable specifies which configuration is selected when the project is built, and is empty by default. Furthermore, flags defined in the CMAKE_CXX_FLAGS variable are selected when BUILD_TYPE is empty. Compiler flags can be observed in your CMake cache either via the terminal or CMake GUI program:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*f7LwQqdjGxK7D-j0hTck4g.gif" /><figcaption>Checking the project’s default compiler flags.</figcaption></figure><p>You will notice CMAKE_CXX_FLAGS_DEBUG and CMAKE_CXX_FLAGS_RELEASE variables defined in the CMake cache and set to some default compiler flags:</p><ul><li>-g : Generates debugging symbols. Should be used for debug builds during development.</li><li>-O3: Sets a high optimization level. Results in slower compile-time but faster production builds.</li><li>-DNDEBUG: Skips calls to assert functions.</li></ul><p>The following subsections will detail how to configure our project to use the debug and release CMake configurations. We will also define our own compiler flags in CMakeLists.txt.</p><h4>Updating the Project Structure</h4><p>Lets update our project’s directory structure to accommodate individual build configurations by having nested debug and release directories in the bin and build directories:</p><pre><strong># build directory</strong><br>$ cd build &amp;&amp; rm -fr *<br>$ mkdir Debug Release</pre><pre><strong># bin directory</strong><br>$ cd ../bin &amp;&amp; rm -fr *<br>$ mkdir Debug Release<br>$ cd ..</pre><p>It is now possible to use the $&lt;CONFIG&gt; generator expression in the set_target_properties command for defining output directories for executables and libraries. The value of $&lt;CONFIG&gt; will be either Debug or Release when the project builds, making it possible to output executables in the correct location. Update the call to set_target_properties in CMakeLists.txt to set up this functionality:</p><pre>set_target_properties(Boilerplate<br>    PROPERTIES<br>    RUNTIME_OUTPUT_DIRECTORY &quot;${PROJECT_SOURCE_DIR}/bin/$&lt;CONFIG&gt;&quot;)</pre><p>Now we must specify which configuration to use when the native build files are generated with CMake. For example, in the debug directory we would set the CMAKE_BUILD_TYPE variable to Debug when generating the native build files:</p><pre>$ cd build/Debug<br>$ cmake -DCMAKE_BUILD_TYPE=Debug ../.. <br>$ cmake --build .<br>$ ../../bin/Debug/Boilerplate<br>    -| Running: ../../bin/Debug/Boilerplate<br>       Version: 1.0<br>       Hello, CMake!</pre><p>Similarly, the CMAKE_BUILD_TYPE variable is set to Release when we want to generate native build files for a the release configuration:</p><pre>$ cd ../Release<br>$ cmake -DCMAKE_BUILD_TYPE=Release ../.. <br>$ cmake --build .<br>$ ../../bin/Release/Boilerplate<br>    -| Running: ../../bin/Debug/Boilerplate<br>       Version: 1.0<br>       Hello, CMake!</pre><p>The symbolic link pointing to the compiler flags database must also be re-linked because the build structure has been modified:</p><pre>$ rm compile_commands.json<br>$ ln -s build/Debug/compile_commands.json compile_commands.json</pre><p>YCM now compiles code using the debug flags in CMAKE_CXX_FLAGS_DEBUG. Restart the YCM server in Vim if necessary by running :YcmRestartServer to see the changes take effect.</p><h4>Setting Compiler Flags</h4><p>Let’s now append some commands to CMakeLists.txt to pass some extra compilation flags to the compiler:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/691e8cba4dcb4a71b01c0f862bae5317/href">https://medium.com/media/691e8cba4dcb4a71b01c0f862bae5317/href</a></iframe><p>Generator expressions are used here to select compiler flags based on the operating system. The flags defined in the gcc_flags variable are selected if a GCC-like compiler is detected on the system such as clang or c++ . We have also instructed CMake to select the -W3 flag if a Microsoft compiler is detected.</p><p>You may also want to add the -Werror and -Wunused compiler flags If your project is adopting very strict coding standards. Refer to the <a href="https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html">GNU GCC documentation</a> if you need to brush up on the various warning options.</p><p>Since target_compile_features now requests the C++ 11 standard features to build the target, the following lines can be removed from CMakeLists.txt:</p><pre>set(CMAKE_CXX_STANDARD 11)<br>set(CMAKE_CXX_STANDARD_REQUIRED True)</pre><h3>Vim Mappings for Automatic Building and Running</h3><p>This section will detail how to set up mappings to build and run the debug and release configurations from within Vim. It would be nice to be able to press a quick key combination to initiate the build process and run executables without having to drop to a command line and type everything manually.</p><p>Our solution relies on the <a href="https://github.com/tpope/vim-dispatch">vim-dispatch</a> plug-in to execute commands asynchronously after a mapping is pressed. This allows us to continue working in the editor while building takes place in another terminal instance behind the scenes. Once a command completes, its corresponding terminal window can be opened to observe output. By doing this, any warnings or errors that may have occurred during the build can be checked.</p><p>Let’s start by adding the vim-dispatch repository to the plug-in list in .vimrc and installing it:</p><pre>Plugin &#39;tpope/vim-dispatch&#39;<br>:source ~/.vimrc<br>:PluginInstall</pre><p>Make sure to restart Vim after vim-dispatch installs. From here we can define some mappings that will automatically build and run our application:</p><pre>&quot; Open vim-dispatch window and scroll to bottom<br>nnoremap    &lt;C-m&gt;m    :Copen&lt;CR&gt; &lt;bar&gt; G</pre><pre>&quot; Build debug and release targets<br>nnoremap    &lt;C-m&gt;bd   :Dispatch! make -C build/Debug&lt;CR&gt;<br>nnoremap    &lt;C-m&gt;br   :Dispatch! make -C build/Release&lt;CR&gt;</pre><ul><li><strong>&lt;C-m&gt;m</strong> opens the vim-dispatch window by invoking the :Copen command. The cursor scrolls to the last line in the window via the G key.</li><li><strong>&lt;C-m&gt;bd</strong> invokes make to build the debug configuration. The -C flag specifies the directory to navigate to before initiating the build process.</li><li><strong>&lt;C-m&gt;br</strong> is similar to &lt;C-m&gt;bd but builds the release configuration.</li></ul><p>Feel free to define your own mappings for these commands. In the example above, initially pressing Ctrl+m signifies we are dealing with a make-related command. The subsequent presses of bd and br translate to &quot;build debug&quot; and &quot;build release&quot;.</p><p>We also need mappings to run the project’s executables in the bin directory. The problem here is that Vim doesn&#39;t know where the executables are located or their corresponding file names. These properties will change depending on the project you are working on.</p><p>Our solution defines a couple of functions using Vimscript that enable us to specify the executable to run after pressing the corresponding key mapping. These functions are:</p><ul><li><strong>SetBinaryDebug(filename)</strong>: The bin/Debug/&lt;filename&gt; executable is run when F6 is pressed.</li><li><strong>SetBinaryRelease(filename)</strong>: The bin/Release/&lt;filename&gt; executable is run when F7 is pressed.</li></ul><p>This setup requires us to call SetBinaryDebug(..) and SetBinaryRelease(..) each time we launch Vim. However, we have full control over which executable is run for either the debug or release configurations. The function implementations in .vimrc are shown in here:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/a768bdf23e92d2d3a0f615f988030e14/href">https://medium.com/media/a768bdf23e92d2d3a0f615f988030e14/href</a></iframe><p>Notice that the /bin/Release/ and /bin/Debug/ strings are appended to the output of the getcwd() function, which returns the current working directory. These functions can now be called at any time to set the executables that will run when F6 and F7 are pressed:</p><pre>:call SetBinaryDebug(&quot;Boilerplate&quot;)<br>    -| <strong>&lt;F6&gt; will run: /home/.../bin/Debug/Boilerplate</strong></pre><pre>:call SetBinaryRelease(&quot;Boilerplate&quot;)<br>    -| <strong>&lt;F7&gt; will run: /home/.../bin/Release/Boilerplate</strong></pre><pre><strong># Can now press F6 and F7 to run the executables</strong></pre><h3>Compiling and Linking a Dynamic Library</h3><p>This section will take a quick look at using CMake to compile a shared library and linking it with the main executable. The shared library will be compiled from a single C++ source file that defines some easing functions which can be used for animating object properties. For example, you could link to this library if you are creating a game engine that requires smooth easing animations. A corresponding header file will also be added for enabling other source files to reference functions contained in the shared library.</p><p>Our project will put the shared library source files in a nested directory under lib. This nested directory will have the same name as the library. In this demonstration, we will name the shared library interp for interpolation. With this knowledge, we can set up our new interp directory and download the source files from this guide’s repository using curl:</p><pre>$ cd lib &amp;&amp; mkdir interp &amp;&amp; cd interp</pre><pre>$ curl -L <a href="https://raw.githubusercontent.com/danebulat/vim-cmake-boilerplate/master/lib/interp/interpolate.h">https://raw.githubusercontent.com/danebulat/vim-cmake-boilerplate/master/lib/interp/interpolate.h</a> <strong>&gt; interpolate.h</strong></pre><pre>$ curl -L <a href="https://raw.githubusercontent.com/danebulat/vim-cmake-boilerplate/master/lib/interp/interpolate.cpp">https://raw.githubusercontent.com/danebulat/vim-cmake-boilerplate/master/lib/interp/interpolate.cpp</a> &gt; <strong>interpolate.cpp</strong></pre><p>Looking at the source code, our library defines nine functions which correspond to a particular easing function. I recommend referring to<a href="https://easings.net/"> this website</a> if you are interested in implementing other easing functions in your projects.</p><p>The next thing we need to do is create a CMakeLists.txt file inside the interp directory that calls the appropriate commands for compiling the interpolate.cpp source file into a shared library:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ac0ae593e2ee121d7b43cba385472dea/href">https://medium.com/media/ac0ae593e2ee121d7b43cba385472dea/href</a></iframe><p>The target_include_directories command exposes the interp directory’s source files to targets that link against the shared library. This enables CMake to locate interpolate.h when it is included in other source files.</p><p>From here, the top-level CMakeLists.txt file needs some modification. The BUILD_SHARED_LIBS CMake variable firstly needs to be turned on to build shared libraries by default. The add_subdirectory command must also be called to make CMake read the lib/interp/CMakeLists.txt file:</p><pre>   set(CMAKE_EXPORT_COMPILE_COMMANDS ON)</pre><pre><strong>&gt;&gt;</strong> option(BUILD_SHARED_LIBS &quot;Build using shared libraries&quot; ON)<br><strong>&gt;&gt;</strong> add_subdirectory(lib/interp)</pre><p>Moving forward, the target_link_libraries command is called after the executable is generated to link the interp shared library:</p><pre>   add_executable(Boilerplate <br>       &quot;${CMAKE_CURRENT_SOURCE_DIR}/src/main.cpp&quot;)</pre><pre><strong>&gt;&gt;</strong> target_link_libraries(Boilerplate interp)</pre><p>We also tell CMake to select the same compiler flags as the ones used to compile the main executable at the bottom of CMakeLists.txt:</p><pre>target_compile_options(interp PRIVATE<br>   &quot;$&lt;${gcc_like_cxx}:$&lt;BUILD_INTERFACE:${gcc_flags}&gt;&gt;&quot;<br>   &quot;$&lt;${msvc_cxx}: $&lt;BUILD_INTERFACE:${msvc_flags}&gt;&gt;&quot;)</pre><p>From here I recommend cleaning your bin and build directories before re-building the whole project:</p><pre><strong># Clean build and bin directories</strong><br>$ cd build &amp;&amp; rm -fr * &amp;&amp; mkdir Debug Release<br>$ cd ..<br>$ cd bin &amp;&amp; rm -fr * &amp;&amp; mkdir Debug Release<br>$ cd ..</pre><pre><strong># Build debug configuration and re-link compiler flags database</strong><br>$ cd build/Debug &amp;&amp; cmake -DCMAKE_BUILD_TYPE=Debug ../..<br>$ cmake --build .<br>$ cd ../..<br>$ rm compile_commands.json<br>$ ln -s build/Debug/compile_commands.json compile_commands.json</pre><pre><strong># Configure release configuration</strong><br>$ cd build/Release<br>$ cmake -DCMAKE_BUILD_TYPE=Release ../..<br>$ cd ../..</pre><p>With the build system set up and shared library generated, we can now access the shared library functions inside src/main.cpp:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/e929f5ab71784f024fd86a8391295519/href">https://medium.com/media/e929f5ab71784f024fd86a8391295519/href</a></iframe><p>Lastly, the ldd tool can be invoked on the command line to confirm that the executable links to the interp shared library:</p><pre>$ ldd bin/Debug/Boilerplate<br>    -| libinterp.so =&gt; <strong>&lt;path to shared library&gt;</strong></pre><h3>Final Tweaks</h3><p>This short section goes over how to download and configure a nice color scheme for Vim. The screen shots and GIFs presented in this guide show Vim rendering the <a href="https://github.com/sainnhe/edge">Edge color scheme</a>. Feel free to add its repository to your plug-in list in .vimrc and install it with the usual commands:</p><pre>Plugin &#39;sainnhe/edge&#39;<br>:source ~/.vimrc<br>:PluginInstall</pre><p>The next code snippet shows my Edge configuration. I have turned off italics and enabled true color rendering (16 million colors) if the terminal running Vim supports it. The bottom section sets the font to Hack, size 11 — make sure to specify a font that is installed on your system. The last line sets Edge as Vim’s default color scheme:</p><pre>let g:edge_style = &#39;aura&#39;<br>let g:edge_enable_italic = 0<br>let g:edge_disable_italic_comment = 1</pre><pre>if has(&#39;termguicolors&#39;)<br>    set termguicolors<br>endif</pre><pre>set guifont=Hack\ 11<br>set background=dark<br>colorscheme edge</pre><p>I also recommend checking out my powerline integration guide if you would like to render a fancy Vim status line:</p><p><a href="https://dane-bulat.medium.com/powerline-on-linux-an-integration-guide-c097831106f6">Powerline on Linux: An Integration Guide</a></p><h3>In Conclusion</h3><p>Vim is now configured to facilitate the development of large-scale C and C++ applications efficiently. A code completion engine is installed that provides IDE features that are fully customizable. In addition, the build system can grow naturally with the project by adding multiple CMakeList.txt files and calling the appropriate CMake commands.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=eb263c0a19a1" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The AWK Programming Language: An Introduction]]></title>
            <link>https://dane-bulat.medium.com/the-awk-programming-language-an-introduction-7035d343cd30?source=rss-24e03e26bd42------2</link>
            <guid isPermaLink="false">https://medium.com/p/7035d343cd30</guid>
            <category><![CDATA[linux-tutorial]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[computing]]></category>
            <category><![CDATA[programming-languages]]></category>
            <category><![CDATA[awk]]></category>
            <dc:creator><![CDATA[Dane Bulat]]></dc:creator>
            <pubDate>Sun, 06 Dec 2020 17:11:27 GMT</pubDate>
            <atom:updated>2020-12-08T13:21:31.336Z</atom:updated>
            <content:encoded><![CDATA[<h4>A hands-on tutorial for learning AWK on Linux</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PmFhIPNKUfFyJCX0YkEOXg.png" /></figure><h3>Introduction</h3><p>The AWK utility is known for running programs that perform text processing and data extraction, and is often pre-installed on Linux and UNIX based operating systems. This article adopts a hands-on approach to introduce the AWK programming language. Many sample programs are provided as new AWK concepts are discussed throughout. After completing this article, you will have a good understanding of how to write AWK programs to process, extract and report data on your system.</p><h4>A Brief History and Installation</h4><p>The first version of the AWK language was created at Bell Labs in 1977. The “AWK” acronym is derived from the surnames of its creators — Alfred <strong>A</strong>ho, Peter <strong>W</strong>einberger, and Brian <strong>K</strong>ernighan — and shares the same pronunciation as the auk bird. The lowercase version, “awk”, refers to the program that runs scripts written in the AWK programming language. This type of program is known as an interpreter.</p><p>Since its initial release, AWK has been forked and developed further by multiple groups of people. As a result, several versions of AWK exist today each with varying implementations. The programs presented in this article are written for the GNU Awk implementation, also known as <a href="https://www.gnu.org/software/gawk/">Gawk</a>.</p><p>Gawk is installed via the gawk package using your distribution’s package manager. It will probably be installed on your Linux distribution already and linked to the awk command. In other words, the awk and gawk commands will invoke the same program.</p><p>Use your distribution‘s package manager to install Gawk if it’s not already on your system:</p><pre>$ sudo pacman -S gawk       <strong># Arch</strong><br>$ sudo apt install gawk     <strong># Debian and Ubuntu</strong><br>$ sudo dnf install gawk     <strong># Fedora and CentOS</strong></pre><p>We can also confirm the install location and the fact awk is a symbolic link that points to the gawk binary:</p><pre><strong># Output gawk location</strong><br>$ which gawk<br>    -| /usr/bin/gawk</pre><pre><strong># Output awk symbolic link location</strong><br>$ which awk<br>    -| /usr/bin/awk</pre><pre><strong># Confirm awk is a symbolic link</strong><br>$ ls -l $(which awk)<br>    -| <strong>l</strong>rwxrwxrwx 1 root root 4 Apr 15  2020 /usr/bin/awk -&gt; <strong>gawk</strong></pre><p>Lastly, run awk with some basic command-line options to get acquainted with the tool:</p><pre><strong># Output version (--version)</strong><br>$ awk -V<br>    -| GNU Awk 5.1.0 ...</pre><pre><strong># Output license (--copyright)</strong><br>$ awk -C<br>    -| Copyright (C) 1989, 1991-2020 Free Software Foundation. ...</pre><pre><strong># Output usage (--help)</strong><br>$ awk -h<br>    -| Usage: gawk [POSIX or GNU style options] -f progfile ...</pre><p>Now you have Gawk installed, let’s proceed to the next section which discusses how to run AWK programs!</p><h3>Running AWK Programs</h3><p>An AWK program is data-driven, which means that it performs a set of actions on the textual data it receives. There are two methods for supplying data to a program: Specifying a list of filenames, and piping output from another command. These approaches can be observed with some pseudo code:</p><pre><strong># Providing input files</strong><br>awk &#39;program&#39; input-file1 input-file2</pre><pre><strong># Piping the output of a command</strong><br>command | awk &#39;program&#39;</pre><p>In addition, the program portion can be either:</p><ul><li>A one-shot throw away program that is quick to type and specified on the command line.</li><li>A script defining a sophisticated program that performs more complex data processing.</li></ul><p>The best approach to take will depend on your program’s overall complexity and whether you want to re-use it in the future. The following pseudo code illustrates how to provide an AWK program:</p><pre><strong># Command line</strong><br>awk &#39;program&#39; input-file1 input-file2 ...</pre><pre><strong># Program file</strong><br>awk -f program-file input-file1 input-file2 ...</pre><p>Notice that a program specified directly on the command line is surrounded in single quotes. Alternatively, you can provide a program file with the -f flag.</p><p>It is also a good idea to append the .awk file extension to your scripts for a couple of reasons. It enables editors to render syntax highlighting for the AWK programming language correctly, and it immediately conveys the nature of the file when it&#39;s listed in a file manager or terminal.</p><h4>Running Throw-Away Programs</h4><p>Here are some quick one-shot example programs invoked directly on the command line. Feel free to run each one in your terminal, and don’t worry if the syntax makes no sense at this point:</p><pre><strong># Output total number of fonts in /usr/share/fonts</strong><br>$ fc-list | awk &#39;END { print &quot;Total fonts:&quot;, NR }&#39;</pre><pre><strong># Output all (hidden) files starting with &quot;.bash&quot;</strong><br>$ ls -la | awk &#39;$NF ~ /^.bash/ { print $NF }&#39;</pre><pre><strong># Output all filenames that contain more than 10 characters <br># (minus file extension)</strong><br>$ ls | awk &#39;{ i=index($0,&quot;.&quot;); i&gt;0?s=substr($0,1,i-1):s=$0; <br>     if(length(s)&gt;10) print }&#39;</pre><p>Throw-away programs can also process text files:</p><pre><strong># Create a text file listing system fonts</strong><br>$ fc-list &gt; font-list</pre><pre><strong># Provide the text file and output total number of fonts</strong><br>$ awk &#39;END { print &quot;Total fonts:&quot;, NR}&#39; font-list</pre><p>Programs specified on the command line can be more than one line, but you will need to use the backslash character to separate each one:</p><pre>$ ls -l | awk &#39;BEGIN { print &quot;About to count files...&quot; } \<br>  { ++count } \<br>  END { print &quot;Total files: &quot;, count }&#39;</pre><p>These types of programs are great for getting information quickly and experimentation. However, writing programs in text files means there is no limit to the complexity or length of your program.</p><h4>Running AWK Scripts</h4><p>We will use a script provided in this article’s corresponding GitHub repository to discuss the concepts in this section. You can download the <a href="https://github.com/danebulat/awk-samples.git">GitHub repository</a> containing this article’s code examples using git on the command line:</p><pre>$ git clone <a href="https://github.com/danebulat/awk-samples.git">https://github.com/danebulat/awk-samples.git</a></pre><p>The program discussed in this section is located at samples/filemod.awk. The script renames files in the current working directory by converting all the characters to either lowercase or uppercase. When no command-line options are specified, all filenames are converted to lowercase:</p><pre><strong># Enter samples directory</strong><br>$ cd awk-samples/samples</pre><pre><strong># Add executable permissions to the scripts</strong><br>$ chmod +x *.awk</pre><pre><strong># Invoke script</strong><br>$ ls -la | ./filemod.awk</pre><p>After piping ls -la to ./filemode.awk you should see some formatted output informing you of any modifications to the filenames:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/849/1*xkmWvLsAcVtaSi1TpLii6A.gif" /><figcaption>Running the samples/filemod.awk program.</figcaption></figure><p>Let’s discuss the commands that were necessary to run the script:</p><ul><li>chmod +x *.awk<strong><br></strong>In order to use the ./program.awk shorthand syntax, the scripts must have executable permissions.</li><li>ls -la | ./filemod.awk<strong><br></strong>The program receives output from ls -la and disregards lines representing hidden files because dotfiles should not be renamed. The initial total line is also skipped.</li></ul><p>If you want to invoke programs using the ./program.awk syntax, a shebang character sequence must also be specified at the top of a script pointing to your AWK binary, followed by the -f flag:</p><pre>#! /usr/bin/awk -f</pre><p>Alternatively, the longer invocation method is to specify the awk utility and pass a script’s filename to the -f flag explicitly:</p><pre>$ ls -la | awk -f filemod.awk</pre><p>Running a script this way doesn’t require it to have executable permissions, nor does the shebang character sequence have to be specified at the top of the file. However, I would recommend using the shorthand syntax because less typing is required.</p><p>Our program also accepts a variable assignment on the command line which instructs it to perform either lowercase or uppercase conversion:</p><pre>$ ls -la | ./filemod.awk -v c=l    <strong># lowercase conversion</strong><br>$ ls -la | ./filemod.awk -v c=u    <strong># uppercase conversion</strong></pre><p>Variables passed to a program from the command line must appear after the -v option. If a value other than l or u is assigned to the c variable, or we don&#39;t assign any variables, the program converts to lowercase by default.</p><h3>Patterns and Actions</h3><p>An AWK program is made up of rules, and rules are made up of a <strong>pattern</strong> with associated <strong>actions</strong>. You therefore build an AWK program by defining a list of rules that have the following structure:</p><pre><strong># AWK program</strong><br>pattern { actions }  <strong># Rule</strong><br>pattern { actions }  <strong># Rule</strong></pre><p>Each rule can specify one pattern in addition to a set of actions. The following subsections will discuss this workflow in more detail.</p><h4>Patterns</h4><p>A rule doesn’t have to include any actions if a pattern is defined. Likewise, it doesn’t need to define a pattern if an action is defined. However, one of them is needed for the program to do some processing on the data it receives.</p><p>A pattern is specified before the action code, and can be either a conditional expression or regular expression:</p><pre><strong># Regular expression examples</strong><br>$ ls -l | awk &#39;/(\.py)&amp;/&#39;<br>$ ls -l | awk &#39;/README/&#39;</pre><pre><strong># Conditional expression examples</strong><br>$ ls -l | awk &#39;$NF == &quot;README.md&quot;&#39;<br>$ ls -la | awk &#39;$NF ~ /^\./ { printf $NF }&#39;</pre><p>Let’s take a look at each pattern in the examples above:</p><ul><li>/(\.awk)$/<strong> </strong><br>Forward slashes are used to denote regular expressions in AWK. Within them, you can input standard regular expression characters to define your pattern. In this example, all files with the .awk extension are matched.</li><li>/README/ <br>No special characters are used within this pattern, which means if an input record contains the text README, it will be matched and the corresponding action block will execute if one is defined.</li><li>$NF == &quot;README.md&quot;<strong><br></strong>A pattern can also be a conditional expression using the same operators found in C and C++. For example, this pattern checks the value inside the predefined AWK variable $NF, which references the last field in the current input record. If $NF contains the string README.md, the pattern matches and the associated action block is executed.</li><li>$NF ~ /^\./ <br>The regular expression /^\./ matches on strings that begin with a period character, representing a hidden file. The ~ operator performs a comparison on the string inside the $NF variable with the regular expression. Note that the !~ operator is also available.</li></ul><p>A pattern needs to return “true” in order for the action block to run. AWK understands truthness in the following way:</p><ul><li>True is defined as any nonzero numeric value, or any nonempty string.<br>I.e. &quot;Hello, world!&quot;, 10, “Three”, &quot;0&quot; are all true.</li><li>False is defined as zero or the null string.<br>I.e. &quot;&quot; and 0 are false.</li></ul><p>With the understanding that a rule is executed when its pattern matches the current input record, let’s proceed to discussing actions next.</p><h4>Actions</h4><p>AWK will always run the action block if a pattern is not provided. If a pattern is provided, it needs to return a true value in order for the action block to run.</p><p>An action is always defined inside curly braces {...} and consists of one or more statements. Either a semi-colon (;) or a newline separate statements in AWK. A one-shot program will use semi-colons to separate statements:</p><pre>$ ls -l | awk &#39;{ print &quot;Filename:&quot;, $NF; print &quot;\tOwner:&quot;, \<br>        ($3 == &quot;&quot; ? &quot;--&quot; : $3) }&#39;</pre><p>However, a script will likely separate statements on separate lines to properly format the code, making it more readable:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/883aab21b221c8e99931e418be8f87f1/href">https://medium.com/media/883aab21b221c8e99931e418be8f87f1/href</a></iframe><p>In the script above, the print function is called to format and send data to the standard output. This program is saved as samples/statements.awk in the source files and can be invoked on the command-line:</p><pre>$ ls -l | ./statements.awk</pre><p>In the event that an action is not specified for a rule, the whole input record is sent to standard output where the following action code is run implicitly:</p><pre>{ print $0 }</pre><p>The $0 here is another predefined variable which stores the string of the current input record. If you are wondering why you are seeing output after not defining any actions in your AWK programs, this is what&#39;s going on behind the scenes.</p><p>Regarding the sample/filemod.awk program, three rules are defined where two of them use special patterns:</p><ul><li>BEGIN { ... }<br>This rule uses the special BEGIN pattern. It causes the rule to execute exactly once when the program starts, before the first input record is read.</li><li>$NF !~ /^\./ &amp;&amp; $1 !~ /^(total)/ { ... }<br>This rule processes the input record providing it doesn&#39;t represent a hidden file or begins with the string total. Two regular expression comparisons are performed with the !~ operator to accomplish this.</li><li>END { ... }<br>This rule uses the special END pattern. It causes the rule to execute exactly once after all input is read and before AWK exists.</li></ul><p>Let’s take a closer look at the BEGIN and END patterns in the next section.</p><h4>BEGIN and END Patterns</h4><p>The BEGIN and END patterns are useful for providing startup and cleanup actions for your program. For example, a BEGIN block could print headings if a program renders table data, and an END block could output summary information. The samples/begin.awk program demonstrates the BEGIN and END patterns:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/79cbe3f2830a91925e9203a10e7944f3/href">https://medium.com/media/79cbe3f2830a91925e9203a10e7944f3/href</a></iframe><p>This script renders Filename and Permission column headings inside the rule containing the BEGIN pattern. Notice that the printf function is used for making the first column a minimum of twenty characters wide.</p><p>Additionally, the rule containing the END pattern outputs the value of the $NR predefined variable, which stores the number of input records that have been processed so far. Feel free to run the script in the following way and observe its output:</p><pre>$ ls -l | ./begin.awk</pre><h3>Input Records and Fields</h3><h4>Records</h4><p>AWK processes data in units called records. Each record is separated by the newline character (\n) — meaning a record is one line of input by default. Additionally, AWK stores the current input record in a predefined variable called $0, which can be referenced in your programs:</p><pre><strong># Output input records</strong><br>$ ls -l | awk &#39;{ print &quot;Current record:&quot; $0 }&#39;</pre><pre><strong># Same behavior with text files</strong><br>$ ls -l &gt; file-list<br>$ awk &#39;{ print &quot;Current record:&quot;, $0 }&#39; file-list</pre><p>The character used for separating records is stored in a predefined variable called the <strong>record separator</strong> denoted by RS in your programs. This variable can be modified in situations where input needs to be separated differently:</p><pre><strong># Treat each directory as an input record</strong><br>$ pwd | awk &#39;BEGIN { RS=&quot;/&quot; } { print &quot;--&gt;&quot;, $0 }&#39;</pre><p>Notice that the record separator character is not actually included in the input record. Another three useful predefined variables are available in regards to record processing:</p><ul><li>FILENAME <br>Contains the name of the current input file.</li><li>FNR (<strong>F</strong>ile <strong>N</strong>umber of <strong>R</strong>ecords) <br>How many records have been read from the current input file.</li><li>NR : (<strong>N</strong>umber of <strong>R</strong>ecords) <br>How many records have been read in total.</li></ul><p>The samples/records.awk program uses these variables to output the first ten records within the text files passed to the program:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/16e233fd5b05fadfb16d3bbe4fa95823/href">https://medium.com/media/16e233fd5b05fadfb16d3bbe4fa95823/href</a></iframe><p>The program checks if FNR contains the value 10 after outputting the current record. If it does, a message is displayed and the nextfile function is called to process the next input file. To invoke the program, simply pass some input files on your system such as /etc/passwd and /etc/group:</p><pre>$ ./records.awk /etc/passwd /etc/group<br>    -| Outputs first 10 records in /etc/passwd and /etc/group</pre><h4>Fields</h4><p>Each record is automatically split into chunks called fields. Similar to records, fields are separated by the value of a predefined variable called the <strong>field separator</strong> denoted in your programs with FS.</p><p>The default value of FS is a space character, which carries the meaning that fields are separated by <strong>runs of white space</strong> - such as leading and trailing space and TAB characters.</p><p>Field separation can be observed by outputting the value of the NF, which is a predefined variable that stores the number of fields each record contains:</p><pre>$ ls -l | awk &#39;{ print &quot;Number of fields in record:&quot;, NF }&#39;<br>    -| Number of fields in record: 2<br>       Number of fields in record: 9<br>       ...</pre><p>Individual fields can also be accessed with special variables in the form of $N where N is the field index. The first field in a record has an index of one, and is therefore accessed with $1. The second field is referenced with $2, and so on. The last field can also be accessed with the $NF variable, which is useful if you don&#39;t know the actual index value. Let&#39;s take a look at an example:</p><pre><strong># Output filenames and modification dates</strong><br>$ ls -l | awk &#39;{ if (NR&gt;1) print $NF, &quot;\tmodified on&quot;, $6, $7 }&#39;</pre><p>The field separator will sometimes need modifying if data received by the program is formatted differently. For example, fields inside the /etc/passwd and /etc/group configuration files are separated with the colon (:) character. To access these fields correctly within an AWK program, the field separator needs to be set accordingly:</p><pre><strong># Output system users and their home directory</strong><br>$ awk &#39;BEGIN { FS=&quot;:&quot; } { printf &quot;%-30s%s\n&quot;, $1, $6 }&#39; /etc/passwd</pre><p>The printf function is used again here to format columns appropriately. As well as within the BEGIN block, FS can also be set on the command line through the -F option:</p><pre>$ awk -F: &#39;{ printf &quot;%-30s%s\n&quot;, $1, $6 }&#39; /etc/passwd</pre><p>As with the record separator, It is possible to assign a regular expression to the field separator where fields are separated by occurrences of characters that match the regular expression. This is an advanced feature but mentioned here for completeness.</p><h4>Record and Field Splitting Summary</h4><p>The following two tables summarize the behavior of the RS (Record Separator) and FS (Field Separator) variables depending on their assigned value. Feel free to refer to them if you need a refresher.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/2bbefe8634d55e8acac6c1cb93c6c19a/href">https://medium.com/media/2bbefe8634d55e8acac6c1cb93c6c19a/href</a></iframe><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/326d920f6159094234d46fa820e7669f/href">https://medium.com/media/326d920f6159094234d46fa820e7669f/href</a></iframe><h3>Printing Output</h3><p>One of the strengths of the AWK programming language is its ability to efficiently format data and produce meaningful reports. This functionality is provided by the print() and printf() functions in combination with a few predefined variables.</p><h4>The print Function</h4><p>The print() function is used to output text with simple, standardized formatting. You will probably use this function if you just want to output some variables and strings in a linear way. The syntax is also very simple:</p><pre><strong># Items separated with commas</strong><br>print item1, item2, ...<br>print(item1, item2, ...)</pre><pre><strong># Items separated without commas</strong><br>print item1 item2 item3<br>print(item1 item2 item3)</pre><p>The syntax presented above illustrates a couple of important points:</p><ul><li>You can provide optional parenthesis when calling print. Parenthesis are only required if redirection is performed with the &gt; operator.</li><li>Items can be separated with or without a comma character.</li></ul><p>While separating items with and without a comma are both syntactically correct, AWK will process your arguments differently depending on which method you choose. let’s take a look at an example:</p><pre><strong># With commas</strong><br>$ awk &#39;BEGIN { print &quot;Hello&quot;, &quot;World&quot; }&#39;<br>    -| Hello World</pre><pre><strong># Without commas</strong><br>$ awk &#39;BEGIN { print &quot;Hello&quot; &quot;World&quot; }&#39;<br>    -| HelloWorld</pre><p>From the output of these commands we can deduce that items are separated with a space character if a comma is used, and simply concatenated if a comma is not used.</p><p>As a matter of fact, you don’t even need to separate items if you just want to concatenate them:</p><pre>$ awk &#39;BEGIN { print &quot;Hello&quot;&quot;World&quot; }&#39;</pre><p>Although this works, I would recommend separating items with a space to make your code more readable!</p><p>The fact that a space is output between items when a comma is specified in print is no coincidence. The space character is actually the default value of a predefined variable called the <strong>output field separator</strong> denoted by OFS in your programs. This variable can be set to any string if you require something other than a space to separate your items:</p><pre><strong># Setting the output field separator</strong><br>$ awk &#39;BEGIN { OFS=&quot; ---&gt; &quot;; print &quot;Hello&quot;, &quot;AWK&quot;, &quot;World&quot;}&#39;<br>    -| Hello ---&gt; AWK ---&gt; World</pre><p>AWK also provides a predefined variable called the <strong>output record string</strong> which is denoted by ORS in your programs. The string stored in this variable is a newline character by default (\n) and is output before the print function exits. The ORS variable can be set to any string if you require something other than a newline character to separate print output:</p><pre><strong># The default ORS is a newline character</strong><br>$ awk &#39;BEGIN { print &quot;Hello&quot;; print &quot;World&quot; }&#39;<br>    |- Hello<br>       World</pre><pre><strong># Setting ORS to a new string</strong><br>$ awk &#39;BEGIN {ORS=&quot; [END OF RECORD]\n&quot;;print &quot;Hello&quot;;print &quot;World&quot;}&#39;<br>    |- Hello [END OF RECORD] <br>       World [END OF RECORD]</pre><p>The last predefined variable we will look at in this section is called the <strong>output format</strong> and is denoted by OFMT in your programs. It stores a string representing a format specifier that controls how numeric output is displayed with the print function. The default value is %.6g:</p><ul><li>%g : Prints a number in either scientific notation or in floating-point notation - which ever uses the fewest characters.</li><li>.6 : A period followed by an integer representing the precision. Signifies the maximum amount of significant digits when used with %g format specifier.</li></ul><p>The OFMT variable can be modified just like ORS and OFS:</p><pre><strong># Default OFMT is &quot;%.6g&quot;</strong><br>$ awk &#39;BEGIN { print 3.141592653, 6.283185306, 12.566370612 }&#39;<br>    -| 3.14159 6.28319 12.5664</pre><pre><strong># Setting OFMT to a new format</strong><br>$ awk &#39;BEGIN { OFMT=&quot;%6.3f&quot;; print 3.141592653, 6.283185306, 12.566370612 }&#39;<br>    -| 3.142 6.283 12.566</pre><p>We set the output format to %4.3f which specifies that:</p><ul><li>A minimum of four digits are output for every number.</li><li>A maximum of three digits are output after a decimal point.</li></ul><p>The recommended practice is to set OFS, ORS and OFMT in the BEGIN block before your input records start processing.</p><h4>The printf Function</h4><p>The printf function is a powerful tool that enables you to format output to match your exact requirements. For example, it is straight forward to output columns of certain widths, as well as formatting a number’s precision.</p><p>The syntax of printf is similar to print, except the first argument is always a format string:</p><pre>printf format, item1, item2, ...<br>printf(format, item1, item2, ...)</pre><p>The format string is a specification of what is to be output by printf. Within it you include <strong>format specifiers</strong> as well as regular text. The subsequent items after the format string are what will be plugged into the format specifiers.</p><p>Let’s take a look at the samples/prinft.awk example program in the GitHub repository:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/aa3db62bb07922e3136d5effd436a920/href">https://medium.com/media/aa3db62bb07922e3136d5effd436a920/href</a></iframe><p>This program is responsible for displaying a list of files and their permissions in a readable way. Invoke it on the command-line by piping the output of an ls -l command:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/849/1*-mxr--znklfV7YCN8fXgGg.gif" /><figcaption>Running the samples/print.awk program.</figcaption></figure><p>The printf function is used to format the output in this program.</p><ul><li>format = &quot;%20-s%s\n&quot;<strong><br></strong>A variable called format caches a format string so we can pass it to multiple invocations of printf. The format string itself contains two %s format specifiers. The first one includes a modifier, 20-, which specifies a minimum width of twenty characters for the string.</li><li>printf format, &quot;Filename&quot;, &quot;Permissions&quot;<strong><br> </strong>printf format, &quot;--------&quot;, &quot;-----------&quot;<br>These calls to printf are executed once in the BEGIN block to output column headings. They receive the format variable in addition to two regular strings for its items.</li><li>printf format, $NF, $1<strong><br></strong>This printf is invoked for each input record the program receives. We provide it the format variable so the column data lines up with the headings appropriately. We also specify $NF and $1 for the items which refer to the filename and permission strings in the input record.</li><li>printf &quot;\n\tFile count: %i\n&quot;, count<strong><br></strong>We call one more printf in the END block and pass it a count variable, which stores the total number of records the program output. The %i format specifier renders count as an integer.</li></ul><p>Notice that printf doesn&#39;t append a newline character to the end of its output; it is therefore necessary to provide one explicitly.</p><p>In additional to %s and %i, many more format specifiers are available to use with printf. A format specifier always starts with the percent character (%) and ends with a <strong>format-control letter</strong> - the most useful of which are listed here:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cdd1b59182bb0be6e4a889af9079df81/href">https://medium.com/media/cdd1b59182bb0be6e4a889af9079df81/href</a></iframe><p>Modifiers are also specified in between a format specifier’s % character and format-control letter. They can be used for things like controlling number precision, padding and widths. Modifiers are completely optional, but I’ll provide a list of some portable ones here for completeness:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d98c64e1423420016e82a28671278f21/href">https://medium.com/media/d98c64e1423420016e82a28671278f21/href</a></iframe><h3>Built-In Functions</h3><p>The AWK programming language provides many built-in functions that can be called in your programs. Built-in functions fall into three categories: numeric, string, and I/O. For example, the samples/filemod.awk program uses the following functions:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1b2af8f5e438e5bc93938c0443114634/href">https://medium.com/media/1b2af8f5e438e5bc93938c0443114634/href</a></iframe><p>Of particular interest here is the system(command) function, which enables us to pass a string representing a shell command that is subsequently executed. The filemod.awk program utilizes this function to invoke a mv command in order to rename files:</p><pre>command = sprintf(&quot;mv %s %s\n&quot;, filename, target)<br>system(command)</pre><p>The sprintf function firstly constructs and returns the command string before it is passed to system(). As you can imagine, the ability to dynamically construct arbitrary commands and execute them within a program is a powerful feature.</p><p>It is important to not leave a space between the function name and its parenthesis, as doing this could confuse the AWK interpreter depending on where the function is being called. I encourage you to check out the <a href="https://www.gnu.org/software/gawk/manual/gawk.html#Functions">AWK programming manual</a> to discover the many functions available to use in your programs.</p><h3>In Conclusion</h3><p>I hope that the content presented in this article has sparked an interest in the AWK programming language. With the fundamental concepts of how AWK works under your belt, I’m sure it will become a useful utility in your Linux toolbox for analyzing and filtering data on your system.</p><p>Keep an eye out for more AWK related articles coming from myself in the future. I also recommend a couple of resources to continue your learning:</p><ul><li><a href="https://www.gnu.org/software/gawk/manual/gawk.html#Index">The GNU Awk User’s Guide</a>: A thorough overview of Gawk and the AWK programming language.</li><li><a href="http://awklang.org/">awklang.org</a>: Τhe site for things related to the awk language</li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7035d343cd30" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Powerline on Linux: An Integration Guide]]></title>
            <link>https://dane-bulat.medium.com/powerline-on-linux-an-integration-guide-c097831106f6?source=rss-24e03e26bd42------2</link>
            <guid isPermaLink="false">https://medium.com/p/c097831106f6</guid>
            <category><![CDATA[vim]]></category>
            <category><![CDATA[bash]]></category>
            <category><![CDATA[computing]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[linux-tutorial]]></category>
            <dc:creator><![CDATA[Dane Bulat]]></dc:creator>
            <pubDate>Fri, 27 Nov 2020 13:21:08 GMT</pubDate>
            <atom:updated>2020-12-09T02:35:11.463Z</atom:updated>
            <content:encoded><![CDATA[<h4>Render clean status lines and prompts in Vim, Bash, i3 and tmux</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GX2sf-adN0xNni6zNzPhng.png" /></figure><h3>Introduction</h3><p>Productivity tools on Linux often include both a status line and a prompt. The function of a status line is to display important information relevant to the program’s current context; and a prompt identifies where a program is expecting some input from the user. Some good application examples that utilize these features include the Bash shell, i3 window manager and Vim.</p><p><a href="https://github.com/powerline/powerline">Powerline</a> is a program that renders status lines and prompts in a consistent way across multiple applications. The powerline configuration files are modified with JSON to set colors, symbols, and even control the type of content that is rendered on an application’s status line or prompt. For example, we might want the shell prompt to output the absolute path of the current working directory, or have the window manager’s status bar display system information such as CPU loads and consumed memory. Powerline can be configured to accomplish both of these tasks.</p><p>This article will detail how to install powerline on Linux along with its dependencies. Subsequent sections will then go through how to integrate powerline with the <a href="https://www.vim.org/">Vim</a> text editor, <a href="https://www.gnu.org/software/bash/">Bash</a> shell, <a href="https://i3wm.org/">i3 window manager,</a> and <a href="https://github.com/tmux">tmux</a> program. Each section will build upon the last by gradually exposing how powerline works behind the scenes and how to configure it effectively. After completing this guide you will be able to create your own layouts and color schemes with powerline for rendering your favorite prompts and status lines on Linux.</p><h4>System Dependencies</h4><p>Powerline is written in the <a href="https://www.python.org/">Python</a> programming language, which means that your system will need to have a recent version of Python installed to run it. This guide recommends having the latest version of Python installed to run powerline. Use your Linux distribution’s package manager to install Python if it is not already on your system:</p><pre>$ sudo pacman -S python      <strong># Arch</strong><br>$ sudo apt install python    <strong># Debian and Ubuntu</strong><br>$ sudo dnf install python    <strong># Fedora and CentOS</strong></pre><p>It is also a good idea to install the git package so powerline can detect and render information about the git repositories on your file system:</p><pre>$ sudo pacman -S git         <strong># Arch</strong><br>$ sudo apt install git       <strong># Debian and Ubuntu</strong><br>$ sudo dnf install git       <strong># Fedora and CentOS</strong></pre><p>Keep in mind that we will need to install a couple of Python packages later to enable rendering of system information, as well as integrating powerline with the i3 window manager. We will install these modules via <a href="https://pypi.org/project/pip/">pip</a> — Python’s very own package manager — which we’ll go through how to set up next.</p><h3>Installation</h3><p>This section will detail how to install powerline with the pip package manager. The installation instructions presented in this section will work on any Linux distribution.</p><p>You may also have seen a powerline package in your distribution’s official repositories. The problem with these packages is that they often include extra scripts that get invoked automatically when launching certain applications. For example, the official <a href="https://www.archlinux.org/packages/?name=powerline">powerline package</a> for Arch Linux installs a global powerline.vim script that Vim automatically loads when it is launched. This will result in a conflict If you have another status line plug-in enabled such as vim-airline. With this in mind, I recommend installing powerline transparently via pip as the first course of action.</p><h4>Installing Pip</h4><p>To set up pip we firstly need to download its installation script called get-pip.py from a remote server. Let’s download this script via curl in an appropriate location:</p><pre>$ cd ~/Downloads<br>$ curl <a href="https://bootstrap.pypa.io/get-pip.py">https://bootstrap.pypa.io/get-pip.py</a> -o get-pip.py</pre><p>Then run get-pip.py via the Python interpreter to install pip on your system:</p><pre>$ python3.8 get-pip.py</pre><pre><strong># Output</strong><br>Successfully installed pip-20.2.4 setuptools-50.3.2 wheel-0.35.1</pre><p>An executable file called pip is created on your file system which will probably be located at ~/.local/bin/pip. Your PATH environment variable may need to be updated if pip cannot be executed directly from the command-line. Let’s firstly confirm where pip is installed with a find command:</p><pre>$ sudo find / -name pip</pre><pre><strong># Output</strong><br>/home/dane/.local/bin/pip</pre><p>The output above confirms that pip is installed locally in the home folder’s .local/bin directory. If necessary, append your location directory to the PATH variable in ~/.bashrc:</p><pre>$ echo $PATH<br>$ vim ~/.bashrc</pre><pre><strong># In bash.rc</strong><br>export PATH=&quot;~/.local/bin:$PATH&quot;</pre><p>Alternatively, If your system has multiple Python installations with each one containing a different version of pip, you could create an alias that directly points to the pip executable you wish to use:</p><pre><strong># Provide an absolute path</strong><br>alias pip=”/home/dane/.local/bin/pip”</pre><p>Lastly, be sure to refresh the bash configuration and confirm the pip command works by checking its version:</p><pre>$ source ~/.bashrc<br>$ pip --version</pre><pre><strong># Output</strong><br>pip 20.2.4 from /home/dane/.local/lib/python3.8/site-packages/pip (python 3.8)</pre><pre><strong># Display pip help<br></strong>$ pip -h</pre><p>We will use pip to download and install the official powerline package in addition to a couple of other important python modules in the next section.</p><h4>Installing Powerline</h4><p>The <a href="https://pypi.org/project/powerline-status/">official powerline package</a> in the Python Package index is called powerline-status. Be careful not to confuse it with the powerline package — which is completely unrelated to the plug-in that we are discussing in this guide!</p><p>Install powerline via pip by running the following command:</p><pre>$ pip install powerline-status</pre><p>Then confirm its install location on your file system using pip’s show command:</p><pre>$ pip show powerline-status</pre><pre><strong># Output</strong><br>...<br>Location: /home/dane/.local/lib/python3.8/site-packages</pre><p>You can issue an uninstall command If you ever want to remove powerline from your system altogether:</p><pre>$ pip uninstall powerline-status</pre><h4>Python Packages</h4><p>Moving forward, two Python packages should also be installed to give powerline some extra functionality:</p><ul><li><strong>psutil</strong>: Required for segments that display system properties such as CPU load percentage and network loads.</li><li><strong>i3ipc</strong>: Required for binding the powerline program with the i3 window manager and handling i3 segments.</li></ul><p>I recommend installing the i3ipc package even if you are not an i3 user because it might come in handy at a later date. Invoke pip to Install these packages:</p><pre>$ pip install psutil i3ipc</pre><pre><strong># Installs to:</strong><br><strong># ~/.local/lib/python3.8/site-packages/psutil</strong><br><strong># ~/.local/lib/python3.8/site-packages/i3ipc</strong></pre><h4>Font Installation</h4><p>The powerline plug-in uses several special glyphs to do things like render its arrow effect. We therefore need to acquire some fonts that include these special symbols. There are two ways of doing this:</p><ul><li>Download and install the PowerlineSymbols.otf fallback font by following the instructions on <a href="https://powerline.readthedocs.io/en/master/installation/linux.html?highlight=PowerlineSymbols.otf#fonts-installation">this page</a>. This font will be used when powerline needs to render a special glyph symbol which the primary font is missing.</li><li>Install a range of patched fonts that are able to render all of powerline’s special glyphs.</li></ul><p>This guide recommends the second option which involves downloading th<a href="https://github.com/powerline/fonts">e powerline fonts</a> git repository and installing its collection of patched fonts. Feel free to browse through the repository’s GitHub page to discover which patched fonts are included along with their corresponding license. You will notice that most patched fonts inherit the same name as the un-patched version, but have for Powerline appended to the end. For example, the patched Source Code Pro font is named Source Code Pro for Powerline.</p><p>Let’s go ahead and download the repository in an appropriate location and run the install.sh script in its root directory. The patched fonts will be copied into your local fonts directory (the place where a user’s fonts are stored) and the font cache will update:</p><pre>$ git clone <a href="https://github.com/powerline/fonts.git">https://github.com/powerline/fonts.git</a> --depth=1 <br>  powerline-fonts</pre><pre>$ cd powerline-fonts<br>$ ./install.sh</pre><pre><strong># Output</strong><br>Copying fonts...<br>Resetting font cache, this may take a moment...<br>Powerline fonts installed to /home/dane/.local/share/fonts</pre><pre><strong># List fonts directory to confirm the copy was successful</strong><br>$ ls ~/.local/share/fonts</pre><p>An uninstall.sh script is also included in the repository’s root directory. You can run this script to remove the patched fonts from your system.</p><p>From here you can open up any application — such as your terminal emulator — and select one of the patched fonts to use for that program. When it comes to choosing a font for powerline, I recommend firstly previewing them in an application called <a href="https://aur.archlinux.org/packages/font-manager/">font-manager</a>:</p><pre>$ yay -S font-manager              <strong># Arch</strong><br>$ sudo apt install font-manager    <strong># Debian and Ubuntu</strong><br>$ sudo dnf install font-manager    <strong># Fedora</strong></pre><pre><strong># Run</strong><br>$ font-manager</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*5-cpuygqadwNU3b2QsJBAA.gif" /><figcaption>Previewing powerline fonts with the Font Manager application.</figcaption></figure><h3>Using Powerline with Vim</h3><p>This section will detail how to enable powerline inside the Vim text editor. The goal for this section is to simply see powerline in action and get a feel for how it changes the overall experience of using an application.</p><p>Your version of Vim must have been compiled with Python in order for powerline to work. Run the following command to check the Python versions your Vim installation supports:</p><pre>$ vim — version | grep +python</pre><pre><strong># Output<br></strong>+python/dyn +python3/dyn</pre><p>You are good to go if you see a line containing either +python or +python3 — meaning that Vim supports Python versions 2 and 3 respectively.</p><p>At this point, all we need to do is add some code to .vimrc to enable powerline:</p><pre>set laststatus=2           <strong>&quot; Always display the status bar</strong><br>set powerline_cmd=&quot;py3&quot;    <strong>&quot; Tell powerline to use Python 3</strong></pre><pre>python3 from powerline.vim import setup as powerline_setup<br>python3 powerline_setup()<br>python3 del powerline_setup</pre><p>No other code is necessary to get powerline working inside of Vim. This means that if you are using a Vim plug-in manager such as Vundle and have a :Plugin directive that references another installation of powerline, that line must be removed. To disable powerline in Vim, simply comment out the code above by preceding each line with a double-quote (&quot;).</p><p>The status line and file tabs will now be rendered via powerline the next time you open Vim. If you switch between the different modes — such as Insert, Visual and Normal — you’ll notice the status line text and colors change appropriately. I encourage you to have a play with some files and observe how the status line changes before proceeding to the next section.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*QvLFEJmb86-ntyetWl8zTg.gif" /><figcaption>The Vim status line being rendered with powerline.</figcaption></figure><h3>Using Powerline with Bash</h3><p>This section will take a look at using powerline to render a custom Bash prompt. Its subsections will detail:</p><ul><li>Modifying .bashrc to initialize powerline and its environment.</li><li>Making a local copy of powerline’s default configuration files for further editing.</li><li>The responsibility of the various configuration files and how they are structured on the file system.</li><li>Creating a theme for the Bash prompt.</li><li>Creating a color scheme for the Bash prompt.</li></ul><p>After completing this section you will be familiar with the “powerline workflow” and will have acquired enough knowledge to configure powerline to work with other applications.</p><h4>Modifying .bashrc</h4><p>An application must invoke a particular binding script when it is launched to enable powerline rendering. Several bindings are shipped with the powerline installation and are stored in the powerline/bindings/ directory. As an example, the bash/powerline.sh script is the binding for bash and must be invoked when bash is launched.</p><p>With this in mind, add the upcoming code to your .bashrc file to initialize some necessary environment variables and invoke the corresponding binding script:</p><pre><strong># Run powerline daemon<br># To locate this file, run: sudo find / -name powerline-deamon</strong><br>/home/dane/.local/bin/powerline-daemon -q</pre><pre><strong># Cache powerline location<br># To locate this directory, run: pip show powerline-status</strong><br>export POWERLINE_LOCATION=&quot;/home/dane/.local/lib/python3.8/site-packages/powerline&quot;</pre><pre><strong># Set up powerline environment</strong><br>export POWERLINE_CONFIG_COMMAND=powerline-config<br>export POWERLINE_BASH_CONTINUATION=1<br>export POWERLINE_BASH_SELECT=1</pre><pre><strong># Invoke binding script</strong><br>. &quot;$POWERLINE_LOCATION/bindings/bash/powerline.sh&quot;</pre><ul><li>/home/dane/.local/bin/powerline-daemon -q<br>We firstly ensure that the powerline daemon is running so it automatically handles requests that are sent from applications rendering with powerline in the background. The daemon process dramatically improves the performance of powerline and should always be running.</li><li>export POWERLINE_LOCATION=&quot;/home/dane/.local/lib/python3.8/site-packages/powerline&quot;<br>An environment variable called POWERLINE_LOCATION is defined to cache the powerline installation directory for convenience. Change this value to your installation path accordingly.</li><li>export POWERLINE_CONFIG_COMMAND=powerline-config<br>export POWERLINE_BASH_CONTINUATION=1<br>export POWERLINE_BASH_SELECT=1<br>Three environment variables are then initialized with recommended values to ensure correct rendering intervals. These values are queried in the bash binding script.</li><li>. &quot;$POWERLINE_LOCATION/bindings/bash/powerline.sh&quot;<br>Invokes the bash binding script. After running this command, powerline will now render your bash prompt.</li></ul><p>When we start modifying configuration files, the powerline daemon process needs to be killed and restarted in order to see changes reflected in the terminal. One way to do this is by running these two commands:</p><pre><strong># Kill the powerline daemon process</strong><br>$ powerline-daemon -k</pre><pre><strong># Start the powerline daemon process quietly</strong><br>$ powerline-daemon -q</pre><p>That’s a lot of typing for something that needs to be done often. One way to go about doing this in a less tedious and error-prone way is to create an alias in .bashrc:</p><pre><strong># in .bashrc</strong><br>alias pd=”powerline-daemon -k &amp;&amp; powerline-daemon -q”</pre><pre><strong># Refresh configuration on the command line</strong><br>$ source ~/.bashrc</pre><p>Now the powerline deamon is restarted whenever we run pd on the command line:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*aE68na5cLBNcJbrNFb-fAQ.gif" /><figcaption>Restarting the powerline daemon with an alias.</figcaption></figure><h4>Setting Up a Configuration Directory</h4><p>The powerline installation contains a directory called config_files which, as you may have guessed, contains a complete set of default configuration files. When a powerline enabled application is launched, the appropriate configuration files are read to render the status line and/or prompt correctly. We modify a set of configuration files to change the behavior and appearance of elements to be rendered.</p><p>The official powerline documentation recommends copying the contents of config_files into your own configuration directory at ~/.config/powerline. This enables us to freely modify and add new configuration files whilst leaving the original installation untouched. Now, if an error occurs somewhere within our configuration that cannot be fixed, we can always revert back to the default configuration by copying over the files again.</p><p>Let’s set up a local configuration directory and copy over the default set of configuration files that ship with powerline:</p><pre>$ mkdir ~/.config/powerline<br>$ cp -fr {powerline_location}/config_files ~/.config/powerline</pre><p>Make sure to replace {powerline_location} with the appropriate path for your installation. Run pip show powerline-status and refer to the Location key if you are unsure about what path to use.</p><p>The next section will take a look at specific configuration files and how they are structured on the file system.</p><h4>The Configuration Directory Structure</h4><p>It is important to understand how the configuration directory is structured as well as the main files that require modification when configuring powerline’s behavior. Take note that all of the configuration files are formatted in JSON.</p><p>The powerline configuration files are organized in a certain way:</p><pre><strong>- powerline/</strong><br>    - config.json           <strong># Main configuration</strong><br>    - colors.json           <strong># Colors and gradient definitions<br></strong>    <strong>- colorschemes/</strong>         <strong># Directory for all color schemes</strong><br>        - default.json<br>        <strong>- &lt;extension&gt;/</strong><br>            - &lt;scheme_name&gt;.json<br>            ...<br>        ...<br>    <strong>- themes/</strong>               <strong># Directory for all themes</strong><br>        - &lt;top_level&gt;.json<br>        <strong>- &lt;extension&gt;/</strong><br>            - &lt;theme_name&gt;.json<br>            ...<br>        ...</pre><ul><li>powerline/config.json<br>The main powerline configuration file which specifies the theme and colorscheme for each powerline-supported application. Themes and color schemes can be hooked up to an application by modifying this file.</li><li>powerline/colors.json<strong><br> </strong>Colors and gradients are defined in this file and are referenced within color scheme configuration files elsewhere in the directory structure. We will define some colors and use them in a color scheme configuration file for theming the bash prompt.</li><li>powerline/colorschemes/<strong><br></strong>All color schemes are saved under this directory. It contains a default.json file that powerlne will fallback to if an application is hooked up to an incomplete color scheme. Color schemes for particular applications are saved in the appropriate &lt;extension&gt; directory. For example, Bash color schemes must be stored in colorschemes/shell/. Browse this directory to see which applications powerline provides color schemes for out of the box.</li><li>powerline/themes/<strong><br></strong>All themes are saved under this directory. The term <em>“theme”</em> in powerline refers to the layout of a prompt or status line, and has nothing to do with color. A theme specifies a list of segments, which are regular Python functions that return string data to be rendered. Powerline provides <a href="https://powerline.readthedocs.io/en/master/configuration/segments.html">many segments</a> that output specific information — such as the date and CPU usage. Themes for particular applications are saved in the appropriate &lt;extension&gt; directory, and a default.json file is usually provided as a base configuration.</li></ul><h4>Creating a Theme for the Bash Prompt</h4><p>Segments are regular Python functions that return some data to be rendered in a prompt or status line. Powerline ships with many segments which are all defined within Python modules. These Modules are stored in the powerline/segments/ directory. We call a segment using a specific syntax in a theme configuration file:</p><pre>&quot;function&quot;: &quot;powerline.segments.module.segment_name&quot;</pre><p>Segments also accept a range of arguments which customize their final output. Every argument is assigned a default value in the segment’s signature, but we will sometimes want to override them. This is done by defining a JSON list called args within the corresponding segment block:</p><pre>&quot;args&quot;: {<br>    &quot;arg1&quot;: &quot;val1&quot;,<br>    &quot;arg2&quot;: &quot;val2&quot;, ...<br>}</pre><p>As an example, the cwd segment is used in our bash theme to render the current working directory in a specifc way:</p><pre>{<br>  &quot;function&quot;: &quot;powerline.segments.common.env.cwd&quot;,<br>  &quot;args&quot;: {<br>    &quot;ellipsis&quot;: &quot;..&quot;,      <strong># String to use for omitted directories</strong> <br>    &quot;dir_shorten_len&quot;: 20, <strong># Max directory character length</strong><br>    &quot;dir_limit_depth&quot;: 2   <strong># Max directory depth to render</strong><br>  },<br>  &quot;after&quot;: &quot; &quot;,            <strong># String to display after the output</strong><br>  &quot;priority&quot;: 20           <strong># Give a high rendering priority</strong><br>},</pre><ul><li>We configure the cwd segment to render up to two parent directories. The ellipsis string .. is used to represent collapsed directories. I.e. ~/.config/powerline/themes/vim will become ../powerline/themes/vim.</li><li>The after key specifies an optional string that is rendered after the output of a particular segment. A single space character is assigned here to adjust the spacing between segments. The before key is also available for rendering a string before the segment output.</li><li>An optional priority value can also be specified for a segment. In cases where a prompt or status line has limited width, segments with low priority values will be rendered ahead of segments with a high priority. Leave out this option to always render the segment.</li></ul><p>Browsing through the <a href="https://powerline.readthedocs.io/en/master/configuration/segments.html">Powerline Segment Reference</a> is the best way to discover new segments. A <a href="https://powerline.readthedocs.io/en/master/develop.html">developer guide</a> is also available if you have some Python programming experience and wish to make your own segments.</p><p>Let’s go ahead and modify the powerline/themes/shell/default_leftonly.json file to layout the segments we want to render for the Bash prompt. The following gist contains the complete code listing for this file:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/7cea375d0c5b43c1bef3726c0f384f2c/href">https://medium.com/media/7cea375d0c5b43c1bef3726c0f384f2c/href</a></iframe><p>This table also describes the segments used by this theme:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/64c2f0d104f2fce1fee385ab4baa3d42/href">https://medium.com/media/64c2f0d104f2fce1fee385ab4baa3d42/href</a></iframe><p>To make the shell binding aware of our modified theme, it must be specified in the main powerline/config.json configuration file within the shell block:</p><pre>&quot;shell&quot;: {<br>     &quot;colorscheme&quot;: &quot;default&quot;,<br>     <strong>&quot;theme&quot;: &quot;default_leftonly&quot;,</strong><br>     &quot;local_themes&quot;: {<br>         &quot;continuation&quot;: &quot;continuation&quot;,<br>         &quot;select&quot;: &quot;select&quot;<br>     }<br>},</pre><p>Remember to restart the powerline-daemon process to see the theme applied in your terminal.</p><h4>Creating a Color Scheme for the Bash Prompt</h4><p>The code presented in this section will apply a dark color scheme to the bash prompt. We will also use a teal background color for the user segment.</p><p>Let’s start things off by defining some new colors in the powerline/colors.json configuration file:</p><pre>“teal”: 6,<br>“navyblue”: 17,<br>“darkblue2”: 18,<br>“slateblue”: 61,<br>“purpleturquoise”: 66,<br>“steelblue”: 67,<br>“gray01”: 234,</pre><p>We also need to create a new directory inside powerline/colorschemes/ named shell/, in addition to a new JSON file within this directory called teal_dark.json:</p><pre>$ cd ~/.config/powerline/colorschemes<br>$ mkdir shell &amp;&amp; cd shell<br>$ touch teal_dark.json</pre><p>The teal_dark.json file must define the highlight groups (what colors to use) for each segment defined in its corresponding theme file. Color schemes use the following syntax to assign colors to segments:</p><pre>&quot;group_name&quot;: { &quot;fg&quot;: &quot;color&quot;, &quot;bg&quot;: &quot;color&quot;, <br>                &quot;attrs&quot;: [&quot;bold&quot;, &quot;italic&quot;, &quot;underline&quot;] }</pre><ul><li>group_name: Refers to the name of a segment or a custom group.</li><li>fg and bg: Specifies a color defined in colors.json to set the foreground and background colors of the group respectively.</li><li>attrs[]: A list containing optional &quot;bold&quot;, &quot;italic&quot; and &quot;underline&quot; strings.</li></ul><p>The complete code listing for the teal_dark.json file is displayed here:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/43094733ebdc9edfbf56db83e4c92bc3/href">https://medium.com/media/43094733ebdc9edfbf56db83e4c92bc3/href</a></iframe><p>To make the shell binding aware of our new color scheme, it must be specified in the main powerline/config.json configuration file within the shell block:</p><pre>&quot;shell&quot;: {<br>    <strong>&quot;colorscheme&quot;: &quot;teal_dark&quot;,</strong><br>    &quot;theme&quot;: &quot;default_leftonly&quot;,<br>    &quot;local_themes&quot;: {<br>        &quot;continuation&quot;: &quot;continuation&quot;,<br>        &quot;select&quot;: &quot;select&quot;<br>    }<br>},</pre><p>Restart the powerline-daemon process to see the color scheme rendered in your terminal emulator. I encourage you to experiment with your configurations until you arrive at a prompt you are happy with. Also check out this <a href="https://jonasjacek.github.io/colors/">color code cheat sheet</a> if you intend to add more colors to the colors.json file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/850/1*tNGMUxXLV8OqeqPhBsicjw.gif" /><figcaption>Our configured Bash prompt rendered with powerline.</figcaption></figure><h3>Using Powerline with i3</h3><p>This section will take a look at configuring powerline to render the status bar used in the i3 window manager environment.</p><p>Firstly modify the bar { .. } block in your ~/.config/i3/config file to invoke the powerline binding for i3. You will also need to replace {powerline_root} with the location of your powerline installation:</p><pre>bar {<br>    status_command python3 {powerline_root}/powerline/bindings<br>      /i3/powerline-i3.py<br>    font pango:Hacker 11<br>}</pre><p>After reloading your i3 configuration, powerline will render the status bar and apply a very basic theme. From here we will add additional segments to the theme configuration file and create a color scheme like we did for the bash prompt.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*GKcqCakd6l41W_J379CLfg.png" /><figcaption>The i3 bar rendered with the default theme and color scheme configurations applied.</figcaption></figure><h4>Installing a Powerline Extension for Memory Segments</h4><p>Since the current version of powerline doesn’t provide any segments for outputting memory usage, we will install a small extension called powerline-mem-segment via pip to get this functionality. Feel free to read the package’s <a href="https://github.com/mKaloer/powerline_mem_segment">GitHub repository homepage</a> to discover its features and usage details.</p><p>All we need to do is run a single command to install the extension:</p><pre>pip install powerline-mem-segment</pre><p>The memory usage segments are now available and will be hooked up to our i3 bar in the next section.</p><h4>Creating a Theme for the i3 Bar</h4><p>Moving forward, let’s add and configure the segments we want to see on the i3 status bar in the powerline/themes/wm/default.json file. The complete code listing for this file is displayed here:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dadf18d933590d19550d081a8f613231/href">https://medium.com/media/dadf18d933590d19550d081a8f613231/href</a></iframe><p>The segments used in this theme are described here:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/514674b165aa164c05e3e57f29c59dad/href">https://medium.com/media/514674b165aa164c05e3e57f29c59dad/href</a></iframe><p>At this point, our configuration still needs to define either the mem_usage or mem_usage_gradient highlight groups for the memory segments to render. This will be covered in the next section.</p><h4>Creating a Color Scheme for the i3 Bar</h4><p>The code presented in this section will apply a dark color scheme to the i3 status bar. Following the same process detailed in the bash section, let’s start by creating a new directory inside powerline/colorschemes/ named wm/, in addition to a JSON file within this directory called i3_dark.json:</p><pre>$ cd ~/.config/powerline/colorschemes<br>$ mkdir wm &amp;&amp; cd wm<br>$ touch i3_dark.json</pre><p>The complete code listing of the i3_dark.json file follows:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/30c447ae0fd15d90b633c03c9c601e24/href">https://medium.com/media/30c447ae0fd15d90b633c03c9c601e24/href</a></iframe><p>Lastly, we must specify the i3_dark color scheme name in the main powerline/config.json configuration file, this time within the wm block:</p><pre>&quot;wm&quot;: {<br>    <strong>&quot;colorscheme&quot;: &quot;i3_dark&quot;,</strong><br>    &quot;theme&quot;: &quot;default&quot;,<br>    &quot;update_interval&quot;: 2<br>}</pre><p>Powerline will now render the i3 bar using our theme and color scheme configuration files:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/746/1*lZVKPAvS-LvcFxnCvwNfiQ.png" /><figcaption>Our configured i3 bar without the external IP address segment.</figcaption></figure><h3>Using Powerline with tmux</h3><p>This last section will detail how to enable powerline in tmux. I will leave the theme and color scheme configuration steps as an exercise for readers who have read up to this point!</p><p>Append some code to your ~/.tmux.conf file to enable powerline rendering:</p><pre><strong># Ensure the powerline daemon is running</strong><br>run-shell &quot;powerline-daemon -q&quot;</pre><pre><strong># Invoke the tmux powerline binding</strong><br>source &quot;{powerline_root}/bindings/tmux/powerline.conf&quot;</pre><p>Make sure to input the correct location of your tmux/powerline.conf binding. Powerline will now render the status line in tmux on its next launch:</p><pre>$ tmux new-session</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/855/1*wGFPF9kGDkc9EN7nBX5ZKQ.png" /><figcaption>Powerline rendering both the tmux status line and Bash prompt.</figcaption></figure><h3>In Conclusion</h3><p>Congratulations on completing this powerline integration guide! A lot of content was presented in concise sections throughout, so I certainly recommend taking some time to play with your configuration and referring back to relevant sections if necessary.</p><p>From here, I recommend checking out the <a href="https://powerline.readthedocs.io/en/latest/usage.html#plugins">other applications</a> powerline supports as well as continuing to configure your perfect status lines and prompts. I also encourage you to browse through the <a href="https://powerline.readthedocs.io/en/master/configuration/segments.html">segment reference documentation</a> to see the full list of what can be rendered with powerline.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c097831106f6" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to Turn Vim Into a Lightweight IDE]]></title>
            <link>https://dane-bulat.medium.com/how-to-turn-vim-into-a-lightweight-ide-6185e0f47b79?source=rss-24e03e26bd42------2</link>
            <guid isPermaLink="false">https://medium.com/p/6185e0f47b79</guid>
            <category><![CDATA[vim]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[linux-tutorial]]></category>
            <category><![CDATA[computing]]></category>
            <category><![CDATA[linux]]></category>
            <dc:creator><![CDATA[Dane Bulat]]></dc:creator>
            <pubDate>Wed, 18 Nov 2020 14:57:30 GMT</pubDate>
            <atom:updated>2021-02-10T12:37:53.381Z</atom:updated>
            <content:encoded><![CDATA[<h4>Build your own editor to facilitate programming with Vim</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SgrpkG99gtzIH7UK60IVQg.png" /></figure><h3>Introduction</h3><p>The Vim text editor is often used as a tool for editing configuration files on a Linux system. For this purpose, Vim does the job exceptionally well by providing unique features such as buffers, macros and bookmarking to name a few.</p><p>As well as a text editor, Vim also ships with a built-in scripting language called Vim script. This powerful feature allows developers to extend the functionality of Vim by writing plug-ins that accomplish specific tasks, such as code completion and interacting with the file system. This means that it is possible to install and configure any number of plug-ins to extend Vim’s functionality and literally build the perfect editor, specifically for you.</p><p>This article will demonstrate how to turn Vim into a lightweight IDE by installing and configuring a range of useful plug-ins. We will adopt a step-by-step approach where you will quickly learn the work flow (or process) of building your own editor with Vim.</p><h4>Task Overview</h4><p>The Vim text editor is very customizable even without the use of plug-ins. It is therefore a good idea to initially tweak some of Vim’s fundamental behavior. With this in mind, we will complete a few tasks before installing any plug-ins:</p><ul><li>Install the <a href="https://github.com/VundleVim/Vundle.vim">Vundle</a> plug-in manager.</li><li>Modify some Vim settings for correct tabbing behavior, searching, and terminal functionality.</li><li>Install some color schemes from the <a href="https://github.com/rafi/awesome-vim-colorschemes">awesome-vim-colorschemes</a> repository.</li></ul><p>At this point, we will take a look at installing and configuring these plug-ins:</p><ul><li><a href="https://github.com/sheerun/vim-polyglot">vim-polyglot</a>: Provides language packs and indentation.</li><li><a href="https://github.com/jiangmiao/auto-pairs">auto-pairs</a>: Pair completion when typing brackets, parenthesis, quotes, etc.</li><li><a href="https://github.com/preservim/nerdtree">NERDTree</a>: An integrated file system explorer.</li><li><a href="https://github.com/preservim/tagbar">tagbar</a>: A panel that displays a source file’s tags.</li><li><a href="https://github.com/dyng/ctrlsf.vim">ctrlsf.vim</a>: Enables asynchronously searching the file system.</li><li><a href="https://github.com/derekwyatt/vim-fswitch">fswitch</a>: For switching between header and implementation files.</li><li><a href="https://github.com/derekwyatt/vim-protodef">vim-protodef</a>: For pulling function prototypes into implementation files.</li></ul><p>Many Vim plug-ins come with sensible configurations and should work adequately out-of-the-box. Excellent documentation normally accompanies a plug-in as well, which is usually accessed via Vim’s :help command.</p><p>Browsing through plug-in documentation is definitely a worthwhile endeavor because you might find a cool feature or setting that you didn’t know about before.</p><h4>Getting the Final Configuration File</h4><p>I have created a <a href="https://github.com/danebulat/vim-config/tree/master/light-ide">GitHub repository</a> containing the final .vimrc file that we will be putting together in this article. Feel free to refer to it at any point if you get stuck or would like installation instructions for using it on your own system.</p><p>You can also download the final Vim configuration file using curl on the command line:</p><pre>$ curl -L https://raw.githubusercontent.com/danebulat/vim-config/master/light-ide/vimrc &gt; vimrc</pre><h3>Installing a Vim Plug-in Manager</h3><p>A Vim plug-in manager is a script that is able to automatically download, install and remove plug-ins by simply reading your .vimrc file. A plug-in manager will download plug-ins into separate directories and enable them every time we launch Vim.</p><p>Since our IDE will require many plug-ins — which may also have external dependencies — we will opt to install the Vundle plug-in manager. Vundle is installed by firstly cloning its Git repository into your ~/.vim/bundle directory:</p><pre>$ mkdir -p ~/.vim/bundle<br>$ cd ~/.vim/bundle<br>$ git clone <a href="https://github.com/VundleVim/Vundle.vim.git">https://github.com/VundleVim/Vundle.vim.git</a> Vundle.vim</pre><p>Then open your .vimrc file and add the following boilerplate code:</p><pre>set nocompatible<br>filetype off</pre><pre><strong>&quot; Set the runtime path to include Vundle and initialize</strong><br>set rtp+=~/.vim/bundle/Vundle.vim</pre><pre><strong>&quot; Download plug-ins to the ~/.vim/plugged/ directory</strong><br>call vundle#begin(&#39;~/.vim/plugged&#39;)</pre><pre><strong>&quot; Let Vundle manage Vundle</strong><br>Plugin &#39;VundleVim/Vundle.vim&#39;</pre><pre>call vundle#end()<br>filetype plugin indent on</pre><p>Now, whenever we want to install a new plug-in that is hosted on GitHub, we specify it using the format Plugin &#39;&lt;github_account&gt;/&lt;repository_name&gt;&#39; between the call vundle#begin(..) and call vundle#end() lines. Also take note that repositories will be downloaded into the ~/.vim/plugged/ directory on the file system.</p><p>Let’s now open Vim and run the :PluginInstall command to enable Vundle:</p><pre>$ vim ~/.vimrc<br><strong># In Vim</strong><br>:PluginInstall</pre><p>A side panel will open after running :PluginInstall showing the installation progress. When the Vim status line displays a Done message, feel free to close the panel by pressing q.</p><p>In order to remove a plug-in, simply comment out the corresponding Plugin line and run the :PluginClean command in Vim. The associated repository will then be removed from your file system.</p><h3>Modifying Some Vim Settings</h3><p>This section will go through modifying some basic Vim settings to make the editor feel more modern and natural to use.</p><p>Open up .vimrc and add some code to enable line numbers and syntax highlighting:</p><pre>set nu     <strong>&quot; Enable line numbers</strong><br>syntax on  <strong>&quot; Enable syntax highlighting</strong></pre><p>Vim will now look for a corresponding syntax/&lt;language&gt;.vim file when it opens up a particular source file. For example, if we open a file called main.cpp, Vim will attempt to load a syntax/cpp.vim file to render C++ syntax correctly. Because we will be installing language packs via a plug-in very shortly, it&#39;s not important to know about which syntax schemes your Vim installation supports - we just need to make sure syntax is turned on.</p><p>Let’s also fix Vim’s tabbing behavior by modifying some settings:</p><pre><strong>&quot; How many columns of whitespace a \t is worth</strong><br>set tabstop=4 </pre><pre><strong>&quot; How many columns of whitespace a &quot;level of indentation&quot; is worth</strong><br>set shiftwidth=4 </pre><pre><strong>&quot; Use spaces when tabbing</strong><br>set expandtab</pre><p>The expandtab setting guarantees that our source code files will always use space characters instead of the tab (\t) character. Encoding tabs as space characters in source files is recommended for a range of programming languages such as Python. We also make sure that tabs take up four columns of white space via the shifttab and tabstop settings.</p><p>Moving forward, Vim’s search function is triggered when the forward slash key is pressed in normal mode. Let’s tweak its behavior to make text searching in source files more intuitive:</p><pre>set incsearch  &quot; <strong>Enable incremental search</strong><br>set hlsearch   &quot;<strong> Enable highlight search</strong></pre><p>The incsearch setting will automatically update matches in a source file as you type a search query. The hlsearch setting turns on highlighting to provide some visual feedback when typing a search term. Run the :noh command to hide any highlighted text in the editor.</p><p>Lastly, let’s configure how the built-in terminal is opened when the :term command is run:</p><pre>set termwinsize=12x0   <strong>&quot; Set terminal size</strong><br>set splitbelow         <strong>&quot; Always split below</strong><br>set mouse=a            <strong>&quot; Enable mouse drag on window splits</strong></pre><p>We provide a sensible default height of twelve rows for the terminal window, and instruct Vim to always open it below the current active window. Mouse functionality is also enabled on the last line, which allows us to resize a window by clicking and dragging on a split.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*e_GsirLW-zSCzHEa1N7gKw.gif" /><figcaption>The configured terminal and window splitting.</figcaption></figure><h3>Plug-in Installations</h3><h4>Language Packs for Syntax Highlighting</h4><p>The <a href="https://github.com/sheerun/vim-polyglot">vim-polyglot</a> plug-in provides correct syntax highlighting and indentation for over 50 languages. When a file is open in Vim, the plug-in will load a corresponding script for that particular file type and apply the correct language settings.</p><p>Let’s add the Polyglot repository to the plug-in list in .vimrc and install it on the command line:</p><pre>Plugin &#39;sheerun/vim-polyglot&#39;<br>  <strong># Install on command line</strong><br>$ vim +PluginInstall +qall</pre><p>Because language packs are loaded on demand, your Vim startup time will not be affected by having many installed on your system. Moreover, we can inspect the storage requirement for the language packs with the du utility:</p><pre>$ du -hcs ~/.vim/plugged/vim-polyglot<br> <br><strong># Output</strong><br>23M .vim/plugged/vim-polyglot/<br>23M total</pre><p>Our IDE now supports syntax highlighting and indentation for over 50 languages. Feel free to open some source files of various languages to check out this new behavior.</p><h4>Color Schemes</h4><p>Although Vim ships with some interesting color schemes, it would be nice to install our own collection. It is possible to preview Vim’s installed color schemes with the :color &lt;scheme-name&gt; command. I recommend tabbing through the scheme names after typing :color .</p><p>If you are not satisfied with the defaults, there is a plethora of excellent color schemes online available for download. A great resource for discovering new color schemes for Vim is the <a href="https://github.com/rafi/awesome-vim-colorschemes">awesome-vim-colorschemes</a> repository, which lists a huge collection of color scheme projects. I encourage you to spend some time looking through the list and choosing your favorites. My configuration will include the following color schemes:</p><ul><li><a href="https://github.com/Badacadabra/vim-archery">archery</a>: A color scheme inspired by Arch Linux colors.</li><li><a href="https://github.com/cocopon/iceberg.vim/">iceberg</a>: Dark blue color scheme.</li><li><a href="https://github.com/arcticicestudio/nord-vim">nord</a>: An arctic, north-bluish clean and elegant theme.</li><li><a href="https://github.com/scheakur/vim-scheakur">scheakur</a>: A professional light/dark color scheme.</li><li><a href="https://github.com/kristijanhusak/vim-hybrid-material">vim-hybrid-material</a>: Material color scheme based on w0ng/vim-hybrid.</li><li><a href="https://github.com/lifepillar/vim-solarized8">vim-solarized8</a>: An optimized solarized theme.</li></ul><p>Add the Git repositories of your chosen color schemes to .vimrc and run the necessary installation command on the command line:</p><pre>Plugin &#39;cocopon/iceberg.vim&#39;<br>Plugin &#39;arcticicestudio/nord-vim&#39;<br>Plugin &#39;Badacadabra/vim-archery&#39;<br>Plugin &#39;kristijanhusak/vim-hybrid-material&#39;<br>Plugin &#39;scheakur/vim-scheakur&#39;<br>Plugin &#39;lifepillar/vim-solarized&#39;<br>  <strong># Install on command line</strong><br>$ vim +PluginInstall +qall</pre><p>From here, I recommend previewing the installed color schemes before deciding on which one to load by default. Open up a source file with plenty of code in order to get a good feel for each of the new color schemes:</p><pre>$ vim test.cpp<br>: color &lt;TAB&gt;</pre><p>At this point we can append a couple of lines to .vimrc to automatically load a color scheme when launching Vim:</p><pre>set background=dark   <strong>&quot; dark or light</strong><br>colorscheme scheakur  <strong>&quot; Your favorite color scheme&#39;s name</strong></pre><p>Set the background variable to either dark or light depending on the style of your default color scheme. Run the :set background=light and :set background=dark commands in Vim If you&#39;re unsure about which one to use.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9wvjaYvkmp27p37MjCGxcA.gif" /><figcaption>Previewing color schemes.</figcaption></figure><h4>Pair Completion</h4><p>The <a href="https://github.com/jiangmiao/auto-pairs">Auto Pairs</a> plug-in provides automatic pair completion when we input characters such as brackets, parenthesis and quotes. It also provides some formatting functionality, such as when you press the Enter key after typing an initial pair character.</p><p>Let’s add the auto-pairs repository to the plug-in list in .vimrc and install it on the command line:</p><pre>Plugin &#39;jiangmiao/auto-pairs&#39;<br>  <strong># Install on command line</strong><br>$ vim +PluginInstall +qall</pre><p>Pair completion is useful when working with programming languages such as C and C++, but it often gets in the way when editing basic configuration files that don’t require pair completion. It would be nice to quickly toggle pair completion on and off via a key mapping. The auto-pairs plug-in provides a variable specifically for this, which we can set in .vimrc:</p><pre>let g:AutoPairsShortcutToggle = &#39;&lt;C-P&gt;&#39;</pre><p>Pair completion is now toggled whenever we press Ctrl+P in normal mode. I encourage you to read the <a href="https://github.com/jiangmiao/auto-pairs">auto-pairs GitHub page</a> to find out more about what this useful plug-in can do.</p><h4>File System Explorer</h4><p>The <a href="https://github.com/preservim/nerdtree">NERDTree</a> plugin is a powerful file system explorer. It provides functionality for browsing the filesystem, opening files for reading or editing, and performing file operations. NERDTree is a feature-rich plug-in and includes many configuration settings, key mappings, as well as an API for adding new features.</p><p>This section will introduce the NERDTree plug-in by detailing the installation process and going through some key features so you can start using it effectively. I also encourage you to check out the excellent documentation by running the :help nerdtree command within Vim after completing this tutorial.</p><p>Let’s add the NERDTree repository to the plug-in list in .vimr and install it on the command line:</p><pre>Plugin &#39;preservim/nerdtree&#39;<br>  <strong># Install on command line</strong><br>$ vim +PluginInstall +qall</pre><p>From here you can run the :NERDTree command in Vim to open up a new tree. The panel can be closed either by pressing the q key or running the :NERDTreeClose command.</p><p>The j and k keys may be used for moving up and down the tree, and you can search for specific nodes (files and directories) with the forward slash key (/) as usual. NERDTree also defines key mappings to navigate and interact with the tree - the most useful of which are detailed in this table:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/69e557c6f7462131cbd5142278b102ba/href">https://medium.com/media/69e557c6f7462131cbd5142278b102ba/href</a></iframe><p>Have a play with these key mappings to get used to navigating the tree. Take note that the NERDTree panel must be in focus for the mappings to work.</p><p>Let’s also take a look at some important commands for opening and closing the NERDTree panel and modifying the tree root:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/970dfe9857dbe3f7b3fd78db3e76b24d/href">https://medium.com/media/970dfe9857dbe3f7b3fd78db3e76b24d/href</a></iframe><p>The root represents the outermost directory of the tree hierarchy. You will likely change roots when switching between projects that you wish to work on at any given moment.</p><p>NERDTree also supports the creation of bookmarks which can be used for navigating to projects and important locations on your file system efficiently. For example, you could create a bookmark for every project on your system — enabling you to jump from project-to-project and work on the corresponding project files efficiently. Providing a tree is in focus, these commands allow you to work with bookmarks:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1da7bc9c8f76fd9299e96b8f83b754b8/href">https://medium.com/media/1da7bc9c8f76fd9299e96b8f83b754b8/href</a></iframe><p>Let’s go ahead and override some configuration variables provided by NERDTree in .vimrc to tweak its behavior and appearance:</p><pre>let NERDTreeShowBookmarks = 1   <strong>&quot; Show the bookmarks table</strong><br>let NERDTreeShowHidden = 1      <strong>&quot; Show hidden files</strong><br>let NERDTreeShowLineNumbers = 0 <strong>&quot; Hide line numbers</strong><br>let NERDTreeMinimalMenu = 1     <strong>&quot; Use the minimal menu (m)</strong><br>let NERDTreeWinPos = “left”     <strong>&quot; Panel opens on the left side</strong><br>let NERDTreeWinSize = 31        <strong>&quot; Set panel width to 31 columns</strong></pre><p>Let’s also set up a mapping that will toggle the NERDTree panel when the F2 key is pressed:</p><pre>nmap &lt;F2&gt; :NERDTreeToggle&lt;CR&gt;</pre><p>Moving forward, I recommend refining your configuration and adding more key mappings as you continue to use NERDTree and its many commands.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Kxc9OKii5HkI0Q5vk9oV8w.gif" /><figcaption>Using the NERDTree file explorer to open and edit files.</figcaption></figure><h4>Source File Outliner</h4><p>The <a href="https://github.com/preservim/tagbar">tagbar</a> plugin provides a way to browse the tags of the currently opened file. What this means is that we can immediately see the structure of the file by viewing its tags which convey class, variable and function identifiers. Key mappings are also provided that allow you to navigate an outline and jump to the corresponding location in the source file.</p><p>The <a href="https://github.com/preservim/tagbar">tagbar GitHub page</a> reveals that it depends on the <a href="http://ctags.sourceforge.net/">Exuberant Ctags</a> program (&gt;= 5.5), which is often wrapped in a package called ctags . This package needs to be installed via your operating system’s package manager to make the tagbar plug-in usable with Vim. For example, ctags can be installed on Arch Linux via pacman:</p><pre>$ sudo pacman -S ctags<br>  <strong># Check Exuberant Ctags or Universal Ctags is installed</strong><br>$ ctags --version</pre><p>With all dependencies taken care of, add the tagbar repository to the plug-in list in .vimrc and install it on the command line:</p><pre>Plugin &#39;preservim/tagbar&#39;<br>  <strong># Install on command line</strong><br>$ vim +PluginInstall +qall</pre><p>The plug-in exposes some useful variables that we can modify to improve our workflow in the editor. The following code shows some overridden tagbar settings in our Vim configuration file:</p><pre><strong>&quot; Focus the panel when opening it</strong><br>let g:tagbar_autofocus = 1</pre><pre><strong>&quot; Highlight the active tag</strong><br>let g:tagbar_autoshowtag = 1</pre><pre><strong>&quot; Make panel vertical and place on the right</strong><br>let g:tagbar_position = &#39;botright vertical&#39;</pre><pre><strong>&quot; Mapping to open and close the panel</strong><br>nmap &lt;F8&gt; :TagbarToggle&lt;CR&gt;</pre><p>Setting the g:tagbar_autofocus variable to 1focuses the tagbar panel upon opening it. Additionally, setting the g:tagbar_autoshowtag to 1 means that the correct symbol is highlighted in the tagbar panel as the cursor is moved within the corresponding source file. A mapping to the :TagbarToggle&lt;CR&gt; command is also defined which allows us to open the sidebar by simply pressing the F8 key.</p><p>The panel’s cursor can be controlled with the standard Vim navigation keys (h, j, k, l) as well as some additional mappings defined by the plug-in. The most important mappings are presented here:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f8244dbd9a54ef0b395e57db73b971d2/href">https://medium.com/media/f8244dbd9a54ef0b395e57db73b971d2/href</a></iframe><p>The tagbar documentation details its complete command set and configuration options. Run the :help tagbar command in Vim to open it up.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rxfEqMiEw7fQhoSqibyQ2Q.gif" /><figcaption>Using the tagbar panel to jump to source code.</figcaption></figure><h4>Searching Files</h4><p><a href="https://github.com/dyng/ctrlsf.vim">CtrlFS</a> is a powerful plug-in that searches for files on the file system asynchronously. Search strings can be based on the current word being hovered over by the cursor, a selection in visual mode, an arbitrary search term we type ourselves, and even a regular expression.</p><p>After performing a search, we are able to quickly preview the resulting files in a pop-up or load them into the current window for editing.</p><p>The CtrlSF Github page states that your system needs to have a code-searching tool installed such as ack or ag in order for the plug-in to work. Therefore, make sure you install one of these tools via your operating system&#39;s package manager. For example, you can install the ack package on Arch Linux via pacman:</p><pre>sudo pacman -S ack</pre><p>After installing necessary dependencies, add the CtrlFS repository to the plug-in list in .vimrc and install it on the command line:</p><pre>Plugin &#39;dyng/ctrlsf.vim&#39;<br>  <strong># Install on command line</strong><br>$ vim +PluginInstall +qall</pre><p>The upcoming configuration specifies how important features of the plug-in behave. I recommend copying this code and tweaking the values after using the tool a few times:</p><pre><strong>&quot; Use the ack tool as the backend</strong><br>let g:ctrlsf_backend = &#39;ack&#39;</pre><pre><strong>&quot; Auto close the results panel when opening a file</strong><br>let g:ctrlsf_auto_close = { &quot;normal&quot;:0, &quot;compact&quot;:0 }</pre><pre><strong>&quot; Immediately switch focus to the search window</strong><br>let g:ctrlsf_auto_focus = { &quot;at&quot;:&quot;start&quot; }</pre><pre><strong>&quot; Don&#39;t open the preview window automatically</strong><br>let g:ctrlsf_auto_preview = 0</pre><pre><strong>&quot; Use the smart case sensitivity search scheme</strong><br>let g:ctrlsf_case_sensitive = &#39;smart&#39;</pre><pre><strong>&quot; Normal mode, not compact mode</strong><br>let g:ctrlsf_default_view = &#39;normal&#39;</pre><pre><strong>&quot; Use absoulte search by default</strong><br>let g:ctrlsf_regex_pattern = 0;</pre><pre><strong>&quot; Position of the search window</strong><br>let g:ctrlsf_position = &#39;right&#39;</pre><pre><strong>&quot; Width or height of search window</strong><br>let g:ctrlsf_winsize = &#39;46&#39;</pre><pre><strong>&quot; Search from the current working directory</strong><br>let g:ctrlsf_default_root = &#39;cwd&#39;</pre><p>It is also strongly recommended that you set up some mappings that enable you to search quickly:</p><pre><strong>&quot; (Ctrl+F) Open search prompt (Normal Mode)</strong><br>nmap &lt;C-F&gt;f &lt;Plug&gt;CtrlSFPrompt </pre><pre><strong>&quot; (Ctrl-F + f) Open search prompt with selection (Visual Mode)</strong><br>xmap &lt;C-F&gt;f &lt;Plug&gt;CtrlSFVwordPath</pre><pre><strong>&quot; (Ctrl-F + F) Perform search with selection (Visual Mode)</strong><br>xmap &lt;C-F&gt;F &lt;Plug&gt;CtrlSFVwordExec</pre><pre><strong>&quot; (Ctrl-F + n) Open search prompt with current word (Normal Mode)</strong><br>nmap &lt;C-F&gt;n &lt;Plug&gt;CtrlSFCwordPath</pre><pre><strong>&quot; (Ctrl-F + o )Open CtrlSF window (Normal Mode)</strong><br>nnoremap &lt;C-F&gt;o :CtrlSFOpen&lt;CR&gt;</pre><pre><strong>&quot; (Ctrl-F + t) Toggle CtrlSF window (Normal Mode)</strong><br>nnoremap &lt;C-F&gt;t :CtrlSFToggle&lt;CR&gt;</pre><pre><strong>&quot; (Ctrl-F + t) Toggle CtrlSF window (Insert Mode)</strong><br>inoremap &lt;C-F&gt;t &lt;Esc&gt;:CtrlSFToggle&lt;CR&gt;</pre><p>The mappings all follow the convention of initially pressing Ctrl+F followed by a unique letter to perform an operation with CtrlSF. Go ahead and test out the mappings above, and refer to the following table to navigate the results panel:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f53315121c118b2abb5cc89968396661/href">https://medium.com/media/f53315121c118b2abb5cc89968396661/href</a></iframe><p>To wrap up our discussion on the CtrlSF plug-in, its full documentation can be opened in Vim by running the :help ctrlsf-options command.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j1-xOuakAh1PoDsX4fTQVQ.gif" /><figcaption>Previewing CtrlSF search results.</figcaption></figure><h4>Switching Between .h and .cpp Files</h4><p>The next plug-in we will take a look at is called <a href="https://github.com/derekwyatt/vim-fswitch">FSwitch</a> and provides functionality for quickly switching between a header and implementation file. The plug-in makes it possible to configure the type of companion files, as well as the search paths that are utilized to find them.</p><p>Firstly add the FSwitch repository to the plug-in list in .vimrc and install it on the command line:</p><pre>Plugin &#39;derekwyatt/vim-fswitch&#39;<br> <strong># Install on command line</strong><br>$ vim +PluginInstall +qall</pre><p>Let’s now jump back into .vimrc and append this code:</p><pre>au! BufEnter *.cpp let b:fswitchdst = &#39;hpp,h&#39;<br>au! BufEnter *.h let b:fswitchdst = &#39;cpp,c&#39;These lines specify that:</pre><p>These lines specify that:</p><ul><li>When the loaded buffer is a .cpp file, the companion is of type hpp or h.</li><li>When the loaded buffer is a .h file, the companion is of type cpp or c.</li></ul><p>The configuration specifies that companion files must have the same name — meaning that only the file type may be different so the connection can be made.</p><p>Let’s also define a mapping that splits the active window vertically and loads the companion file on the right side of the split. This responsibility is given to Ctrl+Z with the line:</p><pre>nmap &lt;C-Z&gt; :vsplit &lt;bar&gt; :wincmd l &lt;bar&gt; :FSRight&lt;CR&gt;</pre><p>FSwitch allows companion files to be placed in separate ./include/ and ./src/ directories relative to the project&#39;s root. If you need the plug-in to search other directories in your project, check out the documentation by running :help fswitch in Vim.</p><h4>Pulling Prototypes Into Implementation Files</h4><p>The <a href="https://github.com/derekwyatt/vim-protodef">vim-protodef</a> plug-in can pull in function prototypes from a C++ header (.h) file into a corresponding implementation (.cpp) file. Let&#39;s try this out by firstly updating .vimrc and installing the plug-in:</p><pre>Plugin &#39;derekwyatt/vim-protodef&#39;<br>  <strong># Install on command line</strong><br>$ vim +PluginInstall +qall</pre><p>By default, the plug-in provides two mappings to pull in the prototypes:</p><pre><strong>&quot; Pull in prototypes</strong><br>nmap &lt;buffer&gt; &lt;silent&gt; &lt;leader&gt; ,PP</pre><pre><strong>&quot; Pull in prototypes without namespace definition&quot;</strong><br>nmap &lt;buffer&gt; &lt;silent&gt; &lt;leader&gt; ,PN</pre><p>The leader key is in fact the backward slash (\) key — meaning that your prototypes will be copied into your implementation file by pressing the \ + PP or \ + PN key combinations. Feel free to adjust these values and consult the documentation for more configuration options by running :help protodef.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q_4ysm-VaUJkDdy9BlP0lw.gif" /><figcaption>Using the FSwitch and Protodef plug-ins.</figcaption></figure><h3>In Conclusion</h3><p>At this point we have a boilerplate configuration with several essential IDE features including syntax highlighting for over 50 languages, a file system explorer, file searching, and an integrated terminal. If you are a developer who prefers to write code in a minimal environment, this setup may be adequate.</p><p>The following list describes some other Vim plug-ins that you could install to create a more feature-rich environment:</p><ul><li><a href="https://github.com/Yggdroot/indentLine">indentLine</a>: Displays indentation levels with vertical lines.</li><li><a href="https://github.com/tpope/vim-fugitive">vim-fugitive</a>: A Git wrapper.</li><li><a href="https://github.com/airblade/vim-gitgutter">vim-gitgutter</a>: Show a Git diff in the gutter.</li><li><a href="https://github.com/matze/vim-move">vim-move</a>: Move lines and selections up and down.</li><li><a href="https://github.com/ycm-core/YouCompleteMe">YouCompleteMe</a>: A powerful code-completion engine.</li></ul><p>From here, I encourage you to read more plug-in documentation and continue to iteratively build upon your IDE. The <a href="https://vimawesome.com/">Vim Awesome</a> website is also a great resource for discovering new plug-ins.</p><h4>Related Articles</h4><p>Other articles I have written that focus on Vim:</p><p><a href="https://dane-bulat.medium.com/vim-setting-up-a-build-system-and-code-completion-for-c-and-c-eb263c0a19a1">Vim: Setting up a Build System and Code Completion for C and C++</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6185e0f47b79" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Archiving and Compression on Linux: An Introduction]]></title>
            <link>https://dane-bulat.medium.com/archiving-and-compression-on-linux-an-introduction-3dc2490e1bb7?source=rss-24e03e26bd42------2</link>
            <guid isPermaLink="false">https://medium.com/p/3dc2490e1bb7</guid>
            <category><![CDATA[computing]]></category>
            <category><![CDATA[linux-tutorial]]></category>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[compression]]></category>
            <category><![CDATA[archiving]]></category>
            <dc:creator><![CDATA[Dane Bulat]]></dc:creator>
            <pubDate>Mon, 09 Nov 2020 15:19:50 GMT</pubDate>
            <atom:updated>2020-12-07T10:11:39.407Z</atom:updated>
            <content:encoded><![CDATA[<h4>Exploring the tar utility and standard compression tools on Linux</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*zXFEBfSujCY88xtTcN4_tA.png" /></figure><h3>Introduction</h3><p>Archiving is the process of combining multiple files into a single package called an archive. This archive can then be easily distributed to another machine, backed up in a repository somewhere, or simply kept on your own machine as a way to organize and cleanup your file system. Archives are also an essential component of the Linux ecosystem, because all the software you install via your distribution’s package manager will initially be downloaded as a compressed archive from a remote repository. Therefore, working with archives is an important aspect of using a Linux based operating system effectively.</p><p>This article will introduce the standard archiving tool used on Linux — which is the <a href="https://en.wikipedia.org/wiki/Tar_(computing)">tar</a> software utility — and demonstrate its usage on the command line to create and work with archives, or <em>tarballs</em>. The tar utility is also able to compress an archive via a compression tool. This usage pattern of firstly creating an archive and then compressing it using a compression tool is often adopted for distributing packages remotely.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c-eMYBCRJNpdKMrezv_a5g.png" /><figcaption>Regular files being added to an archive with <strong>tar</strong> and then compressed with <strong>gzip</strong>.</figcaption></figure><p>Compressing an archive can reduce its storage space requirements and is an important step in preparing it for distribution. With these points in mind, this article will also introduce the major compression tools on Linux, and discuss their usage both with tar and individually on the command line. More specifically, we shall take a look at the <a href="https://en.wikipedia.org/wiki/Gzip">gzip</a>, <a href="https://en.wikipedia.org/wiki/Bzip2">bzip2</a>, <a href="https://en.wikipedia.org/wiki/XZ_Utils">xz</a> and <a href="https://en.wikipedia.org/wiki/Zstandard">zstd</a> compression tools.</p><h4>The tar File Archiver</h4><p>Tar was initially released in January 1979 and has since evolved into the standard file archiver on Linux. As of writing this article in November 2020, the latest stable version of tar was released on February 23, 2019 and therefore is still in active development to this day.</p><p>If tar is not on your system already, go ahead and install it via your Linux distribution’s package manager:</p><pre>$ sudo pacman -S tar      <strong># Arch</strong><br>$ sudo apt install tar    <strong># Debian and Ubuntu</strong><br>$ sudo dnf install tar    <strong># CentOS and Fedora</strong></pre><p>I won’t dive into the intricacies of how tar works under the hood too much in this article, but I will mention some important points to be aware of as we progress through the commands in subsequent sections.</p><h4>Compression Tools</h4><p>There are quite a few compression tools available on Linux, with each one implementing a specific compression algorithm. The compression tools introduced in this article all have a very similar command line interface and usage pattern, which means that if you know how to use one of the tools, you can use them all.</p><p>Go ahead and install the <strong>gzip</strong>, <strong>bzip2</strong>, <strong>xz</strong> and <strong>zstd</strong> compression tools via your Linux distribution’s package manager:</p><pre>$ sudo pacman -S gzip bzip2 xz zstd            <strong># Arch</strong><br>$ sudo apt install gzip bzip2 xz-utils zstd    <strong># Debian and Ubuntu</strong><br>$ sudo dnf install gzip bzip2 xz zstd          <strong># Fedora</strong></pre><p>A later section in the article will describe each compression tool in more detail and present some useful command line options as well. For now, the upcoming table provides an overview of the four compression tools we just installed. Also take note of the <strong>Extension</strong> column since we will need to specify one of these file extensions in order to compress files with the associated compression tool via tar:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/40bce4d14eee0c329dbdbddcba199483/href">https://medium.com/media/40bce4d14eee0c329dbdbddcba199483/href</a></iframe><p>Why do we need to know about so many compression tools? Well, there are a few reasons. Even if package managers adopt a modern compression tool such as zstd or xz, there will still be a plethora of packages and repositories out there compressed with the older gzip or bzip2. In order to decompress and access the files in these archives, the associated compression tool is required. In addition, you may find that an older compression tool such as gzip or bzip2 is sufficient for your needs.</p><h3>Environment Setup</h3><p>This section will demonstrate a couple of simple command line tools that will enable us to identify a file’s type, as well as output a file or directory’s size in a human readable format. These tools might come in handy for inspecting the various archive formats we’ll be working with. I’ll also provide some instructions for setting up a working directory and how to acquire some wallpaper files for readers who wish to follow along with the article and use the same commands.</p><p>Feel free to skip this section if you want to go straight to the archiving and compression commands.</p><h4>Determining the Type of a File</h4><p>The file command outputs a file’s type and should already be installed on your Linux system. To invoke the tool, simply pass the name of a file you want to inspect, such as a JPEG image:</p><pre>$ file Acrylic.jpg</pre><pre><strong># Output</strong><br>Acrylic.jpg: JPEG image data, JFIF standard 1.01, resolution (DPI), density 72x72, segment length 16, progressive, precision 8, 5403x3602, components 3</pre><p>We can use the file command to confirm the type of archives, compressed files and regular files on the system. Feel free to use it if you ever get confused on what type of file you’re working with.</p><h4>Determining the Size of Files and Directories</h4><p>The du command (Disk Usage) outputs the disk space usage of files and directories on a system. Similar in syntax to the file tool, simply specify the files and directories you wish to know the size of:</p><pre><strong># Find size of two JPEG files</strong><br>$ du Acrylic.jpg ColdWarm.jpg</pre><pre><strong># Output</strong><br>4516 Acrylic.jpg<br>3512 ColdWarm.jpg</pre><p>By default, du outputs the size of files in kilobytes — 4516 and 3512 in the example above. The -h flag outputs the file size in a more human-readable format, while the -c flag displays the total size of the files as well:</p><pre>$ du -hc Acrylic.jpg ColdWarm.jpg</pre><pre><strong># Output</strong><br>4.5M Acrylic.jpg<br>3.5M ColdWarm.jpg<br>7.9M total</pre><p>File sizes are now displayed in megabytes and have an M appended to the output. The total size of the files is also displayed at the bottom of the list. In addition to files, directories can also be passed to du. Feel free to check out the manual pages and have a play with the tool before continuing.</p><h4>Downloading Some Files to Work With</h4><p>Let’s go ahead and set up a working directory for us to try out the archiving and compression commands demonstrated in the subsequent sections:</p><pre><strong># Create a directory in the home folder and enter it</strong><br>$ mkdir ~/archive-demos <br>$ cd ~/archive-demos</pre><p>From here, copy over some wallpapers from your /usr/share/backgrounds/ directory. If you do not have any wallpapers on your system, feel free to download the gnome-backgrounds package, which provides some nice wallpapers for your desktop. This package will require about 40MB of space on your system:</p><pre>$ sudo pacman -S gnome-backgrounds</pre><p>Then copy the backgrounds into your working directory:</p><pre>$ cp -fr /usr/share/backgrounds/gnome ~/archive-demos</pre><p>Now we have a place to work along with some files, let’s take a look at some tar commands in the next section.</p><h3>An Archiving and Compression Tutorial</h3><p>Now that we understand what a file archiver and a compression tool is, let’s take a look at some commands to solve a few real-world scenarios.</p><p>When working with tar on the command line, you will often use the same command line options for achieving a specific task — such as creating and extracting archives, as well as listing files within an archive. These frequently used options are detailed here:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4ecec9bcdb052226779a270d61cf3499/href">https://medium.com/media/4ecec9bcdb052226779a270d61cf3499/href</a></iframe><p>The tar utility also provides several options that enable filtering archives through specific compression tools. For example, if you wish to compress an archive with gzip, we would pass the <strong>z</strong> option along with the aforementioned <strong>cf</strong> options. The following table lists the filtering options available to tar that we shall use in this tutorial:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/36aeee575b38efafa6b2cf88270825ce/href">https://medium.com/media/36aeee575b38efafa6b2cf88270825ce/href</a></iframe><p>There are many more filtering options available to tar on the command line, so feel free to explore the manual pages after reading this article.</p><p>Moving forward, let’s enter our working directory and create some archives of desktop wallpapers by executing these commands in the terminal:</p><pre>$ tar cfv wallpapers.tar gnome/<br>$ tar czfv wallpapers.tar.gz gnome/         <strong># gzip</strong><br>$ tar cjfv wallpapers.tar.bz2 gnome/        <strong># bzip2</strong><br>$ tar cJfv wallpapers.tar.xz gnome/         <strong># xz</strong><br>$ tar cfv wallpapers.tar.zst gnome/ --zstd  <strong># zstd<br></strong>$ ls</pre><p>The time it takes to create the compressed archive will differ depending on the filter passed to tar — this is because each filter implements a different compression algorithm. On my machine, the xz filter took the most time to compress the files, whilst the zstd filter performed the fastest.</p><p>Let’s list the files in the archives we’ve just created by using the <strong>-t</strong> option:</p><pre>$ tar tfv wallpapers.tar<br>$ tar tzfv wallpapers.tar.gz                <strong># gzip</strong><br>$ tar tjfv wallpapers.tar.bz2               <strong># bzip2</strong><br>$ tar tJfv wallpapers.tar.xz                <strong># xz</strong><br>$ tar tfv wallpapers.tar.zst --zstd         <strong># zstd</strong></pre><p>Tar needs to firstly decompress the archive before retrieving the file names. Notice that xz’s decompression algorithm performs much faster than its compression algorithm.</p><p>It is also possible to list the individual files and directories within an archive:</p><pre>$ tar tzfv wallpapers.tar.gz gnome/Acrylic.jpg gnome/Wood.jpg</pre><p>If you intend to extract an archive, It is good practice to firstly list the archive’s files before performing the actual extract operation. For example, if you mistakenly downloaded a huge archive and immediately extract all the files before checking it’s content, you could be in some trouble!</p><p>The Vim and Emacs text editors are also able to list the files contained in an archive. To list the files in one of our archives with Vim, run this command:</p><pre>$ vim wallpapers.tar.xz</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/776/1*8gHrsrGQHCYGjTW4xtXbVA.png" /><figcaption>Listing the contents of the <strong>wallpapers.tar.xz</strong> archive with vim.</figcaption></figure><p>Vim’s syntax highlighting and search features might make life easier when browsing through an archive’s files.</p><p>Let’s clean up our working directory by placing all the archives we’ve created in a nested directory called example-01:</p><pre>$ mkdir example-01<br>$ mv wallpapers.* example-01/</pre><p>Thus far, the parent gnome/ directory has been added to our archives. What if we want our archive to only include image files? The <strong>-C</strong> option solves this problem as it allows us to move into a directory before performing an operation with tar:</p><pre>$ tar cfv wallpapers.tar -C gnome/ $(ls gnome/)<br>$ tar czfv wallpapers.tar.gz -C gnome/ $(ls gnome)         <strong># gzip</strong><br>$ tar cjfv wallpapers.tar.bz2 -C gnome/ $(ls gnome)        <strong># bzip2</strong><br>$ tar cJfv wallpapers.tar.xz -C gnome/ $(ls gnome)         <strong># xz</strong><br>$ tar cfv wallpapers.tar.zst -C gnome/ $(ls gnome) --zstd  <strong># zstd</strong></pre><p>Files can also be deleted from an uncompressed.tar archive with the <strong>-delete</strong> option. Therefore, we must firstly decompress an archive before attempting to delete any of its content:</p><pre>  <strong># rename wallpapers.tar to prevent it being overwritten</strong><br>$ mv wallpapers.tar wallpapers-original.tar</pre><pre>  <strong># decompress the .bz2 archive with -d flag</strong><br>$ bzip2 -d wallpapers.tar.bz2</pre><pre>  <strong># delete Wood.jpg and Acrylic.jpg from the archive</strong><br>$ tar f wallpapers.tar --delete Wood.jpg Acrylic.jpg</pre><pre>  <strong># compress the archive again with bzip2</strong><br>$ bzip2 wallpapers.tar</pre><pre>  <strong># list content of the wallpapers.tar.bz2 archive</strong><br>$ vim wallpapers.tar.bz2</pre><p>Notice that we had to use the bzip2 program on the command line to decompress the wallpapers.tar.bz2 archive. We then deleted the files and compressed it with bzip2 once again. Conversely, files are appended to an archive via the <strong>-r</strong> option:</p><pre><strong>  # decompress the .bz2 archive with -d flag</strong><br>$ bzip2 -d wallpapers.tar.bz2</pre><pre>  <strong># add Wood.jpg and Acrylic.jpg back into the archive</strong><br>$ tar rfv wallpapers.tar -C gnome/ Wood.jpg Acrylic.jpg</pre><pre>  <strong># compress the archive again with bzip2<br></strong>$ bzip2 wallpapers.tar</pre><pre>  <strong># list content of the wallpapers.tar.bz2 archive<br></strong>$ vim wallpapers.tar.bz2</pre><p>To extract files from an archive we use tar’s <strong>-x</strong> option. If we extract a file into a directory with an identical file already there, that original file will be overwritten. It is therefore important to be aware of the location you plan to extract files to and the possibility of any files being overwritten. Let’s go ahead and extract one of the archives in our working directory:</p><pre>$ tar xzfv wallpapers.tar.gz  <strong># extract files in gzip archive</strong><br>$ tar tzfv wallpapers.tar.gz  <strong># confirm files still exist in archive</strong><br>$ ls                          <strong># list working directory</strong></pre><p>All of our images were extracted in our working directory! This isn’t very clean — it would be useful if we could extract all the files into a separate folder. This is possible if we specify the <strong>one-top-level</strong> option:</p><pre><strong>  # remove the files we just extracted</strong><br>$ rm *.jpg *.png *.xml</pre><pre>  <strong># create an empty directory<br></strong>$ mkdir top</pre><pre>  <strong># extract files into the top/ directory and list</strong><br>$ tar xzfv wallpapers.tar.gz --one-top-level=top<br>$ ls top</pre><p>Our extracted files were all written into the top directory. If you don’t assign a directory name to the --one-top-level option, files will be extracted into a directory with the same name as the archive:</p><pre>$ rm -fr top<br>$ tar xzfv wallpapers.tar.gz --one-top-level<br>$ ls wallpapers</pre><p>To extract individual files and directories from the archive, simply append the items you’d like to extract to the end of the command:</p><pre>$ tar xfv wallpapers.tar.zst --zstd Wood.jpg        <strong># zstd</strong><br>$ tar xJfv wallpapers.tar.xz Acrylic.jpg Road.jpg   <strong># xz</strong></pre><p>We have built up enough knowledge to work with our own compressed archives in Linux. Let’s go ahead and clean up our working directory before moving onto the last example:</p><pre>$ rm *.jpg<br>$ mkdir example-02<br>$ mv wallpapers.* wallpapers example-02/</pre><p>Lastly, we will look at how to append a tarball to another tarball using the <strong>-A </strong>option:</p><pre><strong>  # create two archives and list their contents</strong><br>$ tar cfv foo.tar -C gnome/ Acrylic.jpg Road.jpg<br>$ tar cfv bar.tar -C gnome/ Dark_Ivy.jpg Wood.jpg<br>$ tar tfv foo.tar &amp;&amp; tar tfv bar.tar</pre><pre><strong>  # append bar.tar to foo.tar and list the contents again</strong><br>$ tar Afv foo.tar bar.tar<br>$ tar tfv foo.tar &amp;&amp; tar tfv bar.tar</pre><pre><strong>  # compress archive with gzip</strong><br>$ gzip foo.tar</pre><p>Notice that tar doesn’t allow you to append compressed archives. You will need to decompress the compressed archives you wish to combine before performing the actual append operation. After appending bar.tar to foo.tar in the above example, foo.tar is compressed with gzip, producing foo.tar.gz.</p><p><em>After compressing a file with </em><strong><em>gzip</em></strong><em>, the original uncompressed archive is discarded. Specify the </em><strong><em>-k</em></strong><em> option (keep) to keep the uncompressed version on the filesystem.</em></p><p>Lets wrap up the tutorial by cleaning up the working directory one more time:</p><pre>$ mkdir example-03<br>$ mv foo.tar.gz bar.tar example-03/</pre><h4>Other Useful tar options</h4><p>Feel free to experiment with some additional tar options presented in the following table:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/901489a77b32fa4951a7c5fc3126a640/href">https://medium.com/media/901489a77b32fa4951a7c5fc3126a640/href</a></iframe><p>I also encourage you to check out the tar manual pages, which will certainly be your best resource for finding new information about the tool:</p><pre>$ man tar</pre><h3>A Closer Look at the Compression Tools</h3><p>This section serves as a quick reference for readers wishing to familiarize themselves on the command line interface of our adopted compression tools.</p><p><strong>Command Line Reference — gzip</strong></p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/9fb846fabbe3264d663856c3fcbb4be5/href">https://medium.com/media/9fb846fabbe3264d663856c3fcbb4be5/href</a></iframe><h4>Command Line Reference — bzip2</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/55783b7e3a4b2c9326590f4e68ce1e0b/href">https://medium.com/media/55783b7e3a4b2c9326590f4e68ce1e0b/href</a></iframe><h4>Command Line Reference — xz</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/647a4f4ce7612dab3ecba3abd619bc2f/href">https://medium.com/media/647a4f4ce7612dab3ecba3abd619bc2f/href</a></iframe><h4>Command Line Reference — zstd</h4><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/f8d0b6db82754e3560a9a82f377c7335/href">https://medium.com/media/f8d0b6db82754e3560a9a82f377c7335/href</a></iframe><h3>In Conclusion</h3><p>This article provided an introduction to working with archives on Linux using the tar file archiver and some standard Linux compression tools — including gzip, bzip2, xz and zstd. I hope the article has provided enough information and hands-on examples such that you can now comfortably create and manage your own archives on Linux.</p><p>From here, I encourage you to read further into the manual pages of tar and the various compression tools available on Linux, as well as looking into the pros and cons that come with using the individual compression algorithms.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=3dc2490e1bb7" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>