<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Maxence Malbois on Medium]]></title>
        <description><![CDATA[Stories by Maxence Malbois on Medium]]></description>
        <link>https://medium.com/@maxencemalbois?source=rss-187dc1e4c7c2------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*9Hzl1VT59rbOnulT.jpg</url>
            <title>Stories by Maxence Malbois on Medium</title>
            <link>https://medium.com/@maxencemalbois?source=rss-187dc1e4c7c2------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 10:31:40 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@maxencemalbois/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Implementing Tiptap in a Rails 7 app with Turbo and Stimulus JS]]></title>
            <link>https://maxencemalbois.medium.com/migrating-from-trix-to-tiptap-in-a-rails-7-app-with-turbo-and-stimulus-js-97f253d13d0?source=rss-187dc1e4c7c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/97f253d13d0</guid>
            <category><![CDATA[tiptap-editor]]></category>
            <category><![CDATA[rails]]></category>
            <category><![CDATA[javascript]]></category>
            <dc:creator><![CDATA[Maxence Malbois]]></dc:creator>
            <pubDate>Sat, 09 Sep 2023 13:37:40 GMT</pubDate>
            <atom:updated>2024-01-29T19:24:48.274Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*qGv5r0mA9-m-h3VjSnlmgg.png" /></figure><p>I recently decided to switch my Rails Rich editor from Trix to Tiptap in order to shape my own Rich Text experience.</p><p><a href="https://tiptap.dev/editor">Tiptap</a> library is based on <a href="https://github.com/prosemirror">ProseMirror</a> and ProseMirror is not a Rich Text Editor (RTE) per se, but rather a Rich Text Editor creation tool. Then you end up one notch higher that a classic ready to use Rich Text Editor. Though the journey to replicate Trix functionalities was rather quick…</p><blockquote><strong>Note</strong>: Trix deals with its own way of producing HTML, especially not producing any &lt;p&gt; tag which was a problem for me.<br>Tiptap/ProseMirror has its own limitation: anything in a list will be a &lt;p&gt; tag (&lt;ul&gt;&lt;li&gt;&lt;p&gt;text&lt;/p&gt;&lt;/li&gt;&lt;/ul&gt;) and this cannot be changed (<a href="https://github.com/ueberdosis/tiptap/issues/118">please see this github case</a>). You have to make your paragraphs inside a list element inlinein the final document displaying your tiptap input</blockquote><h4>Things to know</h4><p>1 — Because Tiptap lets you create your own Rich Rext Editor, everything is modular, then you have to import an extension for literally everything. Even required modules such as text or document need be imported. <br>For a simple editor with 5 buttons : bold, italic, link, unordered list, ordered list, I had to import:</p><pre>import { Controller } from “@hotwired/stimulus”<br>import { Editor } from ‘@tiptap/core’<br>import Document from ‘@tiptap/extension-document’<br>import Paragraph from ‘@tiptap/extension-paragraph’<br>import Text from ‘@tiptap/extension-text’<br>import ListItem from ‘@tiptap/extension-list-item’<br>import BulletList from ‘@tiptap/extension-bullet-list’<br>import OrderedList from ‘@tiptap/extension-ordered-list’<br>import Bold from ‘@tiptap/extension-bold’<br>import Italic from ‘@tiptap/extension-italic’<br>import Link from ‘@tiptap/extension-link’</pre><p>(first import is my Stimulus controller)</p><p>2 — Tiptap cannot be applied to an input field (at least when I tried it). I have then attached it to a simple &lt;div&gt; and noticed it was made contenteditable . Then You have to create a JS listener on submit to populate your field according to the Tiptap instance value. <br>For the sake of performance I have done this when the form was submitted, by preventing the form being sent in a Stimulus controller and using Turbo method to send it.</p><pre>formSubmit(event){<br>    event.preventDefault()<br>    event.stopPropagation()<br>  <br>    const formField = this.element.querySelector(&quot;.tiptap__field-content&quot;)<br><br>    formField.value = this.element.editor.getHTML()<br>    Turbo.navigator.submitForm(this.element)      <br>  }</pre><h3>Final code</h3><p>The HTML is a simple erb (eruby) partial. You can see there is a bit of Tailwind though my styling is mainly SCSS / BEM methodology:</p><pre>&lt;div class=&quot;mt-2 tiptap__editor&quot;&gt;  <br>  &lt;div class=&quot;tiptap__toolbar&quot;&gt;<br>    &lt;div class=&quot;tiptap__button-group&quot;&gt;<br>      &lt;button type=&quot;button&quot; class=&quot;tiptap__button tiptap__button--bold&quot; data-action=&quot;click-&gt;resumeDataTiptap#bold&quot;&gt;<br>        &lt;svg xmlns=&quot;http://www.w3.org/2000/svg&quot; width=&quot;24&quot; height=&quot;24&quot; fill=&quot;currentColor&quot;&gt;&lt;path d=&quot;M15.6 11.8c1-.7 1.6-1.8 1.6-2.8a4 4 0 0 0-4-4H7v14h7c2.1 0 3.7-1.7 3.7-3.8 0-1.5-.8-2.8-2.1-3.4zM10 7.5h3a1.5 1.5 0 1 1 0 3h-3v-3zm3.5 9H10v-3h3.5a1.5 1.5 0 1 1 0 3z&quot;/&gt;&lt;/svg&gt;<br>      &lt;/button&gt;<br>      &lt;button type=&quot;button&quot; class=&quot;tiptap__button tiptap__button--italic&quot; data-action=&quot;click-&gt;resumeDataTiptap#italic&quot;&gt;<br>        &lt;svg xmlns=&quot;http://www.w3.org/2000/svg&quot; width=&quot;24&quot; height=&quot;24&quot; fill=&quot;currentColor&quot;&gt;&lt;path d=&quot;M10 5v3h2.2l-3.4 8H6v3h8v-3h-2.2l3.4-8H18V5h-8z&quot;/&gt;&lt;/svg&gt;<br>      &lt;/button&gt;<br>      &lt;button type=&quot;button&quot; class=&quot;tiptap__button tiptap__button--link&quot; data-action=&quot;click-&gt;resumeDataTiptap#link&quot;&gt;<br>        &lt;svg xmlns=&quot;http://www.w3.org/2000/svg&quot; width=&quot;24&quot; height=&quot;24&quot; fill=&quot;currentColor&quot;&gt;&lt;path d=&quot;M9.88 13.7a4.3 4.3 0 0 1 0-6.07l3.37-3.37a4.26 4.26 0 0 1 6.07 0 4.3 4.3 0 0 1 0 6.06l-1.96 1.72a.91.91 0 1 1-1.3-1.3l1.97-1.71a2.46 2.46 0 0 0-3.48-3.48l-3.38 3.37a2.46 2.46 0 0 0 0 3.48.91.91 0 1 1-1.3 1.3z&quot;/&gt;&lt;path d=&quot;M4.25 19.46a4.3 4.3 0 0 1 0-6.07l1.93-1.9a.91.91 0 1 1 1.3 1.3l-1.93 1.9a2.46 2.46 0 0 0 3.48 3.48l3.37-3.38c.96-.96.96-2.52 0-3.48a.91.91 0 1 1 1.3-1.3 4.3 4.3 0 0 1 0 6.07l-3.38 3.38a4.26 4.26 0 0 1-6.07 0z&quot;/&gt;&lt;/svg&gt;<br>      &lt;/button&gt;<br>    &lt;/div&gt;<br>    &lt;div class=&quot;tiptap__button-group&quot;&gt;<br>      &lt;button type=&quot;button&quot; class=&quot;tiptap__button tiptap__button--ul&quot; data-action=&quot;click-&gt;resumeDataTiptap#unorderedList&quot;&gt;<br>        &lt;svg version=&quot;1&quot; xmlns=&quot;http://www.w3.org/2000/svg&quot; width=&quot;24&quot; height=&quot;24&quot; fill=&quot;currentColor&quot;&gt;&lt;path d=&quot;M4 4a2 2 0 1 0 0 4 2 2 0 0 0 0-4zm0 6a2 2 0 1 0 0 4 2 2 0 0 0 0-4zm0 6a2 2 0 1 0 0 4 2 2 0 0 0 0-4zm4 3h14v-2H8v2zm0-6h14v-2H8v2zm0-8v2h14V5H8z&quot;/&gt;&lt;/svg&gt;<br>      &lt;/button&gt;<br>      &lt;button type=&quot;button&quot; class=&quot;tiptap__button tiptap__button--ol&quot; data-action=&quot;click-&gt;resumeDataTiptap#orderedList&quot;&gt;<br>        &lt;svg xmlns=&quot;http://www.w3.org/2000/svg&quot; width=&quot;24&quot; height=&quot;24&quot; fill=&quot;currentColor&quot;&gt;&lt;path d=&quot;M2 17h2v.5H3v1h1v.5H2v1h3v-4H2v1zm1-9h1V4H2v1h1v3zm-1 3h1.8L2 13.1v.9h3v-1H3.2L5 10.9V10H2v1zm5-6v2h14V5H7zm0 14h14v-2H7v2zm0-6h14v-2H7v2z&quot;/&gt;&lt;/svg&gt;<br>      &lt;/button&gt;<br>    &lt;/div&gt;<br>    <br>  &lt;/div&gt;   <br>  <br>  &lt;div class=&quot;tiptap__link-area hidden&quot;&gt;<br>    &lt;input type=&quot;text&quot; class=&quot;tiptap__link-field&quot; placeholder=&quot;enter URL here ...&quot; data-action=&quot;keydown-&gt;resumeDataTiptap#preventEnter&quot;&gt;&lt;/input&gt;<br>    &lt;button type=&quot;button&quot; class=&quot;tiptap__link-button&quot; data-action=&quot;click-&gt;resumeDataTiptap#setLink&quot;&gt;set&lt;/button&gt;<br>    &lt;button type=&quot;button&quot; class=&quot;tiptap__link-button tiptap__link-button--unset&quot; data-action=&quot;click-&gt;resumeDataTiptap#unsetLink&quot;&gt;unset&lt;/button&gt;<br>  &lt;/div&gt;<br>  &lt;div class=&quot;tiptap__textarea&quot;&gt;&lt;%= (initial_content || &quot;&quot;).html_safe %&gt;&lt;/div&gt;<br>  &lt;%= form.hidden_field field_name, class:&quot;tiptap__field-content&quot; %&gt;<br>&lt;/div&gt;</pre><p>Implementation is easy, just attach the tiptap Stimulus controller to your form :</p><blockquote>my Stimulus controller is named resumeDataTiptap as I am building a resume creation tool (go check it here: <a href="https://www.cvdunk.com/">cvdunk.com</a>), just name it as you want and rename all Stimulus controller actions in the partial</blockquote><pre>&lt;%= form_with(model: [@resume, experience], data: { controller: &quot;resumeDataTiptap&quot;}) do |form| %&gt;</pre><p>and render the partial instead of a classic form field. Data that you need is : the initial content which is the current value of your field, the form, and finaly the field name as a Ruby symbol :</p><pre>&lt;%= render partial: &#39;shared/tiptap_editor&#39;,<br>           locals: { initial_content: experience.experience_body,<br>                     form: form,<br>                     field_name: :experience_body } %&gt;</pre><p>(As you can see I have placed my partial in app/views/shared but it is up to you)</p><p>And the biggest chunk now: the tiptap Stimulus controller. Please note I am not the best Javascripter in town and there may be some room for refactoring :</p><pre>import { Controller } from &quot;@hotwired/stimulus&quot;<br>import { Editor } from &#39;@tiptap/core&#39;<br>import Document from &#39;@tiptap/extension-document&#39;<br>import Paragraph from &#39;@tiptap/extension-paragraph&#39;<br>import Text from &#39;@tiptap/extension-text&#39;<br>import ListItem from &#39;@tiptap/extension-list-item&#39;<br>import BulletList from &#39;@tiptap/extension-bullet-list&#39;<br>import OrderedList from &#39;@tiptap/extension-ordered-list&#39;<br>import Bold from &#39;@tiptap/extension-bold&#39;<br>import Italic from &#39;@tiptap/extension-italic&#39;<br>import Link from &#39;@tiptap/extension-link&#39;<br><br><br>export default class extends Controller {<br><br><br>  // tiptap controller need to be added to the form<br>  // a formSubmit action will be programmatically created<br><br>  connect() {<br>    const masterElement = this.element  <br><br>    if(!masterElement.nodeName == &quot;FORM&quot;){<br>      console.log(&quot;You must add tiptap to a form element&quot;)<br>    }<br>    if(!masterElement.querySelector(&quot;.tiptap__editor&quot;)){<br>      console.log(&quot;We cannot find the tiptap editor&quot;)<br>    }<br><br>    // programmatically adding a Stimulus submit action to the form<br>    // in order to populate form field before submit<br>    masterElement.setAttribute(&quot;data-action&quot;, &quot;submit-&gt;resumeDataTiptap#formSubmit&quot;)<br>    this.initTiptap(masterElement)    <br><br>  }<br><br>  formSubmit(event){<br>    event.preventDefault()<br>    event.stopPropagation()<br>  <br>    const formField = this.element.querySelector(&quot;.tiptap__field-content&quot;)<br><br>    formField.value = this.element.editor.getHTML()<br>    Turbo.navigator.submitForm(this.element)      <br>  }<br><br>  initTiptap(masterElement){<br>    const tiptapEditor = masterElement.querySelector(&quot;.tiptap__editor&quot;)<br>    <br>    const textArea = this.element.querySelector(&quot;.tiptap__textarea&quot;)<br>    const tipTapContent = textArea.innerHTML<br>    textArea.innerHTML = &quot;&quot;<br><br>    const boldButton = this.element.querySelector(&quot;.tiptap__button--bold&quot;)<br>    const italicButton = this.element.querySelector(&quot;.tiptap__button--italic&quot;)<br>    const linkButton = this.element.querySelector(&quot;.tiptap__button--link&quot;)<br>    const olButton = this.element.querySelector(&quot;.tiptap__button--ol&quot;)<br>    const ulButton = this.element.querySelector(&quot;.tiptap__button--ul&quot;)<br><br>    const linkArea = this.element.querySelector(&quot;.tiptap__link-area&quot;)<br>    const linkInput = this.element.querySelector(&quot;.tiptap__link-field&quot;)<br>    const unsetButton = this.element.querySelector(&quot;.tiptap__link-button--unset&quot;)<br><br>    const editor = new Editor({<br>      element: textArea,<br>      extensions: [<br>        Document,<br>        Paragraph,<br>        Text,<br>        ListItem,<br>        BulletList,<br>        OrderedList,<br>        Bold,<br>        Italic,<br>        Link.configure({<br>          openOnClick: false,<br>        })<br>      ],<br>      type: &#39;HTML&#39;,<br>      content: tipTapContent,<br>      autofocus: true,<br>      editable: true,<br>      injectCSS: false,<br>      hideHTMLTags: true,<br>      onSelectionUpdate({ editor }) {<br>        boldButton.classList.toggle(&quot;tiptap__button--selected&quot;, editor.isActive(&#39;bold&#39;))<br>        italicButton.classList.toggle(&quot;tiptap__button--selected&quot;, editor.isActive(&#39;italic&#39;))<br>        linkButton.classList.toggle(&quot;tiptap__button--selected&quot;, editor.isActive(&#39;link&#39;))<br>        <br>        if(!editor.isActive(&#39;link&#39;)){<br>          linkArea.classList.toggle(&quot;hidden&quot;, true)<br>          linkInput.value = &quot;&quot;<br>          unsetButton.classList.toggle(&quot;hidden&quot;, true)<br>        }else{<br>          linkArea.classList.toggle(&quot;hidden&quot;, false)<br>          let previousUrl = editor.getAttributes(&#39;link&#39;).href<br>          linkInput.value = previousUrl<br>          unsetButton.classList.toggle(&quot;hidden&quot;, false)<br>        }<br>        <br>        ulButton.classList.toggle(&quot;tiptap__button--selected&quot;, editor.isActive(&#39;bulletList&#39;))<br>        olButton.classList.toggle(&quot;tiptap__button--selected&quot;, editor.isActive(&#39;orderedList&#39;))<br>        <br>      },<br>    });<br><br>    // attaching tiptap editor to the form so we can destroy it in Stimulus<br>    // disconnect method<br>    masterElement.editor = editor  <br>    <br>  }  <br><br><br>  // The logic happens here<br>  <br>  // When toggling out, I auto select the entire element before changing status <br>  // for example: the cursor is located in a bold bit of text<br>  // but the text is not actually selected -&gt; Tiptat will select automatically<br>  // the whole element<br>  // this is done with &quot;.extendMarkRange&quot;<br><br>  // as you can see it is highly customizable so you can tweak the user<br>  // experience to your tune<br><br>  bold(event){<br>    const button = event.target.closest(&quot;.tiptap__button&quot;)<br>    const editor = this.element.editor<br>    <br>    if(!editor.view.state.selection.empty){<br>      if(editor.isActive(&#39;bold&#39;)){<br>        editor.chain().focus().extendMarkRange(&#39;bold&#39;).toggleBold().run()<br>      }else{<br>        editor.chain().focus().toggleBold().run()<br>      }      <br>      button.classList.toggle(&quot;tiptap__button--selected&quot;, editor.isActive(&#39;bold&#39;)) <br>    }else{<br>      if(editor.isActive(&#39;bold&#39;)){<br>        editor.chain().focus().extendMarkRange(&#39;bold&#39;).toggleBold().run()<br>      }<br>    }    <br>    <br>      <br>  }<br><br>  italic(){<br>    const button = event.target.closest(&quot;.tiptap__button&quot;)<br>    const editor = this.element.editor<br>    <br>    if(!editor.view.state.selection.empty){<br>      if(editor.isActive(&#39;italic&#39;)){<br>        editor.chain().focus().extendMarkRange(&#39;italic&#39;).toggleItalic().run()<br>      }else{<br>        editor.chain().focus().toggleItalic().run()<br>      }      <br>      button.classList.toggle(&quot;tiptap__button--selected&quot;, editor.isActive(&#39;italic&#39;)) <br>    }else{<br>      if(editor.isActive(&#39;italic&#39;)){<br>        editor.chain().focus().extendMarkRange(&#39;italic&#39;).toggleItalic().run()<br>      }<br>    } <br>  }<br><br>  link(){<br>    const button = event.target.closest(&quot;.tiptap__button&quot;)<br>    const editor = this.element.editor<br>    const linkArea = this.element.querySelector(&quot;.tiptap__link-area&quot;)<br>    const previousUrl = editor.getAttributes(&#39;link&#39;).href<br>    const linkInput = this.element.querySelector(&quot;.tiptap__link-field&quot;)<br><br>    if(previousUrl){<br>      linkInput.value = previousUrl<br>    }    <br>    <br>    if(button.classList.contains(&quot;tiptap__button--selected&quot;)){<br>      linkArea.classList.toggle(&quot;hidden&quot;, true)<br>      button.classList.toggle(&quot;tiptap__button--selected&quot;, false)<br>    }else{<br>      if(!editor.view.state.selection.empty){<br>        linkArea.classList.toggle(&quot;hidden&quot;, false)<br>        button.classList.toggle(&quot;tiptap__button--selected&quot;, true)<br>      }<br>    }<br>  }<br><br>  setLink(){<br>    const button = event.target.closest(&quot;.tiptap__button&quot;)<br>    const editor = this.element.editor<br>    const linkInput = this.element.querySelector(&quot;.tiptap__link-field&quot;).value <br>    const previousUrl = editor.getAttributes(&#39;link&#39;).href<br>    const unsetButton = this.element.querySelector(&quot;.tiptap__link-button--unset&quot;)<br><br>    <br>    if(linkInput){<br>      editor.chain().focus().extendMarkRange(&#39;link&#39;).setLink({ href: linkInput }).run()  <br>      unsetButton.classList.toggle(&quot;hidden&quot;, false)    <br>    }else{<br>      editor.chain().focus().extendMarkRange(&#39;link&#39;).unsetLink().run()<br>      unsetButton.classList.toggle(&quot;hidden&quot;, true)<br>    }<br>    <br>  }<br><br>  unsetLink(){<br>    const button = event.target.closest(&quot;.tiptap__button&quot;)<br>    const editor = this.element.editor<br>    const linkInput = this.element.querySelector(&quot;.tiptap__link-field&quot;).value <br>    const previousUrl = editor.getAttributes(&#39;link&#39;).href<br>    const unsetButton = this.element.querySelector(&quot;.tiptap__link-button--unset&quot;)<br><br>    editor.chain().focus().extendMarkRange(&#39;link&#39;).unsetLink().run()<br>    unsetButton.classList.toggle(&quot;hidden&quot;, true)<br><br>  }<br><br>  unorderedList(){<br>    const button = event.target.closest(&quot;.tiptap__button&quot;)<br>    const editor = this.element.editor<br>    const olButton = this.element.querySelector(&quot;.tiptap__button--ol&quot;)<br><br>    if(editor.isActive(&quot;orderedList&quot;) &amp;&amp; !editor.isActive(&quot;bulletList&quot;)){<br>      editor.chain().focus().extendMarkRange(&#39;link&#39;).toggleOrderedList().run()<br>      olButton.classList.toggle(&quot;tiptap__button--selected&quot;, editor.isActive(&#39;orderedList&#39;))<br>    }<br><br>    editor.chain().focus().toggleBulletList().run()<br>    button.classList.toggle(&quot;tiptap__button--selected&quot;, editor.isActive(&#39;bulletList&#39;))<br>  }<br><br>  orderedList(){<br>    const button = event.target.closest(&quot;.tiptap__button&quot;)<br>    const editor = this.element.editor<br>    const ulButton = this.element.querySelector(&quot;.tiptap__button--ul&quot;)<br>    <br>    if(!editor.isActive(&quot;orderedList&quot;) &amp;&amp; editor.isActive(&quot;bulletList&quot;)){<br>      editor.chain().focus().extendMarkRange(&#39;link&#39;).toggleBulletList().run()<br>      ulButton.classList.toggle(&quot;tiptap__button--selected&quot;, editor.isActive(&#39;bulletList&#39;))<br>    }<br><br>    editor.chain().focus().toggleOrderedList().run()<br>    button.classList.toggle(&quot;tiptap__button--selected&quot;, editor.isActive(&#39;orderedList&#39;)) <br><br>  }<br><br>  preventEnter(event){<br>    if (event.key === &#39;Enter&#39; || event.keyCode === 13) {<br>      event.preventDefault()<br>    }<br>  }<br><br><br>  disconnect() {<br>    this.element.editor.destroy()<br>  }<br><br>}</pre><p>And finally the SCSS file is here below. <br>You will require a .hidden class available globally with a single attribute : display: none so that stuff can be toggled in and out when needed.</p><pre>.tiptap{  <br>  border: 1px solid #bbb;<br>  border-radius: 3px;<br>  margin: 0;<br>  padding: 0.4em 0.6em;<br>  outline: none;<br>  min-height: 180px;<br><br>  a{<br>    border-bottom: 2px solid black;<br>  }<br>  ul, ol{<br>    padding-left: 1.5rem;    <br>    list-style-position: outside;<br>  }<br>  ul{<br>    list-style-type: disc;<br>  }<br>  ol{<br>    list-style-type: decimal;<br>  }<br><br>  // link<br>  &amp;__link-area{    <br>    padding: 0.2rem;<br>    transition: all 0.2s;<br>    opacity: 1;<br>    display: flex;<br>    gap: 3px;<br>  }<br>  &amp;__link-area--hidden{<br>    display: none;<br>    <br>  }<br>  &amp;__link-field::placeholder{<br>    font-style: italic;<br>    color: hsl(0, 0%, 45%);<br>  }<br>  &amp;__link-field{<br>    border: 1px solid hsl(0, 0%, 70%); <br>    border-radius: 20px;<br>    padding: 0.2rem 1rem;<br>    width: 100%;<br>    font-size: 0.9rem;<br>  }<br>  &amp;__link-field:focus::placeholder {<br>    color: hsla(0, 0%, 0%, 0);<br>  }<br>  &amp;__link-button{<br>    background-color:  hsl(0, 0%, 91.8%) !important;<br>    padding: 0.2rem 0.5rem;<br>    border-radius: 5px;<br>  }<br>  &amp;__link-button:hover{<br>    background-color: hsl(115.6, 73.2%, 78%) !important;<br>    transition: all 0.2s;<br>  }<br><br>  &amp;__container{<br>    display: flex;<br>    flex-direction: column;<br>  }<br><br>  &amp;__textarea{<br>    <br>  }<br><br>  &amp;__toolbar{<br>    padding: 0 1vw;<br>    display: flex;<br>    gap: 2vw;<br>    white-space: nowrap;<br><br>    border-top-left-radius: 0.5rem;<br>    border-top-right-radius: 0.5rem;<br>    background-color: hsl(0deg, 0%, 92%);<br>    padding-top: 5px;<br>    padding-bottom: 5px;<br>    justify-content: start;<br>  }<br><br>  &amp;__button-row{<br>    display: flex;<br>    flex-wrap: nowrap;<br>    overflow-x: auto;<br>    <br>  }<br><br>  &amp;__button-group { <br>    display: flex;<br>    width: min-content;<br>    gap: 2px;<br>  }<br><br>  &amp;__button {<br>    height: 2rem;<br>    padding: 0 0.7vw;<br>    opacity: 0.6;<br>    border-radius: 5px;<br>  }<br><br>  &amp;__button:hover {<br>    opacity: 1;<br>  }<br><br><br>  &amp;__button--selected{<br>    background-color: hsl(53.85, 85%, 49%) !important;<br>    opacity: 1 !important;<br>  }<br><br>}</pre><p>Don’t hesitate to let me know if something is wrong or can be improved. Also I don’t recommend copying the code but rather doing it by yourself in the first place as if you have decided to go the tiptap / ProseMirror way it’s better to have a good grasp on how it works so you can personalize easily later on.</p><blockquote>Note: I haven’t had time to add A11Y and aria tags, if you do so please let me know or add a link to your blog in the comments</blockquote><p>Best</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=97f253d13d0" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Rails direct to S3 uploads with Shrine, Uppy and Sidekiq (with node modules )]]></title>
            <link>https://maxencemalbois.medium.com/rails-direct-to-s3-uploads-with-shrine-uppy-and-sidekiq-with-node-modules-a3b678fb6c75?source=rss-187dc1e4c7c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/a3b678fb6c75</guid>
            <dc:creator><![CDATA[Maxence Malbois]]></dc:creator>
            <pubDate>Sun, 10 Apr 2022 07:59:53 GMT</pubDate>
            <atom:updated>2022-06-05T10:07:23.460Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*DaOqYcEOQH564WRm" /><figcaption>Photo by <a href="https://unsplash.com/@peterneumann?utm_source=medium&amp;utm_medium=referral">Peter Neumann</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In this older story one major problem happened: <strong>huge increase in web server load when image attachment and versionning was performed.</strong></p><p>See the below metrics of my Hobby web dyno at Heroku while versioning thirty 5 megabytes images into 4 different versions:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*c1L5dQN5JuRg9dxvZTs22Q.png" /><figcaption>response times (Hobby dyno)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uJSkAvmeqsowTh462M216A.png" /><figcaption>dyno load (Hobby dyno)</figcaption></figure><p>We can clearly see an application response time skyrocketing to 30 seconds and a dyno load value reaching 7 (while the maximum value is theoretically 1 for such dyno). Our app is getting blasted !!</p><p><strong>It is not really workable this way: web server should not perform heavy tasks. Web servers are dedicated to serve data in high volume, as fast as possible.</strong></p><p><strong>Good practice would be to async the versionning of images</strong>: it will allow for a more tailored backend service in order to attach/version images as efficiently as possible and keep a high performing app.</p><p>Then, the logic is only shared between the view (Uppy) and the web server (Rails/Shrine) while the load is deferred to Amazon and the worker dyno.</p><p>=&gt; the client facing part of our app is not impacted.</p><p><strong>Though there is still one problem</strong>: in my case I have to return to the user a snapshot of the uploaded images straight after it is processed. And because asyncing means sending the versionning to an independent entity, there is no way to return the data of the uploaded image to the view easily.</p><p>We basically have two options : ping a dedicated action that checks if image has been uploaded and saved or use Actioncable.</p><p>We will use the second option here.</p><h3>1 What we need</h3><h4><strong>Shrine</strong></h4><p>Shrine is a Ruby on Rails modular attachment gem developed by <a href="https://github.com/janko">Janko Marohnić</a>. It is highly efficient and tweakable and is a good alternative to ActiveStorage. Basically the attachment data is saved in the same table as the model it belongs to.</p><p>We just need to add an extra column to the model table. We will use the latest version of the gem ie 3.4.0</p><p><a href="https://github.com/shrinerb/shrine">GitHub - shrinerb/shrine: File Attachment toolkit for Ruby applications</a></p><h4>Uppy</h4><p>Uppy is a front end uploading JS library. It is well maintained and offers everything needed: drag n drop of files, thumbnails of images being uploaded, front end validations etc … I am using version 2.7.0</p><p><a href="https://uppy.io/">Uppy</a></p><h4>Sidekiq</h4><p>Sidekiq is a Ruby on Rails gem that allows asyncing of jobs. You can fire Sidekiq in a Heroku Worker dyno. Also if you have your own asyncing library you will easily know how to adapt the below code to your specific case.</p><p>One important thing to know is that Sidekiq allows broadcasting to an ActionCable stream so that the post-attachment data can be returned to the view.</p><p><a href="https://github.com/mperham/sidekiq">GitHub - mperham/sidekiq: Simple, efficient background processing for Ruby</a></p><h3><strong>2 The logic</strong></h3><ul><li>In order to direct-upload the files to Amazon, Shrine will have to ask S3 for an authorization to upload a file (presign) and S3 will return a JSON containing necessary data to upload a single file.</li><li>Uppy will grab this data and upload a file to AWS S3.</li><li>Once upload is complete, AWS S3 will send to the view a Json containing the file location.</li><li>Uppy will format this information to make understandable to Shrine and pass it to a Rails controller for attachment.</li><li>The Rails controller will fire the data to a Sidekiq worker to async the promotion and the versionning of the file.</li><li>The Sidekiq worker will version the image into different new files at S3, get the new data and finally save the attachment data in the newly created record.</li><li>At this point the uploading and attachment of the original file and the processing of the different versions is now complete. We will return necessary data to display a thumbnail of the uploaded image in the view. We do this through ActionCable.</li><li>The users sees every newly uploaded and versionned image appearing sequentially in the view.</li></ul><h3>3 Shrine</h3><h4>shrine setup</h4><p>First, to make Shrine work you need at least 2 gems: Shrineand image_processing . image_processing offers higher levels commands to two different versionners: Imagemagick and Vips. Imagemagick and Vips are binaries that need to be installed on your computer. Recent Heroku stacks already include Imagemagick but you may need to use a buildpack to install Vips.</p><p>Let me now show you my Shrine initializer and comment it for you.</p><p>app/config/initializers/shrine.rb</p><pre>require &quot;shrine&quot;<br>require &quot;shrine/storage/s3&quot;</pre><pre>s3_options = {<br>  bucket:            Rails.application.credentials.aws[:bucket],<br>  access_key_id:     Rails.application.credentials.aws[:s3_user],<br>  secret_access_key: Rails.application.credentials.aws[:s3_password],<br>  region:            &#39;eu-west-1&#39;,<br>}</pre><pre>Shrine.storages = {<br>  cache: Shrine::Storage::S3.new(prefix: &quot;cache&quot;, **s3_options),<br>  store: Shrine::Storage::S3.new(**s3_options),<br>  public_store: Shrine::Storage::S3.new(public: true, upload_options: { cache_control: &quot;public, max-age=15552000&quot; }, **s3_options)<br>}</pre><pre>Shrine.plugin :activerecord<br>Shrine.plugin :cached_attachment_data <br>Shrine.plugin :restore_cached_data <br>Shrine.plugin :determine_mime_type<br>Shrine.plugin :backgrounding<br>Shrine.plugin :remove_invalid<br>Shrine.plugin :presign_endpoint, presign_options: lambda { |request|  <br>  filename = request.params[&#39;filename&#39;]<br>  type     = request.params[&#39;type&#39;]<br>{<br>    content_disposition: &quot;inline; filename=\&quot;#{filename}\&quot;&quot;, <br>    content_type: type, <br>    content_length_range: 0..(25 * 1024 * 1024)<br>  }<br>}</pre><p>s3_options is basically your AWS S3 credentials. Here grabbed through Rails credentials files.</p><p>Shrine_storages this is the logic for storing the files at S3. Basically it uses the S3 options but you can pass any useful data. The prefix can allow you to store specific files in a default folder ( Here the temporary files will be stored in a folder called cache). Also you can specify upload_options (here I have generated an extra store called public_store that will make the files public and add a cache policy). cache and store are required stores. You can add more stores.</p><p>Shrine.plugin Because Shrine is modular you can decide what functions you need.</p><ul><li>Shrine.plugin :activerecord is self explanatory.</li><li>Shrine.plugin :cached_attachment_data is for retaining the cached file across form redisplays.</li><li>Shrine.plugin :restore_cached_data to re-extract metadata when attaching a cached file.</li><li>Shrine.plugin :determine_mime_type helps determine mime type accurately (you can specify any mime type analyzer)</li><li>Shrine.plugin :backgrounding allows attaching in the background. (Yet using Sidekiq is curving round this plugin)</li><li>Shrine.plugin :remove_invalid restore the old attachment in case you replace a file by an invalid file</li><li>Shrine.plugin :presign_endpoint the plugin that will get authorization from S3 to upload a file<br>As you can see there is specific config here. Especially we can’t presign files above 25MB.</li></ul><p>(Actually I am not using all of these plugins in this example)</p><h4>The uploader</h4><p>Ok, now the initializer is fine. We also need to add specific uploading logic: you won’t do the same when uploading a PDF or a JPEG image. Variants will differ etc …</p><p>Here we are dealing with images and will create 2 versions: small and thumb. Then you can create a new folder app/shrine_uploaders where you will store your specific uploading logic.</p><p>Let’s say this image is a profile pic. In this folder, just create a new uploader named profile_pic_uploader.rb and make it look like this :</p><pre>require ‘image_processing/vips’</pre><pre>class ProfilePicUploader &lt; Shrine<br> <br> plugin :validation_helpers<br> plugin :store_dimensions<br> plugin :default_storage, cache: :cache, store: :public_store<br> plugin :derivatives</pre><pre> Attacher.validate do<br>  validate_max_size 5 * 1024 * 1024<br>  validate_mime_type_inclusion [‘image/jpg’, ‘image/jpeg’, ‘image/png’]<br> end</pre><pre>def generate_location(io, record: nil, name: nil, derivative: nil, **)<br> record_model = record.class.name.to_s.downcase<br> name = super # the default unique identifier</pre><pre> [record_model, name].compact.join(‘/’)<br>end</pre><pre> Attacher.derivatives do |original|<br> <br>  pipeline = ImageProcessing::Vips.source(original)<br> { <br>  small: pipeline.saver(quality: 80, strip: true).resize_to_fill!(400, 400, sharpen: false), <br>  thumb: pipeline.saver(quality: 80, strip: true).resize_to_fill!(200, 200, sharpen: false)<br> }<br> end<br>end</pre><p>We use the :default_storage plugin to tell Shrine to make the uploaded images public at S3 (anyone can access them) by using the public_store that we defined in the initializer.</p><p>The Attacher.validate block allows validation of the uploaded image.</p><p>The generate_location block allows to create specific paths for specific models.<br>Especially putting files in specific folders. Otherwise the files from all models will be stored together into the S3 bucket. (Dont forget that S3 folders are conceptual: files are actually dumped alltogether into the same bucket).</p><p>The Attacher.derivatives block creates the different variants of the original image. The syntax is documented in image_processing gem and is the same whether you use Imagemagick or Vips.</p><h3>3 The Rails Setup</h3><h4>The model</h4><p>Ok now it’s time to implement the above uploader to a Rails model.</p><p>Let’s call our model ProfilePic , we will include the above uploader to this model class.</p><p><strong>models/profile_pic.rb</strong></p><pre>class ProfilePic &lt; ApplicationRecord<br>  include ProfilePicUploader::Attachment.new(:image)<br>end</pre><p>As you can see, we create a virtual attribute :image that will be known to Shrine.</p><p>Then in our migration we will need to add a column image_data (the same name as our virtual attribute + _data suffix) that can be either of text, json or jsonb type.</p><p>In this new column will be stored all related data to our :image attachment: versions locations, store, sizes etc ..</p><h4>The presign endpoint</h4><p>Then we need a presign endpoint in order to generate S3 auth for our files. This can be done in routes.rbthis way :</p><pre>authenticate :pro do<br> # dont forget to pass filename and type<br> mount Shrine.presign_endpoint(:cache) =&gt; “presign/s3/params” <br> mount Shrine.presign_endpoint(:cache) =&gt; “presign” <br>end</pre><p>As you can see here, only authenticated users from model Pro can use this presign endpoint. You can make it available to everyone if you wish.</p><p>If you need authorization on top of authentication you can do that on a per controller basis, just check here : <a href="https://shrinerb.com/docs/plugins/presign_endpoint#calling-from-a-controller">https://shrinerb.com/docs/plugins/presign_endpoint#calling-from-a-controller</a></p><p>Also there is a little quirk here as there is two routes : “presign/s3/params” and “presign”. It is just because our front end library Uppy slightly changes the route to add “s3/params”. So it is not always required to do such.</p><p>When pinging this presign endpoint with a get method, we need to pass filename and filetype (in query string) or an error will be triggered.</p><p>Also some front end library can make a preflight request with the verb options . Don’t ask me too much about the preflight request but this is a kind of request querying the right verb to use. Uppy does that but will default to the get verb if preflight request is not answered.</p><p>Just to make things clean, just add this route to your routes.rb :</p><pre>match ‘*all’, controller: ‘application’, action: ‘cors_preflight_check’, via: [:options]</pre><p>And add the following to your application_controller.rb :</p><pre>def cors_preflight_check<br> return unless request.method == ‘OPTIONS’<br> cors_set_access_control_headers<br> render plain: ‘’<br>end</pre><pre>private</pre><pre>def cors_set_access_control_headers<br> response.headers[‘Access-Control-Allow-Origin’] = ‘*’<br> response.headers[‘Access-Control-Allow-Methods’] = ‘POST, GET, PUT, PATCH, DELETE, OPTIONS’<br> response.headers[‘Access-Control-Allow-Headers’] = ‘Origin, Content-Type, Accept, Authorization, Token, Auth-Token, Email, X-User-Token, X-User-Email, x-xsrf-token’<br> response.headers[‘Access-Control-Max-Age’] = ‘1728000’<br> response.headers[‘Access-Control-Allow-Credentials’] = true<br>end</pre><h3>4 Uppy</h3><p>Ok now we have do deal with Uppy, our front end uploading library. Uppy offers a wealth of different methods to add files. Of course drag’n’drop. But also specific elements. I like Dashboard a lot. It is like a box that shows a thumbnail of the images you have dropped into and the uploading success or error messages.</p><p>Uppy is highly tweakable and can match basically most possible use cases imaginable. So if you need it to have a specific behavior you may dive in Uppy’s documentation to achieve this.</p><p>Uppy will attach to any html element. We will use a simple empty div :</p><pre>&lt;div&gt;&lt;/div&gt;</pre><p>Though we will tweak it a bit with data attributes that will match all our use cases. Also this method is convenient to use Uppy on different pages with different config.</p><pre>&lt;div id=&quot;uppy&quot; data-uppy-gui=&quot;drag&quot; data-form-type=&quot;uppy&quot; data-uppy-instance=&quot;uppy-profile-pic&quot; data-max-file-size=&quot;5&quot; data-presign-url=&quot;/presign&quot;, data-post-url=&quot;&lt;%= profile_pics_path %&gt;&quot;, data-allowed-files=&quot;image/jpg,image/jpeg,image/png&quot; data-rails-model=&quot;profile_pic&quot; data-image-attribute=&quot;image&quot;&gt;&lt;/div&gt;</pre><p>All these data attributes have been set by me. You may use Uppy in a different way. But let me explain what they do :</p><ul><li>data-uppy-gui : just the graphical user interface we want Uppy to use, either dashboard or dragndrop.</li><li>data-form-type : just a hook for Uppy</li><li>data-uppy-instance : own Uppy internal ID that must be different if multiple uppies happen on the same page</li><li>data-max-file-size : the maximum file size that should match our uploader we have set earlier, so that there is no mismatch between front and back</li><li>data-post-url : just the route to our rails app that will promote and version the files (asynchly with Sidekiq). This route implies we have a ProfilePics controller. The Post method is not present here as hard written in the Js (see below)</li><li>data-allowed-files : the file-types accepted (that should again matches our Shrine Uploader config set earlier)</li><li>data-parent-model : the model this image attachment will depend to. We said earlier it is ProfilePic model.</li><li>data-self-model : the attachment name</li></ul><p>Ok now let’s add Uppy. Just do a yarn add uppy in your rails console. This will generate node modules for the latest version of Uppy. You can see that in your package.json file.</p><p>I am using Jsbundling with Esbuild but it works with Jsbundling and any bundler (rollups, webpack..). With Webpacker 5 gem I have experienced issues with the latest versions of Uppy that would not bundle. (Don’t ask me why, webpacker is a weird beast now deprecated and I feel it was better to switch to JSbundler than to debug everything).</p><blockquote>If your app with webpacker doesn’t want to bundle the latest version of Uppy, just use an Uppy 1.x.x version. It has a few security flags but should work. Also there are a few Uppy tweaks, just check Uppy doc for Upgrading to 2.x.x. and use it the other way (I just hope you don’t have to deal with that to be honest)</blockquote><p>Then create file in Javascript folder (or a pack if you use Webpacker) :</p><pre>import Uppy from ‘<a href="http://twitter.com/uppy/core">@uppy/core</a>’<br>import XHRUpload from ‘<a href="http://twitter.com/uppy/xhr-upload">@uppy/xhr-upload</a>’<br>import Dashboard from ‘<a href="http://twitter.com/uppy/dashboard">@uppy/dashboard</a>’<br>import DragDrop from ‘<a href="http://twitter.com/uppy/drag-drop">@uppy/drag-drop</a>’<br>import Informer from ‘<a href="http://twitter.com/uppy/informer">@uppy/informer</a>’<br>import AwsS3 from ‘<a href="http://twitter.com/uppy/aws-s3">@uppy/aws-s3</a>’</pre><pre>import French from ‘<a href="http://twitter.com/uppy/locales">@uppy/locales</a>/lib/fr_FR’<br>import American from ‘<a href="http://twitter.com/uppy/locales">@uppy/locales</a>/lib/en_US’</pre><pre>import ‘<a href="http://twitter.com/uppy/core">@uppy/core</a>/dist/style.css’<br>import ‘<a href="http://twitter.com/uppy/dashboard">@uppy/dashboard</a>/dist/style.css’</pre><pre>function initUppy(uppy_node){</pre><pre>var uppy = new Uppy({<br> id: uppy_node.dataset[“uppyInstance”],<br> autoProceed: true, <br> allowMultipleUploadBatches: true,<br> locale: French,<br> restrictions: {<br> maxFileSize: uppy_node.dataset[“maxFileSize”]*1024*1024,<br> maxNumberOfFiles: null,<br> minNumberOfFiles: null,<br> allowedFileTypes: uppy_node.dataset[“allowedFiles”].split(“,”)<br> }<br> })</pre><pre>if(uppy_node.dataset[“uppyGui”] == “dash”){</pre><pre>uppy.use(Dashboard, {<br> target: uppy_node,<br> inline: true,<br> replaceTargetContent: true,<br> showProgressDetails: true,<br> width: 1200,<br> height: 400, <br> })</pre><pre>} else {</pre><pre>uppy.use(DragDrop, { <br> target: uppy_node,<br> width: ‘100%’,<br> height: ‘100%’,<br> note: null<br> })</pre><pre>}</pre><pre>uppy.use(AwsS3, { <br> companionUrl: uppy_node.dataset[“presignUrl”], <br> })</pre><pre>uppy.on(‘upload-success’, function (file, data) {</pre><pre>// construct uploaded file data in the format that Shrine expects<br> var uploadedFileData = JSON.stringify({<br> id: file.meta[‘key’].match(/^cache\/(.+)/)[1], // object key without prefix<br> storage: ‘cache’,<br> metadata: {<br> size: file.size,<br> filename: file.name,<br> mime_type: file.type,<br> }<br> })</pre><pre>fetch(uppy_node.dataset[“postUrl”], {<br> method: “POST”,<br> credentials: “same-origin”,<br> body: JSON.stringify({ <br> // square brackets are used to make the value of data attribute as JSON attribute<br> [uppy_node.dataset[“railsModel”]]: {<br> [uppy_node.dataset[“imageAttribute”]]: uploadedFileData<br> }<br> }),<br> headers: {<br> ‘X-CSRF-Token’: document.querySelector(‘meta[name=”csrf-token”]’).content,<br> ‘Content-Type’: ‘application/json’ <br> }, <br> })<br> .then(response =&gt; response.json())<br> .then(data =&gt; {</pre><pre>});</pre><pre>})<br>}</pre><pre>function initAllUppies(){<br> const allUppies = document.querySelectorAll(“[data-form-type=’uppy’]”);<br> allUppies.forEach(function(one_uppy){<br> initUppy(one_uppy);<br> })<br>}</pre><pre>if (document.body.contains(document.querySelector(“[data-form-type=’uppy’]”))) { <br> initAllUppies();<br> };<br></pre><p>Ok let’s explain this big block :</p><p>First 6 lines is just us importing all required Uppy modules from our node modules.</p><p>In the nextlines we also import 2 translations : FR and US. Be careful, the translations are part of Uppy but not necessarily added with it. You may also need to yarn add @uppy/locales .</p><p>Then at the bottom of the block there are two functions that will initiate each of our html blocks that contains data-form-type=&#39;uppy&#39; . We pass the uppy node to the Uppy initializer.</p><p>Then we have our real Uppy initializer: function initUppy(uppy_node) . I let you digest it but it is not that complex. Mostly we grab the data-attributes from our original html div and use them to populate Uppy with.</p><p>few explains:</p><ul><li>French : Yeah sorry it defaults to french. I ave not changed my code to pick american depending of the locale. But this is easy to do. Just add an extra data-attribute to the original Uppy html div to pick the right locale. You may need to rename French to “fr” and American to “en” in this case.</li><li>maxFileSize and AllowedFileTypes : we just grab the dataset attributes and fill it appropriately</li><li>Then we have a clever selector between Uppy Dashboard and Uppy Drag’n’drop</li><li>Then the companion part: this is where we ping the presign to get the authorization to upload from S3. the route is the one we have defined earlier. Uppy will add “s3/params” to it and also add the filename and filetype. <br>By the way, to verify your presign route is correct you can replicate that in your browser navbar: presign/s3/params?name=whatever&amp;type=png and you should see a beautiful json with S3 authing data.</li><li>now we grab the message sent by amazon when our file is uploaded and format it fine for Shrine. It is the // construct uploaded part</li><li>then we use this data to pass it to our Rails app for promotion of the file. <br>Shrine will just create our ProfilePic record and we will attach this image details (no file is present here, it is just information about where the file is located at S3, as we have directly uploaded to S3, our app doesn’t deal with files yet)<br>As you can see the param received by our Rails app will be [:profile_pic][:image] which is perfectly matching rails convention : model and model field. As Shrine has created an :image attribute in the ProfilePic model we are all fine. (we will populate the image field, not image_data field)</li></ul><p>I let you check Uppy doc in case you need to understand some specific uppy settings but they are self-explanatory : autoProceed, allowMultipleUploadBatches etc …</p><blockquote>Also as you can see there is CSS files. It should end up in your builds folder if you use JSbundling Rails. You need to add both uppy.js and uppy.css to every page using Uppy. With stylesheet_link_tag , javascript_include_tag etc…</blockquote><h3>5 creation of the image</h3><p>I hope you are still with me. <br>We are nearly done. Image has been uploaded to S3 with the cache store settings (see ProfilePicUploader) and we have received all the info from Amazon about where it sits. We have transformed that data to a good param for our rails app. Now we just have to create the record in a Sidekiq worker and stream back to the view with ActionCable.</p><p><strong>profile_pics_controller.rb :</strong></p><pre>class ProfilePicsController &lt; ApplicationController</pre><pre>def create</pre><pre>   ProfilePicWorker.perform_async(image_details: params[:profile_pic][:image])</pre><pre>end</pre><p><strong>profile_pic_worker.rb</strong></p><pre>class ProfilePicWorker<br> include Sidekiq::Workersidekiq_options queue: ‘critical’<br> sidekiq_options retry: false</pre><pre> def perform(image_details)<br>  profile_pic= current_pro.profile_pic.new(image:image_details)<br></pre><pre>  if profile_pic.valid?<br>   profile_pic.image_derivatives!<br>   profile_pic.save</pre><pre>   // Here some Actioncable to Stream back to the view <br> <br>  end<br> end<br>end</pre><p>And Voila.</p><p>Yet a few notes. Profile pic is a singular resource (yep sorry this might be confusing but I still use plural named controllers even when having singular resource in routes.rb : resource profile_pics). That’s why I don’t need to pass any details about the parent of ProfilePic. I just grab Devise helper current_pro . But in case you want to create records with an image attachment from a plural resource model, just change the upload path in the Uppy html and change data-post-url=&quot;&lt;%= profile_pics_path %&gt;&quot; to data-post-url=&quot;&lt;%= profile_pics_path(@parent) %&gt;&quot; . And your Sidekiq worker will need a reference to the parent too.</p><p>Also I realised that a profile pic is the few cases that may require to do the creation in the controller to make sure it is rendered back to the view asap. You can do that too. Just scrap the Sidekiq worker and do that in the controller. But your app needs to have room for this. As it will block a full Puma Worker while doing the processing. <br>Here it is not a big deal as there are only 2 versions but it may takes longer ..</p><p>Then you may not need to explicitely pass params[:profile_pic][:image] to the image attribute but rather whitelist image.</p><p>I hope this thread is useful to you. And now you have an Uppy.js file that can be used in a wide variety of cases thanks to the data attributes on the html element.</p><p>Best</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a3b678fb6c75" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Understanding image types and optimizing them]]></title>
            <link>https://maxencemalbois.medium.com/images-for-the-web-developer-98b084ff9378?source=rss-187dc1e4c7c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/98b084ff9378</guid>
            <category><![CDATA[image]]></category>
            <category><![CDATA[image-compression]]></category>
            <category><![CDATA[image-processing]]></category>
            <dc:creator><![CDATA[Maxence Malbois]]></dc:creator>
            <pubDate>Wed, 17 Nov 2021 18:30:08 GMT</pubDate>
            <atom:updated>2022-04-10T08:10:53.233Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*xf6fHnylywCDZeCv" /><figcaption>Photo by <a href="https://unsplash.com/@lazycreekimages?utm_source=medium&amp;utm_medium=referral">Michael Dziedzic</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Handling images when you are a web developer is not an easy task. Speed of connection to the internet is constantly improving yet it is still important to deliver optimized images. Because some users may still experience getting a decent connection speed or sometimes smartphones can only access degraded connection.</p><h3>The big picture</h3><p>Among all the different types of images available for the webdev, there are mostly 2 main families: bitmap and vector.</p><h4>Bitmap images</h4><p>The bitmap images is the most common image type: just a sequence of dots (or pixels) recorded independently.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/538/1*U5bnqssgNFOAnfDJ0HauNQ.jpeg" /><figcaption>9 pixels</figcaption></figure><p>Each pixel holds enough data for the software that displays it (image viewer, browser..) to recompose the final color properly.</p><p>The data each pixel holds depend of the color model. For displaying on a computer screen (or a phone screen), the color model often used is RGB which stands for RED — GREEN — BLUE (basically primitive colors).</p><blockquote>A lot of different color models exist and have their own use. For example the model CMYK which stands for CYAN — MAGENTA — YELLOW — BLACK is the preferred model for printing on paper. Why ? Just because press printers have a slot for black ink which allows to have deep blacks printed on paper.</blockquote><p><em>“ Ok ! So now we know bitmap images are made of pixels in RGB color model. But what does it mean practically ? And how does it translate into my computer memory ? “</em></p><p>Well, if you have some knowledge of computers, you probably know that the storage capacity of a computer is counted in Bytes. And 1 Byte is a sequence of 8 bits.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*FrwVz8YiQmj9b4f5k6LSuw.png" /><figcaption>one byte</figcaption></figure><p>Each bit value can be 1/0 or True/False.. It is basically binary.. Then One Byte can take any value between 0 (8 bits with a value of 0) to 255 (8 bits with a value of 1).</p><p>Basically to know the possible values of single Byte, just multiply each bit possible result by every other bit possible value : 2*2*2*2*2*2*2*2 = 256</p><p><strong><em>— The depth of an image —</em></strong></p><p><em>“ Ok ! So a single pixel is made of Bytes ! And Bytes have 256 different values. But how does this translates into different shades of colors ? “</em></p><p>Well this is easy : one pixel assigns a single Byte to each channel of the color mode it belongs to.</p><p>We said earlier images used on screens were using RGB color mode, then a single byte is assigned to each composite color of the color mode:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*aGVS1zud4HT_FkBYAKx6ng.png" /><figcaption>red green and blue bytes</figcaption></figure><p>Then now you understand something :</p><ul><li>there can be 256 shades of red</li><li>256 shades of green</li><li>and 256 shades of blue</li></ul><p>Which gives 256*256*256 = 16777216 different colors !!</p><p>Yes, with only 256 shades of each primitive colors, we can make more than 16.5 million colors. <strong>Then the depth of the image is “3 bytes of 8 bits” or “24 bits”.</strong></p><p><strong>— RGBA or 32 bits —</strong></p><p><em>“ Right this is very interesting but sometimes I see a different color mode: RGBA ! And sometimes I see images with a depth of 32 bits ? ”</em></p><p>RGBA color mode is just about stretching the above concept. It adds a fourth byte that stands for “ALPHA” or “Alpha channel” which basically is the transparency of the pixel.</p><p><em>“ Why do you talk about transparency ? Why do I need it ? “</em></p><p>Well, transparency is something that has appeared with Image Editing decades ago and is very important today.</p><p>Let’s say you are building a banking website with a certain CSS background color. Go for grey! And now you want to add a nice image of a credit card as an illustration. But your credit card has been photographed laying on a white table … It won’t really fit in..</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*QsZTPVtp_3zZa8Q_" /><figcaption>Photo by <a href="https://unsplash.com/@markuswinkler?utm_source=medium&amp;utm_medium=referral">Markus Winkler</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Now open your image editor and remove the background :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/984/1*qWkco9A_Mjk9mxeC9sPe-g.png" /><figcaption>With transparency</figcaption></figure><p>By removing the background, you have just pushed the value of the ALPHA byte to 255 to the pixels you have “erased”. And the edges of the credit card probably have a transparency between 0 and 255 so it blends smoothly with the grey background of your website.</p><p>You can now change your website CSS background color at will. Switch from grey, to white, to yellow: the cutout image will alway look good. With the surrounding of the credit card being filled up by your website CSS background color value.</p><p>Basically your browser will compute what it displays by calculating the different layers of images overlapping each other and accounting for their Alpha channels.</p><p>Now You have learnt about Bitmap images. We will cover compression later .. It’s time for vector images now !!</p><h4>Vector images</h4><p>Vector images is a completely different beast. Here we have no dot or pixel. The vector image is a sequence of instructions that is repeated by the software that displays the image.</p><p>The instructions of a vector image could translate in human language to something like :</p><p>“ draw a line with black ink starting on top of page and going south for 15 millimeters. Then draw a circle with a red ink with a diameter of 30 millimeters and a fill color as pink…. etc …”</p><p>As you can imagine a vector image doesn’t look at all like a bitmap image. It rather looks like a cartoon, a sign, a drawing, a sketch …</p><p>Here is an example of a vector image (yet converted to a bitmap image as Medium website doesn’t accept vector images to be used in Medium articles) :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/300/1*_xF8-ynNPjgeGDDruQ4rxA.png" /><figcaption>a lovely plane</figcaption></figure><p><em>I have added a pink background to this lovely plane so you can see it better.</em></p><p>and here is the full code for this plane:</p><pre>&lt;?xml version=”1.0&quot; encoding=”UTF-8&quot; standalone=”no”?&gt;<br>&lt;! — Created with Inkscape (<a href="http://www.inkscape.org/">http://www.inkscape.org/</a>) →</pre><pre>&lt;svg<br> xmlns:dc=”<a href="http://purl.org/dc/elements/1.1/">http://purl.org/dc/elements/1.1/</a>&quot;<br> xmlns:cc=”<a href="http://creativecommons.org/ns#">http://creativecommons.org/ns#</a>&quot;<br> xmlns:rdf=”<a href="http://www.w3.org/1999/02/22-rdf-syntax-ns#">http://www.w3.org/1999/02/22-rdf-syntax-ns#</a>&quot;<br> xmlns:svg=”<a href="http://www.w3.org/2000/svg">http://www.w3.org/2000/svg</a>&quot;<br> xmlns=”<a href="http://www.w3.org/2000/svg">http://www.w3.org/2000/svg</a>&quot;<br> xmlns:sodipodi=”<a href="http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd">http://sodipodi.sourceforge.net/DTD/sodipodi-0.dtd</a>&quot;<br> xmlns:inkscape=”<a href="http://www.inkscape.org/namespaces/inkscape">http://www.inkscape.org/namespaces/inkscape</a>&quot;<br> width=”33.660679mm”<br> height=”20.716875mm”<br> viewBox=”0 0 33.660679 20.716875&quot;<br> version=”1.1&quot;<br> id=”svg1227&quot;<br> inkscape:version=”0.92.3 (2405546, 2018–03–11)”<br> sodipodi:docname=”avion1.svg”&gt;<br> &lt;defs<br> id=”defs1221&quot; /&gt;<br> &lt;sodipodi:namedview<br> id=”base”<br> pagecolor=”#ffffff”<br> bordercolor=”#666666&quot;<br> borderopacity=”1.0&quot;<br> inkscape:pageopacity=”0.0&quot;<br> inkscape:pageshadow=”2&quot;<br> inkscape:zoom=”0.35&quot;<br> inkscape:cx=”-539.24641&quot;<br> inkscape:cy=”67.721442&quot;<br> inkscape:document-units=”mm”<br> inkscape:current-layer=”layer1&quot;<br> showgrid=”false”<br> fit-margin-top=”0&quot;<br> fit-margin-left=”0&quot;<br> fit-margin-right=”0&quot;<br> fit-margin-bottom=”0&quot;<br> inkscape:window-width=”1920&quot;<br> inkscape:window-height=”944&quot;<br> inkscape:window-x=”1920&quot;<br> inkscape:window-y=”27&quot;<br> inkscape:window-maximized=”1&quot; /&gt;<br> &lt;metadata<br> id=”metadata1224&quot;&gt;<br> &lt;rdf:RDF&gt;<br> &lt;cc:Work<br> rdf:about=””&gt;<br> &lt;dc:format&gt;image/svg+xml&lt;/dc:format&gt;<br> &lt;dc:type<br> rdf:resource=”<a href="http://purl.org/dc/dcmitype/StillImage">http://purl.org/dc/dcmitype/StillImage</a>” /&gt;<br> &lt;dc:title&gt;&lt;/dc:title&gt;<br> &lt;/cc:Work&gt;<br> &lt;/rdf:RDF&gt;<br> &lt;/metadata&gt;<br> &lt;g<br> inkscape:label=”Layer 1&quot;<br> inkscape:groupmode=&quot;layer&quot;<br> id=”layer1&quot;<br> transform=”translate(-107.9018,-146.03442)”&gt;<br> &lt;g<br> id=”g1206&quot;<br> transform=”matrix(0.35277777,0,0,-0.35277777,84.412131,243.85155)”&gt;<br> &lt;path<br> d=”m 88.7091,262.6309 -22.1242,14.646 95.4161,-0.266 z”<br> style=”fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none;stroke-width:0.1&quot;<br> id=”path50&quot;<br> inkscape:connector-curvature=”0&quot; /&gt;<br> &lt;path<br> d=”m 88.7091,262.6309 -8.2969,-20.031 15.1684,9.977 66.4204,24.434 -73.2919,-14.38&quot;<br> style=”fill:#f6f2d8;fill-opacity:1;fill-rule:nonzero;stroke:none;stroke-width:0.1&quot;<br> id=”path52&quot;<br> inkscape:connector-curvature=”0&quot; /&gt;<br> &lt;path<br> d=”m 80.4122,242.5999 19.2426,1.503 -4.0742,8.474 z”<br> style=”fill:#dcd9c0;fill-opacity:1;fill-rule:nonzero;stroke:none;stroke-width:0.1&quot;<br> id=”path54&quot;<br> inkscape:connector-curvature=”0&quot; /&gt;<br> &lt;path<br> d=”m 95.5806,252.5769 10.6613,-34.025 55.7591,58.459 z”<br> style=”fill:#ffffff;fill-opacity:1;fill-rule:nonzero;stroke:none;stroke-width:0.1&quot;<br> id=”path56&quot;<br> inkscape:connector-curvature=”0&quot; /&gt;<br> &lt;/g&gt;<br> &lt;/g&gt;<br>&lt;/svg&gt;</pre><p>As you can see it is real instructions !! On the opposite, if you try to read a bitmap image into a text reader, you will only get funny characters or gibberish as it is not intended to be read by a human.</p><p>A vector image is perfectly readable and understandable. Vector images are really just instructions !</p><p>Actually it is going a step further : this “plane” vector image is of SVG format and you can see that it really looks like HTML : it embeds tags like &lt;svg&gt;&lt;/svg&gt;&lt;path/&gt; etc ..<br>Then, instead of adding your vector image as an image file such as <strong>plane.svg </strong>in-between HTML “image” tags, <strong>you can just pick some elements and add them in your HTML straight away!!</strong></p><p>Pretty insane !!! :)<br>You can even add some CSS to them: animation, transparency, rotation …</p><h3>A quick comparison</h3><p>In the real world, vector images will rather be used as small illustrations and Bitmap images for real life pictures (like taken with a proper camera or your smartphone camera).</p><p>Regarding the limitations and advantages :</p><h4>vector images</h4><ul><li>vector images can be pretty small if the illustration is simple.</li><li>they can be scaled at will without losing any quality : there is no pixel</li><li>they are highly integratable into HTML</li></ul><h4>Bitmap images</h4><ul><li>they are used for real life pictures</li><li>they can be sometimes very big, it is important to compress them</li><li>they are complementary to vector images: just different uses …</li></ul><p>Ok the tricky part now : you know the two different possible types of images but we also want to make your website faster. Then we need to optimize and discriminate.</p><h3>Vector file types optimization</h3><p>— file types —</p><p>Vector images can come in various flavors. We have learnt they are just instructions but the software that you use may “package” these instructions into different formats.</p><p>The open source extensions for vector images are : .svg .eps .ps or even .html.</p><p>And a beautiful open source software to create vector images is for example <a href="https://inkscape.org/">Inkscape</a>. Don’t hesitate to check it.</p><p><strong>At the moment 99.8% of browsers can read .svg files </strong>( <a href="https://caniuse.com/?search=svg">https://caniuse.com/?search=svg</a> )</p><p>Copyrighted file formats can be : .ai (Adobe illustrator), .cdr (Coreldraw) , and probably many others ..</p><p>These formats cannot be read by browsers as far as I know. So if you are using Adobe Illustrator to make images for the web, just export to an open source format such as SVG. (widely supported by browsers)</p><p>— optimization —</p><p>Vector files usually don’t need optimization. They are fairly small. But it is still possible to remove all the overhead. One good tool to do such is <a href="https://jakearchibald.github.io/svgomg/">SVGOMG</a>. SVGOMG will trim everything that is not an image instruction to make your file as light as possible.</p><p>Also there is the case you have a very complex drawing : lots of elements, lots of colors, lots of pen drawings … Then it is a good thing to compare the size of the vector file to its Bitmap equivalent.</p><p>Though because vector files can be scaled at will without changing their weight, it also depends of the dimensions of the final image you need. If you have a 100KB vector file. Converting it into a 150px * 150px bitmap image will probably produce a much smaller file. But if you need a bitmap of 1500px * 1500px then your bitmap will be much heavier …</p><p>Then always try to convert any complex SVG (bigger than 70KB) to its Bitmap equivalent.</p><h3>Bitmap file types and optimiziation</h3><p>Ok now take a deep breath: this is the tricky part of Bitmap file formats and optimization :D</p><p>I have explained you that a bitmap image is made of pixels. Those pixels are a blend of primitive colors: 3 RGB Bytes or 4 RGBA bytes.</p><p>But all Bitmap file formats don’t work like that. Also imagine a bitmap image of 200px * 200px in size with a thin red border of 1 pixels all around it.</p><p>Basically, the top red border will be the same 200 pixels : { Red: 255, Blue: 0 , green: 0} or { Red: 255, Blue: 0 , Green: 0, Alpha: 0}.</p><p><em>“ 200 hundred times the same data written next to each other ? is it efficient ?”</em></p><p>Not really. It would indeed be better to write in our image : “repeat 200 times the pixel with those colors : { Red: 255, Blue: 0 , Green: 0, Alpha: 0}” and we would save a lot of memory space!!</p><p>This is compression ! (though compression algorithms are much more advanced than this)</p><p>So basically all Bitmap files on the internet nowadays are compressed. The original format .bmp is not really used anymore. <br>Some algorithm can even compress more than what we expect .. but then the image quality become degraded (we call it a lossy compression).</p><p>So let’s browse all file formats :</p><blockquote>Disclaimer : I try to give general information as close to possible to what they really are. But some file formats are very deep and highly twistable. Please bear with me in case some information is not accurate. Also the big lines should be right.</blockquote><h4>— BMP</h4><p>As far as I know this one has little or no compression. Images are very heavy. <a href="https://en.wikipedia.org/wiki/BMP_file_format">https://en.wikipedia.org/wiki/BMP_file_format</a><br>Yet the wiki says some compression can be added and also it can have an alpha channel. (see wiki link to examine all the different available image depths and options)</p><p>Not much used nowadays.</p><h4>— JPG or JPEG</h4><p>Probably the most popular bitmap image format for the past couple of decades. The compression algorithm is destructive: it will degrade the image quality (you won’t be able to recover initial image quality by no mean) and add artifacts.</p><p>Artifacts are tiny glitches in the image, especially in contrasty areas : around object edges..</p><p>The compression algorithm is quite complex and work with chunks (tiles) of the original images.</p><p><strong><em>=&gt; This is a very good format for the web. Noteworthy you can select the quality value (or compression amount ) you desire and find the proper balance between image weight and image quality. But be carefull: a JPEG image with a quality of 90% may give different sizes depending on the software you use. The quality value is relative, not only to the JPEG algorithm, but also to the software that will compress.</em></strong></p><h4>— PNG</h4><p>PNG is non destructive. It’s compression algorithm is very good and decrease a BMP image by a lot without actually changing the image at all.</p><p>PNG is great for working copies : an image you will save on your computer to work on and alter later.</p><p>But for the web the weight of an PNG image may still be too high for a speedy delivery of your HTML pages.</p><h4>— TIFF</h4><p>The preferred format for photographers. It is lossless of course. And very advanced. <br>I told you earlier that a Byte is made of 8 bits. And our RGB image has one Byte per color channel. <br>TIFF format can accept channels made of 16 bits !!!! <a href="https://photo.stackexchange.com/questions/13893/is-tiff-really-of-higher-quality-than-png-24">https://photo.stackexchange.com/questions/13893/is-tiff-really-of-higher-quality-than-png-24</a><br>Which means we not only get 16 millions colors per pixel but billions of colors !! (Also your eye can’t probably see them, and modern monitors can’t even show them except very expansive ones)</p><p>It is loved by photographers because cameras now have sensors so sensitive they can have 14 bits per channels. Also when you do HDR or image enfusing you may need 16 bits channels to edit an image. Even if your output image will be 8 bits per channel ..</p><p>So it is like a “Deluxe working copy” of an image. (TIFF images can be Gigabytes big ! whereas we web developers, try to make our images as small as possible and our images are most often below 100KB)</p><h4>— GIF</h4><p>Gif is a different beast as each pixel is not assigned 3 or 4 different channels. Instead it works with a palette.</p><p>When you try to compress an image into GIF, the compression software will create a palette of 256 of the most used colors in the image.</p><p>And each pixel, instead of having a shade of blue, a shade of green and a shade of red, will be assigned a single color from the palette.</p><p>This allows GIF to be a rather well compressed format ( only one Byte is necessary for each pixel )</p><p>Also there is the possibility to have a transparency value. Yet this is not an alpha channel (with 256 gradients of transparency) but rather a boolean value: transparent — or not transparent.</p><p>It is a good format for images with splats of colors.</p><h4>— PNG 8bit</h4><p>PNG is a bit like GIF. It works with a palette of 256 colors but can have a real alpha channel. Also the compression algorithm is much better than GIF. Prefer PNG 8 bit to GIF.</p><p><strong><em>=&gt; It is excellent for logos and simple drawings. It can produce extremely light files while preserving quality.</em></strong></p><p>(Be careful PNG 8 bit is destructive as it shrinks the number of colors to 256 when PNG 32 bits is a lossless format)</p><h4>— WEBP</h4><p>WEBP is a fairly recent format with incredible compression ratios. Some say it is 25% lighter than any JPEG for similarly looking compressed images.</p><p>It is indeed very good. And has less artifacts than JPEG. But it drastically smoothens out details .. So you better prefer JPEG for some images and use WEBP in some other cases.</p><p>WEBP is fairly well supported by browsers.<br>Approximately 95% of browsers can display them : <a href="https://caniuse.com/?search=webp">https://caniuse.com/?search=webp</a></p><p><strong><em>=&gt; a contender to JPEG replacement</em></strong></p><h4>— Other formats</h4><p>There are a wealth of newer file formats in the pipes: HEIF, AVIF, JPEGXL with their qualities and drawbacks.. I will not dive too much into them at the moment as they are not fully supported by browsers at the moment (If at all).</p><p>So I keep this section open for more additions and reviews of these new formats.</p><h4>now WEBP vs JPEG, who wins ?</h4><p>As Isaid WEBP is being considered the clear winner “on the paper”. It makes files smaller than JPEG.</p><p>Though compression algorithm doesn’t give similar results :</p><ul><li>webp : smoothen out detail</li><li>jpeg : adds some artifacts</li></ul><p>In order to help you decide, here is a quick comparison between an original image in the <strong>PNG format</strong> (that Medium may compress into Jpeg 90%) with a weight of 325KB, a <strong>Jpeg with a quality of 50%</strong> and a weight of 21 KB, and finally a <strong>Webp with a quality of 70%</strong> and a similar weight of 21 KB.</p><p>All images are originally 400px * 400px and stitched together into a single image.</p><p><em>( I have not taken the same quality for both JPEG and WEBP as I think taking pictures with a similar weight is the fairest )</em></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Alzp4lGuAkGibYIFnWfM_Q.png" /><figcaption>original 321 KB — JPEG 50% 21 KB — WEBP 70% 21 KB</figcaption></figure><blockquote>edit: click on the above image to enlarge it before checking fine details between images. But basically Medium compression kinda kills the ability to compare.</blockquote><p>First thing we see is that all images are rather acceptable, but The two images on the right are 15 times lighter in weight than the original.</p><p>here is a crop of the JPEG and WEBP 32 px * 32px around the flowers :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/720/1*sn_xRsCXslqDN1nMon5tnA.png" /><figcaption>JPEG detail — WEBP detail</figcaption></figure><p>As you can see it is clear that JPG adds a lot of artifacts (but still try to preserve original details as much as possible) and WEBP smoothens out details without adding artifacts. <br>If you have a look at the flower branch on the right, the WEBP version is much cleaner but shows only shades of RED / BROWN as the JPEG image still shows some GREENISH colors.</p><p>For this very image WEBP looks cleaner. But sometimes you might try JPEG or PNG 8bit, and play with compression quality to get what you desire.</p><h3>Q &amp; A</h3><p><em>“ thank you for telling me all that. But well, how do I compress in the end ? “</em></p><p>Well it depends what you have :</p><ul><li><strong>“I have a JPEG”</strong></li></ul><p>A JPEG is technically already compressed with a quality ratio from 0% to 100% (100% being close to the original but it is still a bt lossy). You can try decreasing the quality ratio even-though it is always better to compress to JPEG from a lossless format.</p><ul><li><strong>“I have a PNG”</strong></li></ul><p>PNG is lossless. A working copy. You can compress to JPEG, WEBP or maybe PNG 8bits.</p><p>There are very good tools to compress:</p><p><a href="https://squoosh.app/">Squoosh App</a> is very good and can compress to the newest formats if needed. Though it is front ended: your own computer is compressing the image. Don’t try to use Squoosh App from a Raspberry Pi or even a low tier smartphone. You will get funny results.<br>Squoosh app is made by <a href="https://github.com/GoogleChromeLabs/squoosh">Google Labs</a> and source code is public.</p><p>There are also tons of other compressors for you to browse...</p><p>One I like (because I made it) is <a href="https://www.picfairy.com/">picfairy.com</a> : it will compress your image into various different images quality increments for JPEG and WEBP and also try to compress to PNG 8bits. So you can compare visual quality with final image weight. It can also convert SVG to Bitmap.</p><ul><li><strong>“ my image is JPEG 70% quality but is still too big”</strong></li></ul><p>Just make sure you use an image with the right dimensions: if your image is 1000px * 1000px but you add the CSS rule width: 200px; to your image tag then you can probably squeeze your image a bit.</p><p>No need for too big images. Even if users have Retina displays, you can safely resize your image to 400px * 400px and use the above mentioned CSS rule.</p><p>In a perfect world, you would provide your users different images sizes depending on the fluidity of your HTML page. (thank you Bootstrap and Tailwind haha)</p><p>Well I hope this Article will be useful to you. <br>I will improve it over time.</p><p>Best</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=98b084ff9378" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The Ruby on Rails database connection pool]]></title>
            <link>https://maxencemalbois.medium.com/the-ruby-on-rails-database-connections-pool-4ce1099a9e9f?source=rss-187dc1e4c7c2------2</link>
            <guid isPermaLink="false">https://medium.com/p/4ce1099a9e9f</guid>
            <category><![CDATA[ruby-on-rails]]></category>
            <dc:creator><![CDATA[Maxence Malbois]]></dc:creator>
            <pubDate>Fri, 13 Dec 2019 16:41:55 GMT</pubDate>
            <atom:updated>2024-03-04T15:30:11.268Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*8bT9h6EfYgKzFrTM" /><figcaption>Photo by <a href="https://unsplash.com/@jblesly?utm_source=medium&amp;utm_medium=referral">Lesly Juarez</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Maybe as a Rails coder you have already experienced error messages such as</p><blockquote><em>could not obtain a connection from the pool within 5.000 seconds (waited 5.002 seconds); all pooled connections were in use (ActiveRecord::ConnectionTimeoutError)</em></blockquote><p>that left you puzzled on how to fix it…</p><p>Well it happens to every Rails coder at some point and can give headaches. So I will try to summarize here the big outlines to help you fix this.</p><h3>How many connections can my database handle ?</h3><p>This is the first thing to know. It is usually not a limiting factor but it’s something to keep in mind as your Puma server(s) will eat on those connections.</p><ul><li>in development</li></ul><p>If you work like me with a Postgres database, this is easy to know, the answer is here : <a href="https://www.postgresql.org/docs/current/runtime-config-connection.html">https://www.postgresql.org/docs/current/runtime-config-connection.html</a></p><p>“The default is typically 100 connections, but might be less if your kernel settings will not support it (as determined during initdb)”</p><p>You can obviously tweak this number of connections if your computer is powerful enough. (not covered here)</p><ul><li>in production</li></ul><p>It is probably different for your production environment. I am hosted at Heroku and my “hobby” plan offers 20 connections.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/560/1*hDxP9-kb4eIsIuZYwT6k_w.png" /><figcaption>my postgres plans at Heroku offers 20 connections</figcaption></figure><p>The next available plan “Standard 0” offers 120 connections.</p><h3>What about my rails app ? The principles:</h3><p>Well if you host like me through a PAAS service like Heroku, you have to allocate the database connections to your Heroku dynos.</p><p>Well let’s start with a quick tutorial of Heroku and the Dynos.</p><p>One dyno is basically a Virtual Machine that can run a single instance of your server (Puma in my case, if you run a different server, you may need to find appropriate information for it.)</p><p>Then, Puma (in “clustered mode”) run workers in order to serve your website to users. Each worker can run multiple threads: threads are the processes that can serve your app (concurrently).</p><p>The configuration of Puma is done in the following config file:</p><p>config/puma.rb</p><pre>workers Integer(ENV[‘WEB_CONCURRENCY’] || 2)<br>threads_count = Integer(ENV[‘RAILS_MAX_THREADS’] || 5)<br>threads threads_count, threads_count</pre><p>Workers are set through the environment variable WEB_CONCURRENCY and threads are set through the environment variable RAILS_MAX_THREADS</p><p>For our example, let’s say we have WEB_CONCURRENCY = 2 and RAILS_MAX_THREADS = 2</p><p>This would give the following schema :</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JV-9sqCOIIscIjbAnf6Eug.png" /><figcaption>disclaimer: author of SVG dinosaur illustration is Freepik</figcaption></figure><p>In the above situation we have 2 Puma workers and 2 threads (per worker). Each worker is a full version of your app, like a small server on its own, then your Rails connection pool apply to every worker of your Puma server instance.</p><blockquote><strong>The GVL</strong></blockquote><blockquote>I don’t want to add too much complexity at this stage but increasing threads count may not increase concurrency in the same fashion (or arithmetically). <br>One thread can serve one user, but only one thread at a time can process Ruby code inside a specific worker . Then adding a lot of threads to a Puma worker may not increase that much your application capabilities. <br>Adding more workers though does increase the computational capabilities of your app… but Puma Workers require more resource than threads. <br>Please read here for more details : <a href="https://www.speedshop.co/2020/05/11/the-ruby-gvl-and-scaling.html">https://www.speedshop.co/2020/05/11/the-ruby-gvl-and-scaling.html</a></blockquote><p>We then only need to count the number of threads per Puma worker to populate database.yml</p><pre>production: <br> pool: 2</pre><p>Each thread can handle a single request at a time. If a thread is getting requests from multiple users, it will answer requests but serve them sequentially. Because of this, each thread is assigned a single database connection, no more..</p><p>To know the number of connections taken at database level, we have to multiply the number of Puma workers by the number of threads… then 2*2 ie 4 here. These 4 connections are counted towards the total amout of connections your database can handle (20 as seen above)</p><blockquote><strong><em>Side note about this configuration real world:</em></strong><em> The above configuration of 2 workers and 2 threads should work without issue on a small Heroku dyno (such as hobby) and easily serve multiple simultaneous users with good performance (actually 4 users at a time, but if many more users connect at the same time, serving them should still look smooth). Provided the webserver is not delegated heavy tasks such as image versionning etc ...</em></blockquote><p>2 heroku blog references:</p><ul><li><a href="https://devcenter.heroku.com/articles/deploying-rails-applications-with-the-puma-web-server">Deploying Rails Applications with the Puma Web Server</a></li><li><a href="https://devcenter.heroku.com/articles/concurrency-and-database-connections#connection-pool">Concurrency and Database Connections in Ruby with ActiveRecord</a></li></ul><p>and also this stackoverflow thread :</p><p><a href="https://stackoverflow.com/questions/24280743/what-is-the-difference-between-workers-and-threads-in-puma">https://stackoverflow.com/questions/24280743/what-is-the-difference-between-workers-and-threads-in-puma</a></p><h3>config/database.yml</h3><p>Because we know that the connection pool equates the number of threads we can set database.yml like this :</p><pre>production:<br>pool: &lt;%= ENV[‘RAILS_MAX_THREADS’] %&gt;</pre><h3>The additional stuff that can complexify the above process</h3><ol><li>Sidekiq</li></ol><p>Sidekiq is a process that execute some tasks in the background of your app. Sidekiq is run on an independent Heroku dyno: a worker dyno.</p><p>It is possible to set Sidekiq to perform multiple tasks concurrently.</p><p>sidekiq.yml</p><pre>development:<br> :concurrency: 5<br>production: <br> :concurrency: 2<br>:queues:<br> — critical<br> — default</pre><p>As you can see here, in production, Sidekiq is set up to run 2 threads concurrently.</p><p>These threads may or may not make request to the database. In real world, these threads are very likely to make calls to the database : statistics gathering saved in database, sending emails to users in database, cleaning old images which attachment data is stored in database etc ..</p><p>Then it is safe to allow a single database connection to each of the concurrent Sidekiq threads.</p><p>And in the former part of this thread we have set environment variable RAILS_MAX_THREADS to 2.</p><p><strong>Because Sidekiq uses the same configuration as the webserver</strong>, it will grab the same db pool as the webserver , which actually matches perfectly our Sidekiq needs here ..</p><p><strong>Though</strong> now let’s say running 2 Sidekiq threads is not enough and leaves the background dyno permanently busy and not doing things at the pace you desire. Also the dyno would have the technical capacity of doing more simultaneous tasks. Then you feel it is a bit stupid to not use it to the tune it can potentially achieve ...</p><p>Let’s set Sidekiq concurency to 4 :</p><pre>development:<br> :concurrency: 5<br>production: <br> :concurrency: 4<br>:queues:<br> — critical<br> — default</pre><p>Now we have a problem, because the number of available database connections is still 2 and basically exogenous to Sidekiq.</p><p>One solution is to make the number of Sidekiq processes an environment variable, let’s say envSIDEKIQ_CONCURRENCY and compare this new environment variable to envRAILS_MAX_THREADS :</p><p>sidekiq.yml</p><pre>development:<br> :concurrency: 5<br>production: <br> :concurrency: &lt;%= ENV[&#39;SIDEKIQ_CONCURRENCY&#39;] %&gt;<br>:queues:<br> — critical<br> — default</pre><p>database.yml</p><pre>production: <br> &lt;% if ENV[&#39;SIDEKIQ_CONCURRENCY&#39;] &gt; ENV[&#39;RAILS_MAX_THREADS&#39;] %&gt;<br>pool: &lt;%= ENV[&#39;SIDEKIQ_CONCURRENCY&#39;] %&gt;<br>&lt;% else %&gt;<br>pool: &lt;%= ENV[&#39;RAILS_MAX_THREADS&#39;] %&gt;<br>&lt;% end %&gt;</pre><p>Though now let’s say SIDEKIQ_CNCURRENY is 4 and we are still running two threads per Puma worker. The db pool will be 4 in order to serve Sidekiq gracefully. But we have 2 extra database connections that we don’t use per worker at web dyno level.</p><p>web dyno : 2*4 = 8 connections</p><p>background dyno: 4 connections</p><p>With these two dynos we eat 12 connections from the 20 total connections of the database. And 4 of these connections may never be used…</p><p>So it is always delicate to make a good setup in order to feed your Puma threads and Sidekiq instances properly but also effectively.</p><blockquote>Here is a github thread for those interested about Sidekiq configuration: <a href="https://github.com/mperham/sidekiq/issues/2985">https://github.com/mperham/sidekiq/issues/2985</a></blockquote><blockquote>Also one cool thing to do is to have Sidekiq use the Rails max threads value and boot as many Sidekiq instances until you have reached the desired concurrency.</blockquote><blockquote>For example my Foreman Procfile looks like this at the moment:</blockquote><blockquote>sidekiq1: bundle exec sidekiq<br>sidekiq2: bundle exec sidekiq<br>web: rails s -p 3000<br>webpack: bin/webpack-dev-server</blockquote><blockquote>I have 2 Puma workers and 3 threads per worker. As Sidekiq uses Puma max threads value, Sidekiq instances have also a concurrency of 3. And because I have started two of them. The total concurrency is 6 and no database connection is wasted. (Also this technique is mostly good for home or bare metal servers as more Dynos would be required at Heroku in this case)</blockquote><blockquote>Another solution to not waste database connection is to start Sidekiq with a specific environment variable at the procfile level and then you can discriminate between app workers and async workers. Just check for the presence of this environment variable (that could be named SIDEKIQ_APP=true or something similar) and match Pool with the Sidekiq concurrency in the database.yaml file, without having to compare to the number of threads per Puma worker.<br>I have not tried this solution myself but it should work fine. Let me know your experience with it as it may be the best solution. Yet your app is not “idempotent”, the same app cannot be booted exactly the same for Puma and Sidekiq. If you forget the environment variable for the back end app, Sidekiq may silently start but break later.</blockquote><p>2. Ruby threads</p><p>One more things to factor in, is the ability of a Puma thread to make more calls to the database than anticipated.</p><p>If you remember the above, a Puma thread can serve a single user at a time. So if a Puma thread receive 2 internet requests at the same time, it will serve them one after the other. The database cannot choke.</p><p>Well …. not if you use Ruby threads. Ruby threads are pieces of code that are executed on top of your application (or should I say alongside your application). A Ruby thread block is entirely removed from your app flow.</p><p>My database actually choked when I implemented the below code :</p><pre><a href="http://twitter.com/imagetodestroy">@imagetodestroy</a> = Photo.find_by(hashed_id: params[:hashed_id])<br> <br> Thread.new do <br> <a href="http://twitter.com/imagetodestroy">@imagetodestroy</a>.destroy<br> end</pre><p>Why using Ruby threads ?</p><p>Well in order to make my website snappier and more responsive I had delegated the destruction of an image to a Ruby thread. (my app got much snappier in the process !! )</p><p>Destroying an image in my case takes around 500ms (because, for each image, there are multiple variants stored, then multiple files to delete at S3). The view was then rendered in about 600ms.</p><p>When delegating to a ruby thread my view was rendered in about 100ms. It was a huge improvement. When making the above call AJAX, my app went even smoother !!</p><p><strong>Though one major problem : </strong>because the Ruby thread works beside your normal app, if the user click very fast on a new image to destroy it, two threads blocks can end up making calls to the database at the same time ! And your database then choke !</p><p>This is a problem that can be easily seen if you run a single Puma thread per Worker. If you run 2 puma threads or more, then you have at least 2 database connections available, then there is a chance that the database connection dedicated to another Puma thread is idle and your app can use it while it is not in use.</p><p>One solution is to add one or two more connections to the db pool in order to give a bit of breathing to your app (if you happen to use those Ruby threads).</p><pre>production:<br>pool: &lt;%= ENV[‘RAILS_MAX_THREADS’].to_i + 2 %&gt;</pre><p>Though one advice regarding those threads blocks : never instantiate them in a loop ! You can choke database in milliseconds.</p><blockquote>Also this example is not a good design. What would be better real world, is to have a column in the database that helps marking an image as live or destroyed. It is much faster to update a column than deleting remote images. Then a Sidekiq worker could later do the cleaning depending on that column value.</blockquote><h3>Conclusion</h3><p>It’s not very complicate to deal with the database connections but you have to have a good snapshot of your code in mind, and also a clear vision of the design of your app, with a knowledge of what could make calls to the database. Whether code, gems …</p><p>Once you know that and have the basic knowledge of how it works, it’s easy to tweak database.yml in order to have everything working gracefully.</p><p>Best</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4ce1099a9e9f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>