<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Rikito Taniguchi on Medium]]></title>
        <description><![CDATA[Stories by Rikito Taniguchi on Medium]]></description>
        <link>https://medium.com/@tanishiking?source=rss-c6345405a4be------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/2*j7KS7YpE7oE1OU5YyWTzDw.jpeg</url>
            <title>Stories by Rikito Taniguchi on Medium</title>
            <link>https://medium.com/@tanishiking?source=rss-c6345405a4be------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Wed, 18 Nov 2026 18:54:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@tanishiking/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Building a simple job queue using Go with no third-party dependencies]]></title>
            <link>https://medium.com/@tanishiking/building-a-simple-job-queue-using-go-with-no-third-party-dependencies-2ba0d0c03d8b?source=rss-c6345405a4be------2</link>
            <guid isPermaLink="false">https://medium.com/p/2ba0d0c03d8b</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[job-queue]]></category>
            <dc:creator><![CDATA[Rikito Taniguchi]]></dc:creator>
            <pubDate>Sat, 27 Apr 2019 12:43:32 GMT</pubDate>
            <atom:updated>2019-04-27T12:43:32.005Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/255/1*l7lKbJ1crOGKliTF90zXnQ.png" /></figure><p>A job queue is a data structure that maintains the jobs: submitted jobs resides on the job queue until they can be scheduled to run.</p><p>There are many kinds of job queue software such as RabbitMQ, Celery, AWS SQS, and so on. They offer job queue’s functionality with high standards of durability and scalability. However, in some cases, you may want to use job queue functionality without any external dependencies, and any libraries that provide that functionality doesn’t match your requirement.</p><p>In this article, I’m going to show how easy it is to build a simple job queue that has the following features using Go.</p><ul><li>Run(dequeue) the jobs concurrently</li><li>Restrict the concurrency</li><li>Buffered Queue</li><li>Graceful stop</li><li>Do not stop until the users explicitly notify the job queue</li></ul><p>First, define a data structure that represents a job to run that contains a payload on its fields. And define an interface of Worker who has a method Work that receive a Job as an argument.</p><p>In this example, I’m going to makeJob have a URL as its payload, but you can change it.</p><pre>type Job struct {<br>    URL string<br>}</pre><pre>type Worker interface {<br>    Work(j *Job)<br>}</pre><p>Next, define a data structure that represents a job dispatcher.</p><pre>type Dispatcher struct { <br>    sem       chan struct{}<br>    jobBuffer chan *Job<br>    worker    Worker<br>    wg        sync.WaitGroup<br>}</pre><pre>func NewDispatcher(worker Worker, maxWorkers int, buffers int) *Dispatcher {<br>    return &amp;Dispatcher{<br>        sem:       make(chan struct{}, maxWorkers),<br>        jobBuffer: make(chan *Job, buffers),<br>        worker:    worker,<br>    }<br>}</pre><p>semrepresents a <strong>counting semaphore</strong> <strong>for restricting the number of goroutines (job’s concurrency)</strong>, we can use a buffered channel for this purpose.</p><p>jobBuffer <strong>plays the role of a job queue</strong>, and it can <strong>restrict the length of the queue </strong>by defining it as a buffered channel.</p><p>wg will be used for blocking the procedure until this dispatcher stops.</p><p>Before building the main logic of a job queue, I’m going to add some methods to start a job queue, enqueue a job, wait until the job queue will stop, and stop the job queue.</p><pre>func (d *Dispatcher) Start(ctx context.Context) {<br>    d.wg.Add(1)<br>    go d.loop(ctx) // loop will be defined later<br>}</pre><pre>func (d *Dispatcher) Wait() {<br>    d.wg.Wait()<br>}</pre><pre>func (d *Dispatcher) Add(job *Job) {<br>    d.jobBuffer &lt;- job<br>}</pre><pre>func (d *Dispatcher) stop() {<br>    d.wg.Done()<br>}</pre><p>ctx will be used for notifying the signals to the instance.</p><p>It’s time to write the main logic of a job queue: dequeue a job and run it.</p><p>In this loop method, the dispatcher will dequeue a job from jobBuffer and run it in the infinite loop. This infinite loop will end when it receives a value from ctx.Done() .</p><pre>func (d *Dispatcher) loop(ctx context.Context) {<br>Loop:<br>    for {<br>        select {<br>            case &lt;-ctx.Done():<br>                break Loop<br>            case job := &lt;-d.jobBuffer:<br>                // Decrement a semaphore count<br>                d.sem &lt;- struct{}{}<br>                go func(job *Job) {<br>                    defer func() { &lt;-d.sem }()<br>                    // After the job finished,<br>                    //increment a semaphore count<br>                    d.worker.Work(job)<br>                }(job)<br>        }<br>    }<br>    d.stop()<br>}</pre><p>In addition to that, I’m going to add a <strong>graceful restart</strong> feature using sync.WaitGroup .</p><pre>func (d *Dispatcher) loop(ctx context.Context) {<br>    var wg sync.WaitGroup<br>Loop:<br>    for {<br>        select {<br>            case &lt;-ctx.Done():<br>                // wait until all jobs finish<br>                wg.Wait()<br>                break Loop<br>            case job := &lt;-d.jobBuffer:<br>                wg.Add(1)<br>                d.sem &lt;- struct{}{}<br>                go func(job *Job) {<br>                    defer wg.Done()<br>                    defer func() { &lt;-d.sem }()<br>                    d.worker.Work(job)<br>                }(job)<br>        }<br>    }<br>    d.stop()<br>}</pre><p>Now, we have a full implementation for a simple and customizable job queue!</p><p>This could be applied for various purposes like web crawler (enqueue a URL as Job, and Worker will access to the URL) and so on.</p><p>Here’s a full source code including the easy usage.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/98a38ad49ac900a1b6c4023b5467382c/href">https://medium.com/media/98a38ad49ac900a1b6c4023b5467382c/href</a></iframe><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2ba0d0c03d8b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Avoid deadlock caused by a conflict of transactions that (accidentally) acquire gap lock in InnoDB]]></title>
            <link>https://medium.com/@tanishiking/avoid-deadlock-caused-by-a-conflict-of-transactions-that-accidentally-acquire-gap-lock-in-innodb-a114e975fd72?source=rss-c6345405a4be------2</link>
            <guid isPermaLink="false">https://medium.com/p/a114e975fd72</guid>
            <category><![CDATA[innodb]]></category>
            <category><![CDATA[mysql]]></category>
            <category><![CDATA[sql]]></category>
            <category><![CDATA[programming]]></category>
            <dc:creator><![CDATA[Rikito Taniguchi]]></dc:creator>
            <pubDate>Mon, 08 Apr 2019 01:45:06 GMT</pubDate>
            <atom:updated>2019-04-10T07:03:25.864Z</atom:updated>
            <content:encoded><![CDATA[<h3>A quick introduction to Gap locking</h3><blockquote>A gap lock is a lock on a gap between index records, or a lock on the gap before the first or after the last index record.</blockquote><p><a href="https://dev.mysql.com/doc/refman/8.0/en/innodb-locking.html">MySQL :: MySQL 8.0 Reference Manual :: 17.7.1 InnoDB Locking</a></p><p>In addition to foreign-key constraint checking and duplicate key checking, gap locking is enabled for searches and an index scan if the transaction isolation level is above <em>Repeatable Read.</em></p><p><strong>This lock mechanism helps to prevent other transactions from inserting to the gap while the transaction reads the range. As a result, InnoDB can prevent <em>Phantom-Read </em>anomaly even if it’s transaction isolation level is <em>Repeatable Read.</em></strong></p><p>(For more details about ANSI SQL Isolation Levels, see this awesome blog post).</p><p><a href="https://blog.acolyer.org/2016/02/24/a-critique-of-ansi-sql-isolation-levels/">A Critique of ANSI SQL Isolation Levels</a></p><p>As described above, gap locking prevents from occurring <em>Phantom-Read</em> anomalies. However, if you develop an application that concurrently INSERTs a bunch of data without knowing this locking mechanism, numerous of deadlock may appear.</p><h3>A problematic case that will occur deadlocking</h3><p>Assume that you are developing an application that executes the following SQL in a single transaction concurrently (and you cannot use INSERT ... ON DUPLICATE KEY UPDATE for some reasons).</p><pre>SELECT * FROM `blog` WHERE id = ... FOR UPDATE;</pre><pre>-- the following query will executed only when<br>-- the row was not found in the first query.</pre><pre>INSERT INTO `blog` (id, title, content) VALUES (...);</pre><p>where the database schema is something like this.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/8593042cdafa7238f820c607e17c1aff/href">https://medium.com/media/8593042cdafa7238f820c607e17c1aff/href</a></iframe><p>(We use SELECT ... FOR UPDATE for this example, but<strong> any </strong><a href="https://dev.mysql.com/doc/refman/8.0/en/glossary.html#glos_locking_read"><strong>locking read queries</strong></a><strong> can be applied to this case</strong> like UPDATE and DELETE).</p><h4>Why deadlock will occur</h4><p>As long as you successfully find a row in the first query, it’s no problem. However, if the first search query couldn’t find any rows, the problem will happen because</p><ul><li>if the locking read statement uses a unique index with a unique search condition, InnoDB locks only the index record found.</li><li><strong>However, if no records found by the statement, InnoDB acquire the (shared) gap lock that covers the key described by search condition.</strong></li></ul><p>As a result, a deadlock will happen in the following case (assume that the <strong>records whose id is 3 and 6 exist</strong>).</p><ul><li><strong>Transaction1</strong>: SELECT * FROM blog WHERE id = 4 FOR UPDATE; , and locks the gap between id=3 and id=6 (because it failed to find the record).</li><li><strong>Transaction2</strong>: SELECT * FROM blog WHERE id = 5 FOR UPDATE; , and locks the gap between id=3 and id=6 (it won’t be blocked because these gap locks are shared lock).</li><li><strong>Transaction1</strong>: INSERT INTO blog (id, ...) VALUES (4, ...); , this statement will wait because<strong> Transaction2 </strong>has already locked the gap between id=3 and id=6.</li><li><strong>Transaction2</strong>: INSERT INTO blog (id, ...) VALUES (5, ...); , this statement will wait because<strong> Transaction1 </strong>has already locked the gap between id=3 and id=6.</li><li><strong>Deadlock occurred!</strong></li></ul><p>For more details about lock mechanisms in InnoDB, see the documentation.</p><p><a href="https://dev.mysql.com/doc/refman/8.0/en/innodb-locks-set.html">MySQL :: MySQL 8.0 Reference Manual :: 15.7.3 Locks Set by Different SQL Statements in InnoDB</a></p><h3>Resolve (avoid) deadlock</h3><p>So, what can we do for avoiding this deadlock?</p><p>The straightforward way is just <strong>retrying the failed transaction</strong>. This is the most reliable way, but sometimes this retrying cost cannot be ignored especially for the application that requires high processing performance.</p><p>One wired way for resolving this problem is executing the following SQL in a single transaction instead of SELECT ... FOR UPDATE then INSERT.</p><pre>INSERT INTO blog (...) VALUES (...) ON DUPLICATE KEY UPDATE id = id;<br>SELECT * FROM blog WHERE id = ... FOR UPDATE;</pre><p>The first query tries to INSERT a record and do nothing if the entry is duplicated. This query doesn’t acquire a gap lock that blocks locking read query in another transaction as far as the unique key isn’t duplicate.</p><p>(Technically, INSERT query does get a gap lock called <a href="https://dev.mysql.com/doc/refman/8.0/en/innodb-locking.html#innodb-insert-intention-locks"><strong>Insert Intention Locks</strong></a>).</p><p>Therefore, this transaction can execute the same logic as <em>“SELECT and INSERT if the row is not found”</em> with no deadlocking.</p><p>Note that, unfortunately, this technique cannot be always appliable. For example, if there are foreign-key constraint on the unique key.</p><h3>Investigate Deadlock</h3><p>For investigating the cause of deadlock, <a href="https://dev.mysql.com/doc/refman/8.0/en/innodb-enabling-monitors.html">InnoDB lock Monitor</a> will help you :)</p><p>Happy locking life!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a114e975fd72" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[jpp: JSON pretty printer that occupies a minimal number of lines]]></title>
            <link>https://medium.com/@tanishiking/jpp-json-pretty-printer-that-occupies-a-minimal-number-of-lines-9a7322e630da?source=rss-c6345405a4be------2</link>
            <guid isPermaLink="false">https://medium.com/p/9a7322e630da</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[pretty-print]]></category>
            <category><![CDATA[json]]></category>
            <dc:creator><![CDATA[Rikito Taniguchi]]></dc:creator>
            <pubDate>Wed, 13 Mar 2019 14:03:08 GMT</pubDate>
            <atom:updated>2019-03-13T14:03:08.785Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://github.com/tanishiking/jpp">tanishiking/jpp</a></p><p>jpp is a JSON Prettier Printer that occupies a minimal number of lines while pretty-printing given JSON, and it lets us browse a large json file comfortably.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*o3b5t3BE0ucA2QNDWoF7sQ.png" /><figcaption>output of jpp / original / output of jq</figcaption></figure><h3>Background</h3><p>I’ve been utilized <a href="https://stedolan.github.io/jq/">jq</a> or <a href="https://github.com/jmhodges/jsonpp">jsonpp</a> for pretty printing json on CLI environment.</p><p>Though they are quite helpful, I’ve been felt inconvenience with pretty-printed json with them when the input json is too large (e.g. the mapping for elasticsearch) because their output tends to have too many lines.</p><p>This is because they add newlines after each value even though the terminal has enough width to put the values in a single line. As a result, the resulting json has too many lines, and it is sometimes inconvenient to browse.</p><p><strong>jpp</strong> resolves this inconvenience by pretty-printing input json by trying to occupy a minimal number of lines.</p><h3>Demo</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/693/1*xl-DJ_JKHp0N2H_x-yojgQ.gif" /></figure><h3>Installation</h3><p><strong>Homebrew</strong></p><pre>$ brew install tanishiking/jpp/jpp</pre><p><strong>Download from GitHub releases</strong></p><p><a href="https://github.com/tanishiking/jpp/releases">https://github.com/tanishiking/jpp/releases</a></p><p><strong>Build from source</strong></p><pre>$ go get -u github.com/tanishiking/jpp/cwd/jpp</pre><h3>Usage</h3><p>jpp takes json from standard input and sends a pretty printed json to standard output.</p><p>Since jpp is UNIX friendly, it is possible to pipe data in and out of it. This can be useful for using jpp as a filter or interacting with other tools.</p><p><strong>Options</strong></p><p>jpp accepts some CLI options: -w and -i .</p><ul><li>-w : terminal width used when it calculates the layout of resulting json (default: your terminal width). Note that this command does not guarantee there are no lines whose columns are longer than the width. It just attempts to keep lines within this length when possible.</li><li>-i : Indent string (default: 2 spaces).</li></ul><p><strong>Output colors</strong></p><p>We can change the output color of jpp using environment variables. For more details, see the <a href="https://github.com/tanishiking/jpp/blob/master/README.md">README</a>.</p><h3>Technical background</h3><p>jpp calculates the layout of resulting json using the pretty printing algorithm that is presented in <a href="https://homepages.inf.ed.ac.uk/wadler/papers/prettier/prettier.pdf">A prettier printer by Philip Wadler</a>.</p><p>This algorithm is quite simple but extremely powerful, and it is widely used for numerous pretty printers like <a href="https://github.com/prettier/prettier">prettier/prettier</a> which is an excellent pretty printer for javascript, jsx, and so on.</p><p>The golang implementation of the algorithm is now available!</p><p><a href="https://github.com/tanishiking/prettier">tanishiking/prettier</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9a7322e630da" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>