<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Yavuz Selim Yesilyurt on Medium]]></title>
        <description><![CDATA[Stories by Yavuz Selim Yesilyurt on Medium]]></description>
        <link>https://medium.com/@ysyesilyurt?source=rss-b4bdeffb225------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*xgvuOKR1v187QjVDsUuJ_g.jpeg</url>
            <title>Stories by Yavuz Selim Yesilyurt on Medium</title>
            <link>https://medium.com/@ysyesilyurt?source=rss-b4bdeffb225------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 01:19:26 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ysyesilyurt/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Development with Git Hooks]]></title>
            <link>https://medium.com/picus-security-engineering/development-with-git-hooks-18be38cc4780?source=rss-b4bdeffb225------2</link>
            <guid isPermaLink="false">https://medium.com/p/18be38cc4780</guid>
            <category><![CDATA[git]]></category>
            <category><![CDATA[gitbook]]></category>
            <category><![CDATA[hooks]]></category>
            <category><![CDATA[ci-cd-pipeline]]></category>
            <dc:creator><![CDATA[Yavuz Selim Yesilyurt]]></dc:creator>
            <pubDate>Wed, 27 Oct 2021 08:13:45 GMT</pubDate>
            <atom:updated>2021-10-27T22:25:18.934Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/450/1*bAJrSit_8HoM5sy7ydw0zw.png" /></figure><p>Nowadays, having a decent CI/CD platform is a must. A robust CI/CD Pipeline provides a comprehensive set of tools that help your developers and engineers, in the context of both <em>Dev</em> &amp; <em>Ops,</em> package and deliver software successfully to the end users.</p><p>Your Pipeline may consist of many tasks that gather around the <em>ultimate CI /CD jobs — build</em>, <em>test</em> and <em>deploy</em>. Depending on the pipeline, the <strong>automated</strong> structure of the pipelines directly provides a <em>Rapid Application Development </em>platform and lots of cool benefits such as ability to utilize <em>Automated Testing </em>tools <em>and </em>various deployment methods such as <em>blue/green</em>, <em>canary </em>and <em>linear.</em></p><p>These automated tasks take a huge burden off developers’ shoulders when it comes to validation &amp; verification of the implementations we’ve done. But sometimes these automated pre-merge checks are just <strong>not enough</strong> and some additional checks <strong>before submission</strong> of the implementation to the pipeline may come in really handy.</p><p>Recently, I’ve experienced a problem that made me go for a solution that involves such an additional check mechanism I described above. Let us discuss it below.</p><h3>Problem</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*j3xPaS_nN-2IGk6o" /><figcaption>Photo by <a href="https://unsplash.com/@yancymin?utm_source=medium&amp;utm_medium=referral">Yancy Min</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In <a href="https://www.picussecurity.com/">Picus</a>, we have a CI/CD pipeline and when it is time to merge any development branch these pipelines automatically run. Among many other tasks, our pipelines include a step for running our (any kind of) tests.</p><p>Trusting the pipelines, I was <em>pushing</em> my implementations incrementally to the remote repository and let the pipelines assess the code. It takes sometime for the pipelines to finish, so after waiting a while for them I saw that the tests running on the pipelines failed. After checking what went wrong in the tests I realized that I forgot to <em>regenerate the mocks </em>that we use in our tests <strong>🤦</strong>.</p><p>I fixed that case at the time but this forgetfulness of mine kept happening again and again, making me waste tiny bit of a time on each occurence and making me go back and fix those tiny problems and re-push, potentially coming at some cost.</p><h3>Solution</h3><p>I needed to come up with a solution that handles all such tasks before my pushes to the remote repository (doesn’t matter if I forget or not) in an <strong>automated</strong> and <strong>generic</strong> manner. Obviously I needed a script that handles the tasks when run and a mechanism that allows me to run this script before every push.</p><h4>Pre-Push Hook</h4><p>After a short research step I realized that <a href="https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks">Git Hooks</a> are just great tools that one can use for such purposes. There are a number of types of these hooks (you can see the full list from <a href="https://git-scm.com/docs/githooks#_hooks">here</a>) and pre-push was the one that I was looking for to get rid of my problem. Using pre-push hook one can control the complete flow of any push operation for the codebase on underlying git repository.</p><p>For instance <strong>Running (any kind of) Tests </strong>could be a good starting point. Using that, one can ensure that <strong>no code that fails on tests is pushed to the remote repository.</strong> Another good example (especially for my case :) could be <strong>Regenerating all outdated mocks</strong> and with that, making sure that there exists no outdated auto-generated mocks on the remote. In addition to these two, we may also want to execute some other tasks for push convenience or, again, something that contributes to fail-free pipelines in the remote VCS.</p><p>The hook is just a bash script that is called by Git after a git push command but before anything has been pushed. If this bash script exits with a non-zero status git ensures nothing will be pushed. A sample pre-push script could be as follows:</p><pre>REMOTE<strong>=</strong>&quot;<strong>$</strong>1&quot;<br>URL<strong>=</strong>&quot;<strong>$</strong>2&quot;<br><strong>read</strong> LOCAL_REF LOCAL_SHA REMOTE_REF REMOTE_SHA</pre><pre><strong>function</strong> run_tests() {<br> echo &quot;Running all tests...&quot;<br> <br> go test ./go/... -p 1<br> <br> <strong>if</strong> [ $? <strong>-eq</strong> 0 ]; <strong>then</strong> <br>  echo &quot;All tests run successfully 🎉🎉🎉&quot;<br> <strong>else</strong><br>  echo &quot;Tests failed ❌ ❌ ❌ aborting push!&quot;<br>  <strong>exit</strong> 1<br> <strong>fi</strong><br>}</pre><pre><strong>function</strong> main() {<br> echo &quot;Starting Custom pre-push hook...&quot;<br> <br> <strong>if</strong> [ <strong>-z</strong> &quot;$REMOTE_SHA&quot; ] <strong>&amp;&amp;</strong> [ <strong>-z</strong> &quot;$LOCAL_SHA&quot;]; <strong>then</strong><br>  echo &quot;No new push is on the way, skipping pre-push tasks...&quot;<br> <br> <strong>elif</strong> [ ! <strong>-z</strong> &quot;$REMOTE_SHA&quot; <strong>-a</strong> &quot;$REMOTE_SHA&quot; <strong>!=</strong> &quot; &quot; ] <strong>&amp;&amp;</strong> [ ! -z &quot;$LOCAL_SHA&quot; <strong>-a</strong> &quot;$LOCAL_SHA&quot; <strong>!=</strong> &quot; &quot; ]; <strong>then</strong><br>  echo &quot;New commits detected since last push, better run tests...&quot;<br>  run_tests<br> <br> <strong>else</strong><br>  echo &quot;Unknown pre-push hook usage ❌ please consult to the author of the hook with your use case...&quot;<br> <strong>fi</strong><br> <br> echo &quot;Successfully Executed pre-push hook, now pushing...&quot;<br>}</pre><pre>main<br><strong>exit</strong> 0</pre><p>Under your repository’s .git/hooks folder you can find samples for each type of Git Hook with descriptions of the parameters and inputs passed by Git to the bash scripts.</p><h3>My Custom Pre-Push Hook</h3><p>To promote the ease of adding/ordering/removing pre-push tasks and to have the minimal maintenance cost possible of the hook in future, I have created a Task Queue and aTask Template structure in my custom Git Hook implementation in collaboration with the way that Git executes these hook scripts. Related parts of the hook is given below in bash syntax (some parts omitted for brevity).</p><pre>/* ... */</pre><pre><strong>function</strong> run_all_tasks() {<br>   /* ... */<br>   <br>   i<strong>=</strong>0<br>   TASKS<strong>=</strong>$1<br>   <strong>for</strong> TASK <strong>in</strong> &quot;${TASKS[@]}&quot;; <strong>do</strong><br>      ((i++))<br>      $TASK $i<br>   <strong>done</strong></pre><pre>   /* ... */<br>}<br><br><strong>function</strong> get_tasks() {<br>   /* ... */</pre><pre>   TASKS<strong>=</strong>(check_basename check_upstream)<br><br>   BACKEND_TASKS<strong>=</strong>(auto_generate_mocks run_unit_tests run_integration_tests commit_backend_mock_changes)</pre><pre>   FRONTEND_TASKS<strong>=</strong>(run_cypress_tests)<br><br>   /* ... */</pre><pre>        <strong>if</strong> (git diff --stat $(git rev-parse --abbrev-ref --symbolic-full-name @{u}) | grep -q &quot;.go&quot;); <strong>then</strong><br>           ## Changes detected for Backend, Adding Backend tasks ##<br>           <br>           <strong>for</strong> BET <strong>in</strong> &quot;${BACKEND_TASKS[@]}&quot;; <strong>do</strong><br>              TASKS<strong>+=</strong>($BET)<br>           <strong>done</strong><br>        <strong>fi</strong><br><br>        <strong>if</strong> (git diff --stat $(git rev-parse --abbrev-ref --symbolic-full-name @{u}) | grep -q &quot;.ts\|.tsx\|.js\|.jsx\|.css\|.html&quot;); <strong>then</strong><br>          ## Changes detected for Frontend, Adding Frontend tasks ##<br>           <br>        <strong>for</strong> FET <strong>in</strong> &quot;${FRONTEND_TASKS[@]}&quot;; <strong>do</strong><br>              TASKS<strong>+=</strong>($FET)<br>           <strong>done</strong><br>        <strong>fi<br></strong>   <strong>fi</strong><br><br>   run_all_tasks $TASKS<br>}<br><br><strong>function</strong> main() {<br>   /* ... */</pre><pre>   ## Run pre-push tasks only if a new push is on the way ##<br>   <strong>if</strong> [ -z &quot;$REMOTE_SHA&quot; ] <strong>&amp;&amp;</strong> [ -z &quot;$LOCAL_SHA&quot;]; <strong>then</strong><br>      ## No new commits detected, skip pre-push tasks ##</pre><pre>   <strong>elif</strong> [ $LOCAL_SHA <strong>==</strong> $NEW_BRANCH_SHA ]; <strong>then</strong><br>      ## Remote Branch Deletion, skip pre-push tasks ##<br>   <br>   <strong>elif</strong> [ ! -z &quot;$REMOTE_SHA&quot; -a &quot;$REMOTE_SHA&quot; <strong>!=</strong> &quot; &quot; ] <strong>&amp;&amp;</strong> [ ! -z &quot;$LOCAL_SHA&quot; -a &quot;$LOCAL_SHA&quot; <strong>!=</strong> &quot; &quot; ]; <strong>then</strong><br>      ## New commits detected, run pre-push tasks ##<br>      get_tasks</pre><pre>   <strong>else</strong><br>      ## Invalid Usage, skip pre-push tasks ##<br>   <strong>fi</strong><br>}<br><br>main</pre><p>In addition to <a href="https://github.com/ysyesilyurt/git-hooks-template">this</a> hook template, I’ve also uploaded a <a href="https://github.com/ysyesilyurt/git-hooks-template/tree/main/hooks">sample</a> hook with following sample tasks (most of which are the ones that I’ve used to solve my actual problem mentioned at introduction:) :</p><ul><li>check_basename Checks whether working directory is set correctly by Git, <em>Common</em> Task.</li><li>check_upstream Checks whether current HEAD is behind upstream or not, for any push to go through, the local HEAD should be ahead of the upstream branch, <em>Common</em> Task.</li><li>auto_generate_mocks Checks the files that has committed changes since last push, if detects interfaces with an existing mock and has changes, regenerates the mock interfaces for them, <em>Backend</em> Task.</li><li>run_unit_tests and run_integration_tests are pretty self explanatory I believe :), <em>Backend</em> Tasks.</li><li>commit_backend_mock_changes Commits all the regenerated mock interfaces (if any), <em>Backend</em> Task.</li><li>run_cypress_tests Run tests written for Frontend with cypress, <em>Frontend</em> Task.</li></ul><p>If any of the above tasks fail, <strong>push will not succeed</strong> and an error will be thrown. Above tasks can be modified, enriched, appended or prepended with minimal effort using the general pattern I’ve used for existing tasks. Just implement a new function for the new task and add it to the corresponding queue with correct ordering.</p><p>For the full implementation check out <a href="https://github.com/ysyesilyurt/git-hooks-template">this</a> repository which contains a template sample form of the actual pre-push hook I’ve implemented for our Git repositories at <a href="https://www.picussecurity.com/">Picus</a>.</p><h4>How do I Enable it?</h4><p>The default path for Git hooks is .git/hooks and Git looks for special-named scripts in there, namely looks for a script named as pre-push in our case. If such a script exist in there then git pre-push hook is enabled and Git is going to run that script before each push.</p><h4>I Can’t Share it Though</h4><p>But as you can see this folder is under .git so the hook will not be visible to the others if you work on the repository collaboratively. If you want to get your hook on the remote repository as well then you need to configure the core.hooksPath of your local git repository as follows:</p><pre>git config core.hooksPath hooks</pre><p>Above command sets the hooks folder for Git to a folder named hooks in the current directory, you can set to any path you wish. Now Git is going to look for hooks in this folder instead of .git/hooks .</p><h4>I’ve Enabled But How do I Bypass it?</h4><p>If you don’t want your push to be reviewed by pre-push hook then you can use:</p><pre>git push --no-verify</pre><p>This is going to push your new commits to remote directly.</p><h3>Bonus: Pre-Commit 🎉</h3><p>I loved using hooks. I found them very useful from certain aspects, hope you also find them useful and actually try them sometime. After my pre-push implementation we have noticed that a tiny script that runs before every commit could also be very useful.</p><p>Therefore we have then implemented a <strong>pre-commit</strong> hook that takes care of the pre-commit tasks such as <em>formatting</em> the code etc. I have uploaded a <a href="https://github.com/ysyesilyurt/git-hooks-template/blob/main/go-git-hooks-sample/hooks/pre-commit">sample</a> for pre-commit too in my sample hooks <a href="https://github.com/ysyesilyurt/git-hooks-template">repository</a> I mentioned earlier.</p><p>Thanks for reading. Throughout this story I tried to describe my experience with git pre-push hook, what my problem was and how did I solve them using the hook. I hope the story was clear, informative and, most importantly, fun :) Please feel free to share your opinions on comments section.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=18be38cc4780" width="1" height="1" alt=""><hr><p><a href="https://medium.com/picus-security-engineering/development-with-git-hooks-18be38cc4780">Development with Git Hooks</a> was originally published in <a href="https://medium.com/picus-security-engineering">Picus Security Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Custom JSON Marshaller in Go and Common Pitfalls]]></title>
            <link>https://medium.com/picus-security-engineering/custom-json-marshaller-in-go-and-common-pitfalls-c43fa774db05?source=rss-b4bdeffb225------2</link>
            <guid isPermaLink="false">https://medium.com/p/c43fa774db05</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[json]]></category>
            <category><![CDATA[software-development]]></category>
            <dc:creator><![CDATA[Yavuz Selim Yesilyurt]]></dc:creator>
            <pubDate>Fri, 22 Oct 2021 07:54:21 GMT</pubDate>
            <atom:updated>2021-10-22T14:00:40.022Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N5uQspIr6maYJt2GIlNz-w.png" /><figcaption>Gopher is exercising. From <a href="https://gophercises.com/">gophercises.com</a></figcaption></figure><p>A couple of weeks ago, for some reason, the type of a <em>very fundamental</em> enumeration in our Go codebase has been changed from string to int value. That change required further changes for some other dto s which then required further changes in some other part of the code that was using the value and so on (You know the story, ripple effect of tiny change on fundamental types…).</p><p>For the most part, we managed to refactor such usages but when it was time to deal with the usages of value on the frontend code we suddenly realized that it would be best if we find a way to keep usages of this value as string values in the frontend code since int values for an enum value does not mean a lot for the frontend code and it would actually be best if these enum values are self-explanatory in the code and do not require the programmer to make additional checks for simple enumeration values.</p><p>To overcome this problem we did not want to inject a mapping logic on the frontend, because, for us, frontend code should be as <em>dummy</em> as possible, namely it should not dive into additional complex logic implementations. It should only manage how the resulting data should be displayed.</p><p>Therefore, instead of explicit mappings we wanted to implement a custom marshaller for the dto s that contains the enumeration value to help us customize the way that json.Marshal() choose to (de)serialize our dto.</p><h4>Marshaler and Unmarshaler Interfaces</h4><p>It is pretty straightforward in Go to implement custom marshallers for your custom types. You just need to implement <a href="https://pkg.go.dev/encoding/json#Marshaler">Marshaler</a> interface by overriding its MarshalJSON method and implement <a href="https://pkg.go.dev/encoding/json#Unmarshaler">Unmarshaler</a> interface by overriding its UnmarshalJSON method, which both are located in encoding/json library. So let us give it a try using the example dto I gave below.</p><pre>/* Integer Dummy Status Enumeration that we want to marshal as string and unmarshal as Status in our DTOs */<br><strong>type</strong> Status <strong>int</strong></pre><pre><strong>const</strong> (<br> Waiting Status = iota<br> Queued<br> Completed<br> Error<br>)</pre><pre>/* Stringer implementation for Bar */<br><strong>func</strong> (s Status) String() string {<br> statuses := [...]string{&quot;Waiting&quot;, &quot;Queued&quot;, &quot;Completed&quot;, &quot;Error&quot;}<br> <strong>if</strong> len(statuses) &lt; int(s) {<br>  <strong>return</strong> &quot;&quot;<br> }<br> <strong>return</strong> statuses[s]<br>}</pre><pre>/* Util function to convert string version of Status type to its corresponding Status Enum type */<br><strong>func</strong> ToStatus(b string) Status {<br> <strong>switch</strong> b {<br> <strong>case</strong> &quot;Waiting&quot;:<br>  <strong>return</strong> Waiting<br> <strong>case</strong> &quot;Queued&quot;:<br>  <strong>return</strong> Queued<br> <strong>case</strong> &quot;Error&quot;:<br>  <strong>return</strong> Error<br> <strong>case</strong> &quot;Completed&quot;:<br>  <strong>return</strong> Completed<br> <strong>default</strong>:<br>  <strong>return</strong> -1<br> }<br>}</pre><pre>/* Composite Foo type that contains a Bar and a Status */<br><strong>type</strong> Foo <strong>struct</strong> {<br> FooID     string `json:&quot;fooId&quot;`<br> FooStatus Status `json:&quot;fooStatus&quot;`<br> FooBars   []Bar  `json:&quot;fooBars&quot;`<br>}</pre><pre>/* Bar type that contains a StatusEnum */<br><strong>type</strong> Bar <strong>struct</strong> {<br> BarID     string `json:&quot;barId&quot;`<br> BarStatus Status `json:&quot;barStatus&quot;`<br>}</pre><p>As can be seen from the snippet above we have <em>three</em> types, one of which is Status which we are going to use as the <em>enumeration that we want to</em> (un)marshal <em>customly</em>. The others, Foo and Bar, are the types that we are going to <em>write a custom </em>(un)marshallers<em> for</em> since they both contain the Status field. In addition, note that Foo is a composite type, i.e. it contains a Bar array in itself.</p><h4>Our Goal</h4><p>To make things clear, at the end of this story we want to marshal types Foo and Bar such that the Status the field they contain does not get marshaled as int but instead, it gets marshaled as string .</p><p>Namely instead of</p><pre>{<br>  &quot;fooId&quot;: &quot;this is the foo&quot;,<br>  &quot;fooStatus&quot;: 1,<br>  &quot;fooBars&quot;: [<br>    {<br>      &quot;barStatus&quot;: 1,<br>      &quot;barId&quot;: &quot;this is bar-1&quot;<br>    },<br>    {<br>      &quot;barStatus&quot;: 2,<br>      &quot;barId&quot;: &quot;this is bar-2&quot;<br>    }<br>  ]<br>}</pre><p>we want to have the following serialization in the end:</p><pre>{<br>  &quot;fooStatus&quot;: &quot;Queued&quot;,<br>  &quot;fooId&quot;: &quot;this is the foo&quot;,<br>  &quot;fooBars&quot;: [<br>    {<br>      &quot;barStatus&quot;: &quot;Queued&quot;,<br>      &quot;barId&quot;: &quot;this is bar-1&quot;<br>    },<br>    {<br>      &quot;barStatus&quot;: &quot;Completed&quot;,<br>      &quot;barId&quot;: &quot;this is bar-2&quot;<br>    }<br>  ]<br>}</pre><p>Well, those seem cool so we should start with our custom implementations.</p><h4>Implementations</h4><p>So a basic Marshaler implementation for Foo could be as one in below.</p><pre><strong>func</strong> (f *Foo) MarshalJSON() ([]byte, error) {<br> <strong>return</strong> json.Marshal(&amp;struct {<br>  FooStatus string `json:&quot;fooStatus&quot;`<br>  *Foo<br> }{<br>  FooStatus: f.FooStatus.String(),<br>  Foo:    f,<br> })<br>}</pre><p>Defining and using a new struct definition on the fly may seem confusing at first but it’s the only way to construct a new representation for Foo that allows us to have Status field marshaled as string rather than int .</p><h4>Are We Done?</h4><p>At the first look, it may seem like it has no problems and would give us want we want but <strong>think twice </strong>because if you were to use this implementation then you would get fatal error: stack overflow in runtime and the program will crash.</p><p>The reason for that is json.Marshal requires to Go to simply call Marshaler implementations for all the fields in the struct which will lead to an <strong>infinite loop</strong> as f also inherits Foo&#39;s MarshalJSON method. To fix this issue we actually need a workaround such as the one below.</p><pre><strong>func</strong> (f *Foo) MarshalJSON() ([]byte, error) {<br> <strong>type</strong> Alias Foo<br> <strong>return</strong> json.Marshal(&amp;<strong>struct</strong> {<br>  FooStatus string `json:&quot;fooStatus&quot;`<br>  *Alias<br> }{<br>  FooStatus: f.FooStatus.String(),<br>  Alias:     (*Alias)(f),<br> })<br>}</pre><p>… and probably we would also need to unmarshal our objects back to their original form so it would be nice if we also implement Unmarshaler for Foo. We can use an auxiliary struct and partially unmarshal Foo then use ToStatus util to recover our original struct:</p><pre><strong>func</strong> (f *Foo) UnmarshalJSON(data []byte) error {<br> <strong>type</strong> Alias Foo<br> aux := &amp;<strong>struct</strong> {<br>  FooStatus string `json:&quot;fooStatus&quot;`<br>  *Alias<br> }{<br>  Alias: (*Alias)(f),<br> }<br> <br> <strong>if</strong> err := json.Unmarshal(data, &amp;aux); err != nil {<br>  <strong>return</strong> err<br> }<br> <br> f.FooStatus = ToStatus(aux.FooStatus)<br> <strong>return</strong> nil<br>}</pre><p>Also for Bar:</p><pre>/* Custom Marshaler for Bar */<br><strong>func</strong> (b *Bar) MarshalJSON() ([]byte, error) {<br> <strong>type</strong> Alias Bar<br> <strong>return</strong> json.Marshal(&amp;struct {<br>  BarStatus string `json:&quot;barStatus&quot;`<br>  *Alias<br> }{<br>  BarStatus: b.BarStatus.String(),<br>  Alias:     (*Alias)(b),<br> })<br>}</pre><pre>/* Custom Unmarshaler for Bar */<br><strong>func</strong> (b *Bar) UnmarshalJSON(data []byte) error {</pre><pre><strong>type</strong> Alias Bar</pre><pre>aux := &amp;struct {<br>  BarStatus string `json:&quot;barStatus&quot;`<br>  *Alias<br> }{<br>  Alias: (*Alias)(b),<br> }</pre><pre><strong>if</strong> err := json.Unmarshal(data, &amp;aux); err != nil {<br>  <strong>return</strong> err<br> }</pre><pre>b.BarStatus = ToStatus(aux.BarStatus)<br> <strong>return</strong> nil<br>}</pre><h4>Now everything is ready! Or are they?</h4><p>Let’s test our implementations using the driver code below.</p><pre>log.Printf(&quot;--- Starting (un)marshalling test ---&quot;)</pre><pre>bar1 := dto.Bar{<br> BarID:     &quot;this is bar-1&quot;,<br> BarStatus: dto.Queued,<br>}</pre><pre>bar2 := dto.Bar{<br> BarID:     &quot;this is bar-2&quot;,<br> BarStatus: dto.Completed,<br>}</pre><pre>foo := dto.Foo{<br> FooID:     &quot;this is the foo&quot;,<br> FooStatus: dto.Queued,<br> FooBars:   []dto.Bar{bar1, bar2},<br>}</pre><pre>marshalled, err := json.Marshal(foo)<br><strong>if</strong> err != nil {<br> log.Fatalf(&quot;Oops! Error when marshalling %v\n&quot;, err)<br>}</pre><pre>log.Printf(&quot;This is marshalled = %s\n&quot;, marshalled)</pre><pre><strong>var</strong> unmarshalled dto.Foo</pre><pre>err = json.Unmarshal(marshalled, &amp;unmarshalled)<br><strong>if</strong> err != nil {<br> log.Fatalf(&quot;Oops! Error when unmarshalling %v\n&quot;, err)<br>}</pre><pre>log.Printf(&quot;Success! This is unmarshalled = %s\n&quot;, unmarshalled)</pre><p>When we try to run this driver code the result is going to be as follows:</p><pre><strong>2021/02/28 22:09:14</strong> --- Starting (un)marshalling test ---</pre><pre><strong>2021/02/28 22:09:14</strong> This is marshalled = {&quot;fooId&quot;:&quot;this is the foo&quot;,&quot;fooStatus&quot;:1,&quot;fooBars&quot;:[{&quot;barStatus&quot;:&quot;Queued&quot;,&quot;barId&quot;:&quot;this is bar-1&quot;},{&quot;barStatus&quot;:&quot;Completed&quot;,&quot;barId&quot;:&quot;this is bar-2&quot;}]}</pre><pre><strong>2021/02/28 22:09:14</strong> Oops! Error when unmarshalling json: cannot unmarshal number into Go struct field .fooStatus of type string<br>exit status 1</pre><p>As can be seen in the 3rd row of the driver code log above, our driver program seems to be unable to unmarshal our type Foo back to its original form due to its fooStatus field. Let us debug further to see why our custom (un)marshalers do not work:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*mVDt8r5c90FVG7lMtKFmHw.png" /><figcaption>Apparently, our Marshaler implementation for Foo ends up not being called</figcaption></figure><p>The reason of this failure seems to be related with our marshaler since fooStatus seems to be marshaled with its enumeration value instead our custom marshaling method which is responsible for marshaling the field as string. On the other hand, we can see that the same problem has not been an issue for the Bar&#39;s inside the Foo, i.e. their barStatus has been marshaled as string successfully. Using this information we can infer that the Go has managed to call the marshaler we wrote for Bar but unable to call the one we have written for Foo.</p><h4>Solution</h4><p>The reason for that is actually a basic receiver issue. We have a pointer receiver on MarshalJSON and since we initialize our Foo object as a struct variable instead of a pointer to it, the method ends up not getting called. If we were to convert this one:</p><p><strong>func</strong> (f *Foo) MarshalJSON() ([]byte, error) { ... }</p><p>to this one:</p><p><strong>func</strong> (f Foo) MarshalJSON() ([]byte, error) { ... }</p><p>our problems would vanish.</p><h4>Mixed Usage of Receivers!?</h4><p>Now to achieve consistency we should also convert our Unmarshaler to a value receiver (as Effective Go suggests) but we can not do so because UnmarshalJSON needs pointer to the value and returns error. But in this case, we end up having a Marshaler implementation with value receiver and an Unmarshaler implementation with pointer receiver. So it would be more convenient if keep the Marshaler signatures the same but use json.Marshal them with value’s reference. Namely, we should modify our driver code as follows:</p><pre>log.Printf(&quot;--- Starting (un)marshalling test ---&quot;)</pre><pre>... // same as earlier</pre><pre>foo := dto.Foo{<br> FooID:     &quot;this is the foo&quot;,<br> FooStatus: dto.Queued,<br> FooBars:   []dto.Bar{bar1, bar2},<br>}</pre><pre>marshalled, err := json.Marshal(&amp;foo)<br><strong>if</strong> err != nil {<br> log.Fatalf(&quot;Oops! Error when marshalling %v\n&quot;, err)<br>}<br>log.Printf(&quot;This is marshalled = %s\n&quot;, marshalled)</pre><pre>... // same as earlier</pre><h4>Great! Now we have a Hidden Logic!</h4><p>But please note that this kind of usage injects a <em>hidden logic</em> for the json.Marshal usage of Foo type through the codebase and all developers of this codebase needs to be actively aware of this logic. Instead of initializing our variable as struct value we could use pointer values and never experience such problems.</p><p>So instead of:</p><pre>foo := dto.Foo{<br> FooID:     &quot;this is the foo&quot;,<br> FooStatus: dto.Queued,<br> FooBars:   []dto.Bar{bar1, bar2},<br>}</pre><p>we could use:</p><pre>foo := &amp;dto.Foo{<br> FooID:     &quot;this is the foo&quot;,<br> FooStatus: dto.Queued,<br> FooBars:   []dto.Bar{bar1, bar2},<br>}</pre><p>then we would have no problems with any kind of json.Marshal usage. But this initializing convention is going to introduce the famous discussion of <em>using pointers vs using struct values </em>into the codebase and it’s a completely different story to think about. So for that discussion, I strongly suggest reading <a href="https://medium.com/a-journey-with-go/go-should-i-use-a-pointer-instead-of-a-copy-of-my-struct-44b43b104963">this</a> medium story which approaches this issue in detail with comparisons.</p><h4>Finally, what do we have now?</h4><p>After all these problems and solutions discussed above (including a tiny bit of <em>using pointers vs using struct values </em>discussion in comments) the resulting implementation for our problem (with its driver code) is given below.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ece2abb28784307dcfbefb39eb00db2e/href">https://medium.com/media/ece2abb28784307dcfbefb39eb00db2e/href</a></iframe><h4>We’re done. But wait, there was an easy way!</h4><p>Instead of implementing a custom marshaler and unmarshaler for all the types that include the enumeration the value that we want to marshal customly, we could have only implemented one decent (un)marshaler for the <em>enumeration itself</em> (in our case Status) and Go would use those implementations for all json.Marshal calls on all types that contain the enumeration. In fact, in the end, we have used this way to solve our problem since this is simpler and neater.</p><pre>/* Status definition, implementations and ToStatus() function omitted for brevity */</pre><pre>/* Custom Marshaller for Status */<br>func (s Status) MarshalJSON() ([]byte, error) {<br>   return json.Marshal(s.String())<br>}<br><br>/* Custom Unmarshaller for Status */<br>func (s *Status) UnmarshalJSON(data []byte) error {<br>   var statusStr string<br>   if err := json.Unmarshal(data, &amp;statusStr); err != nil {<br>      return err<br>   }<br>   *s = ToStatus(statusStr)<br>   return nil<br>}</pre><p>The reason behind why I’ve postponed this way to solve our problem was to illustrate as many pitfalls as possible 🙂.</p><p>Thanks for reading. Please feel free to add your opinions about the problems, pitfalls, and solutions that have been discussed here. Also, don’t hesitate to add any other potential pitfalls/tips that may come into your mind about this topic in the comments section :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c43fa774db05" width="1" height="1" alt=""><hr><p><a href="https://medium.com/picus-security-engineering/custom-json-marshaller-in-go-and-common-pitfalls-c43fa774db05">Custom JSON Marshaller in Go and Common Pitfalls</a> was originally published in <a href="https://medium.com/picus-security-engineering">Picus Security Engineering</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Logging Field Builder in Go]]></title>
            <link>https://ysyesilyurt.medium.com/logging-field-builder-in-go-76064c663d8f?source=rss-b4bdeffb225------2</link>
            <guid isPermaLink="false">https://medium.com/p/76064c663d8f</guid>
            <category><![CDATA[logging]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[structured-logging]]></category>
            <category><![CDATA[go]]></category>
            <category><![CDATA[builder-pattern]]></category>
            <dc:creator><![CDATA[Yavuz Selim Yesilyurt]]></dc:creator>
            <pubDate>Wed, 20 Oct 2021 14:33:43 GMT</pubDate>
            <atom:updated>2021-10-22T08:17:00.535Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*62wD5wDsGQdjFYz8Y3X_mg.jpeg" /><figcaption>Gophers at logging work. From the <a href="https://golang.org/doc/gopher/">Golang website</a> (with tiny bit of modification from me :).</figcaption></figure><p>It is often believed that one of the most important things to consider when developing an application is <em>Logging </em>especially when it comes to providing a good level of confidence for promoting the application to production. This process of <em>capturing the flow of our code as it executes </em>passively urges the programmer to think about the design of their program and makes them focus more on how their code gets the job done together with what it is actually written for.</p><p>Proper logging is one of those aspects that differentiates intermediate or advanced developers from a newbie. If you have a robust logging infrastructure that audits the execution of your application in production it so much easier to debug it when unexpected things (that would probably make you uncomfortable 😕) pop up.</p><p>Nowadays there are lots of options in pretty much every programming language to help programmers to build a nice logging infrastructure for their programs. For instance in Go, <a href="https://github.com/sirupsen/logrus">logrus</a>, <a href="https://github.com/golang/glog">glog</a> and <a href="https://github.com/uber-go/zap">zap</a> are popular logging frameworks (for full list of such frameworks you can check <a href="https://github.com/avelino/awesome-go#logging">here</a>). I personally prefer <a href="https://github.com/uber-go/zap">zap</a> since it does provide a nice flexibility for both structured logging with fields and formatted logging with <a href="https://github.com/uber-go/zap#quick-start">sugar</a>. On top of those it is also <a href="https://github.com/uber-go/zap#performance">fast</a>.</p><h4>Structured Logging</h4><p>Having covered basic importance of <em>logging </em>let us move on to a section for <strong>how to </strong>part on logging. This is something that you, as the potential debugger of your program in the future when unexpected things happen in your program, should consider with more care.</p><p>There are basically 2 types of logging when it comes down to this: <em>Basic</em> Logging and <em>Structured</em> Logging.</p><p>Any application’s typical log entries should be as informative as possible about the context that it’s being used in and should not be repetitive. Those should be some of the most important concerns when implementing logging for any program. Programmers can cover those needs with simple logging lines (that would possibly both log to application console and files) in textual format, spread around the codebase. That would simply be Basic Logging.</p><p>But sometimes, we may want to use algorithms to categorize, index and search through log files based on specific parameters (by number, date, user, etc.) that would help us understand the context better or use automated processing to investigate log files. That’s when Structured Logging kicks in. A descriptive example (that uses <a href="https://github.com/uber-go">uber-go</a>/<a href="https://github.com/uber-go/zap">zap</a>) would be:</p><pre>log.Warnf(&quot;No handler function found for the emitted %s event&quot;, e.eventType) // Basic Version</pre><pre>// Outputs: No handler function found for the emitted TakeOff event</pre><pre>log.Warn(&quot;No handler function found for the emitted event type&quot;,<br>zap.String(&quot;event-type&quot;, string(e.eventType))) // Structured Version</pre><pre>// Outputs: No handler function found for the emitted event type {&quot;event-type&quot;: &quot;TakeOff&quot;}</pre><p>If structured version was chosen then any searching, filtering, sorting or indexing operation on event-type field on logs will be available for any purpose later on. Another example that uses <a href="https://github.com/uber-go">uber-go</a>/<a href="https://github.com/uber-go/zap">zap</a> from their own documentation:</p><pre>log.Info(&quot;failed to fetch URL&quot;,<br>  // Structured context as strongly typed Field values.<br>  zap.String(&quot;url&quot;, url),<br>  zap.Int(&quot;attempt&quot;, 3),<br>  zap.Duration(&quot;backoff&quot;, time.Second)<br>)</pre><h4>Wait a second, Am I repeating stuff too much?</h4><p>While having a good structured logging pattern adopted in your whole codebase is great, you would notice that logging fields you use along your codebase are prone to get inconsistent (as long as you are not using pre-defined constant as fields) and, most importantly, repetitive with high verbosity. For example you may use a common timestamp field in lots of along the codebase which means you would probably repeat something similar to the line below on each instance in lots of places in the codebase:</p><pre>zap.Time(&quot;timestamp&quot;, time)</pre><p>… and possibly with lots of fields combined:</p><pre>log.Debug(&quot;Firing Event&quot;,<br>   zap.Int(&quot;event-id&quot;, id),<br>   zap.String(&quot;event-type&quot;, &quot;TakeOff&quot;),<br>   zap.String(&quot;issuer&quot;, user.getName()),<br>   zap.Int(&quot;account-id&quot;, user.getAccountId()),<br>   zap.Time(&quot;timestamp&quot;, time.Now().Format(time.<em>RFC3339</em>)))</pre><p>Think of the case where lots of logging lines with lots of fields like above is highly populated everywhere along the codebase.</p><h3>Enter Field Builder</h3><p>To overcome such repetitiveness and verbosity I have came up with a simple Builder solution that would let me keep my structured logging lines simpler, neater and cleaner. <a href="https://github.com/ysyesilyurt/zap-fieldbuilder-template">The implementation</a> is not a library-kind of thing but rather can be seen as a <strong>Generic </strong><strong>FieldBuilder Template for </strong><a href="https://github.com/uber-go/zap"><strong>uber-go/zap</strong></a><strong> .</strong></p><p>Using this template, a simple FieldBuilder constructor and a couple of <em>reusable</em> Builder methods are all you need. For instance the logging line example above could be easily written as:</p><pre>fields := logger.FieldBuilder().<br>   EventId(id).<br>   EventType(&quot;TakeOff&quot;).<br>   Issuer(user.getName()).<br>   AccountId(user.getAccountId()).<br>   Timestamp(time.Now().Format(time.<em>RFC3339</em>))</pre><pre>/* ... */</pre><pre>log.Debug(&quot;Firing Event...&quot;, fields.Build()...)</pre><p>The fields variable’s definition could be placed anywhere before logging line and when those fields is needed, they can be built simply by calling Build() method on the fields and laying out the resulting []zap.Field slice.</p><p>Above all, having this way of building logging fields, provides a <em>neater</em> and <em>cleaner</em> way to programmer for differentiation of the fields that are being logged and <em>how</em> are they being done since this way <em>promotes the control</em> over the <em>consistency</em> of all the same fields that are being logged.</p><p>On the other hand this way of building fields gives the programmer the ability of defining <em>flexible</em> fields that can be <em>reused </em>or <em>replaced </em>very easily along multiple code blocks.</p><pre>/* Earlier definitions omitted for brevity */</pre><pre>start := time.Now()<br><br>requestLogFields := logger.FieldBuilder().<br>   RequestId(newUUID()).<br>   Issuer(&quot;ysyesilyurt&quot;).<br>   AccountId(123).<br>   Url(rawUrl).<br>   Method(http.<em>MethodPost</em>).<br>   Timestamp(start.Format(time.<em>RFC3339</em>))<br><br>log.Debug(&quot;Sending Awesome Request...&quot;, requestLogFields.Build()...)<br><br>reqErr := req.Post()</pre><pre>end := time.Now()<br>duration := end.Sub(start)<br><br>responseLogFields := requestLogFields.<br>   Status(reqErr.GetStatusCode()).<br>   DurationMs(duration.Milliseconds()).<br>   Timestamp(end.Format(time.<em>RFC3339</em>))<br><br>if reqErr != nil {<br>   log.Info(&quot;Oops! Request failed.&quot;, responseLogFields.Build()...)<br>} else {<br>   log.Info(&quot;Yaay! Request completed.&quot;,outgoingLogFields.Build()...)<br>}</pre><p>Of course programmer is responsible for keeping track of the existing or new field definitions as they need to have dedicated Builders. Those Builder methods should be added to the FieldBuilder as needed when needed.</p><h4>Zap FieldBuilder Template</h4><p>You can have a look this template, check some sample fields with usages and maybe play around a little bit from <a href="https://github.com/ysyesilyurt/zap-fieldbuilder-template">zap-fieldbuilder-template</a> repository.</p><p>This FieldBuilder template can be used in Go programs where <em>Structured Logging</em> is adopted with <a href="https://github.com/uber-go/zap">zap</a>, by adding custom Builders for your custom fields according to the needs of your programs (As I do in my Go projects).</p><p>The template is built for <a href="https://github.com/uber-go/zap">zap</a> and assumes underlying logging framework is zap but practically same logic can be built in anywhere that has structured logging.</p><p>Thanks for reading. Please feel free to share your opinions about the topics and solutions that has been discussed here on comments section :)</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=76064c663d8f" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>