<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Boyan Barakov on Medium]]></title>
        <description><![CDATA[Stories by Boyan Barakov on Medium]]></description>
        <link>https://medium.com/@bbarakov?source=rss-897ba0e94236------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*STlhHV72CgWHbABhU2uqpA.jpeg</url>
            <title>Stories by Boyan Barakov on Medium</title>
            <link>https://medium.com/@bbarakov?source=rss-897ba0e94236------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 15:35:18 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@bbarakov/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Understanding dimensions in PyTorch]]></title>
            <link>https://medium.com/data-science/understanding-dimensions-in-pytorch-6edf9972d3be?source=rss-897ba0e94236------2</link>
            <guid isPermaLink="false">https://medium.com/p/6edf9972d3be</guid>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[tensor]]></category>
            <category><![CDATA[numpy]]></category>
            <dc:creator><![CDATA[Boyan Barakov]]></dc:creator>
            <pubDate>Thu, 11 Jul 2019 14:16:32 GMT</pubDate>
            <atom:updated>2021-04-16T16:12:13.138Z</atom:updated>
            <content:encoded><![CDATA[<h4>A better intuition for PyTorch dimensions by visualizing the process of summation over a 3D tensor</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*g8PjYqOligrJoIVE" /><figcaption>Photo by <a href="https://unsplash.com/@crissyjarvis?utm_source=medium&amp;utm_medium=referral">Crissy Jarvis</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>When I started doing some basic operations with PyTorch tensors like summation, it looked easy and pretty straightforward for one-dimensional tensors:</p><pre>&gt;&gt; x <strong>=</strong> torch.tensor([1, 2, 3])</pre><pre>&gt;&gt; torch.sum(x)</pre><pre>tensor(6)</pre><p>However, once I started to play around with 2D and 3D tensors and to sum over rows and columns, I got confused mostly about the second parameterdimof torch.sum.</p><p>Let’s start by what the official documentation says:</p><blockquote><strong>torch.sum(input, dim, keepdim=False, dtype=None) → Tensor</strong></blockquote><blockquote>Returns the sum of each row of the input tensor in the given dimension dim.</blockquote><p>I don’t quite understand this explication. We can sum up over columns so why do one mention that it just “returns the sum of each row” ? This was my first incomprehension.</p><p>However, the more important problem was, as I said, the direction of each dimension. Here’s what I mean. When we describe the shape of a 2D tensor, we say that it contains some <strong>rows </strong>and some<strong> columns. </strong>So for a <em>2x3</em> tensor we’ve 2 rows and 3 columns:</p><pre>&gt;&gt; x = torch.tensor([<br>     [1, 2, 3],<br>     [4, 5, 6]<br>   ])</pre><pre>&gt;&gt; x.shape</pre><pre>torch.Size([2, 3])</pre><p>We specify at first the rows (2 rows) and then the columns (3 columns), right? That brought me to the conclusion that the first dimension (<em>dim=0</em>) stays for rows and the second one (<em>dim=1</em>) for columns. Following the reasoning that the dimension <em>dim=0</em> means row-wise, I expected <em>torch.sum(x, dim=0) </em>to result in a <strong><em>1x2</em></strong> tensor (1 + 2 + 3 and 4 + 5 + 6 for an outcome of tensor[6, 15]). But it turned out I got something different: a <strong><em>1x3</em></strong> tensor.</p><pre>&gt;&gt; torch.sum(x, dim=0)</pre><pre>tensor([5, 7, 9])</pre><p>I was surprised to see that the reality was the opposite of what I’ve expected because I finally got the result tensor[6, 15] but when passing the parameter <em>dim=1</em>:</p><pre>&gt;&gt; torch.sum(x, dim=1)</pre><pre>tensor([6, 15])</pre><p>So why is that? I found out <a href="https://medium.com/@aerinykim/numpy-sum-axis-intuition-6eb94926a5d1">an article</a> of <a href="https://medium.com/u/1d8994ad0efc">Aerin Kim 🙏</a> tackling down the same confusion but for NumPy matrices where we pass a second parameter called <em>axis</em>. NumPy sum is almost identical to what we have in PyTorch except that <em>dim</em> in PyTorch is called <em>axis</em> in NumPy:</p><blockquote>numpy.sum(a, axis=None, dtype=None, out=None, keepdims=False)</blockquote><p>The key to grasp how <em>dim</em> in PyTorch and <em>axis</em> in NumPy work was this paragraph from Aerin’s article:</p><blockquote>The way to understand the “<strong>axis</strong>” of numpy sum is that it <strong><em>collapses</em></strong> the specified axis. So when it collapses the axis 0 (the row), it becomes just one row (it sums column-wise).</blockquote><p>She explains very well the functioning of the <em>axis</em> parameter on <em>numpy.sum. </em>However, it becomes trickier when we introduce a third dimension. When we look at the shape of a 3D tensor we’ll notice that the new dimension gets prepended and takes the first position (in <strong>bold</strong> below) i.e. the third dimension becomes dim=0.</p><pre>&gt;&gt; y = torch.tensor([<br>     [<br>       [1, 2, 3],<br>       [4, 5, 6]<br>     ],<br>     [<br>       [1, 2, 3],<br>       [4, 5, 6]<br>     ],<br>     [<br>       [1, 2, 3],<br>       [4, 5, 6]<br>     ]<br>   ])</pre><pre>&gt;&gt; y.shape</pre><pre>torch.Size([<strong>3</strong>, 2, 3])</pre><p>Yes, it’s quite confusing. That’s why I think some basic visualizations of the process of summation over different dimensions will greatly contribute to a better understanding.</p><p>The first dimension (<em>dim=0</em>) of this 3D tensor is the highest one and contains 3 two-dimensional tensors. So in order to sum over it we have to collapse its 3 elements over one another:</p><pre>&gt;&gt; torch.sum(y, dim=0)</pre><pre>tensor([[ 3,  6,  9],<br>        [12, 15, 18]])</pre><p>Here’s how it works:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/354/1*XvGIb1XxF6r9FYHQ3yYPTw.gif" /></figure><p>For the second dimension (<em>dim=1</em>) we have to collapse the rows:</p><pre>&gt;&gt; torch.sum(y, dim=1)</pre><pre>tensor([[5, 7, 9],<br>        [5, 7, 9],<br>        [5, 7, 9]])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/415/1*e608vZqp7jNIdsarKbeunQ.gif" /></figure><p>And finally, the third dimension collapses over the columns:</p><pre>&gt;&gt; torch.sum(y, dim=2)</pre><pre>tensor([[ 6, 15],<br>        [ 6, 15],<br>        [ 6, 15]])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/384/1*pf9K7VX2M6_1p39JUmOkLQ.gif" /></figure><p>If you’re like me, recently started to learn PyTorch or NumPy, I hope these basic animated examples will help you to get a better understanding of how dimensions work, not only for <em>sum</em> but for other methods as well.</p><p>Thanks for reading!</p><p>References:</p><p>[1] A. Kim, <a href="https://medium.com/@aerinykim/numpy-sum-axis-intuition-6eb94926a5d1">Numpy Sum Axis Intuition</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=6edf9972d3be" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/understanding-dimensions-in-pytorch-6edf9972d3be">Understanding dimensions in PyTorch</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>