<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Stephen Darlington on Medium]]></title>
        <description><![CDATA[Stories by Stephen Darlington on Medium]]></description>
        <link>https://medium.com/@sdarlington?source=rss-bed65416e445------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*27_lZ88bKSfJbw8n.jpg</url>
            <title>Stories by Stephen Darlington on Medium</title>
            <link>https://medium.com/@sdarlington?source=rss-bed65416e445------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sat, 21 Nov 2026 16:49:30 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@sdarlington/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Apache Ignite: “IgniteCheckedException: No clients found”]]></title>
            <link>https://medium.com/@sdarlington/apache-ignite-ignitecheckedexception-no-clients-found-47af0f56708d?source=rss-bed65416e445------2</link>
            <guid isPermaLink="false">https://medium.com/p/47af0f56708d</guid>
            <category><![CDATA[apache-ignite]]></category>
            <category><![CDATA[debugging]]></category>
            <dc:creator><![CDATA[Stephen Darlington]]></dc:creator>
            <pubDate>Fri, 16 Aug 2019 08:55:23 GMT</pubDate>
            <atom:updated>2019-08-16T08:55:23.721Z</atom:updated>
            <content:encoded><![CDATA[<p>You know that thing where you’re trying to debug a problem and you just know that <em>you’re</em> the culprit, that past-you did something stupid, and you just can’t figure out what?</p><p>Welcome to my day.</p><p>Anyway, I’m documenting my stupidity so you don’t have to suffer as long as I did.</p><p>The background: in order to debug an application that runs on a cluster can be a challenge. The way I tend to do it is to run a server in debug mode in my IDE and connect to that same server using a client. The server is super-simple:</p><pre><strong>object </strong>Server <strong>extends </strong>App {<br>  <strong>val </strong><em>ignite </em>= {<br>    <strong>val </strong>igniteConfiguration = <strong>new </strong>IgniteConfiguration()<br>    igniteConfiguration.setPeerClassLoadingEnabled(<strong>true</strong>)<br><br>    <strong>val </strong>ipFinder = <strong>new </strong>TcpDiscoveryVmIpFinder()<br>    <strong>val </strong>addresses = util.Arrays.<em>asList</em>(<strong>&quot;127.0.0.1&quot;</strong>)<br>    ipFinder.setAddresses(addresses)<br><br>    <strong>val </strong>discoverySpi = <strong>new </strong>TcpDiscoverySpi()<br>    discoverySpi.setIpFinder(ipFinder)<br>    igniteConfiguration.setDiscoverySpi(discoverySpi)<br><br>    Ignition.<em>start</em>(igniteConfiguration)<br>  }<br>}</pre><p>I click the “Debug” button in my IDE and away it goes.</p><p>My client starts basically the same way, except I have the Ignition.setClientMode(true) statement first. There’s more stuff, but it never gets there.</p><p>Instead I get a stack trace:</p><pre>Aug 15, 2019 4:56:41 PM org.apache.ignite.logger.java.JavaLogger error<br>SEVERE: Failed to send message to remote node [node=TcpDiscoveryNode [id=d7248cbc-39b5-496a-a4e4-cf1f49eb5cb3, consistentId=127.0.0.1,192.168.1.16:47500, addrs=[127.0.0.1, 192.168.1.16], sockAddrs=[/192.168.1.16:47500, /127.0.0.1:47500], discPort=47500, order=1, intOrder=1, lastExchangeTime=1565884601101, loc=false, ver=8.7.6#20190704-sha1:6449a674, isClient=false], msg=GridIoMessage [plc=2, topic=TOPIC_CACHE, topicOrd=8, ordered=false, timeout=0, skipOnTimeout=false, msg=GridDhtPartitionsSingleMessage [parts=null, partCntrs=null, partsSizes=null, partHistCntrs=null, err=null, client=true, exchangeStartTime=100453829886160, finishMsg=null, super=GridDhtPartitionsAbstractMessage [exchId=GridDhtPartitionExchangeId [topVer=AffinityTopologyVersion [topVer=2, minorTopVer=0], discoEvt=DiscoveryEvent [evtNode=TcpDiscoveryNode [id=591e6724-f22d-4bad-a906-819862178db2, consistentId=591e6724-f22d-4bad-a906-819862178db2, addrs=[0:0:0:0:0:0:0:1%lo0, 127.0.0.1, 192.168.1.16], sockAddrs=[/192.168.1.16:0, /0:0:0:0:0:0:0:1%lo0:0, /127.0.0.1:0], discPort=0, order=2, intOrder=0, lastExchangeTime=1565884600967, loc=true, ver=8.7.6#20190704-sha1:6449a674, isClient=true], topVer=2, nodeId8=591e6724, msg=null, type=NODE_JOINED, tstamp=1565884601135], nodeId=591e6724, evt=NODE_JOINED], lastVer=GridCacheVersion [topVer=0, order=1565884600540, nodeOrder=0], super=GridCacheMessage [msgId=1, depInfo=null, lastAffChangedTopVer=AffinityTopologyVersion [topVer=-1, minorTopVer=0], err=null, skipPrepare=false]]]]]<br>class org.apache.ignite.IgniteCheckedException: No clients found<br> at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createTcpClient(TcpCommunicationSpi.java:3536)<br> at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.createNioClient(TcpCommunicationSpi.java:3035)<br> at org.apache.ignite.spi.communication.tcp.TcpCommunicationSpi.reserveClient(TcpCommunicationSpi.java:2915)</pre><p>The important bit here is the “IgniteCheckedException: No clients found” part. The server does see the client, and logs the fact that there’s now a new client.</p><pre>[16:56:34] Topology snapshot [ver=1, locNode=d7248cbc, servers=1, clients=0, state=ACTIVE, CPUs=8, offheap=3.2GB, heap=4.0GB]<br>[16:56:41] Topology snapshot [ver=2, locNode=d7248cbc, servers=1, clients=1, state=ACTIVE, CPUs=16, offheap=3.2GB, heap=8.0GB]<br>[16:57:11] Topology snapshot [ver=3, locNode=d7248cbc, servers=1, clients=0, state=ACTIVE, CPUs=8, offheap=3.2GB, heap=4.0GB]</pre><p>So discovery worked… but it can’t connect. Firewalls? Some weird permissions? (Just me that thinks of <a href="https://www.youtube.com/watch?v=TKQzqwn-jIM">this routine</a>? “It’s <em>here!</em>”)</p><p>I created a plain Java version of the server and… that worked! I’m really confused now.</p><p>Scala just kicks off the Java code, so it can’t be the difference in language. Which leaves… how we call it.</p><p>On the server, long ago, I’d set the following as the options for the VM:</p><pre>-Djava.net.preferIPv4Stack=true -XX:+UseG1GC -XX:+DisableExplicitGC</pre><p>My newly created client did not. I <em>guess</em> it was trying to connect to the IPv6 address/port and that’s why it couldn’t find the server node; I didn’t delve too deeply.</p><p>Instead, I just added the -Djava.net.preferIPv4Stack=true option to the client and everything worked.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=47af0f56708d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ignite from Python (part 2)]]></title>
            <link>https://medium.com/@sdarlington/ignite-from-python-part-2-8a9054da6612?source=rss-bed65416e445------2</link>
            <guid isPermaLink="false">https://medium.com/p/8a9054da6612</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[apache-ignite]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Stephen Darlington]]></dc:creator>
            <pubDate>Mon, 28 Jan 2019 17:54:50 GMT</pubDate>
            <atom:updated>2019-01-28T17:54:50.688Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*5lvAOM2lOqmU95Q2" /><figcaption>Photo by <a href="https://unsplash.com/@maxcodes?utm_source=medium&amp;utm_medium=referral">Max Nelson</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>In <a href="https://medium.com/@sdarlington/ignite-using-python-c3492431e307">part 1</a> we talked about how to connect to <a href="https://ignite.apache.org/">Apache Ignite</a> using Python and do some basic operations, like connecting to a cluster and running SQL queries. Today we’ll build on that and see how we can use the key-value store APIs.</p><p>To make this piece fairly stand-alone, we’re going to start from an empty cache. Let’s create one:</p><pre>&gt;&gt;&gt; from pyignite import Client<br>&gt;&gt;&gt; ignite = Client()<br>&gt;&gt;&gt; ignite.connect(&quot;127.0.0.1&quot;,10800)<br>&gt;&gt;&gt; ignite.sql(&quot;create table ignite (id integer, name varchar, primary key(id));&quot;)</pre><p>Then we’ll use SQL to add a couple of rows:</p><pre>&gt;&gt;&gt; ignite.sql(&quot;insert into ignite values (1, &#39;Arthur Dent&#39;), (2, &#39;Ford Prefect&#39;);&quot;)</pre><p>(You’ll see why I add some rows this way shortly. Don’t worry, you can use put as well but we need to lay some foundations first.)</p><p>As we found last time, we get some weird data back:</p><pre>&gt;&gt;&gt; from pyignite.datatypes import IntObject<br>&gt;&gt;&gt; ig_cache = ignite.get_cache(&#39;SQL_PUBLIC_IGNITE&#39;)<br>&gt;&gt;&gt; val1 = ig_cache.get(1, key_hint=IntObject)<br>&gt;&gt;&gt; val1<br>SQL_PUBLIC_IGNITE_1a083dcd_199b_4501_8156_ff805f4ef27f(NAME=&#39;Arthur Dent&#39;, version=1)</pre><p>What I didn’t say was that you can access the columns just by giving their name.</p><pre>&gt;&gt;&gt; for r in ig_cache.scan():<br>...   print (r[0], r[1].NAME)<br>...<br>1 Arthur Dent<br>2 Ford Prefect</pre><p>The obvious next question is: how do you insert a value using the key-value API? Two ways, the cheater (easy) way is to just copy the class that we got back last time:</p><pre>Ignite = val1.__class__<br>ig_cache.put(3, Ignite(NAME=&#39;Zaphod Beeblebrox&#39;), key_hint=IntObject)</pre><p>Top tip: note the use of the key_hint? You’ll find that the call succeeds without error if you don’t use it, but you’ll get a long value in Ignite which is going to give you grief later on.</p><p>That’s great, but how do you populate an empty cache? Sadly there doesn’t seem to be a way to pull the structure from Ignite’s metadata, so we’re going to have to do some typing to tell it.</p><p>Since we can’t rely on the API, we’re going to have to figure out that metadata ourselves. First, we need the cache name. We can get a list of available caches by doing this:</p><pre>&gt;&gt;&gt; ignite.get_cache_names()<br>[&#39;SQL_PUBLIC_IGNITE&#39;, &#39;PUBLIC&#39;]</pre><p>So we can see that our table “Ignite” is actually called SQL_PUBLIC_IGNITE ‘under the hood.’ (If you’ve used Ignite before, this shouldn’t be a surprise.)</p><p>Next, we also need the data type name. If you created the cache with an explicit type this is pretty easy, but if you used a regular SQL CREATE statement without a WITH clause… it’s… not something you’d guess.</p><p>This code strikes me as a bit crazy, so perhaps I’m missing something… but it works:</p><pre>&gt;&gt;&gt; from pyignite.datatypes import *<br>&gt;&gt;&gt; ig_cache.settings[prop_codes.PROP_QUERY_ENTITIES][0][‘value_type_name’]<br>&#39;SQL_PUBLIC_IGNITE_1a0fe58f_1be8_4daa_af6a_805291ca5070&#39;</pre><p>You can get a bunch of other interesting information from PROP_QUERY_ENTITIES; have a look yourself.</p><p>So, now we have the cache name and the class name – I’m assuming you know the columns – we can finally get started.</p><pre>&gt;&gt;&gt; from<strong> </strong>pyignite.binary import<strong> </strong>GenericObjectMeta<br>&gt;&gt;&gt; from<strong> </strong>collections import<strong> </strong>OrderedDict<br>&gt;&gt;&gt; class Ignite2(metaclass=GenericObjectMeta,type_name=’SQL_PUBLIC_IGNITE_1a0fe58f_1be8_4daa_af6a_805291ca5070&#39;, schema=OrderedDict([<br>... (‘NAME’, String)<br>... ])):<br>...   pass<br>...<br>&gt;&gt;&gt; ig_cache.put(7,Ignite2(NAME=’Mr Prossor’), key_hint=IntObject)</pre><p>One possibly surprising thing: you don’t need to include columns that are part of the key (‘id’ in this case). Otherwise, whether you know Python or not, I don’t think there’s anything too controversial in this code.</p><p>Of course, if ‘put’ works, there’s the rest of the standard key-value store API too, including put_all(), get_all) and get_and_put_if_absent().</p><p>In explaining my thought-process as I worked through this, I have possibly made using the Python API look harder than it actually is. In practice, you can connect to Ignite, load data and interrogate it using a REPL with just a few lines of code. It’s a great, lighter-weight alternative to Spark.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=8a9054da6612" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ignite using Python]]></title>
            <link>https://medium.com/@sdarlington/ignite-using-python-c3492431e307?source=rss-bed65416e445------2</link>
            <guid isPermaLink="false">https://medium.com/p/c3492431e307</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[apache-ignite]]></category>
            <dc:creator><![CDATA[Stephen Darlington]]></dc:creator>
            <pubDate>Fri, 18 Jan 2019 20:55:55 GMT</pubDate>
            <atom:updated>2019-05-16T06:36:39.334Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*LJjxRfWo-XDq2i7P" /><figcaption>Photo by <a href="https://unsplash.com/@hiteshchoudhary?utm_source=medium&amp;utm_medium=referral">Hitesh Choudhary</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>Previously I talked about <a href="https://medium.com/@sdarlington/loading-data-into-apache-ignite-c0cb7c065a7?source=your_stories_page---------------------------">accessing Ignite using Python via Spark</a>. It has advantages but it’s also a little convoluted. In the most recent version of Ignite we might be able to do better since it now comes with a native Python API. Let’s see how that works.</p><p>The Python client is a <em>thin</em> client, which means it connects to the grid using a binary protocol, but doesn’t join the cluster as a peer (unlike thick clients, including the existing Spark integration). One plus is that it’s easier to install.</p><pre>pip install pyignite</pre><p>If you’re not familiar with it, PIP is the Python package manager, like Ruby GEM or CPAN for Perl.)</p><p>And, oh dear:</p><pre>Collecting pyignite<br>  Using cached https://files.pythonhosted.org/packages/db/95/32338b940d6b38089476bd1e2c71d8f493950c0b56c6176fe47dd41d868d/pyignite-0.3.0.tar.gz<br>    Complete output from command python setup.py egg_info:<br>    Traceback (most recent call last):<br>      File &quot;&lt;string&gt;&quot;, line 1, in &lt;module&gt;<br>      File &quot;/private/var/folders/vf/vv29cdg10wn1t8ts9bj869y00000gp/T/pip-install-pCx6rf/pyignite/setup.py&quot;, line 20<br>        def is_a_requirement(line: str) -&gt; bool:<br>                                 ^<br>    SyntaxError: invalid syntax<br>    <br>    ----------------------------------------<br>Command &quot;python setup.py egg_info&quot; failed with error code 1 in /private/var/folders/vf/vv29cdg10wn1t8ts9bj869y00000gp/T/pip-install-pCx6rf/pyignite/</pre><p>If I’d read the documentation I would have known that you need Python 3.4 or higher. I’m using a Mac, so I can use HomeBrew to install the new version of Python and then use <em>that</em> to install the Ignite client.</p><pre>brew install python3<br>pip3 install pyignite</pre><p>Then we can start, just like it says in the documentation:</p><pre>&gt;&gt;&gt; from pyignite import Client<br>&gt;&gt;&gt; ignite = Client()<br>&gt;&gt;&gt; ignite.connect(&#39;127.0.0.1&#39;, 10800)<br>&gt;&gt;&gt; new_cache = ignite.create_cache(&#39;ignite&#39;)<br>&gt;&gt;&gt; new_cache.put(1,&#39;Arthur Dent&#39;)<br>&gt;&gt;&gt; new_cache.put(2,&#39;Ford Prefect&#39;)<br>&gt;&gt;&gt; new_cache.get(1)<br>&#39;Arthur Dent&#39;<br>&gt;&gt;&gt; for k,v in new_cache.scan():<br>...   print (k, &#39; -&gt; &#39;, v)<br>...<br>1  -&gt;  Arthur Dent<br>2  -&gt;  Ford Prefect</pre><p>Nice.</p><p>But what happens if you try to talk to an existing cache?</p><p>Let’s create a table using sqline:</p><pre>create table ignite2 (id integer, name varchar, primary key(id));<br>insert into ignite2 values (1, &#39;Arthur Dent&#39;), (2, &#39;Ford Prefect&#39;);</pre><p>And then use Python to access the same data:</p><pre>&gt;&gt;&gt; existing_cache = ignite.get_cache(&#39;SQL_PUBLIC_IGNITE2&#39;)<br>&gt;&gt;&gt; existing_cache.get(1)<br>&gt;&gt;&gt; print (existing_cache.get(1))<br>None<br>&gt;&gt;&gt;</pre><p>What? <em>I literally just put it there!</em></p><p>This took some digging to get to the bottom of, but the problem is data types. On the Ignite server, the field is of type java.lang.Integer but the Python client seems to default to a Long. Luckily we can give the client some hints.</p><pre>from pyignite.datatypes import IntObject<br>&gt;&gt;&gt; existing_cache.get(1, key_hint=IntObject)<br>SQL_PUBLIC_IGNITE2_1a083dcd_199b_4501_8156_ff805f4ef27f(NAME=&#39;Arthur Dent&#39;, version=1)</pre><p>(I found this a bit confusing. You can import the mapping for date with DateObject and the mapping for long and string with LongObject and StringObject. But for… reasons… you abbreviate integer to IntObject.)</p><p>Anyway, we can now find the correct key and value. The bad news is that it’s in weird format. (If you look on the Ignite-side, you’ll see the same thing. When you create a table without specifying a POJO this is just what happens, so it’s entirely expected, but you can’t tell me that the name isn’t weird!)</p><p>There <em>is</em> a way of working with these structures but for today I’m going to give you a cheat: use SQL.</p><pre>&gt;&gt;&gt; result = ignite.sql(&#39;select * from ignite2&#39;)<br>&gt;&gt;&gt; for x in result:<br>...   print (x[0] , &#39; -&gt; &#39;, x[1])<br>...<br>1  -&gt;  Arthur Dent<br>2  -&gt;  Ford Prefect</pre><p>Stay tuned for how to access the “weird” structured data.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c3492431e307" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[The trick to successfully integrating Apache Ignite and PySpark]]></title>
            <link>https://medium.com/@sdarlington/the-trick-to-successfully-integrating-apache-ignite-and-pyspark-890e436d09ba?source=rss-bed65416e445------2</link>
            <guid isPermaLink="false">https://medium.com/p/890e436d09ba</guid>
            <category><![CDATA[apache-ignite]]></category>
            <category><![CDATA[apache-spark]]></category>
            <dc:creator><![CDATA[Stephen Darlington]]></dc:creator>
            <pubDate>Fri, 11 Jan 2019 18:07:06 GMT</pubDate>
            <atom:updated>2019-01-11T18:07:06.297Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*1dROhOzcOOAh-TZt" /><figcaption>Photo by <a href="https://unsplash.com/@stephaniemccabe?utm_source=medium&amp;utm_medium=referral">Stephanie McCabe</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>I’ve seen a few people struggling with integrating Spark and Ignite in the mailing lists. As with so much of this stuff, it’s actually quite straightforward if the stars align. But getting those stars to align can take some trial and error. Here’s my experience so, hopefully, you can skip some of it.</p><p>First, the caveats. My use case here is mainly just playing around with data. I’m not setting up a cluster of Spark nodes (though the principles apply). One of the strengths of Spark is the ability to use a DataFrame from Python, making it easy do lots of <em>ad hoc</em> queries. But Ignite has a faster SQL engine and the ability to operate over a whole clusters-worth of data. I find the combination quite effective!</p><p>Step one: let’s start PySpark, as they call the Python version of Spark, with the ability to connect to an Ignite cluster. I’m sure there are more minimal ways of doing it, but this is what I use:</p><pre>bin/pyspark --jars $IGNITE_HOME/libs/*.jar,$IGNITE_HOME/libs/optional/ignite-spark/*jar,$IGNITE_HOME/libs/ignite-spring/*jar,$IGNITE_HOME/libs/ignite-indexing/*jar</pre><p>I’m using Spark 2.3 here, which is one version behind the current one. I didn’t dig too deeply, but I was getting weird errors when I used 2.4 that just went away when I went back a version. On the Ignite side I’m using 2.7 (the current version) and, yes, there are lots of stars in the command-line. I’m aiming to minimise typing rather than memory. When going into production you might want to filter the list of JARs.</p><p>If you’re using a Spark cluster, the spark-submit command looks very similar.</p><p>So we can play around, let’s load in a CSV file:</p><pre>input = spark.read.format(&quot;csv&quot;)<br>                  .option(&quot;header&quot;,&quot;true&quot;)<br>                  .option(&quot;delimiter&quot;,&quot;;&quot;)<br>                  .load(&quot;TickDataCashBBBA.csv&quot;)</pre><p>Let’s write that CSV file to Ignite:</p><pre>import os</pre><pre>configFile = os.environ[&#39;IGNITE_HOME&#39;] + &quot;/config/default-config.xml&quot;</pre><pre>input.write.format(&quot;ignite&quot;)<br>           .option(&quot;table&quot;,&quot;TickData&quot;)<br>           .option(&quot;primaryKeyFields&quot;,&quot;date,isix,entryTime&quot;)<br>           .option(&quot;config&quot;,configFile)<br>           .save()</pre><p>I’m running my Ignite node and Spark client on the same machine but that’s not a requirement. As long as Spark can “see” the cluster — and you tell it how using the ‘config’ parameter — it should be okay. You might want to run the count() command on the resulting DataFrame to make sure you got the primary key right.</p><p>And then let’s pull it back in again!</p><pre>ignite = spark.read.format(&quot;ignite&quot;)<br>                   .option(&quot;config&quot;,configFile)<br>                   .option(&quot;table&quot;,&quot;TickData&quot;)<br>                   .load()</pre><p>Now we have two DataFrames, one in Spark’s memory, the other in Ignite, and both pretty much look the same from a programming point of view. For example, we can do things like:</p><pre>input.select(input[&quot;isix&quot;], input[&quot;entryPrcAsk&quot;] - input[&quot;entryPrcBid&quot;]).groupby(&quot;isix&quot;).mean().show()</pre><pre>ignite.select(ignite[&quot;isix&quot;], ignite[&quot;entryPrcAsk&quot;] - ignite[&quot;entryPrcBid&quot;]).groupby(&quot;isix&quot;).mean().show()</pre><p>And they both return similar values.</p><p>(Why <em>similar</em> and not <em>identical</em>? Well, they’re different SQL engines and the floating point numbers are stored slightly differently, meaning that calculations have slightly different errors. If I’d used integers or strings I would not have expected the difference.)</p><p>You’ll generally find that the more complex the SQL expression and the more data you have, the faster the Ignite version will go relative to the pure-Spark one.</p><p>Even without the performance boost, I love the fact that I can do so much more with Ignite without having to compile any code.</p><p>Once you get the basics right, you can starting tuning it to your particular use case. For that you should check out the <a href="https://apacheignite-fs.readme.io/docs/ignite-for-spark">official documentation here</a>. Good luck!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=890e436d09ba" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[LivenessProbe for Ignite pods]]></title>
            <link>https://medium.com/@sdarlington/livenessprobe-for-ignite-pods-9ebeed63658e?source=rss-bed65416e445------2</link>
            <guid isPermaLink="false">https://medium.com/p/9ebeed63658e</guid>
            <category><![CDATA[apache-ignite]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Stephen Darlington]]></dc:creator>
            <pubDate>Thu, 03 Jan 2019 14:41:16 GMT</pubDate>
            <atom:updated>2019-01-03T14:41:16.377Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ukHq0GFn7JZRgzcd" /><figcaption>Photo by <a href="https://unsplash.com/@erwanhesry?utm_source=medium&amp;utm_medium=referral">Erwan Hesry</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>The last couple of posts have been about using Apache Ignite on Kubernetes, and this one follows in that vein.</p><p>We’ve got our cluster up and running; we’ve got the ability to scale it using StatefulSets; we’ve <a href="https://medium.com/p/afbed40c7e53">activated it using a job</a>. How do we check that it’s still up and running?</p><p>Kubernetes has a feature called a “<a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/">liveness probe</a>” that we should be using, that way it can restart the pod if something untoward happens.</p><p>I’m sure there are all kinds of clever ways of making it work. We need one that’s a balance between being quick to call while telling us something useful and not requiring significant resources. Checking that the log file is still being written to might be easy, but we’d have to parse it to decide if it’s happy. Similarly, checking that the JVM is up and running doesn’t tell us whether Ignite is able to respond to requests.</p><p>Is there a way to use some of Ignite’s built-in functionality to define a simple test? I think so. Let’s try the REST API:</p><pre>    containers:<br>      - name: ignite-node<br>        image: apacheignite/ignite:2.7.0<br>        env:<br>        - name: OPTION_LIBS<br>          value: ignite-kubernetes,ignite-rest-http<br>...<br>        livenessProbe:<br>          httpGet:<br>            path: /ignite?cmd=version<br>            port: 8080<br>          initialDelaySeconds: 30<br>          periodSeconds: 10</pre><p>There are more options you might like to tweak (failureThreshold, for example, defines how many times it’s allowed to fail before the pod is restarted, defaulting to three), but those last six lines are all we need. InitialDelaySeconds allows a grace-period before the check is executed — we don’t want to ping it while the node is still starting up.</p><p>You could also change the end-point to do something more sophisticated, but be careful because you want the status of <em>this</em> Ignite node and not that of the whole cluster. Don’t, for example, query for a particular value in a cache as you won’t know which node it’s stored on!</p><p>That’s it for this time, but stay tuned as I’m hoping to make these posts a little more frequent this year.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9ebeed63658e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Ignite 2.7 on Kubernetes]]></title>
            <link>https://medium.com/@sdarlington/ignite-2-7-on-kubernetes-7dc32bca9461?source=rss-bed65416e445------2</link>
            <guid isPermaLink="false">https://medium.com/p/7dc32bca9461</guid>
            <category><![CDATA[docker]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[apache-ignite]]></category>
            <dc:creator><![CDATA[Stephen Darlington]]></dc:creator>
            <pubDate>Mon, 10 Dec 2018 14:16:22 GMT</pubDate>
            <atom:updated>2018-12-10T14:16:22.772Z</atom:updated>
            <content:encoded><![CDATA[<p>While I’m writing about running Apache Ignite on Kubernetes… it turns out that there’s a bug in the latest version (2.7) that came out last week. Starting Ignite from the standard Docker image with ignite-kubernetes enabled <a href="http://apache-ignite-users.70518.x6.nabble.com/ignite-kubernetes-seems-to-be-missing-the-jackson-annotations-dependency-td25670.html">will result in a NoClassDefFoundError exception</a>.</p><p>There’s a simple workaround: you can add ignite-aws or ignite-rest-http to your list of optional packages. That doesn’t commit you to using them; they just happen to include the missing dependency.</p><pre>...<br>    containers:<br>      - name: ignite-node<br>       image: apacheignite/ignite:2.7.0<br>        env:<br>        - name: OPTION_LIBS<br>          value: ignite-kubernetes,ignite-rest-http</pre><p>There’s a <a href="https://issues.apache.org/jira/browse/IGNITE-10577">ticket for this</a>. Hopefully it will be fixed soon.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7dc32bca9461" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Activating an Apache Ignite cluster on Kubernetes]]></title>
            <link>https://medium.com/@sdarlington/activating-an-apache-ignite-cluster-on-kubernetes-afbed40c7e53?source=rss-bed65416e445------2</link>
            <guid isPermaLink="false">https://medium.com/p/afbed40c7e53</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[apache-ignite]]></category>
            <dc:creator><![CDATA[Stephen Darlington]]></dc:creator>
            <pubDate>Fri, 07 Dec 2018 16:03:26 GMT</pubDate>
            <atom:updated>2018-12-07T16:03:26.072Z</atom:updated>
            <content:encoded><![CDATA[<p>A couple of weeks ago I did a talk at the OpenStack conference about deploying <a href="https://www.openstack.org/videos/berlin-2018/distributed-database-devops-dilemmas-kubernetes-to-the-rescue">Apache Ignite on the Kubernetes</a> container orchestration system. As with any talk, with only so much time to work with, some things ended up on the cutting room floor.</p><p>One of those things was how to activate the cluster when using Ignite’s native persistence support. I mentioned that you needed to do it — since it’s one of the things that complicates using a distributed database in these environments — but I didn’t really say <em>how</em>. In the demo, I just logged straight into one of the already running pods and ran control.sh --activate. You could argue that this is not an unreasonable thing to do. Activating the cluster is something you only do once so a bit of a hack, something that’s not very Kubernetes-ish, wouldn’t be the <em>worst</em> thing you could do.</p><p>But, we can do better.</p><p>One option might be to use kubectl exec. You can run a command something like this:</p><pre>kubectl exec -it ignite-0 --namespace=ignite -- /opt/ignite/apache-ignite-fabric/bin/control.sh --activate</pre><p>That would work. But to my way of thinking this is no better than directly <em>ssh</em>-ing into the pod and manually running a command. Having said that, one nice thing about this command is that it relies on one of the features of <a href="https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/">StatefulSets</a>: unique and predictable names for all the pods. The first pod is always going to have a zero postfix.</p><p>My second thought was to expose the appropriate ports in the cluster and connect from the client using Ignite’s --host and --port options. <a href="http://apache-ignite-users.70518.x6.nabble.com/cluster-activation-on-k8s-when-native-persistence-is-enabled-td25599.html">I was pretty confident that this would do the trick</a>:</p><pre>./control.sh --host service --port 30000 --activate</pre><p>I was wrong.</p><p>In hindsight it was pretty obvious why. While the service was exposing itself (pardon the expression) on port 11211, the script was trying to access it on port 30000. That would be confusing enough, but there were probably other ports that it was trying to access (47100, 47500), all of which would have been available on the wrong port. Even using an <em>ssh</em> tunnel I couldn’t get it working. That’s not to say that it couldn’t be made to work, but we can do better.</p><p>The <em>real</em> way to do it is to create a Kubernetes job.</p><p>I was trying to avoid this since I thought it might be quite tricky but it turned out to be fairly straightforward.</p><pre>apiVersion: batch/v1<br>kind: Job<br>metadata:<br> name: ignite-activate<br>spec:<br> ttlSecondsAfterFinished: 100<br> template:<br> spec:<br> containers:<br> — name: ignite-image<br> image: apacheignite/ignite:2.6.0<br> command: [“/opt/ignite/apache-ignite-fabric/bin/control.sh”, “ — activate”, “ — host ignite”]<br> restartPolicy: Never</pre><p>Then you kick it off with kubectl create -f ignite-activate.yaml.</p><p>It’s not rocket-science, but here a couple of small things that I missed on my first pass. Number one, you and I know that it only needs to be run once, that it’s not a daemon. Kubernetes doesn’t know that, we need to tell it. That’s the restart policy. Without that line, it’ll keep kicking off the job.</p><p>Next, the “ttlSecondsAfterFinished” line. This is currently in alpha, so, depending on the version of the server that you’re running, it either might not work at all or you’ll need to enable experimental features. But it’s neat so I thought I should include it. After running the job, it waits 100 seconds — after you’ve had time to check the logs — it tidies up after itself, deleting pods, and any other resources it used. Without that flag, you’ll have to do that step yourself. So much for automation.</p><p>In the end, the “best” way might also actually be the easiest. How often does that happen?</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=afbed40c7e53" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Apple Watch Series 4 first thoughts]]></title>
            <link>https://medium.com/@sdarlington/apple-watch-series-4-first-thoughts-5c3993e0ca8e?source=rss-bed65416e445------2</link>
            <guid isPermaLink="false">https://medium.com/p/5c3993e0ca8e</guid>
            <category><![CDATA[review]]></category>
            <category><![CDATA[apple-watch]]></category>
            <category><![CDATA[gadgets]]></category>
            <category><![CDATA[exercise]]></category>
            <dc:creator><![CDATA[Stephen Darlington]]></dc:creator>
            <pubDate>Mon, 01 Oct 2018 17:02:27 GMT</pubDate>
            <atom:updated>2018-10-01T17:02:27.463Z</atom:updated>
            <content:encoded><![CDATA[<h3>Apple Watch Series 4</h3><figure><img alt="" src="https://cdn-images-1.medium.com/proxy/1*te3iBfLu-CJurvqfX9TCwQ.jpeg" /></figure><p>This was going to be my first thoughts on the new Apple Watch, posted about a day after I unboxed it. But then life got in the way and I’ve now been using it for a week. What I’d planned to be a little more than a “hot take” is now probably a little closer to an actual review!</p><p>From the picture above you can see what I was using before. It’s not an Apple Watch, it’s not a competing smart watch. It doesn’t even have a battery. (To be clear, my new watch isn’t a replacement. I still plan to wear the Marloe.)</p><p>I’ll be honest: until recently I wasn’t very interested in the Apple Watch. It was too big, didn’t look nice enough, was another thing to recharge every night and none of the things it did were compelling enough to overcome those annoyances.</p><p>Here’s the thing. The Series 4 is still too big — about twice as thick as the Marloe — and doesn’t look as good as a traditional watch. But I’ve been on a health kick for the last year, so partly I bought it as a new gadget, a reward for keeping going for twelve months, and partly, because of that, the health features became compelling enough.</p><p>As you’d expect for Apple, the hardware looks great and is well presented. The finish is beautiful, the controls click and rotate in a satisfying way. I found it odd that it came without the strap attached but I was able to get it up and running pretty quickly.</p><p>I’ve <em>read</em> about the Watch before but I’ve never got beyond playing around with one in a store. It surprised me how complicated it is. By which don’t mean it’s hard to use, rather that there are a lot of choices to make. Choices that you probably won’t know the answer to.</p><p>The new Infographic watch faces <em>sound</em> great, but what am I supposed to do with eight complications? Not having used the Watch before, I don’t know the best way of accessing apps versus having the information available instantly on the watch face. Should I use the app swarm? The most recently used list? Complications? Notifications? Then you find that <em>some</em> complications only appear in <em>some</em> locations. How am I supposed to know this stuff? It’s confusing. I’m sure I’ll figure this stuff out but I feel information overload just as I’m trying to play with my new toy. (Having started writing a Watch app I think I understand it better, but should I have to?)</p><p>I guess I was expecting an iPhone OS 1.0-type experience — just the bare minimum — but I suppose that’s an unfair comparison and an unfair expectation. We’re now at watchOS 5, so it’s had time to evolve. And in hardware terms the Watch is way faster than than the first few iPhones. (It is incredible that you can get a dual core 64-bit CPU on your wrist!)</p><p>Having waited until now to get a Watch, I’ve also <em>seen</em> features that I was concerned about. The main one: the not-always-on screen. Would I forever be wiggling my wrist to see the time?</p><p>No, is the short answer. It’s not 100% successful turning the screen on at the right times but it’s pretty close. The problems I’ve had have been more along the lines of the screen not staying switched on for quite long enough. Maybe I was too slow to read a notification, or I was showing my wife something neat, but I do occasionally find myself wiggle my wrist around just as I feared. Fortunately the times it happen lead me to think that, as I use the watch more, these occasions will happen less and less. I could be wrong, but I think this will work out fine.</p><p>Another example of complexity: if I want to go running, what app do I start? Strava is the obvious one. That’s what I use on my phone. But what about the Workouts app? Do I need that instead? As well? One of the features that attracted my to the S4 was the ability to see my pace for the last kilometre… but how do I get that while recording my GPS trail? Clearly this isn’t an insurmountable problem, but it’s one without an obvious solution. Some experimentation will be needed.</p><p>However, putting the niggles and unexpected complexity aside, how has it been? My experience in this first week has been great. Being able to quickly glance at notifications without pulling out my phone has been very handy. Walking around a strange city with it tapping my wrist with the directions is way more convenient, as are the notifications telling me which gate my flight boards. And being able to record a run without clutching my phone or have it strapped to my arm is surprisingly liberating.</p><p>I’ve heard people say that Watch apps are not great. Maybe that’s true on average but I’ve been impressed with <a href="https://www.strava.com/">Strava</a> and, especially, <a href="https://overcast.fm/">Overcast</a>.</p><p>All these things are slight reductions in friction or increases in convenience and are not necessarily compelling alone but when you add it all up it’s impressive. On paper, I thought the features were nice but not quite worth the asking price. I would still say it’s a luxury rather than a must have but the utility is certainly there.</p><p><em>First published </em><a href="https://www.zx81.org.uk/blog/apple-watch-series-4.html"><em>on my blog</em></a><em>.</em></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5c3993e0ca8e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Loading data into Apache Ignite]]></title>
            <link>https://medium.com/@sdarlington/loading-data-into-apache-ignite-c0cb7c065a7?source=rss-bed65416e445------2</link>
            <guid isPermaLink="false">https://medium.com/p/c0cb7c065a7</guid>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[apache-spark]]></category>
            <category><![CDATA[scripting]]></category>
            <category><![CDATA[apache-ignite]]></category>
            <dc:creator><![CDATA[Stephen Darlington]]></dc:creator>
            <pubDate>Thu, 13 Sep 2018 19:16:46 GMT</pubDate>
            <atom:updated>2018-09-13T19:16:46.823Z</atom:updated>
            <content:encoded><![CDATA[<p><a href="https://ignite.apache.org/">Ignite</a> has all kinds of helpful APIs to make it simple load data in, most notably <a href="https://apacheignite.readme.io/docs/streaming--cep">Data Streamers</a>. But, sometimes you don’t want to start your favourite IDE and start bashing out code just to load in a text file.</p><p>The rest of this piece shows a number of ways that you can achieve this. One caveat I want to share before we get started: you probably don’t want to use any of these solutions in a production environment. These are mostly for experimentation. Maybe you want to play with another feature and need to quickly load in some reference data (how about taking a look at the machine learning APIs?). Or perhaps you want to try load in some data so you can ping it with one of the third-party integrations?</p><h3>Loading files using SQL</h3><p>Probably the simplest method is to use Ignite’s SQL interface.</p><p>If your data is already in CSV format, you can use the <a href="https://apacheignite-sql.readme.io/docs/copy">COPY FROM</a> command. For example:</p><pre>sqlline -u jdbc:ignite:thin://127.0.0.1<br>0: jdbc:ignite:thin://127.0.0.1&gt;COPY FROM &quot;file.csv&quot; INTO tablename (col1, col2) FORMAT CSV;</pre><p>Sadly, that command doesn’t have any options or levers to tweak if your data isn’t in exactly the right format.</p><p>As any savvy Unix user will tell you, you can do wonders with a few commands in the shell. If you know the magic incantations.</p><p>You can do simple add/remove quotes using sed or your favourite text editor. You can easily remove columns (or reorder them) using cut.</p><p>But what if your data file is not anything like CSV?</p><p>If your file is in JSON format, you can convert into CSV with something like the following:</p><pre>jq &#39;(map(keys) | add | unique) as $cols | map(. as $row | $cols | map($row[.])) as $rows | $cols, $rows[] | @csv&#39; &lt; file.json &gt; file.csv</pre><p>Thanks to <a href="https://stackoverflow.com/a/32965227/2998">Stack Overflow</a> for that piece of magic. (If you’re on a Mac, you can install the super-useful jq using <a href="https://brew.sh/">brew</a>.)</p><p>Once your command line gets to a point where it looks like line noise, I usually decide that I want a simpler solution.</p><h3>Loading files using Spark</h3><p><a href="https://spark.apache.org/">Spark</a> might strike you as a bit of a sledge-hammer to crack the data loading nut, but it has a number of advantages. First, it has a REPL, which mean you can experiment much more quickly than with a typical code-compile-run cycle. Secondly, if you use PySpark, there are a lot of useful libraries you can call on.</p><p>Out of the box, Spark doesn’t know about Ignite so we have to start it with a couple of extra parameters:</p><pre>bin/pyspark --jars $IGNITE_HOME/libs/ignite-spring/*.jar,$IGNITE_HOME/libs/optional/ignite-spark/ignite-*.jar,$IGNITE_HOME/libs/*.jar,$IGNITE_HOME/libs/ignite-indexing/*.jar</pre><p>Next, load the data into a Spark DataFrame:</p><pre>b = spark.read.format(&#39;json&#39;).load(&#39;filename.json&#39;)</pre><p>The nice thing about this is that Spark reads and understands the structure of the JSON:</p><pre>&gt;&gt;&gt; b.printSchema()<br>root<br> |-- description: string (nullable = true)<br> |-- extended: string (nullable = true)<br> |-- hash: string (nullable = true)<br> |-- href: string (nullable = true)<br> |-- meta: string (nullable = true)<br> |-- shared: string (nullable = true)<br> |-- tags: string (nullable = true)<br> |-- time: string (nullable = true)<br> |-- toread: string (nullable = true)</pre><p>You can define the schema if it’s got anything wrong (in this case, it’s taken the time column to be a string rather than a timestamp). Depending on your level of experience, you may prefer to fix these errors in Spark or later on in Ignite; whatever works.</p><p>(In case you were wondering, the JSON file I’ve been using is an export of all my bookmarks from <a href="https://pinboard.in/">pinboard.in</a>.)</p><p>You can do any manipulation you like at this point — adding or removing columns, computing new values, changing data types, filtering records — and then write it out to Ignite:</p><pre>b.filter(&#39;href is not null&#39;) \<br>.drop(&#39;hash&#39;, &#39;meta&#39;) \<br>.write.format(&#39;ignite&#39;) \<br>.option(&#39;config&#39;,&#39;default-config.xml&#39;) \<br>.option(&#39;table&#39;,&#39;bookmarks&#39;) \<br>.option(&#39;primaryKeyFields&#39;,&#39;href&#39;) \<br>.mode(&#39;overwrite&#39;) \<br>.save()</pre><p>And that’s it. No manipulation using sed or awk, just read in JSON, write to Ignite, complete with a schema. Nice.</p><h3>Using IgniteDataStreamer</h3><p>Okay, so I started this piece saying that we were looking for quick, easy ways of loading data into Ignite without coding, but I’d be remiss not to mention that using the “official” method really isn’t <em>that</em> hard. There’s even a nice example included with Ignite: <a href="https://github.com/apache/ignite/blob/master/examples/src/main/java/org/apache/ignite/examples/streaming/StreamTransformerExample.java">StreamVisitorExample.java</a></p><h3>Conclusion</h3><p>In conclusion, the easiest way to get data into Ignite without coding is to have the data already in CSV format. But failing that, with a combination of Unix shell tools and higher level tools like Spark you can go a long way without writing a single line of Java.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c0cb7c065a7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[C25K Diary (Part 3)]]></title>
            <link>https://medium.com/@sdarlington/c25k-diary-part-3-fa40a2ffdf65?source=rss-bed65416e445------2</link>
            <guid isPermaLink="false">https://medium.com/p/fa40a2ffdf65</guid>
            <category><![CDATA[running]]></category>
            <category><![CDATA[exercise]]></category>
            <category><![CDATA[c25k]]></category>
            <dc:creator><![CDATA[Stephen Darlington]]></dc:creator>
            <pubDate>Sat, 24 Mar 2018 10:28:44 GMT</pubDate>
            <atom:updated>2018-03-24T10:28:44.892Z</atom:updated>
            <content:encoded><![CDATA[<p>To recap: I started doing the Couch to 5K programme to improve my general fitness. I picked it because it provided a goal and because it didn’t require any specialised gear. I had a few struggles along the way (see parts <a href="https://medium.com/@sdarlington/c25k-diary-7d4cd21c7911">1</a> and <a href="https://medium.com/@sdarlington/c25k-diary-part-2-fbb4a30a238e">2</a>) but eventually completed week 6 of 8. It should be simple to complete the last two weeks, right?</p><p>Sadly, no.</p><p>I finished part two with the following advice to myself:</p><blockquote>the lesson was to push myself but not to push <em>too</em> hard, to back off a little and not to be afraid to repeat either individual days or whole weeks.</blockquote><p>Of course, I didn’t follow my own advice for week 7.</p><p>There was a reason for my cavalier attitude: it was coming up to Christmas and I wasn’t sure how much I would be able to keep jogging over the break. I ambitiously thought that I <em>might</em> be able to complete a 5km run before the end of the year. (Spoiler alert: I didn’t.)</p><p>But, whatever the reason, I surprised myself by managing it. I successfully completed week 7, day 3 on December 18. I did slow down bit by bit over the week, but I achieved my goal of jogging for the whole time (which was 25 minutes without stopping at this point).</p><p>But I’m writing this in March! What happened?</p><p>First: life.</p><p>I went away for Christmas so I wasn’t able to continue my normal routine. I <em>did</em> bring my jogging gear with me and slightly surprised myself by actually going out. I didn’t even try to start week 8 at this point, instead I just set a rough target of being out for about thirty minutes. I didn’t even really plot a route in advance which was probably a mistake. You don’t get hills and traffic lights and pedestrians on a treadmill!</p><p>Running on real roads and paths gave a nice sense of camaraderie that you don’t get in the gym, with people smiling and saluting as I lumbered and sweated along.</p><p>Back in the UK in January I tried to pick up where I left off but immediately hit difficulties. Given that I’d not maintained my routine for a few weeks this was not entirely unexpected, but the amount of time it took to get back on track did.</p><p>After a number of failed runs and false starts I ended up going all the way back to week 6. I settled on day 2 for a while (jog ten minutes, walk three, jog ten) for a few runs, feeling unable even to progress to the last day (22 minutes run).</p><p>And this is where things stayed for over a month. Not only was I not really progressing, I was also not very consistent. One time I would manage twenty minutes straight, the next ten minutes was a challenge.</p><p>I’d gradually inched up my speed over the previous month, so I tried slowing down again but it didn’t have much effect. It seemed to be the amount of time running rather than strictly the speed.</p><p>I was still getting way more exercise than I was previously, so while I wasn’t <em>failing</em> exactly, I <em>did</em> want to have the satisfaction of completing the programme. This lack of progress was getting frustrating.</p><p>Then I hit on a theory: I wasn’t resting enough between runs. At this point I had been continuing to run three times a week, typically on Monday, Wednesday and Friday. While I hadn’t really been recording my progress, anecdotally my better runs were on Mondays.</p><p>To test my theory I switched my runs to Mondays and Thursday. It turns out my hunch was right. It took a couple of weeks from this point but on 3 March I completed the programme, running for the full thirty minutes. A week later I managed it again but at the higher speed.</p><p>So <em>technically</em> I’ve still not run 5km yet. It turns out the programme is for 5km <em>or</em> thirty minutes (I guess the full name is less snappy). My exercise <em>session</em>, including the warm-up and cool-down, is over 5km now so I don’t feel that I’m cheating too much!</p><p>So, what now? I have two goals. First, I want to improve my consistency. I still have some better and some worse days. I think mostly I just need to keep at it. Secondly, I want to up my speed. For the next few months I want to get to doing the full thirty minutes at 10km/h so I can say that I actually did a full 5km.</p><p>Non-goals: running further or longer. I only have so many hours in the day!</p><p>Anyway, I kind of amazed myself by finishing. If you’ll excuse me, I’m tired and need a nap.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=fa40a2ffdf65" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>