<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Nabil Servais on Medium]]></title>
        <description><![CDATA[Stories by Nabil Servais on Medium]]></description>
        <link>https://medium.com/@nabil.servais?source=rss-c9e8b7766c1e------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*Hh1KoGTmL2UnNtP2</url>
            <title>Stories by Nabil Servais on Medium</title>
            <link>https://medium.com/@nabil.servais?source=rss-c9e8b7766c1e------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 12:14:56 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@nabil.servais/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Serverless Spatial Analysis with DuckDB and AWS Lambda — Part 1 “Making it work”.]]></title>
            <link>https://medium.com/@nabil.servais/serverless-spatial-analysis-with-duckdb-and-aws-lambda-part-1-making-it-work-e2664295624b?source=rss-c9e8b7766c1e------2</link>
            <guid isPermaLink="false">https://medium.com/p/e2664295624b</guid>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[duckdb]]></category>
            <category><![CDATA[big-data]]></category>
            <dc:creator><![CDATA[Nabil Servais]]></dc:creator>
            <pubDate>Fri, 17 Feb 2023 21:27:02 GMT</pubDate>
            <atom:updated>2023-02-18T21:23:26.568Z</atom:updated>
            <content:encoded><![CDATA[<h3>Serverless Spatial Analysis with DuckDB and AWS Lambda — Part 1 “Making it work”.</h3><p>And it’s not expensive.</p><p>Geospatial analysis is not limited to maps, it is present every day in many areas of our lives (TODO examples with illustration). However, the specificities make more complicated the storage and the calculation related to the spatial functions, how to represent a point or a polygon? How to detect on a point is or how to calculate the distance between a point and a polygon? How to take into account that the map is flat but the earth is a patatoid.<br>All the answers to these questions can be found in JTS and its derivatives (like GEOS). Moreover, one of the derivatives made it possible to bring spatial data into PostgreSQL via its PostGIS extension.<br>Even OLAP databases integrate spatial components (such as bigquery or its competitor Redshift based on PostgreSQL) and even the latest DuckDB since very recently (based on PostGIS).<br>An interesting feature of DuckkDB is that it is truly serverless like SQLite. No server, no infrastructure needed to use. In addition, it is now possible to decouple storage and computation like BigQuery.<br>DuckDB can query CSV or Parquet files available on an S3 bucket or locally.<br>In a context where keeping a PostgreSQL instance costs a certain amount and consumes energy, we will see how to use DuckDB and its Geo extension in a Serverless context.</p><p>After an excellent experience with Cloud Run, I felt it was time to explore AWS Lambda.</p><p>disclaimer: my employer Mangrove (AWS partner) is not involved in this project and it is entirely possible to replicate this architecture and the work done with Cloud Run (GCP) or Azure Function (Azure) or even other FaaS providers.</p><p>The initial data provided for this project will be the Land Value Applications in Open Data. They have been adapted and reformatted by CEREMA to be usable by the mapping tools (many thanks to them and to all those who have made the use of this data possible).</p><p>The architecture is quite simple : an object storage, an AWS lambda and an http interface. Where a more classical architecture would be more complicated and costly.</p><p>Simplified architecture diagram</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/748/1*DFT1StgFXYq2CxhM6XotMQ.png" /></figure><p>This gives on AWS</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/469/0*iM73xp4IbMCerUt5" /></figure><p>How can you have such a simple cloud architecture?<br>First of all, DuckDB can be considered as the SQLite of OLAP databases, that is to say . Like its equivalent, it can be embedded in memory in its code or be standalone. I really invite you to read the documentation and to look at the state of DuckDB .</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2FrdnPkLSkoyU%3Ffeature%3Doembed&amp;display_name=YouTube&amp;url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DrdnPkLSkoyU&amp;image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FrdnPkLSkoyU%2Fhqdefault.jpg&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=youtube" width="854" height="480" frameborder="0" scrolling="no"><a href="https://medium.com/media/37eccc3aa6ec526a8473bd75bf3761e0/href">https://medium.com/media/37eccc3aa6ec526a8473bd75bf3761e0/href</a></iframe><p>DuckDB like most other OLAP databases can query parquet or CSV files (in addition to its internal storage but I’ll test that another time).<br>This also has an impact in terms of cost, an RDS database to perform this kind of operation would cost $189 not to mention the ecological impact (the database server needs to be turned on constantly).</p><p>DuckDB’s Geo extension is largely based on PostGIS, which allows many of the functions available in PostGIS to be reused.</p><p><a href="https://github.com/handstuyennn/geo">GitHub - handstuyennn/geo</a></p><p>The extension compiles quite easily (just like DuckDB), however it is still under active development which does not exclude some compile bugs, so do not hesitate to give feedback.<br>Before starting the application, it is first necessary to test if spatial analysis is possible.<br>First step, the transformation of gpkg files into parquet. As mentioned before, DuckDB only reads parquet, CSV and JSON files (sqlite is supported natively as an external extension). Fortunately, the Swiss army knife of map files natively supports the parquet format.</p><p>Now the file is ready, you can now perform data mining on your computer without having to bring out the heavy artillery.<br>DuckDB has the good idea to sign its extensions, however Geo is not yet an official extension, it will be necessary to disable the checks.<br>Queries can be made directly on the parquet file.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*dkF5A_vTlqfC-wXO" /></figure><p>Or in memory after import.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*2tqoNyVsaHKYrqY6" /></figure><p>Determining the average price of goods as well as other indicators according to the distance to a point is quite easy (it could also be done according to a polygon).<br>All that remains is to upload the file to S3 and we are ready.<br>As simple as the analysis part was, the compilation for AWS Lambda will be much less simple.<br>First try — Python layer (spoiler: it was a failure)<br>AWS Lambda may be a serverless service, but it still requires an OS to run. And when the OS in question is based on a slightly dated Linux distribution, it makes packaging the application more complex.<br>For reasons that escape me, building the layer proved impossible, impossible for the function to find these libraries. I think I’ll come back to this in more detail (and in English this time).<br>Second try — What about Go or Rust?<br>I could have gone for Go and Rust, two excellent, efficient languages, but I chose to be safe by choosing a language officially supported by the DuckDB team.<br>Now I have an application able to execute DuckDB queries in an AWS Lambda. I will also need to compile the Amazon Linux 2 extension, as the dependencies are already available, the compilation will (if any AWS Lambda developers are reading me, please upgrade to Amazon Linux 2022).<br>Shortly after my success, a user pointed out the existence of this repo which would have saved me several hours of work (fortunately the macbook M1 is really powerful).</p><p><a href="https://github.com/tobilg/duckdb-nodejs-layer">GitHub - tobilg/duckdb-nodejs-layer: Packaging DuckDB for Lambda functions. Example application: https://github.com/tobilg/serverless-duckdb</a></p><p>Also, the article from the same author go deeper to how to deploy DuckDB on AWS Lambda.</p><p><a href="https://tobilg.com/using-duckdb-in-aws-lambda">Using DuckDB in AWS Lambda</a></p><p>Remember to strip the extension before packing it into its function, copy all the functions into the directory and you are ready.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/928/0*Ju4SOx6bRBrMcY4-" /></figure><p>To make the requests, nothing very magical, only javascript code, quickly done badly to prove that it works.</p><p>We are instantiating our libraries.</p><pre>const AWSXRay = require(&#39;aws-xray-sdk-core&#39;);<br>const duckdb = require(&#39;duckdb&#39;)<br>var collect = require(&#39;@turf/boolean-intersects&#39;);<br>var db = new duckdb.Database(&#39;:memory:&#39;, {&quot;allow_unsigned_extensions&quot;:&quot;true&quot;});;</pre><p>We configure DuckDB.</p><pre>const bucket = process.env.S3_BUCKET</pre><pre>db.run(`SET threads TO 4;`, function (err, res) {</pre><pre>if (err) {</pre><pre>throw err;</pre><pre>}</pre><pre>console.log(res);</pre><pre>});</pre><pre>db.run(`PRAGMA temp_directory=&#39;/tmp/&#39;` ,</pre><pre>function(err, res) {</pre><pre>if (err) {</pre><pre>throw err;</pre><pre>}</pre><pre>console.log(res);</pre><pre>});<br></pre><pre>const conn = new duckdb.Connection(db);</pre><pre>conn.run(`LOAD &#39;./geo.duckdb_extension&#39;;`,</pre><pre>function(err, res) {</pre><pre>if (err) {</pre><pre>throw err;</pre><pre>}</pre><pre>console.log(res);</pre><pre>});<br></pre><pre>conn.run(`LOAD &#39;./parquet.duckdb_extension&#39;;`,</pre><pre>function(err, res) {</pre><pre>if (err) {</pre><pre>throw err;</pre><pre>}</pre><pre>console.log(res);</pre><pre>});<br></pre><pre>conn.run(&quot;LOAD &#39;./httpfs.duckdb_extension&#39;;&quot;,</pre><pre>function(err, res) {</pre><pre>if (err) {</pre><pre>throw err;</pre><pre>}</pre><pre>console.log(res);</pre><pre>});</pre><p>To &quot;speed up&quot; the search, the departments are indexed like an index but I will come back to this in part II (make it fast).</p><pre>conn.run(&quot;CREATE TABLE index_dep AS SELECT * EXCLUDE geometry, ST_GEOGFROMWKB(geometry) AS geom FROM &#39;s3://&quot;+ bucket +&quot;/immobiliers/departements.parquet&#39;&quot;</pre><pre>function(err, res) {</pre><pre>if (err) {</pre><pre>throw err;</pre><pre>}</pre><pre>console.log(res);</pre><pre>});</pre><p>And finally we assemble our request with the javascript code (I promise if this post is more than 10 likes, I’ll start using typescript).</p><pre>let response</pre><pre>exports.handler = async (event, context) =&gt; {</pre><pre>try {</pre><pre>let r = [];</pre><pre>const body = JSON.parse(event.body);</pre><pre>console.log(&quot;EVENT: \n&quot; + JSON.stringify(event));</pre><pre>console.log(body);</pre><pre>const fc = body.feature;</pre><pre>const distance = body.distance;</pre><pre>console.log(distance);</pre><pre>let query_dep = &quot;SELECT code FROM index_dep WHERE ST_CONTAINS(geom, ?)&quot;;</pre><pre>const stream_dep = conn.stream(query_dep, JSON.stringify(fc.features[0].geometry));</pre><pre>for await (const row of stream_dep) {</pre><pre>console.log(row);</pre><pre>r.push(row);</pre><pre>}</pre><pre>if (r.length == 0)</pre><pre>return {statusCode:200, body: JSON.stringify({resuls:0})}</pre><pre>num_dep = r[0].code;</pre><pre>console.log(num_dep);</pre><pre>r = []</pre><pre>let query_bien = `</pre><pre>WITH biens as (</pre><pre>SELECT libtypbien,</pre><pre>anneemut,</pre><pre>ST_PERIMETER(ST_GEOGFROMWKB(geom), true) AS surface,</pre><pre>valeurfonc,</pre><pre>ST_DISTANCE(?, ST_GEOGFROMWKB(geom)) as dist,</pre><pre>? as distance,</pre><pre>FROM &#39;s3://` + bucket + `/immobiliers/mutations_d` + num_dep + `.parquet&#39;</pre><pre>WHERE dist &lt; distance</pre><pre>)</pre><pre>SELECT libtypbien AS &quot;type de bien&quot;,</pre><pre>anneemut AS &quot;annnée de vente&quot;,</pre><pre>count(1) as &quot;nombre de biens&quot;,</pre><pre>ROUND(SUM(surface), 2) AS &quot;surface totale&quot;,</pre><pre>ROUND(AVG(valeurfonc), 2) AS &quot;valeur fonciere moyenne&quot;,</pre><pre>ROUND(MAX(valeurfonc), 2) AS &quot;valeur fonciere max&quot;,</pre><pre>ROUND(MIN(valeurfonc),2) AS &quot;valeur fonciere min&quot;,</pre><pre>MIN(dist),</pre><pre>MAX(dist),</pre><pre>FROM biens GROUP BY all ORDER BY &quot;nombre de biens&quot;;</pre><pre>`</pre><pre>console.log(fc.features);</pre><pre>const stream = conn.stream(query_bien, JSON.stringify(fc.features[0].geometry), distance);</pre><pre>for await (const row of stream) {</pre><pre>console.log(row);</pre><pre>r.push(row)</pre><pre>}</pre><pre>response = {</pre><pre>statusCode: 200,</pre><pre>body: JSON.stringify({</pre><pre>results: r,</pre><pre>}),</pre><pre>};</pre><pre>} catch (err) {</pre><pre>console.log(err);</pre><pre>return err;</pre><pre>}<br></pre><pre>return response;</pre><pre>};<br></pre><pre>;</pre><p>Now that I have an API to query my data, I need a user interface. Fortunately there are good data providers outside of Google Maps, and the choice fell on Maptiler which supports maplibre, a graphical web map viewer.</p><p>AND THERE you can click on the map and get an analysis of property prices over a given distance (I think I’ll add the polygon mapping soon enough).</p><p><a href="https://gnopdata-public.s3.eu-west-1.amazonaws.com/index.html">https://gnopdata-public.s3.eu-west-1.amazonaws.com/index.html</a></p><p>Feel free to use it and give feedback.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=e2664295624b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Boosting your analytics application with connector-x and DuckDB]]></title>
            <link>https://medium.com/@nabil.servais/boosting-your-analytics-application-with-connector-x-and-duckdb-9c75e528157b?source=rss-c9e8b7766c1e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9c75e528157b</guid>
            <category><![CDATA[open-source]]></category>
            <category><![CDATA[data-engineering]]></category>
            <dc:creator><![CDATA[Nabil Servais]]></dc:creator>
            <pubDate>Mon, 27 Jun 2022 21:34:27 GMT</pubDate>
            <atom:updated>2022-06-27T21:34:27.352Z</atom:updated>
            <cc:license>https://creativecommons.org/licenses/by-nc/4.0/</cc:license>
            <content:encoded><![CDATA[<p><em>You are waiting Alloy and Unistore ? Try now to use analytics in your application with connector-x and DuckDB (Python only for now).</em></p><h3>OLTP vs OLAP</h3><p>There is a lot of articles but my favorite explication about the difference between OLTP and OLAP is from <a href="https://clickhouse.com/docs/en/intro#why-column-oriented-databases-work-better-in-the-olap-scenario">Clickhouse</a>. You should really read this page before reading the rest of the article.</p><p>But in most case, transactional databases are still the main source for good reasons : easier to insert data, better supports, more safety for the data…</p><p>The usage of data are changing and most users want to have the best of both worlds : a fast database for analytics and easy to use.<br>and that why we saw TimescaleDB, the Clickhouse’s pgsql and mysql engines, alloy or unistore.</p><p>This is also possible with Apache Arrow, DuckDB and Connector-X. Three simple and very powerful libraires.</p><h3>Apache Arrow and friends</h3><p>Apache Arrow is a game changer in the world of data. We have a library for normalize the columnar formats in most languages (C++, Python, Java,Rust…). <br>It also permits to share in a fast way data in applications (JSON wasn’t an option).</p><p>There is a lot of avantage to use Arrow in your analytics application and most of application use it out of the box. It can act as a glue between applications of other librairies.</p><h3>DuckDB and Connector-x</h3><p>For creating data applications, you have a lot of choice. If you want to have 0 maintenance for the database and a fast engine, the best choice is DuckDB.</p><p><a href="https://duckdb.org/">DuckDB</a> supports parquet or CSV files and also scanning arrow tables as a source.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/274/1*ysNJ13RtyvYb-VHILWpo2w.png" /></figure><p>For transfer data, we need to transform the data in arrow. And the connector-x allow you to do that. <a href="https://github.com/sfu-db/connector-x/">Connector-x</a> is based on <a href="https://www.pola.rs/">polars</a> (a rust dataframe library) and <a href="https://github.com/launchbadge/sqlx">sqlx</a>. All is written in rust and it’s damn fast and propose python binding.</p><p>Installation</p><blockquote>pip install pyarrow duckdb connectorx</blockquote><p>And as you can see in the examples, arrow is used as intermediaries between DuckDB and the SQLite throw connector-x.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/1f616e62c4407c5d02d31762846d433d/href">https://medium.com/media/1f616e62c4407c5d02d31762846d433d/href</a></iframe><p>Tadaaaa! In most case it could help to start analytics stuff.</p><p>Connector-x is compatible with a lot of database engine.</p><h3>What’s next ?</h3><p>Now we know how to query a OLTP, we will see how to use streamlit with duckDB.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9c75e528157b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How parse and aggregate a big CSV in python, when you have a slow computer and Internet connection.]]></title>
            <link>https://medium.com/@nabil.servais/how-parse-and-aggregate-a-big-csv-in-python-when-you-have-a-slow-computer-and-internet-connection-9030e74a9bdc?source=rss-c9e8b7766c1e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9030e74a9bdc</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[big-data]]></category>
            <category><![CDATA[aws]]></category>
            <dc:creator><![CDATA[Nabil Servais]]></dc:creator>
            <pubDate>Sun, 18 Jul 2021 11:08:01 GMT</pubDate>
            <atom:updated>2021-07-18T11:08:01.395Z</atom:updated>
            <content:encoded><![CDATA[<h3>A little blog post for help people tor parse and also aggregate data from a CSV in Python without panda/numpy or other crazy and powerful library.</h3><p>In this example, we will use the classic taxis database, and calculate the total lines and the average of tips.</p><p><a href="https://s3.amazonaws.com/carto-1000x/data/yellow_tripdata_2016-01.csv">https://s3.amazonaws.com/carto-1000x/data/yellow_tripdata_2016-01.csv</a></p><p>For parse and aggregate, here the classic solution :</p><ul><li>Download the 6.5GB of data,</li><li>Load the file in memory,</li><li>Parse the file,</li><li>Aggregate the data,</li><li>Expose the results to the user.</li></ul><p>With this kind of solutions, you will need a big internet connection (not my case), a lots of disks and memory (the new M1 macbook is awesome but Slack and VSCode eats most of my RAM).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/480/0*97hv-2ajUVx5t2tT.gif" /></figure><p>The best solutions could optimize the time of downloading the data by parsing and pre-aggregate the data, or download only the needed data.</p><h3>First solution, the Netflix way.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*ITrPXXr_cO1G-mV-.jpg" /></figure><p>I think the future of data is the same of video: streaming. (you should read <a href="https://dev.to/owulveryck/a-streaming-communication-platform-for-the-data-mesh-proof-of-value-ke6)">https://dev.to/owulveryck/a-streaming-communication-platform-for-the-data-mesh-proof-of-value-ke6)</a></p><p>Every modern langage support streaming why not use it.</p><p>We need to download, parse and pre-aggregate data.</p><p>Let see the code :</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/16c4dcfa4d0fa5d15c075127d6f57e7b/href">https://medium.com/media/16c4dcfa4d0fa5d15c075127d6f57e7b/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/768/1*fIrj01jo2VFuul922Z2WPA.png" /><figcaption>I used a t4g.micro because I was watching a movie on Netflix.</figcaption></figure><p>No need of pandas or numpy, just good and old python stuff.</p><p>I could change csv.DictReader by csv.parse but I love python dict and if I need to change</p><h3>Solution 2, the “use some-else computing capacities” way.</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/0*ZHssX3LIWj-zyG9-.png" /><figcaption>Sometime we need someone else’s computer</figcaption></figure><p>Wait what ?</p><p>Remember my Internet connection is slow and Slack and VSCode are eating my RAM and CPU. If I could borrow some computing, it will be faster and cheaper.</p><p>disclaimer : for this solution, I will cheat by using an external libraries. But I will not load the file in a Big Database.</p><p>Did you know S3 SELECT ?</p><p>S3 Select is a way to querying CSV, JSON, or Apache Parquet file stored in a AWS S3 bucket. This is really a serveless service to query data stored in S3 bucket with good and old SQL.</p><p>The documentation is here : <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-glacier-select-sql-reference-select.html.">https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-glacier-select-sql-reference-select.html.</a></p><p>Always read a documentation before doing stuff, it could save a lot of time.</p><p>First install boto3, the AWS SDK for Python.</p><p>In the first implementations we will select only the tip_amount column. It will save a lot of bandswitch.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/3237fa2a9a0ffd81ca96dac475c33615/href">https://medium.com/media/3237fa2a9a0ffd81ca96dac475c33615/href</a></iframe><p>But with this request I will have to cast the result from string to float and count again the number of lines.</p><p>S3 provides <a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/s3-glacier-select-sql-reference-aggregate.html">aggregate functions and casting functions</a>. Next time, I will read the whole documentation before starting to code.</p><p>It will give the following code</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/be68e5590eac3572831adddaf4cab40a/href">https://medium.com/media/be68e5590eac3572831adddaf4cab40a/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*dDHXbzL0pJZZ_c8Eo7fV9A.png" /><figcaption>It’s really fast.</figcaption></figure><p>It even faster and cost efficient (I was finops in an other life).</p><p>Remember read documentation and be lazy, sometime it’s fun.</p><p>If you have better ideas, I will love to read it.</p><p>Thanks to carto for the exercice and the data (and also the processing time for s3, it will cost you some cents).</p><p>Thanks to AWS for S3, it really awesome.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9030e74a9bdc" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[A new python geohash library released!]]></title>
            <link>https://medium.com/@nabil.servais/a-new-python-geohash-library-released-a0d13f965b28?source=rss-c9e8b7766c1e------2</link>
            <guid isPermaLink="false">https://medium.com/p/a0d13f965b28</guid>
            <category><![CDATA[rust]]></category>
            <category><![CDATA[gis]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[geohash]]></category>
            <dc:creator><![CDATA[Nabil Servais]]></dc:creator>
            <pubDate>Thu, 23 Jul 2020 20:30:44 GMT</pubDate>
            <atom:updated>2020-07-23T20:30:44.744Z</atom:updated>
            <content:encoded><![CDATA[<h4>But based on rust.</h4><h4>The goal of this article is to explain how you can create your own python library with rust. Rust is not magic but the language have a lot of fresh and new features. Maybe I made a lot of mistakes, don’t hesitate to correct in the coments.</h4><h3>The situation</h3><p>For a crazy side project, I need geohash encoding/decoding but after writing my code I discover georust and its geohash implementations.</p><p>Geohash is a way to encode a position for more information the <a href="https://en.m.wikipedia.org/wiki/Geohash">Wikipedia</a> article is a good starting point to understand how it works.</p><p>In python, you have 3 libraries :</p><ul><li><a href="https://github.com/wdm0006/pygeohash">py-geohash</a>,</li><li>geohash,</li><li>geohashcx (a <a href="https://aldnav.com/blog/writing-a-geohash-wrapper-using-cffi/">source of inspiration</a> for my module).</li></ul><p>After testing this 3 librairies for my side project I was not satisfied to 100%. And after trying geohash-rust, I would like to have the same in my python app without rewrite in rust.</p><p>Rust allow ffi to be used in Python. Ffi is not very fun in action. Fortunately there is Pyo3, rust bindings for python3.</p><p>Pyo3 purpose to use your rust code in python or python with rust. It also have a complete API for binding python code and rust code.</p><p>First step is to prepare your project like other rust project.</p><pre>$ cargo new geohashrs</pre><p>Then prepare the cargo.toml for compilation and the publication. Don’t hesitate to add the metadata.</p><p>Now, we are ready to code.</p><pre>[package]<br>name = &quot;geohashrs&quot;<br>version = &quot;0.1.4&quot;<br>authors = [&quot;nabil@eml.cc&quot;]<br>edition = &quot;2018&quot;<br>license = &quot;Apache-2.0&quot;<br>repository = &quot;https://github.com/blackrez/geohashrs&quot;<br>homepage = &quot;https://github.com/blackrez/geohashrs&quot;<br>readme = &quot;README.md&quot;<br><br>[lib]<br>name = &quot;geohashrs&quot;<br>crate-type = [&quot;cdylib&quot;, &quot;rlib&quot;]<br><br>[package.metadata.maturin]<br>maintainer = &quot;Nabil Servais&quot;<br>maintainer-email = &quot;nabil@eml.cc&quot;<br>requires-python = &quot;&gt;=3.5&quot;<br>classifier = [&quot;Programming Language :: Python&quot;]<br><br>[dependencies]<br>geohash = &quot;0.9.0&quot;<br><br>[dependencies.pyo3]<br>version = &quot;0.11.1&quot;<br>features = [&quot;extension-module&quot;]</pre><p>The API will be simple for the first version. Just encode and decode, usage of neighbours and direction will be added in the future.</p><h3>Pyo3 a great binding</h3><p>Pyo3 permits to <a href="https://pyo3.rs/v0.11.1/conversions.html">bridge</a> python and rust, this is a huge advantage when you code for 2 langages. Just use the type in rust and use it Python.</p><pre>/// Encode coords into geohash<br>#[pyfunction]<br>fn geohash_encode(x: f64, y: f64, level: usize) -&gt; PyResult&lt;String&gt; {<br>    let c = Coordinate { x: x, y: y };<br>    let geohash_string = geohash::encode(c, level).expect(&quot;Invalid coordinate&quot;);<br>    Ok(geohash_string)<br>}<br><br>// Decode geohash into Python list<br>#[pyfunction]<br>fn geohash_decode(hash: &amp;str) -&gt; PyResult&lt;Vec&lt;f64&gt;&gt; {<br>    let (c, _, _) = decode(&amp;hash).expect(&quot;Invalid hash string&quot;);<br>    let vec = vec![c.x, c.y];<br>    Ok(vec)<br>}</pre><p>The wrappers use macro for pyo3, it is also managed by the lib at the build</p><pre>#[pymodule]<br>fn geohashrs(py: Python, m: &amp;PyModule) -&gt; PyResult&lt;()&gt; {<br>    m.add_wrapped(wrap_pyfunction!(geohash_encode))?;<br>    m.add_wrapped(wrap_pyfunction!(geohash_decode))?;<br>    Ok(())<br>}</pre><p>After writing the code, we need to build the library.</p><h3>Maturin</h3><p>Cargo works with your libs but it will generate libgehashrs.so and not geohashrs. And you need to specify a setuptools file…</p><p>But the Pyo3 have a way to</p><p><a href="https://github.com/PyO3/maturin/">Maturin</a> is a great tools for generate the <a href="https://www.python.org/dev/peps/pep-0427/">wheels</a> and also build the libs.</p><pre>$ pip install maturin<br>$ maturin build</pre><p>And you will find your lib.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*xL9heCXaN9_x0hZHgu-mAQ.png" /></figure><p>Don’t hesitate to use a virtualenv. Maturin is written in rust but it works in python.</p><h3>And the publication</h3><p>Just use :</p><pre>$ maturin publish</pre><p>Maturin will publish after asking some questions.</p><p>It also possible to use a docker images for generate more wheels version and use twine.</p><p>Now I can use my libs in my application using pip.</p><p><a href="https://pypi.org/project/geohashrs/">https://pypi.org/project/geohashrs/</a></p><h3>What’s next ?</h3><ul><li>Add tests</li><li>Add CI/CD</li><li>Add neighbours and directions</li></ul><p>Thanks for the reading.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a0d13f965b28" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using Github actions to test your go code on ARM64]]></title>
            <link>https://medium.com/@nabil.servais/using-github-actions-to-test-your-go-code-on-arm64-ce581e646cb?source=rss-c9e8b7766c1e------2</link>
            <guid isPermaLink="false">https://medium.com/p/ce581e646cb</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[gorgonia]]></category>
            <category><![CDATA[arm]]></category>
            <category><![CDATA[devops]]></category>
            <category><![CDATA[go]]></category>
            <dc:creator><![CDATA[Nabil Servais]]></dc:creator>
            <pubDate>Fri, 10 Jul 2020 20:05:11 GMT</pubDate>
            <atom:updated>2020-07-10T20:05:11.224Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/394/1*N34adkGJKbfBHZ4eg9eycg.jpeg" /></figure><p>Architecture ARM is getting a thing, not only for embbeded hardware like phone, robots or industrial machine. Now servers and computers already have ARM CPU, the <a href="https://www.pine64.org/pinebook/">pinebook</a>, the <a href="https://aws.amazon.com/fr/blogs/aws/new-m6g-ec2-instances-powered-by-arm-based-aws-graviton2/">AWS Graviton2</a> or the future <a href="https://www.theverge.com/2020/7/10/21319772/apple-silicon-arm-based-macbook-pro-air-imac-2020-2021-ming-chi-kuo-report-prediction">MacBook</a>.</p><p>So testing your code on ARM is really important.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/627/1*3O63Dji0xxWFRaOjpRcUCw.png" /></figure><p>As an contributor of <a href="https://gorgonia.org/">gorgonia</a> (a machine learning library in go), we migrate from travis to github action.</p><p>Travis was a great CI but the service was becoming so slow. But if Travis support arm build, this is not the case of github actions.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/280/1*-C_wHr8Rvhsk2FQbNjiqnw.png" /><figcaption>hello github actions</figcaption></figure><p>Github actions purpose to use a self-hosted runner and like I said in introduction AWS purpose ARM based isntance.</p><p>First start an instance with your favorite tool, it could be the console, terraform or pulumi. In our case, we choose ubuntu because it works.</p><p>Connect to ssh to your new instance, upgrade and restart your server if needed.</p><p>Then comeback to github setting of your project, and go to the “action”, verify if self-hosted runner is enabled.</p><p>Choose “Add a runner”, follow the instructions prompted.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wvwD4NlpGFrDKBSLNG-2cA.jpeg" /><figcaption>The instructions from github.</figcaption></figure><p>You can add other labels to identify your runner.</p><p>For more informations, don’t hesitate to check the documentation :</p><p><a href="https://docs.github.com/en/actions/hosting-your-own-runners/using-self-hosted-runners-in-a-workflow">https://docs.github.com/en/actions/hosting-your-own-runners/using-self-hosted-runners-in-a-workflow</a></p><p>Add in your yaml file the section for tests for ARM architecture.</p><pre>on: [pull_request]<br>name: test and build<br>env:<br>  GOPROXY: &quot;https://proxy.golang.org&quot;<br>jobs:<br>  test_amd64:<br>    strategy:<br>      matrix:<br>        go: [1.13, 1.14]<br>    runs-on: ubuntu-latest<br>    steps:<br>    - name: Install Go ${{ matrix.go }}<br>      uses: actions/setup-go@v2<br>      with:<br>        go-version: ${{ matrix.go }}<br>    - name: Checkout code<br>      uses: actions/checkout@v2<br>    - name: Run tests<br>      run: go test -v<br>     <br>  test_arm:<br>    strategy:<br>      matrix:<br>        go: [1.13, 1.14]<br>    runs-on: [self-hosted, linux, ARM64]<br>    steps:<br><br>    - name: Install Go ${{ matrix.go }}<br>      uses: actions/setup-go@v2<br>      with:<br>        go-version: ${{ matrix.go }}<br>    - name: Checkout code<br>      uses: actions/checkout@v2<br>    - name: Run tests<br>      run: go test -v<br><br>  </pre><p>And with this new tests, we discover that we have to fix somes tests failing.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/626/1*wSl6WuwC0wQd9A0UkxoAZQ.png" /></figure><p>Next step, fix this bug. If you have any idea, you’re welcome to help gorgonia team. Finds us on <a href="https://gophers.slack.com/messages/gorgonia/">slack</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=ce581e646cb" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Monitoring the temperature of your room with prometheus (and timescaledb)]]></title>
            <link>https://medium.com/@nabil.servais/monitoring-the-temperature-of-your-room-with-prometheus-and-timescaledb-4b67eb9a193e?source=rss-c9e8b7766c1e------2</link>
            <guid isPermaLink="false">https://medium.com/p/4b67eb9a193e</guid>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[raspberry-pi]]></category>
            <category><![CDATA[prometheus]]></category>
            <dc:creator><![CDATA[Nabil Servais]]></dc:creator>
            <pubDate>Mon, 29 Apr 2019 18:16:00 GMT</pubDate>
            <atom:updated>2019-04-30T06:13:28.930Z</atom:updated>
            <cc:license>http://creativecommons.org/publicdomain/zero/1.0/</cc:license>
            <content:encoded><![CDATA[<p>Nothing original here, just an another method to use your raspberry pi 3 and your temperature sensor. You can find a plenty of methods to do it on the web. ith Cacti, influxdb, mqtt, python script and other stuff. This time, I explain my methodology for choosing and implementing the solution.</p><p>Before choosing a technical solution, I have some constraints. It must cheap (in money and time), easy to maintain (for me), compatible with everything and opensource (I don’t want a sensor stop working because I don’t pay the abonnement or the company stop the support).</p><p>It excludes de facto “IoT” stuff found on aliexpress and amazon and big domotic solutions (remember I just want the temperature of a room).</p><p>I have a lot of solutions to exclude A LOT… I will take more time to exclude solutions than to implement and get fun.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/577/0*JYam6MwfvI_5Drh7" /></figure><p>So in software industry, what we choose today when we need storing time based data and alerting ? We used Prometheus or the TICK stack, right?</p><p>I’m more a Prometheus guy but I love TICK they are doing a great job, next time I will give a try.</p><p>Prometheus is time series database, a metrics scaper and a monitoring system. It scrape a http url with metrics. It is not 100% accurate but I didn’t need it for now.</p><p>Let’s go for Prometheus, let’s buy a cheap sensor. The raspberry pi 3 is a gift from my employer (think.fr) and see what will happen.</p><p>And I want to test timescaledb from long time storage and special analytics (I will develop in a later post).</p><p>Of course, Grafana is the best viewer for time series data.</p><p>For hosting, I choose Digital Ocean because VM is fast and cheap (if you want to you click on this <a href="https://m.do.co/c/25d68fd9d02d">link</a> to give more credits).</p><p>The program will push data on a pushgateway because prometheus can’t scrape an exporter in my network. It is easy compare to MQTT but not scalable, not a problem there is only one sensor for now.</p><p>The pushgateway is only a pass-through to prometheus, it permits to expose metrics to prometheus when it not possible (batch, multi-processus program…) .</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/800/0*yMETN7BeqS8glFG2" /><figcaption>The final architecture.</figcaption></figure><p>The harder part of this solution was to find a sensor easily manageable with a raspberry Pi and golang (I love golang since the first public version).</p><p>My first idea was to use a TMP125 but people on forum don’t recommend it and prefer the DS18B20.</p><p>I’m not a specialist of electronics, so I trust people who tried it before me, maybe it is stupid but I just want the temperature of a room.</p><p>The hardest part was to branch to the raspberry pi. Don’t laugh I destroyed one of sensor because I branch the wrong wire at the wrong GPIO.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fgiphy.com%2Fembed%2FDtXfTSHi6mHFS%2Ftwitter%2Fiframe&amp;url=https%3A%2F%2Fmedia.giphy.com%2Fmedia%2FDtXfTSHi6mHFS%2Fgiphy.gif&amp;image=https%3A%2F%2Fi.giphy.com%2Fmedia%2FDtXfTSHi6mHFS%2Fgiphy.gif&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=giphy" width="435" height="326" frameborder="0" scrolling="no"><a href="https://medium.com/media/72674c8b26bd7e3e3026ef8a26209919/href">https://medium.com/media/72674c8b26bd7e3e3026ef8a26209919/href</a></iframe><p>After getting the sensor working (follow this <a href="http://www.circuitbasics.com/raspberry-pi-ds18b20-temperature-sensor-tutorial/">link</a> ), now just write the golang program to get the temperature from the sensor.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*BMjAWbtLquDIKCqK" /><figcaption>Finally, it’s working!</figcaption></figure><p>Unfortunately, the sensor is not well supported by the great golang library periph (<a href="https://periph.io/">https://periph.io/</a>) but I found a library doing the job and working well <a href="https://github.com/yryz/ds18b20">https://github.com/yryz/ds18b20</a>.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5489aeeae5dd61fcaf8b07ab03a48107/href">https://medium.com/media/5489aeeae5dd61fcaf8b07ab03a48107/href</a></iframe><p>Nothing complicated in this golang code, then use the classic go get and go build (you have to build the golang app on raspiberry Pi).</p><p>note :I will do a correct repo soon.</p><p>Next article will cover the deployment of the prometheus stack with grafana and timescaleDB.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=4b67eb9a193e" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Machine learning in the cloud, the easy way]]></title>
            <link>https://medium.com/@nabil.servais/machine-learning-in-the-cloud-the-easy-way-9636061d12e?source=rss-c9e8b7766c1e------2</link>
            <guid isPermaLink="false">https://medium.com/p/9636061d12e</guid>
            <category><![CDATA[knative]]></category>
            <category><![CDATA[google-cloud-platform]]></category>
            <category><![CDATA[golang]]></category>
            <category><![CDATA[onnx]]></category>
            <dc:creator><![CDATA[Nabil Servais]]></dc:creator>
            <pubDate>Thu, 11 Apr 2019 15:29:06 GMT</pubDate>
            <atom:updated>2019-04-11T15:29:06.618Z</atom:updated>
            <content:encoded><![CDATA[<p>Google Cloud Platform released a new serverless service based on knative: <a href="https://cloud.google.com/run/">cloud run</a>.</p><p>As an ops, it is a very good news, deploying serverless application on multicloud become easier (this is the goal on knative).</p><p>As a developer, it is also a great news, I don’t care on how it will work, I just have to expose a HTTP port and make my applications stateless.</p><p>Let show how it can simplify development and putting quickly in production some code.</p><p>Olivier Wulveryck (<a href="https://twitter.com/owulveryck">@owulveryck</a>) is one of contributor of gorgonia and the main contributor of onnx-go. He made a great demo of his works on his <a href="https://blog.owulveryck.info/2019/04/03/from-a-project-to-a-product-the-state-of-onnx-go.html">blog</a> using webasm generated by golang.</p><p>The demo is a drawing application guessing the number. There is no magic, it is based on gorgonnia, a library that helps facilitate machine learning in Go, and onnx-go, the Go Interface to <a href="https://onnx.ai/">Open Neural Network Exchange (ONNX)</a>.</p><p>The concept is very simple, you load a onnx schema (in this case the <a href="https://en.wikipedia.org/wiki/MNIST_database">mnist</a>) , load it with onnx-go and run it with a backend. In this case gorgnonia. For more details, read (again) <a href="https://blog.owulveryck.info/2018/08/14/my-journey-with-onnx-and-go---the-beginning.html">the article blog</a> of Olivier, he is better than me for this kind of explaination.</p><p>For running it in cloud run, you have to containerize it. Nothing difficult, gorgonia and onnx-go haven’t any Cgo dependencies, the only thing needed is a classic build image for golang.</p><p>If you use the gcloud build image service, you can use a multistage images.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/babdcb252a51e2019c91afca6f222803/href">https://medium.com/media/babdcb252a51e2019c91afca6f222803/href</a></iframe><p>For the build, we can take time to build the image but for runtime we need the smallest images possible. Don’t forget a container can be destroyed or respawn anywhere, anytime.</p><p>For golang apps without cgo, Alpine is a good choice and if you don’t have any external dependencies, you could just use “FROM scratch”. I choose to store images generated from the draw in google storage for a future usage, that why the image needed ‘ca-certificates’.</p><p>Ok now we have a container ready to use. But I made some modifications so I have to fork.</p><p>I also add a solution for store images (it could use goroutines in the next version) and support cors. The static files are uploaded to a bucket and go for it.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/909/0*GsZeYJsc8_VZ4Ce1" /><figcaption>The result : <a href="https://storage.googleapis.com/demo-onnx-gorgonia/index.html">https://storage.googleapis.com/demo-onnx-gorgonia/index.html</a></figcaption></figure><p>The code is located here :</p><p><a href="https://github.com/blackrez/onnx-go/tree/cloud-run/example/gorgonia/mnist-reader">https://github.com/blackrez/onnx-go/tree/cloud-run/example/gorgonia/mnist-reader</a></p><figure><img alt="" src="https://cdn-images-1.medium.com/max/484/0*PGkrnlrub2UbezV9" /><figcaption>The final yaml!</figcaption></figure><p>I have a lot of works to do for a better industrialisation and I want a some cool features like evaluation of the results. But with in a reduced time frame and with little means, I can make a quick and rapid prototype of a application using machine learning with autoscaling.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*L2OWip8HjyXrW4ik" /><figcaption>Easy, simple and quickly way for deploy apps.</figcaption></figure><p>Thanks to Olivier and gorgania developers to make this application and thanks to the knative team (and GCP teams in this case) to help developers for publishing apps more and more easily.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=9636061d12e" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>