<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Justin Güse on Medium]]></title>
        <description><![CDATA[Stories by Justin Güse on Medium]]></description>
        <link>https://medium.com/@guese-justin?source=rss-5ceff1a6b0a7------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*v7VLeTq4mjxwkob6</url>
            <title>Stories by Justin Güse on Medium</title>
            <link>https://medium.com/@guese-justin?source=rss-5ceff1a6b0a7------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Sun, 22 Nov 2026 18:54:10 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@guese-justin/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Was sind die Vorteile von selbst gehosteten KI Modellen?]]></title>
            <link>https://guese-justin.medium.com/was-sind-die-vorteile-von-selbst-gehosteten-ki-modellen-a661ef075a8b?source=rss-5ceff1a6b0a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/a661ef075a8b</guid>
            <category><![CDATA[saas]]></category>
            <category><![CDATA[ai-tools]]></category>
            <category><![CDATA[saas-tools]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[künstliche-intelligenz]]></category>
            <dc:creator><![CDATA[Justin Güse]]></dc:creator>
            <pubDate>Tue, 09 Apr 2024 13:03:13 GMT</pubDate>
            <atom:updated>2024-04-09T13:04:27.950Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*7k9PsZCxIZGY9-nA.jpg" /></figure><p>Der gesamte Artikel ist auf <a href="https://doku-chat.de/de/blog/vorteile-von-selbst-gehosteten-ki-modellen/">https://doku-chat.de/de/blog/vorteile-von-selbst-gehosteten-ki-modellen/</a> zu sehen.</p><h3>Einleitung</h3><p>In der Welt der künstlichen Intelligenz (KI) haben sich große Sprachmodelle wie GPT (Generative Pre-trained Transformer) als leistungsstarke Werkzeuge für eine Vielzahl von Anwendungen etabliert. Unternehmen sollten jedoch vorsichtig sein, sich zu stark auf diese Modelle zu verlassen. Während sie Genauigkeit und Skalierbarkeit bieten, haben sie auch ihre Grenzen. Sie können aufgrund von Einschränkungen bei den Trainingsdaten voreingenommene oder falsche Antworten liefern und sind nicht in der Lage, den menschlichen Kontext und Feinheiten vollständig zu verstehen.</p><p>In diesem Blog-Beitrag werden wir uns mit den Vorteilen von selbst gehosteten KI-Modellen auseinandersetzen. Selbst gehostete Modelle bieten Unternehmen eine größere Kontrolle über ihre KI-Infrastruktur, was zu einer besseren Leistung, Anpassungsfähigkeit und Datenschutz führen kann. Indem sie das Modell auf ihren eigenen Servern hosten, können Unternehmen sicherstellen, dass ihre proprietären Daten unter ihrer Kontrolle bleiben und Bedenken hinsichtlich Datensicherheit und Datenschutz adressiert werden.</p><p>Allerdings birgt die Selbst-Hosting auch Herausforderungen. Es erfordert erhebliche Investitionen in Hardware und Fachkenntnisse, um die Infrastruktur zu verwalten und zu warten. Darüber hinaus liegt die Verantwortung für die Aktualisierung des Modells, um neue Informationen zu berücksichtigen, und für den Schutz vor Sicherheitslücken vollständig beim Unternehmen. Angesichts des raschen Fortschritts in der KI und der Cybersicherheit kann dies eine anspruchsvolle Aufgabe sein.</p><p>Insgesamt repräsentieren große Sprachmodelle einen bedeutenden Fortschritt in der KI und bieten Unternehmen das Potenzial für Innovation und Effizienzsteigerung. Doch ihre Grenzen und die Komplexität von Feinabstimmung und Selbst-Hosting erfordern einen vorsichtigen Ansatz. Unternehmen müssen die Vorteile dieser leistungsstarken Werkzeuge gegen die Risiken einer übermäßigen Abhängigkeit, Ungenauigkeiten und ethischen Bedenken abwägen. Indem sie dies tun, können sie das Potenzial von selbst gehosteten KI-Modellen nutzen und gleichzeitig die Risiken im Zusammenhang mit diesen wegweisenden Technologien mindern.</p><h3>Warum nicht einfach ein Cloud-Modell verwenden?</h3><p>Es gibt mehrere Gründe, warum Unternehmen und Privatpersonen sich für das Selbsthosting von KI-Modellen entscheiden. Einer der Hauptgründe ist die Datensicherheit. Bei der Verwendung von Modellen von Anbietern wie OpenAI besteht die Möglichkeit, dass die eingegebenen Daten als Trainingsmaterial verwendet werden. Durch das Selbsthosting können Unternehmen sicherstellen, dass ihre sensiblen Daten unter ihrer Kontrolle bleiben und nicht für Trainingszwecke verwendet werden.</p><blockquote><em>Tipp: Um die Nutzung Ihrer Daten durch die amerikanischen AI Anbieter wie OpenAI zu vermeiden, sollten Sie ein „selbstgehostetes“ AI Modell betreiben. Dazu benötigen Sie einen Server mit einer Nvidia GPU, und einem entsprechenden „torch“ setup. Für DIY empfiehlt sich z.B. das Open Source Projekt </em><a href="https://ollama.com/"><em>https://ollama.com</em></a><em> . Alternativ ist ein deutscher KI Provider nötig, wie z.B. </em><a href="https://doku-chat.de/"><em>https://doku-chat.de</em></a><em> oder </em><a href="https://datafortress.cloud/"><em>https://datafortress.cloud</em></a><em> . Doku-Chat.de bietet außerdem die Unterteilung nach Projekten, nützliche Sharing/Team Features und viel mehr!</em></blockquote><p>Ein weiterer wichtiger Aspekt ist die Kostenersparnis. Selbst gehostete Modelle sind in der Regel kostenfrei oder erfordern nur geringe Lizenzgebühren im Vergleich zu den hohen Kosten, die mit der Nutzung von Modellen von Drittanbietern verbunden sein können. Dies macht selbst gehostete Modelle zu einer wirtschaftlich attraktiven Option für Unternehmen jeder Größe.</p><p>Der Schutz der Privatsphäre ist ein weiterer entscheidender Faktor. Durch das Selbsthosting behalten Unternehmen die Kontrolle über ihre Daten und können sicherstellen, dass sie nicht an Dritte weitergegeben werden. Dies ist insbesondere in Branchen mit strengen Datenschutzbestimmungen von großer Bedeutung.</p><p>Zusammenfassend bietet das Selbsthosting von KI-Modellen Vorteile wie verbesserte Datensicherheit, Kostenersparnis und Wahrung der Privatsphäre. Unternehmen können die volle Kontrolle über ihre KI-Infrastruktur behalten und gleichzeitig von den Vorteilen dieser leistungsstarken Technologie profitieren.</p><h3>Was benötige ich um ein selbst gehostetes KI-Modell zu betreiben?</h3><p>Um ein selbst gehostetes KI-Modell zu betreiben, benötigen Sie einige spezifische Ressourcen. Ein wichtiger Bestandteil ist ein Nvidia-GPU-Server, da GPUs für das Training und die Ausführung von KI-Modellen eine hohe Rechenleistung bieten. Es ist jedoch wichtig zu beachten, dass Nvidia-GPU-Server ziemlich teuer sein können und eine beträchtliche Investition erfordern können.</p><p>Glücklicherweise gibt es Open-Source-Projekte wie Ollama, die das Hosting von KI-Modellen erleichtern. Ollama ist ein Projekt auf GitHub (<a href="https://github.com/ollama/ollama%29">https://github.com/ollama/ollama)</a> , das eine einfache und benutzerfreundliche Möglichkeit bietet, Modelle selbst zu hosten. Es bietet Tools und Ressourcen, um den Prozess des Selbsthostings zu vereinfachen und zu automatisieren.</p><p>Mit Ollama und einem geeigneten Nvidia-GPU-Server können Sie Ihr eigenes KI-Modell betreiben und von den Vorteilen des Selbsthostings profitieren, ohne die volle finanzielle Belastung eines dedizierten Servers tragen zu müssen.</p><p>Es ist wichtig zu beachten, dass das Betreiben eines selbst gehosteten KI-Modells weiterhin technisches Fachwissen erfordert. Es kann hilfreich sein, über Kenntnisse in den Bereichen Maschinelles Lernen und Serververwaltung zu verfügen oder Experten hinzuzuziehen, um Ihnen bei der Einrichtung und Wartung zu helfen.</p><p>Mit den richtigen Ressourcen und Tools wie Ollama können Sie Ihr eigenes KI-Modell betreiben und die Vorteile des Selbsthostings nutzen, um Kontrolle, Datensicherheit und Kostenersparnis zu gewährleisten.</p><h3>Und was für einen Server brauche ich für die Modelle?</h3><p>Zuerst sollte man abwegen, was man von dem Modell erwartet. “Coding Assistenten” á la Github Copilot brauchen weniger ressourcen und kommen mit einer GPU mit ~8GB VRAM aus. Für größere Modelle welche sich ähnlich ChatGPT verhalten braucht man schon mindestens 16 GB VRAM. Erwartet man eine Performance von GPT-4 muss man schon mehr Geschütze auffahren, und sollte sich an einem GPU Cluster mit mindestens 64 GB VRAM orientieren.</p><p>Eine kleine Übersicht der Modelle, und wie viel Speicherplatz sie benötigen findet man hier:</p><p><a href="https://github.com/ollama/ollama">https://github.com/ollama/ollama</a></p><h4>Und was für eine GPU ist das konkret?</h4><p>Leider ist NVIDIA derzeit im Monopol was AI GPUs angeht. Man kann auch mit technischen “Hacks” eine AMD GPU verwenden, jedoch ist die Performance und Kompatibilität nicht mit allen Projekten gegeben, bzw. muss man die benötigte Arbeitszeit vs. die Kosten für eine NVIDIA GPU abwägen.</p><p>Die Server GPU Linie von NVIDIA kommt mit einem stolzen Preis daher, weshalb es durchaus Sinn machen kann die Gaming-GPU’s von NVIDIA zu verwenden.</p><p>Da es nicht nur auf die VRAM, sondern auch die Tensor-Cores (“AI Cores”) ankommt, macht es durchaus Sinn die 4000er Reihe von NVIDIA zu verwenden anstatt der 3000er Modelle.</p><p>NVIDIA GPU NameVRAMGeeignet für …Preis<a href="https://www.amazon.de/gp/search?ie=UTF8&amp;tag=qaau-21&amp;linkCode=ur2&amp;linkId=94e67979004555721e100e26a68f7c05&amp;camp=1638&amp;creative=6742&amp;index=computers&amp;keywords=rtx+4070">RTX 4070</a>12 GBCoding Completion, sehr simple Tasks (fasse Zusammen, korrigiere, sortiere, …)~648€<a href="https://www.amazon.de/gp/search?ie=UTF8&amp;tag=qaau-21&amp;linkCode=ur2&amp;linkId=94e67979004555721e100e26a68f7c05&amp;camp=1638&amp;creative=6742&amp;index=computers&amp;keywords=rtx+4080">RTX 4080</a>16 GBSchreibe einen Text über, generiere Code, Leistung ~60% von gpt-3.5~1.100€<a href="https://www.amazon.de/gp/search?ie=UTF8&amp;tag=qaau-21&amp;linkCode=ur2&amp;linkId=94e67979004555721e100e26a68f7c05&amp;camp=1638&amp;creative=6742&amp;index=computers&amp;keywords=rtx+4090">RTX 4090</a>24 GBSchreibe einen Text über, generiere Code, Leistung ~80% von gpt-3.5~1.900€</p><blockquote><em>Das klingt zu stressig? Oder wäre besser als Hosted API? Nutzen Sie </em><a href="https://doku-chat.de/de/"><em>doku-chat.de</em></a><em> oder </em><a href="https://datafortress.cloud/de/contact/"><em>lassen sie sich in einem kostenlosen Erstgespräch von uns beraten</em></a><em> .</em></blockquote><h3>Die Vor- und Nachteile des selbst gehosteten KI-Modellen</h3><p>Das selbst gehostete KI-Modell bietet eine Reihe von Vor- und Nachteilen im Vergleich zu anderen Hosting-Optionen wie Cloud-basierten Plattformen. Einer der Vorteile des selbst gehosteten Modells ist die Skalierbarkeit. Unternehmen haben die volle Kontrolle über ihre Infrastruktur und können das Modell an ihre spezifischen Anforderungen und Ressourcen anpassen. Dies ermöglicht eine bessere Skalierbarkeit und Leistungsoptimierung.</p><p>Ein weiterer Vorteil ist die Flexibilität. Durch das selbst gehostete Modell haben Unternehmen die Freiheit, eigene Anpassungen vorzunehmen und das Modell nach Belieben anzupassen. Dies ermöglicht eine größere Anpassungsfähigkeit an spezifische Anwendungsfälle und Geschäftsanforderungen.</p><p>Jedoch gibt es auch einige Nachteile beim selbst gehosteten Modell. Ein wesentlicher Aspekt ist der Wartungsaufwand. Unternehmen sind dafür verantwortlich, das Modell auf dem neuesten Stand zu halten, Patches und Updates einzuspielen und die Sicherheit zu gewährleisten. Dies erfordert technisches Fachwissen und Ressourcen für die Serververwaltung.</p><p>Ein weiterer Faktor ist die Kostenfrage. Obwohl das selbst gehostete Modell langfristig kosteneffizient sein kann, erfordert es zunächst eine erhebliche Investition in Hardware und Expertise. Dies kann für kleinere Unternehmen oder Start-ups eine finanzielle Herausforderung darstellen.</p><p>Es ist wichtig, die Vor- und Nachteile gründlich abzuwägen und die individuellen Anforderungen des Unternehmens zu berücksichtigen. Ein selbst gehostetes Modell bietet mehr Kontrolle und Anpassungsfähigkeit, erfordert aber auch mehr Verantwortung und Ressourcen. Cloud-basierte Plattformen können hingegen eine schnellere Bereitstellung und geringeren Wartungsaufwand bieten, jedoch mit weniger Kontrolle und Anpassungsmöglichkeiten. Es ist ratsam, die spezifischen Bedürfnisse und Möglichkeiten des Unternehmens zu analysieren, um die beste Hosting-Option zu wählen.</p><blockquote><em>Das klingt zu stressig? Oder wäre besser als Hosted API? Nutzen Sie </em><a href="https://doku-chat.de/de/"><em>doku-chat.de</em></a><em> oder </em><a href="https://datafortress.cloud/de/contact/"><em>lassen sie sich in einem kostenlosen Erstgespräch von uns beraten</em></a><em> .</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=a661ef075a8b" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Maximizing Cost Savings in the private Cloud: How to Run Kubernetes without a Load Balancer]]></title>
            <link>https://guese-justin.medium.com/maximizing-cost-savings-in-the-private-cloud-how-to-run-kubernetes-without-a-load-balancer-7fc37fd99c74?source=rss-5ceff1a6b0a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/7fc37fd99c74</guid>
            <category><![CDATA[aws]]></category>
            <category><![CDATA[hosting]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[private-cloud]]></category>
            <dc:creator><![CDATA[Justin Güse]]></dc:creator>
            <pubDate>Fri, 10 Feb 2023 17:26:24 GMT</pubDate>
            <atom:updated>2023-02-10T17:26:24.346Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/500/1*fkVYCcoMFPduzmjAqoDK1g.jpeg" /></figure><p>When it comes to running Kubernetes in the cloud, one of the most significant costs can come from using a load balancer for each service. With prices starting at around $15 per month per load balancer, it’s easy for costs to quickly add up, especially if you have a large number of pods. However, what if we told you that there’s a way to run Kubernetes without the need for a load balancer and still get the benefits of high availability and automatic failover? In this article, we’ll explore how you can achieve cost savings in the cloud by running Kubernetes without a load balancer.</p><h3>Version 1: Nginx Ingress with Cert Manager to just use one Load Balancer</h3><p>One cost-saving strategy is to use only one load balancer for the entire cluster, rather than a separate load balancer for each service. This can be achieved by using an <a href="https://kubernetes.github.io/ingress-nginx/">Nginx Ingress</a>, which acts as a single entry point for all external traffic to the cluster. The Nginx Ingress then distributes the traffic to the appropriate pods based on the domain name, instead of creating one load balancer per service. The best part: If you are adding <a href="https://cert-manager.io/docs/installation/helm/">cert manager</a>, you will get free LetsEncrypt SSL certificates!</p><p>The easiest way to deploy the Nginx Ingress in your cluster, is by using Helm:</p><pre>helm upgrade --install ingress-nginx ingress-nginx \<br>  --repo https://kubernetes.github.io/ingress-nginx \<br>  --namespace ingress-nginx --create-namespace</pre><p>After that, to ensure https and SSL running for your services, you need to deploy Cert Manager with the following command:</p><pre>helm install \<br>        cert-manager jetstack/cert-manager \<br>        --namespace cert-manager \<br>        --create-namespace \<br>        --set installCRDs=true</pre><p>Then, you need to create a “ClusterIssuer” in order to tell LetsEncrypt who you are. Create a file “clusterissuer.yaml” with the content adapted to your Email:</p><pre>apiVersion: cert-manager.io/v1<br>kind: ClusterIssuer<br>metadata:<br>  name: letsencrypt-prod<br>spec:<br>  acme:<br>    server: https://acme-v02.api.letsencrypt.org/directory<br>#change to your email<br>    email: EMAIL<br>    privateKeySecretRef:<br>       name: letsencrypt-prod<br>    solvers:<br>    - http01:<br>        ingress:<br>          class: nginx # public or nginx depending on version</pre><p>After that, apply it to your cluster withkubectl apply -f clusterissuer.yamland you are set up!</p><h3>Creating an Ingress</h3><p>Now, you need to grab the name of your service that you want to expose to the world. You can get it withkubectl get service.</p><p>In this example, let’s assume your service is named “nginx” in the default namespace. To route your domain “test.datafortress.cloud” to it, you need to create the following testdf-ingress.yaml:</p><pre>apiVersion: networking.k8s.io/v1<br>kind: Ingress<br>metadata:<br>  name: ingress-nginx-test<br>  namespace: default<br>  annotations:<br>    cert-manager.io/cluster-issuer: &quot;letsencrypt-prod&quot;<br>spec:<br>  tls:<br>  - hosts:<br>#change to your domain<br>    - test.datafortress.cloud<br>    secretName: tls-secret<br>  rules:<br>#change to your domain<br>  - host: test.datafortress.cloud<br>    http:<br>      paths:<br>        - path: /<br>          pathType: Prefix<br>          backend:<br>            service:<br>              name: nginx<br>              port:<br>                number: 80</pre><p>Apply it withkubectl apply -f testdf-ingress.yaml, and point the domain you used to the load balancer used by your node. Soon you should see it spin up in your cluster, and see your service at the domain you stated in the ingress. To debug, have a look at the nginx pod or certificates. Struggling? <a href="https://datafortress.cloud/contact">Contact us</a> and we will help you out!</p><p>While this solution may save you money on your cloud bill, it’s important to note that using no load balancers can come with its own set of challenges. For example, if a node fails, the traffic won’t be automatically routed to a healthy server, leading to downtime for your service. In many cases, having a load balancer is still the best option as it provides automatic failover and ensures that your services remain available to your customers. It’s up to you to weigh the cost savings against the potential risks and make an informed decision on the best solution for your needs.</p><h3>Version 2: Saving even more money with 0 load balancers!</h3><p>If you’re looking to save even more money on your cloud bill, there’s an alternative to the above solution that uses no load balancers at all! Instead of using Nginx Ingress, this solution utilizes the nodePorts 80 and 443 on the server to redirect traffic to the appropriate pods. This eliminates the need for any load balancers, which can significantly reduce your cloud costs. Let’s dive into the details of this solution.</p><p>To do this, we will upgrade our existing nginx ingress to use NodePorts instead of Load Balancers:</p><pre>helm upgrade --install ingress-nginx ingress-nginx/ingress-nginx<br>    --namespace ingress-nginx \<br>    --create-namespace \<br>    --set controller.service.type=NodePort \<br>    --set controller.service.ports.http=80 \<br>    --set service.annotations.&quot;metallb.universe.tf/address-pool&quot;=singlenode \<br>    --set controller.service.ports.https=443</pre><p>You should soon see your ingress-nginx service to switch to Nodeport, and the load balancer should disappear.</p><p>While using load balancers provides stability to a cluster, it is a legitimate way to save money with one-node Kubernetes distributions. By eliminating the need for load balancers and relying solely on the node ports 80 and 443, you can reduce your monthly costs significantly. However, it’s important to keep in mind that not having a load balancer also means that in case of a failing node, the traffic will not be automatically routed to a healthy server. If you are operating a one-node Kubernetes distribution, this trade-off between stability and cost savings is worth considering.</p><h3>Verdict: Is it worth it?</h3><p>In conclusion, there are multiple ways to save costs in your Kubernetes cluster, from using only one load balancer with nginx ingress to using 0 load balancers and relying on nodeports. While a load balancer has its benefits in terms of stability and ensuring traffic routing, there are alternatives that can help reduce costs. If you’re still unsure of the best way to save money in your Kubernetes cluster, <a href="https://datafortress.cloud/contact">consider taking advantage of the cost-effective shared Kubernetes clusters offered by DataFortress.cloud, or seek our assistance in managing your cluster costs</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=7fc37fd99c74" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Running Storj on Kubernetes to earn money for unused disk space]]></title>
            <link>https://guese-justin.medium.com/running-storj-on-kubernetes-to-earn-money-for-unused-disk-space-982b6b1eac84?source=rss-5ceff1a6b0a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/982b6b1eac84</guid>
            <category><![CDATA[kubernetes]]></category>
            <category><![CDATA[storj]]></category>
            <category><![CDATA[sysadmin]]></category>
            <category><![CDATA[datahorder]]></category>
            <category><![CDATA[web3]]></category>
            <dc:creator><![CDATA[Justin Güse]]></dc:creator>
            <pubDate>Tue, 18 Oct 2022 16:04:32 GMT</pubDate>
            <atom:updated>2022-10-18T16:04:32.871Z</atom:updated>
            <content:encoded><![CDATA[<p>You might have heard of <a href="https://www.storj.io/storj-home">Storj</a>, a distributed S3 compatible cloud storage like AWS S3.</p><p>If not, imagine everyone being able to offer their (private) disk space in exchange for money. Sounds insecure? Actually not, as their technology builds a zero trust environment where every file is encrypted at the client and split into 80 pieces, distributed in multiple copies among Storj nodes, leading to a 99.95% availability, 11 nines durability and a download speed of up to 24 Gb/s.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DfEItm6qIWh7wQ9I0cC_iA.png" /><figcaption>Storj website</figcaption></figure><p>Same as me, you might be running a Kubernetes cluster where a lot of disk space on the nodes is still empty, either because you are using volumes provided by your cloud provider, or because there is still a lot of overhead.</p><p>This might be a good time to mention that I am not affiliated with storj — I’m a Data Engineer offering Cloud solutions (<a href="https://datafortress.cloud/">https://datafortress.cloud/</a>).</p><p>Now what if you could bring that empty disk space to a good use? Briefly said earnings are not amazing, but if that hardware is running anyways it’s a nice bonus! The way Storj is built it will reward you the longer your node is up and running. Expect not much return in the first year, but after 2–3 years you might be able to earn ~4$/TB/month if you have a good internet connection (October 22). This might go up as soon as the “Crypto winter” is over, so who knows.</p><p>Setting up an identity file</p><p>Now, in order for the Pod to know who you are, you first need to create four identity files. This can take some time, so you might have to do this on a strong machine and then copy the files afterward.</p><p>Check the how to <a href="https://docs.storj.io/node/dependencies/identity/">here</a>, or use my brief abbreviation:</p><pre>curl -L <a href="https://github.com/storj/storj/releases/latest/download/identity_linux_amd64.zip">https://github.com/storj/storj/releases/latest/download/identity_linux_amd64.zip</a> -o identity_linux_amd64.zip<br>unzip -o identity_linux_amd64.zip<br>chmod +x identity</pre><pre>./identity create storagenode</pre><pre>./identity authorize storagenode &lt;email:characterstring&gt;</pre><p>After this, you will find four files (ca.cert, ca.key, …) in the folder ~/.local/share/storj/identity/storagenode (check your output, this might be different).</p><p>Kubernetes setup</p><p>Let us first create a namespace with</p><pre>kubectl create namespace storj</pre><p>Then, we will create a secret mounting the files above. You might need to change the directory to match your system</p><pre><em>kubectl -n storj create secret generic storj --from-file=</em>~/.local/share/storj/identity/storagenode/</pre><p>Check your secret with</p><pre>kubectl describe secret storj -n storj</pre><p>The output should look like this:</p><pre>Name:         storj<br>Namespace:    storj<br>Labels:       &lt;none&gt;<br>Annotations:  &lt;none&gt;</pre><pre>Type:  Opaque<br>Data<br>====<br>identity.key:   241 bytes<br>ca.cert:        558 bytes<br>ca.key:         241 bytes<br>identity.cert:  1096 bytes</pre><p>After that, copy this deployment yaml and edit the matching values:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/800f952a1989341fb6c094e9fce62455/href">https://medium.com/media/800f952a1989341fb6c094e9fce62455/href</a></iframe><p>You have to change some values:<br>1. Address: Change this to your IP. Leave the port as is<br>2. Bandwith: Calculate how much TB your Download speed can offer per month<br>3. Email: Enter your email <br>4. Storage: Enter how much disk space you want to be used max<br>5. Wallet: Enter your Wallet id. This is where your reward will be transacted to. If you don’t have one, or want to say thanks feel free to use mine :)<br>6. Change YOURLOCALDIRECTORY to the local filepath of that node which you want to be used for Storj.</p><p>And that’s all! Just</p><p>kubectl apply -f ./storj-kubernetes.yaml as usual, and watch it spin up! Check the logs for more info, or follow the <a href="https://www.storj.io/host-a-node">“setup an interface” tutorial </a>on how to access the Web GUI for information on your earnings etc.</p><p>If you enjoyed this tutorial and experiment, feel free to head over to my <a href="https://datafortress.cloud/">website </a>(<a href="https://datafortress.cloud/">https://datafortress.cloud/</a>) to find more topics about cloud, big-data and AI!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=982b6b1eac84" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Using sklearn’s decision trees to form simple trading decisions for trading-bots]]></title>
            <link>https://guese-justin.medium.com/using-sklearns-decision-trees-to-form-simple-trading-decisions-for-trading-bots-5da2f2123744?source=rss-5ceff1a6b0a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/5da2f2123744</guid>
            <category><![CDATA[python]]></category>
            <category><![CDATA[sklearn]]></category>
            <category><![CDATA[trading]]></category>
            <category><![CDATA[trading-bot]]></category>
            <category><![CDATA[time-series-forecasting]]></category>
            <dc:creator><![CDATA[Justin Güse]]></dc:creator>
            <pubDate>Sun, 09 Oct 2022 07:44:00 GMT</pubDate>
            <atom:updated>2022-10-09T07:44:00.087Z</atom:updated>
            <content:encoded><![CDATA[<p>I have been writing and developing analytics and trading bots for quite some time now, usually involving very complex LSTM (deep learning) models, which get harder and harder to handle.</p><p>This is when I asked myself if I can capture a similar performance using simple if/else-based conditions. I like simple things, and the more an algorithm generalizes, the less risk of overfitting occurs. This has been an issue recently, as highly specialized trading bots performed quite badly during the “corona crashes”, which they could simply not handle because they were overly specialized.</p><p>So what “simple” trading decisions could possibly exist?<br>Everyone might know the “standard” youtube algorithms like “golden”- or “death-cross”, where you buy/sell depending on if the 50-day SMA crosses over the 200-day SMA. This works nicely on some examples but usually fails to capture the more complex market movements happening. <br>Additionally, a lot of people worked out simple correlations of RSI combined with trend lines, but usually, these perform equally badly.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*cOngXsn8aMvMGnbrAyLdZw.png" /><figcaption>The result in the end</figcaption></figure><p>This is where I got the idea to use simple ML to figure out possible correlations or decisions. Whilst recent network models are performing quite well, they also represent their logic in hidden layers which are hard to understand as a human, which is why I have decided to go with the most basic of basic models and see how well they perform: decision trees</p><p>Find the whole Jupyter <a href="https://github.com/JustinGuese/tradingbot22-tradingbots/blob/main/jupyternotebooks/decision-tree-tradingbot.ipynb">https://github.com/JustinGuese/tradingbot22-tradingbots/blob/main/jupyternotebooks/decision-tree-tradingbot.ipynb</a></p><h4>What are decision trees?</h4><p>A decision tree is simply put in a range of if/else statements like if relative-strength-index (RSI) is below 30, do this, if not, do that.</p><p>Sklearn can help us with this using from sklearn.tree.DecisionTreeClassifier</p><h4>Preparing the data</h4><p>To use this decision tree classifier, I want to “tag” data into classes. The most simple classes with trading data are “uptrend” and “downtrend”, or “buy” and “sell”.</p><p>But first, let us get some data including technical indicators.</p><p>Downloading data</p><p>I am using the Python package “yfinance” to download stock data.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/719abb7dcf384c7c5f9dfa88a5ab28e7/href">https://medium.com/media/719abb7dcf384c7c5f9dfa88a5ab28e7/href</a></iframe><p>This function downloads daily data for a given ticker that is listed on <a href="https://finance.yahoo.com/">https://finance.yahoo.com/</a>, adds all kinds of Technical Indicators using the ta library, adds a simple moving average (SMA), and fills nan values backward.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/552/1*HnvrKOHigxXgXcsen6feQQ.png" /><figcaption>Example data of MSFT</figcaption></figure><h4>How to create the target up- and downtrend values?</h4><p>But how can we tell from the above chart what an uptrend and downtrend is?</p><p>Many people work with “pct_change”/percent change and tagging up or down compared to the previous timestep, but I found that this is too finely grained, especially if we want to create simple trading decisions. Therefore I would prefer if we can somehow capture the general “flow”, or trend.</p><p>One way to do this is so first smoothen the “line”, which can be done with an SMA, which averages the values and therefore smoothens the line, and then get the local maxima and minima of this line.</p><p>For this, we can use the numpy function “argrelextrema”</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/d5d0bf6f234f2f47fdafb8a1b3dccbe0/href">https://medium.com/media/d5d0bf6f234f2f47fdafb8a1b3dccbe0/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/553/1*NYeKXY9CmR_qudTo0EiK0g.png" /><figcaption>The result of np.argrelextrema</figcaption></figure><p>As we can see, this nicely tags downtrends as “-1” / “sell”, and uptrends as “1” / “buy” in the DataFrame column “signal”.</p><h4>Creating a baseline</h4><p>It is always a good idea to create some kind of comparison or baseline for your algorithm. Therefore let us just calculate how much you would have earned if you would have just bought all the stocks you could get on the first day of data, and then sold everything on the last. I have grabbed <strong>MSFT </strong>(Microsoft) data from 2010 until 2022.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/94bb92a1529ac8721289667876288f15/href">https://medium.com/media/94bb92a1529ac8721289667876288f15/href</a></iframe><pre>with just holding you would have made 96545.12$ <br>or 74.57% per year <br>or 621.40$ per month</pre><p>Not too bad! Keep in mind that we could not have known back in 2010 that Microsoft will grow so much -&gt; classic survivorship bias.</p><p>Now we are having a baseline that our algorithm should beat, otherwise, it does not make sense.</p><h4>Simulating a trading environment</h4><p>To simulate a trading environment, we need to loop through the data and make a trading decision on each day. For this I have written the “oneRun” function:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dc2759a09a4c44da096d1922836a0aca/href">https://medium.com/media/dc2759a09a4c44da096d1922836a0aca/href</a></iframe><p>Additionally, we are setting a variable “COMMISSION”, which represents the commission on each trade. For interactive brokers this is 0,00025%. Set this to the fee you are paying with COMMISSION = 0.00025.</p><p>X is the DataFrame we created earlier.</p><p>You might notice the “model” that is passed. This should be a model containing a “predict” function, which returns 1 for “buy” and “-1” for sell.</p><p>Let us use a “Fake”-Model for now, which randomly returns something.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4c8ba985e01650406a377fed44bf519a/href">https://medium.com/media/4c8ba985e01650406a377fed44bf519a/href</a></iframe><p>As you can see, it will return an array of the predictions, as sklearn models would do it.</p><p>Executing the “oneRun” function with the FakeModel will result in:</p><pre>with the random model you would have made 1461.83$<br>or 1.13% per year <br>or 9.41$ per month</pre><p>As you might have guessed, randomly trading would end in a bad result (still positive though 👌).</p><h4>Using a decision tree classifier</h4><p>Using a decision tree classifier is quite straightforward.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/314780ef714716b63d446daa4efa77d8/href">https://medium.com/media/314780ef714716b63d446daa4efa77d8/href</a></iframe><pre>          precision    recall  f1-score   support<br>-1.0       0.85      0.90      0.87      1178          <br>1.0          0.94      0.91      0.92      2035      </pre><pre>accuracy                           0.91      3213    <br>macro avg       0.90      0.90      0.90      3213 <br>weighted avg       0.91      0.91      0.91      3213  <br>[[1058  120]  [ 185 1850]]</pre><p>Woohoo, we will be rich!! 0.91 accuracy?! As the seasoned Data Scientist in you might have noticed, we completely ignored train/test splits, therefore massively overfitting our model and getting these false results.</p><p>Why are we doing it then? We just want to use the model to create trading ideas/observations, and not use the model to .predict() something.</p><p>This is why we are going to limit the depth and leaves in the next steps, to reduce the model down to “general” and basic trading ideas.</p><p>One reason it performs quite well right now is, that for example really really specific logics as “if price == 30.8: buy” are in there, which are not at all realistic for real-time prediction. This is because the model “knows” the future, and can easily say <em>“yeah on the 20.6.12 we will get an uptrend, so go all in”.</em></p><h4>Generalizing and “trimming down” the decision tree</h4><p>We will generalize and “trim” the decision tree using two techniques:</p><ol><li>Grabbing data from different stocks -&gt; This is how we will avoid specific assumptions for a stock, like “if volume &gt; 32000: buy”</li><li>Limiting the depth and leaves of the decision tree</li></ol><p>This will hurt performance, but the objective of this post is to find generalized models which can be represented by brief “if/else” statements, and can tolerate unknown market situations quite well.</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/4aace2de4ec096e51e5b1e185e1902f9/href">https://medium.com/media/4aace2de4ec096e51e5b1e185e1902f9/href</a></iframe><p>The bigDf will contain data on a lot of stocks. If you want to build a resistant model you should include all kinds of stocks. I was lazy and just included tech stocks I like.</p><p>Next let us create a function that “trims down” the decision tree</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/ea53e0f211070c5ca233076efc1e0f3d/href">https://medium.com/media/ea53e0f211070c5ca233076efc1e0f3d/href</a></iframe><p>We are basically favoring smaller trees, if the win of the simulated trade is basically (95%) more than the last biggest.</p><pre>the best depth is 15, best leave 70. with a win of 31677.42$</pre><p>The accuracy of the model is still 0.88, which is not too bad!</p><p>Next, let us use graphviz to visualize our decision tree:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/5a20f072e078560243d97174cac7f8f9/href">https://medium.com/media/5a20f072e078560243d97174cac7f8f9/href</a></iframe><p>For visualization purposes, I limited the model to depth=10 and leaves=10</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*dkncdLvIR3OOvHTwq5eOCQ.png" /><figcaption>Decision tree result with depth 10 and leaves 10</figcaption></figure><p>Nice! As we can see a lot of statistics represent more generalized and not individualized metrics like volatility, aaron-trend indicator and others.</p><p>If you would want to decide according to this logic, you would</p><ol><li>Check the Aaron trend indicator. If it is less or equal to 10.0, go down the left path, if not go down the right path</li><li>If you went down the left path, the likeliest decision is “sell”/”downtrend”, which can be further narrowed down by looking at “volatility_dcp” …</li></ol><h4>Let the magic happen: Transform a decision tree to if/else statements</h4><p>Now let us do some magic and convert the decision tree to if/else statements!</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/cae0fb3a7c5d16e3a9fc44070e3d9eed/href">https://medium.com/media/cae0fb3a7c5d16e3a9fc44070e3d9eed/href</a></iframe><p>The result will look like this:</p><iframe src="" width="0" height="0" frameborder="0" scrolling="no"><a href="https://medium.com/media/dcc60d1890bd5cb787eeea5e960bf399/href">https://medium.com/media/dcc60d1890bd5cb787eeea5e960bf399/href</a></iframe><p>Cool! Keep in mind that you still need to convert the “0” to “-1”/”sell”/”downtrend”.</p><p>Congrats, this will be a tradingbot that will be hyper-efficient without needing any libraries (once you have the getData() )!</p><p>Meaning if you want to get a trading decision for the current data, you can simply run</p><pre>df = getData(&quot;BTC-USD&quot;, date(2022, 5, 1)<br>decision = getDecision(df.iloc[-1])<br>if decision == 1:<br>    print(&quot;i say buy!&quot;)<br>elif decision == 0:<br>    print(&quot;i say sell!&quot;)</pre><p><strong>This is just for statistical purposes, do not form your trading decisions based on this data!</strong></p><p>If you liked this article leave a like and comment, if I get enough feedback I will show you how to build bots that are actually performing better using “lookback” windows and “sanity checks” on the decisions.</p><p>Visit my website <a href="https://datafortress.cloud/">https://datafortress.cloud/</a> for more, or find the whole Jupyter notebook at <a href="https://github.com/JustinGuese/tradingbot22-tradingbots/blob/main/jupyternotebooks/decision-tree-tradingbot.ipynb">https://github.com/JustinGuese/tradingbot22-tradingbots/blob/main/jupyternotebooks/decision-tree-tradingbot.ipynb</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5da2f2123744" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Top 7 Kubernetes Security Misconceptions and Myths]]></title>
            <link>https://guese-justin.medium.com/top-7-kubernetes-security-misconceptions-and-myths-b40cda2298d7?source=rss-5ceff1a6b0a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/b40cda2298d7</guid>
            <category><![CDATA[infrastructure-as-code]]></category>
            <category><![CDATA[hosting]]></category>
            <category><![CDATA[kubernetes-cluster]]></category>
            <category><![CDATA[sysadmin]]></category>
            <category><![CDATA[kubernetes]]></category>
            <dc:creator><![CDATA[Justin Güse]]></dc:creator>
            <pubDate>Fri, 05 Aug 2022 09:29:15 GMT</pubDate>
            <atom:updated>2022-08-05T09:29:15.017Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Pxh4ErZBIltCvLWhPaCWhQ.png" /></figure><p>Kubernetes can be complex, but once you get to know the basics, it is a wonderful tool allowing you to get up and running quickly. Once you have deployed your first services, there are common pitfalls in terms of security that you might miss. This article touches on the Top 7 misconceptions.</p><h3>1. Kubernetes Namespaces isolate containers</h3><p>Thinking that namespaces isolate containers is a common misconception. They are only showing up in that specific namespace, but are not isolated from each other. Let us say you are running a postgresql deployment in the “default” namespace as service name “psql-deployment”. If you start up another container, you are able to connect to it connecting to “psql-deployment:5432”. Now, if you move your second deployment to another namespace, you will not be able to connect anymore.</p><p>Now it might be easy to think that namespaces isolate network resources, but the truth is only the routing changed, and the pod can still communicate with the psql-deployment. We could try this out by using the internal DNS routing called “psql-deployment.default.svc.cluster.local”. As you can see the structure is [SERVICE].[NAMESPACE].svc.cluster.local, and every service can be accessed with that.</p><p>This poses a huge security risk but can be avoided by using special networking plugins in Kubernetes. Common plugins are Calico, Flannel, Weave, and Cilium. They can be set up to isolate namespaces, and containers, and specify different Security Groups and Access Control Lists as you might know from other cloud providers.</p><h3>2. Inter-Node communication in Kubernetes is encrypted</h3><p>A similar issue happens with the communication between Kubernetes nodes. Let us say you are encrypting your WordPress (or any other) traffic with automated Cert-Manager and Letsencrypt. All good, right? Wrong — because traffic in between nodes is unencrypted by default. But what does that mean?</p><p>If one node is located in Frankfurt, and another in Munich for availability reasons (good on you!), the communication between nodes still happens unencrypted. This means, that anyone gaining access to the data center connection could read your traffic!</p><p>Of course, this is quite strongly prevented if your hosting provider is reliable, but especially if you are using self-hosted setups you should be careful in doing this. And what can you do against that? You could either manually use self-signed certificates for the pods, or — easier — use Kubernetes plugins for that. The previously mentioned networking plugins offer a lot of functionality. The plugins implement a so-called “service mash”, which deploys a sidecar along with your pod, that is taking care of TLS termination with mTLS enabled between pods.</p><p>Calico does not implement this by default, but Linkerd and Istio can use traffic encryption in combination with Calico. Linkerd is the easiest plugin in these regards, as it implements mTLS out of the box.</p><h3>3. Velero Backups are secured and encrypted</h3><p>Say you have Backups implemented, and save them to AWS S3 for safekeeping. A good step for sure in terms of disaster recovery, but did you know, that Velero does not encrypt your backups by default? This can have serious implications if you are running an isolated self-hosted Kubernetes cluster because you are from the finance industry or store medical data, but the backup is stored in true form on an American cloud provider.</p><p>Many European countries specify to not use American cloud providers due to security risks involved, and many still keep on using S3 for backups. If someone accesses these object storages in the public cloud, there is no encryption on the files stored.</p><p>Unfortunately, at the time of writing, there is no Backup service offering encryption. My solution was to create custom Kubernetes Jobs, that dump the data, encrypt it in the Job, and then store it in object storage provider interfaces.</p><h3>4. Kubernetes secrets are encrypted</h3><p>You might have created secrets in Kubernetes, and might have taken a look at them in your running cluster. “aWxvdmV0ZW5kaWVz”, wow that looks quite encrypted! But again the hard truth is: Wrong! Kubernetes saves Secrets and a lot of other variables as base64 encoded strings. Base64 is just an encoding, and no encryption, meaning we can easily, without any keys, “translate” the above string into the true form. Try it out yourself with tools like e.g. <a href="https://www.base64encode.org/">https://www.base64encode.org/</a>! This means, that if someone gets access to your Kubernetes cluster, he can just read all your secrets. This is currently not completely preventable, but good solutions exist.</p><p>For example “Sealed Secrets”, allow you to encrypt your secrets at rest and in transit on creation. The decryption key is stored in the cluster itself, which is where the secrets get decrypted. This still means though, that if the cluster itself gets compromised, the encryption key is compromised, but at least during creation and when storing the secrets on Github, they remain safe.</p><p>A slightly more complex approach by Bitnami, the “Helm secrets”, can use a Key Management system like AWS KMS or Google CKM to encrypt and decrypt secrets. That way, you do not have to share the same encryption key with all your developers, but only specific developers have access to the encryption keys of their applications, even in the cluster.</p><p>Still, the secrets remain “pure” in the cluster itself, which is another reason to pay close attention to not losing your Admin Kubeconfig.</p><h3>5. Using the same user for everything</h3><p>This brings us to our next point. When creating new clusters your provider usually gives you one Kubeconfig file. This is the admin file, and you can do anything with it. Even though it is tempting, you should never use it except for creating other users and permissions. That way, if one user and role for a specific app or namespace gets compromised, it has no control over the others. The solution is called RBAC (role based access control).</p><p>Number 6 is only available in our <a href="https://shop.easycloudhost.de/product/top-7-downlaods-etc/">free PDF Download of this article!</a>!</p><h3>7. Ignoring resource limits</h3><p>I know, it is tempting to just write your deployments and publish them. But you should at least set namespace limits, or even better deployment resource limits, as this prevents you from a lot of headaches further down the road.</p><p>Kubernetes managed resources automatically, which is amazing, but if someone attacks one specific service of yours with a DDOS attack (basically a lot of traffic), the resources for that deployment will consume almost all Kubernetes resources and all the other applications will fail.</p><p>This can be troublesome if your organization uses one huge cluster without these limits. It does not even have to be an attack. It can be an application with bugs, that enters an infinite loop or just eats all the resources. The solution is to define deployment — or at least — namespace limits. That way, if a namespace reaches that hard limit, the application will be contained, and the other applications keep on running.</p><p>I hope this list gave you a nice picture of what you can do to improve security in your Kubernetes deployment. What was new to you? Did I miss something? Let me know in the comments, and feel free to reach out to me to discuss your Kubernetes projects. Did you know that I am offering a worry-free managed Kubernetes called “<a href="https://easycloudhost.de/services/easykube/">EasyKube? Go and have a look!</a>”</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b40cda2298d7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Enhancing stock data for your Python Algorithmic Trading Model]]></title>
            <link>https://guese-justin.medium.com/enhancing-stock-data-for-your-python-algorithmic-trading-model-b40c668e4087?source=rss-5ceff1a6b0a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/b40c668e4087</guid>
            <category><![CDATA[trading]]></category>
            <category><![CDATA[python]]></category>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[pandas]]></category>
            <category><![CDATA[algorithmic-trading]]></category>
            <dc:creator><![CDATA[Justin Güse]]></dc:creator>
            <pubDate>Wed, 27 Jan 2021 15:35:34 GMT</pubDate>
            <atom:updated>2021-01-27T15:35:34.013Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*on9sQbkT4nE9W2-jm8Wf5A.png" /><figcaption>Image by author</figcaption></figure><h3>Your first steps building a trading algorithm</h3><p>Let us say you are planning to build your own algorithmic trading model.</p><p>You will most likely use only price (Close) data for your model and algorithm, but soon you will discover, that your model does not perform well.</p><p>Soon you will use typical OHLCV data, which refers to Open, High, Low, Close, Volume, which is already better, but the model does not seem to perform quite well enough.</p><p>What can you do?</p><p>Handy Colab Notebook to follow along: <a href="https://colab.research.google.com/drive/1ywqti1TuTDY_Z11ry0x4ITclCwxnXAeI?usp=sharing">https://colab.research.google.com/drive/1ywqti1TuTDY_Z11ry0x4ITclCwxnXAeI?usp=sharing</a></p><p>Gist:</p><p><a href="https://gist.github.com/JustinGuese/019e0e71100abe6555f78c32fd0b10a9">https://gist.github.com/JustinGuese/019e0e71100abe6555f78c32fd0b10a9</a></p><h3>What does a Machine Learning Trading Bot like?</h3><p>The typical machine learning algorithm can only work with the data it got. It (usually) can not create new features or interpretations like “If volume increases and the 3rd derivative of price rises, the price will most likely go up”, but instead can only “look” at the data it got. These would be calculations like “if the price is above 100 USD, and the volume above 2000, the price will most likely go up”.</p><p>Newcomers to Machine Learning are now trying to solve this problem by training for decades, or using more and more GPUs, but a way more efficient way would be to feed that algorithm with additional data, such that it can use more resources to infer it’s calculations.</p><p>This can be achieved by:</p><ol><li>Getting more data (a bigger timespan)</li><li>Adding statistical metrics</li><li>Adding your own signals and interpretations</li></ol><h3>Hands-on: Enhancing your data</h3><h3>First steps — Getting your data</h3><p>First, let us grab some basic OHLCV data. I like the yfinance (<a href="https://pypi.org/project/yfinance/">https://pypi.org/project/yfinance/</a>) module for its simplicity. It is not comparable to live-streams of data, but on the other hand, it’s free and great for experimentation!</p><pre>pip install yfinance pandas numpy matplotlib ta</pre><p>Import that module as well as Pandas Numpy Matplotlib</p><pre>import yfinance as yf<br>import matplotlib.pyplot as plt<br>import pandas as pd</pre><p>Get some stock data, interval and period refer to the timeranges.</p><p>interval accepts values like 1m,2m,5m,15m,30m,60m,90m,1h,1d,5d,1wk,1mo,3mo</p><p>period accepts values like 1d,5d,1mo,3mo,6mo,1y,2y,5y,10y,ytd,max</p><p>Not all combinations work, like 1m (1 Minute) intervals are only working with 7d max, 1h (1 hour) with 3 months max (needs to be written as 90d). But let’s work with daily data first</p><pre>df = yf.download(&quot;MSFT&quot;,period=&quot;5y&quot;,interval=&quot;1d&quot;)<br>df.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/863/0*Xa_QjzeaD1IM326r.png" /><figcaption>Image by author</figcaption></figure><p>Quick excourse: What the heck is Open, High, Low, Close, Adj Close, and Volume? Where is the price?!</p><p>There is no “one price” in the stock market! You can imagine OHLCV data as “buckets” or “bins” in time, summarizing all the trades that occurred in that window. The typical “line” chart you know usually refers to the “Close” price of that stock in time range X, e.g. the value the stock had at the close of the time range.</p><p>If we are looking at daily data, “Open” refers to the average (!) stock price at open of the markets, whilst “Close” refers to the average (!) price the stock had at the close of markets. If looking at hourly data, the “Open” refers to the beginning of that hour, like 11 am, and the “Close” to the close of that hour, so 11:59:59 am.</p><p>Equally “High” is the highest trade/price recorded in that time frame, and “Low” the lowest one. Volume refers to the number of assets or stocks traded in that time range.</p><p>Meaning if e.g. “Low” and “Close” of a column are close to each other, we are most likely seeing a downwards trend as the close is the current low. Also if Volume is high, there is a lot of trades going on, so if e.g. the Volume is higher than usual there seems to be something going on in the market. But anyway, head over to <a href="https://www.investopedia.com/">https://www.investopedia.com/</a> for that, we will continue coding now!</p><h3>What is “Adj Close”?</h3><p>This is important, as most ML algorithms are terribly confused by “normal” close-data. If there is a split in the stock, the data will look like the price has an insane drop.</p><p>The reason is, simply said, that if a stock get’s too expensive, the company decides to “split” the stock in two. Does that mean my investment halves? Of course not, you will simply get double the stocks you are holding, so on paper you still hold the same value of that stock.</p><p>Interestingly a split usually causes prices to rise (silly humans!), and if your machine learning algorithm sees a huge drop in price it will most likely sell the hell out of that stock.</p><p>For this reason, you should always use “adjusted” values, which can be imagined as “cleaned” price data, keeping in mind the splits, dividends, and all other events not affecting the true value of data. Therefore try to always use adjusted data for your algorithms!</p><p>In the case of yfinance it is easy to do, as we can just use “Adj Close” instead of “Close”.</p><h3>Plotting the data</h3><p>Looking at the data we can already see a nice, well-known curve</p><pre>plt.plot(df[&quot;Adj Close&quot;])</pre><h3>Step two: Enhancing your data with statistical data</h3><p>As said above, we need to create more information from our data for the algorithm to use, as it can not do this on its own.</p><p>I like to use the library ta (<a href="https://github.com/bukosabino/ta">https://github.com/bukosabino/ta</a>) as again, it is super easy to use and contains 100+ statistical calculations.</p><p>Install and import it with</p><pre>pip install ta<br>from ta import add_all_ta_features<br>from ta.utils import dropna</pre><p>Now if you already know which values you want to use you can only pick these, oooor we just smash all 100+ at our data:</p><pre>df = dropna(df) # clean nans if present<br>df = add_all_ta_features(df,open=&quot;Open&quot;, high=&quot;High&quot;, low=&quot;Low&quot;, close=&quot;Adj Close&quot;, volume=&quot;Volume&quot;)</pre><p>So what have we done?</p><pre>df.columns<br>Index([&#39;Open&#39;, &#39;High&#39;, &#39;Low&#39;, &#39;Close&#39;, &#39;Adj Close&#39;, &#39;Volume&#39;, &#39;volume_adi&#39;,<br>       &#39;volume_obv&#39;, &#39;volume_cmf&#39;, &#39;volume_fi&#39;, &#39;volume_mfi&#39;, &#39;volume_em&#39;,<br>       &#39;volume_sma_em&#39;, &#39;volume_vpt&#39;, &#39;volume_nvi&#39;, &#39;volume_vwap&#39;,<br>       &#39;volatility_atr&#39;, &#39;volatility_bbm&#39;, &#39;volatility_bbh&#39;, &#39;volatility_bbl&#39;,<br>       &#39;volatility_bbw&#39;, &#39;volatility_bbp&#39;, &#39;volatility_bbhi&#39;,<br>       &#39;volatility_bbli&#39;, &#39;volatility_kcc&#39;, &#39;volatility_kch&#39;, &#39;volatility_kcl&#39;,<br>       &#39;volatility_kcw&#39;, &#39;volatility_kcp&#39;, &#39;volatility_kchi&#39;,<br>       &#39;volatility_kcli&#39;, &#39;volatility_dcl&#39;, &#39;volatility_dch&#39;, &#39;volatility_dcm&#39;,<br>       &#39;volatility_dcw&#39;, &#39;volatility_dcp&#39;, &#39;volatility_ui&#39;, &#39;trend_macd&#39;,<br>       &#39;trend_macd_signal&#39;, &#39;trend_macd_diff&#39;, &#39;trend_sma_fast&#39;,<br>       &#39;trend_sma_slow&#39;, &#39;trend_ema_fast&#39;, &#39;trend_ema_slow&#39;, &#39;trend_adx&#39;,<br>       &#39;trend_adx_pos&#39;, &#39;trend_adx_neg&#39;, &#39;trend_vortex_ind_pos&#39;,<br>       &#39;trend_vortex_ind_neg&#39;, &#39;trend_vortex_ind_diff&#39;, &#39;trend_trix&#39;,<br>       &#39;trend_mass_index&#39;, &#39;trend_cci&#39;, &#39;trend_dpo&#39;, &#39;trend_kst&#39;,<br>       &#39;trend_kst_sig&#39;, &#39;trend_kst_diff&#39;, &#39;trend_ichimoku_conv&#39;,<br>       &#39;trend_ichimoku_base&#39;, &#39;trend_ichimoku_a&#39;, &#39;trend_ichimoku_b&#39;,<br>       &#39;trend_visual_ichimoku_a&#39;, &#39;trend_visual_ichimoku_b&#39;, &#39;trend_aroon_up&#39;,<br>       &#39;trend_aroon_down&#39;, &#39;trend_aroon_ind&#39;, &#39;trend_psar_up&#39;,<br>       &#39;trend_psar_down&#39;, &#39;trend_psar_up_indicator&#39;,<br>       &#39;trend_psar_down_indicator&#39;, &#39;trend_stc&#39;, &#39;momentum_rsi&#39;,<br>       &#39;momentum_stoch_rsi&#39;, &#39;momentum_stoch_rsi_k&#39;, &#39;momentum_stoch_rsi_d&#39;,<br>       &#39;momentum_tsi&#39;, &#39;momentum_uo&#39;, &#39;momentum_stoch&#39;,<br>       &#39;momentum_stoch_signal&#39;, &#39;momentum_wr&#39;, &#39;momentum_ao&#39;, &#39;momentum_kama&#39;,<br>       &#39;momentum_roc&#39;, &#39;momentum_ppo&#39;, &#39;momentum_ppo_signal&#39;,<br>       &#39;momentum_ppo_hist&#39;, &#39;others_dr&#39;, &#39;others_dlr&#39;, &#39;others_cr&#39;],<br>      dtype=&#39;object&#39;)</pre><p>Well, that should be enough for now!</p><p>Also, there is still a lot of nans in there, as some values are only calculated if enough time passed. In my experience filling them with zeros works quite well, even though there are more advanced techniques for that.</p><pre>df = df.fillna(0)</pre><h3>Step three: Creating your own signals</h3><p>Now it is time to translate your weird trade ideas into numbers!</p><p>Let us start with the classic Moving Average cross. The idea is as follows: If a short moving average crosses a slower moving average, it either indicates a price rise or fall, depending on the direction of the cross.</p><p>Again, have a look at investopedia for details: <a href="https://www.investopedia.com/articles/active-trading/052014/how-use-moving-average-buy-stocks.asp">https://www.investopedia.com/articles/active-trading/052014/how-use-moving-average-buy-stocks.asp</a></p><p>Our goal is to first calculate the SMA’s, and then to formulate crosses as 1 and -1, and 0 to signal no cross.</p><h3>Creating Simple Moving Averages</h3><pre># creating simple moving averages<br>averages = [1,2,5,10,15,20,25,50,100]<br>for average in averages:<br>  df[&#39;SMA_%d&#39;%average] = df[&quot;Adj Close&quot;].rolling(window=average).mean()</pre><pre># visualize only SMAs <br>filter_col = [col for col in df if col.startswith(&#39;SMA&#39;)]<br>df[filter_col].tail()</pre><p>And some visualization:</p><pre># results in bigger figures<br>plt.rcParams[&quot;figure.figsize&quot;] = (20,20)<br>for filter in filter_col:<br>  plt.plot(df[filter],label=filter)<br>plt.legend()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*e7FOFZzPB-ZmQIAX.png" /><figcaption>Image by author</figcaption></figure><h3>Creating a crossover signal</h3><p>Let us use a little helper function</p><pre>def createCross(data,fastSMA,slowSMA):<br>  fast = &#39;SMA_%d&#39;%fastSMA<br>  slow = &#39;SMA_%d&#39;%slowSMA<br>  crossname = &quot;cross_%d_%d&quot;%(fastSMA,slowSMA)<br>  previous_fast = data[fast].shift(1)<br>  previous_slow = data[slow].shift(1)<br>  neg = ((data[fast] &lt; data[slow]) &amp; (previous_fast &gt;= previous_slow))<br>  pos = ((data[fast] &gt; data[slow]) &amp; (previous_fast &lt;= previous_slow))<br>  data[crossname] = 0<br>  data.loc[neg,crossname] = -1<br>  data.loc[pos,crossname] = 1<br>  return data</pre><p>And now you could either insert custom values or follow our example and just take the cross products:</p><pre>for fast in averages:<br>  for slow in averages:<br>    if fast != slow and slow &gt; fast:<br>      df = createCross(df,fast,slow)</pre><p>This created a perfect classification signal signaling an upwards cross with 1, and a downwards cross with -1, with 0 being a neutral (no-cross).</p><p>This is just one example of what further signals you can provide.</p><h3>Creating a percentage change column</h3><p>And to add another example, if you are trying to predict the percentage change you will need a column showing the percentage change to the previous time range. This can luckily be easily done using pandas.</p><pre>df[&quot;pct_change&quot;] = df[&quot;Adj Close&quot;].pct_change()</pre><pre>Date<br>2021-01-21    0.013363<br>2021-01-22   -0.004463<br>2021-01-25    0.000538<br>2021-01-26    0.009754<br>2021-01-27   -0.010484<br>Name: pct_change, dtype: float64</pre><p>What a perfect signal for a regressional model!</p><h3>Summary</h3><p>Before we added our enhancements the dataframe just contained 5 columns, not much for a machine learning model!</p><p>In the end, after adding statistical values and our own signals, we already reached 135 features and columns of our dataframe!</p><p>So much better for your model!</p><p>What are your thoughts about this process? Did I miss something? Comment below!<br>Do you want to read more articles by Justin? Head over to my website for more!</p><p><a href="https://www.datafortress.cloud/blog/enhancing-stock-data-for-your-python-algorithmic-trading-model/">https://www.datafortress.cloud/blog/enhancing-stock-data-for-your-python-algorithmic-trading-model/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=b40c668e4087" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How to set up Anaconda and Jupyter Notebook the right way]]></title>
            <link>https://medium.com/data-science/how-to-set-up-anaconda-and-jupyter-notebook-the-right-way-de3b7623ea4a?source=rss-5ceff1a6b0a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/de3b7623ea4a</guid>
            <category><![CDATA[tensorflow]]></category>
            <category><![CDATA[jupyter]]></category>
            <category><![CDATA[jupyter-notebook]]></category>
            <category><![CDATA[anaconda]]></category>
            <category><![CDATA[conda]]></category>
            <dc:creator><![CDATA[Justin Güse]]></dc:creator>
            <pubDate>Mon, 25 Jan 2021 11:27:30 GMT</pubDate>
            <atom:updated>2021-01-25T13:21:20.799Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*v-UK7dWMxFGvC5BLeQgi2w.png" /><figcaption>Image by Author</figcaption></figure><p>If Anaconda (conda) and Jupyter Notebook (Jupyter Lab) are set up the right way the combination of them can become the perfect team, where you are able to easily switch between Deep Learning conda environments.</p><p>Some programs requiring Tensorflow 1.15, others Tensorflow 2.0? No problem! Just switch environments and Tensorflow versions with a simple click.</p><p>Also, did you ever install Jupyter Notebook extensions in each conda environment? Do not worry anymore, we are going to install the extensions once, and have them available in each environment!</p><h3>How are we going to achieve that?</h3><ol><li>Install Anaconda or Miniconda</li><li>Install Jupyter Notebook / Lab in the base environment</li><li>Install a new environment</li><li>Activate the environment for Jupyter Notebook</li></ol><h3>How to install Anaconda or Miniconda?</h3><p>Anaconda is a nice package containing a lot of Python packages already and allows for an easy start into the world of Python. Additionally, it allows creating environments in python, which contain different versions of your Python packages. E.g. if a program only runs with Python 2.7 or older versions of Matplotlib, you can create an own workspace for this program and switch back to Python 3 with a click of a button. Furthermore switching between Tensorflow 2.0 and Tensorflow 1.15 becomes easy as well, finally allowing you to switch between versions easily (which can be quite a headache otherwise).</p><p>Miniconda is a barebones version of Anaconda and can be nice if you are e.g. working on a server, where disk space is limited.</p><p>To install Anaconda or Miniconda head over to their website (<a href="https://www.anaconda.com/products/individual#Downloads">https://www.anaconda.com/products/individual#Downloads</a>), or if you are using Linux just copy the following commands.</p><p>The first link crawls the website for the newest version and writes it to the LATEST_ANACONDA variable.</p><pre>cd ~/Downloads<br>LATEST_ANACONDA=$(wget -O - https://www.anaconda.com/distribution/ 2&gt;/dev/null | sed -ne &#39;s@.*\(https:\/\/repo\.anaconda\.com\/archive\/Anaconda3-.*-Linux-x86_64\.sh\)\&quot;&gt;64-Bit (x86) Installer.*@\1@p&#39;)<br>wget $LATEST_ANACONDA<br>chmod +x Anaconda3*.sh # make it executable<br>./Anaconda3*.sh # execute the installer</pre><p>Follow the dialogue, and just agree on the defaults.</p><h3>Checking and switching the conda environments</h3><p>If conda is installed correctly (might need a logout and login, or restart), you should be able to see the output when typingcondainto your terminal.</p><p>To list the currently installed environments just typeconda env list</p><p>It should currently just show the “base” environment installed.</p><p>Switching between environments works as simply as typingconda activate [NAME]and if done with it deactivating it (and going back to the base environment) withconda deactivate.</p><p>The base environment is activated by default.</p><h3>Install Jupyter Notebook / Lab in the base environment</h3><p>Jupyter Notebook can easily be installed using conda. Our plan is to only install it in the base environment, and then just switch between sub-environments to avoid setting up Jupyter Lab in each environment.</p><h3>Installing Jupyter Notebook (default)</h3><pre>conda install -c conda-forge notebook<br>conda install -c conda-forge nb_conda_kernels</pre><h3>Installing Jupyter Lab</h3><pre>conda install -c conda-forge jupyterlab<br>conda install -c conda-forge nb_conda_kernels</pre><h3>Installing Jupyter Notebook extensions</h3><p>I really like Jupyter Notebook extensions, which support a lot of autocompletion, additional information, and in general things that make your life easier. A good default setting is included with the following install command:</p><pre>conda install -c conda-forge jupyter_contrib_nbextensions</pre><p>A good overview over other extensions: <a href="https://towardsdatascience.com/jupyter-notebook-extensions-517fa69d2231">https://towardsdatascience.com/jupyter-notebook-extensions-517fa69d2231</a></p><h3>(Optional) Installing pip package manager</h3><p>In my opinion it is a good idea to add the pip package manager to the base (and each sub-) environment, as not all packages are supported by conda install. Also, if pip is not installed in each sub-environment the package might just be installed in the “base” conda environment, causing an error where the package is not found in your sub-environment.</p><pre>conda install pip</pre><h3>Creating environments in conda and Jupyter Notebook</h3><p>Let us say you want to install both Tensorflow 2.0 and Tensorflow 1.15 in Jupyter Notebook.</p><p>For this example first, agree if you want to use the GPU or CPU version of Tensorflow. For using the GPU version add “-gpu” to TensorFlow, and otherwise, just leave it as is.</p><p>To create a new conda environment we can run</p><p>conda create --name tf-2.0</p><p>If you already plan to install some packages with it just add them to the end, like:</p><pre>conda create -n tf-2.0 tensorflow-gpu pip ipykernel</pre><p>I recommend installingpipfor package installation, andipykernelwill be needed to switch environments using Jupyter Notebook</p><p>To install an environment using TensorFlow 1.15 use the following:</p><pre>conda create -n tf-1.15 tensorflow-gpu==1.15 pip ipykernel</pre><p>If done successfully, you should be able to see three environments when executing the following command:</p><pre>conda env list</pre><ol><li>base</li><li>tf-2.0</li><li>tf-1.15</li></ol><h3>Start Jupyter Notebook and check the environments and extensions</h3><pre>jupyter notebook</pre><p>Running Jupyter Notebook in the base environment should allow you to see a tab containing “Extensions”, as well as “conda”/“environments”. Head over to Extensions and activate whichever extensions you like, and if you are ready, create a new notebook using the “new” button. Here, you should be able to choose between your base, tf-2.0 and tf-1.15 environment.</p><p>Attention: You always need to run jupyter notebook in the base environment. Runconda deactivateto leave your current environment and return to the base one.</p><p>If you need to install further packages activate an environment usingconda activate [NAME], run your commands likeconda install Xorpip install X, and leave the environment usingconda deactivate.</p><p>Let me know if this worked for you, it helped me a lot and I wished I would have known about this earlier!</p><p>Find this and other articles on my website: <a href="https://www.datafortress.cloud/blog/how-to-set-up-anaconda-and-jupyter-notebook-the-right-way/">https://www.datafortress.cloud/blog/how-to-set-up-anaconda-and-jupyter-notebook-the-right-way/</a></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=de3b7623ea4a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-set-up-anaconda-and-jupyter-notebook-the-right-way-de3b7623ea4a">How to set up Anaconda and Jupyter Notebook the right way</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[How To — Ditching Ubuntu in favor of Arch Linux for a Deep Learning Workstation]]></title>
            <link>https://medium.com/data-science/how-to-ditching-ubuntu-in-favor-of-arch-linux-for-a-deep-learning-workstation-c7ffae120f32?source=rss-5ceff1a6b0a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/c7ffae120f32</guid>
            <category><![CDATA[data-engineering]]></category>
            <category><![CDATA[machine-learning-tools]]></category>
            <category><![CDATA[arch-linux]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[deep-learning-framework]]></category>
            <dc:creator><![CDATA[Justin Güse]]></dc:creator>
            <pubDate>Tue, 29 Sep 2020 12:20:48 GMT</pubDate>
            <atom:updated>2020-09-29T12:52:51.125Z</atom:updated>
            <content:encoded><![CDATA[<h3>How To — Ditching Ubuntu in favor of Arch Linux for a Deep Learning Workstation</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*sDxCgIjxu4MDE2Y_YBBxlQ.png" /></figure><h3>Why should I ditch Ubuntu?</h3><p>Most of you might be using Ubuntu for their workstations, and that is fine for the more inexperienced users. One of the issues I had with Ubuntu and the Tensorflow/CUDA though, has been that handling the different drivers and versions of CUDA, cudnn, TensorFlow, and so on has been quite a struggle. I’m not sure about you, but once I had a working Tensorflow 1.15 or 2.0 environment, I usually did not touch it anymore being scared to mess up this holy configuration.</p><p>Working with different programs it would be nice to have a way of switching between the two most used TensorFlow versions of 1.15 and 2.0 like you can do with Google Colab in a single command, but installing a different TensorFlow version usually messed up my system again.</p><p>Additionally, Arch has always been on my To-Do list, as it is the most “barebone” Linux distro you can get, meaning you are working way closer on the hardware compared to “higher abstractions” like Ubuntu. In their own words, Ubuntu is built to “work out of the box and make the installation process as easy as possible for new users”, whilst the motto of Arch Linux is “customize everything”. Being way closer to the hardware Arch is insanely faster compared to Ubuntu (and miles ahead of Windows), for the cost of more Terminal usage.</p><p>When I have been using Arch in the past weeks, RAM usage usually halved compared to Ubuntu, and installing Machine Learning packages is a breeze. I can have both TensorFlow 1.15 and 2.0 working together, switching the versions with Anaconda environments. Also, the system works quite stable, as I am using the LTS (long term support) kernels of Linux, and usually updates to the famous AUR (user-made packages in Arch) are coming out a month ahead of the Debian (Ubuntu) packages.</p><p>All in all, I can only recommend setting up an Arch Linux Deep Learning station as it is:</p><ol><li>Faster, like packages will install super fast, deep learning is supercharged, …</li><li>More stable</li><li>Easier to switch between TensorFlow versions compared to Ubuntu.</li></ol><p>I will split the how-to in two parts, the first one being “How to I install Arch Linux” and the second one being “How to install the Deep Learning workstation packages”.</p><p>For the general <a href="https://www.datafortress.cloud/blog/howto-install-arch-linux-the-easy-way/">“How to install Arch Linux”, head over to this article</a>.</p><p>If Arch is too complex for now, you could try out <a href="https://https//manjaro.org/">Manjaro</a>, which is a user-friendly version of Arch, even though I can not guarantee that all packages will work the same, as they are slightly different. All in all it should work the same though.</p><p>I was thinking about creating a ready to install Image (iso or img), if enough people are interested leave a comment below or message me!</p><h3>Installing the Deep Learning (TensorFlow, CUDA, CUDNN, Anaconda) setup on a fresh Arch Linux installation</h3><p>Once you are <a href="https://www.datafortress.cloud/blog/howto-install-arch-linux-the-easy-way/">done with the Arch installation (phew!)</a>, let us first change some settings such that our system works more stable.</p><h3>1. Switching to the fastest mirrors</h3><p>Software is downloaded from so-called “mirrors”, which are servers containing all the Arch libraries. If not done automatically, it could happen that your servers are not optimized yet. Therefore, we are going to install a small tool that finds and saves the fastest servers called “reflector”</p><p>Install reflector using</p><blockquote><em>sudo pacman -S reflector</em></blockquote><p>Find and download the best servers</p><blockquote><em>reflector –verbose -l 20 -n 20 –sort rate –save /etc/pacman.d/mirrorlist</em></blockquote><p>Check the output if it makes sense, e.g. if the domains are close to your location. If not, you could add the country tag to get more precise results, e.g. for Germany and Austria:</p><blockquote><em>reflector -c “AT,DE” –verbose -l 20 -n 20 –sort rate –save /etc/pacman.d/mirrorlist</em></blockquote><p>Update your installation</p><blockquote><em>sudo pacman -Syyu</em></blockquote><h3>2. Changing the Desktop Environment</h3><p>If you are using Manjaro or chose the “Gnome” Desktop environment as you know it from Ubuntu, it might be worth it to think about changing it as Gnome is known to eat more RAM than Chrome, and we surely need RAM in our Deep Learning setup.</p><p>If you like Gnome, feel free to skip this step. Otherwise, I can recommend the Xfce desktop as it is a good combination of lightweight and full of features.</p><p>Download Xfce</p><blockquote><em>sudo pacman -S xfce4 xfce4-goodies lxdm</em></blockquote><p>Lxdm is a display manager that allows you to use multiple desktops.</p><p>Log out of your current session and press Alt + F2 (or Alt + F3 if it does not work) to get a terminal. First disable Gnome and afterward “activate” Xfce:</p><p>Deactivate and uninstall gnome:</p><blockquote><em>sudo systemctl disable gdm<br>sudo pacman -R gnome gnome-extras</em></blockquote><p>Activate Xfce</p><blockquote><em>sudo systemctl enable lxdm<br>sudo systemctl start lxdm</em></blockquote><p>If the new Xfce desktop does open just login and explore, if not try to reboot (sudo reboot). If that does not help proceed to crying and rolling on the floor, and send me a message or comment afterward.</p><h3>3. Installing the LTS (long term support) Linux kernels for better stability</h3><p>Arch is famous for being really close to the current Linux kernels, which is good if you always want the newest packages and Linux features, but a bad idea if you are building a Deep Learning Workstation.</p><p>That is why I switched to the LTS kernels, which are basically kernels that receive more support and are more stable than the newer versions of the Linux Kernel.</p><p>Luckily switching kernels is super easy in Arch. First we will download the kernels, and afterward tell our boot manager which kernel to choose.</p><p>First download the LTS kernels:</p><blockquote><em>sudo pacman -S linux-lts linux-lts-headers</em></blockquote><p>Have a look at your current kernel versions:</p><blockquote><em>ls -lsha /boot</em></blockquote><p>One kernel should be named vmlinuz-linux.img and initramfs-linux.img (your current versions) and the LTS ones the same with -lts at the end.</p><p>If you are seeing two kernels you can now proceed to delete the old kernels:</p><blockquote><em>sudo pacman -R linux</em></blockquote><p>Now a more advanced part is that you will need to tell your bootloader which kernel to choose. The question is which bootloader you are using, but in most cases it is Grub. If you followed my Arch installation tutorial your bootloader is systemd-boot.</p><p>My recommendation is try the Grub instructions, and if that does not work proceed to the others.</p><h4>Changing the Grub bootloader for the LTS linux kernels</h4><blockquote><em>grub-mkconfig -o /boot/grub/grub.cfg</em></blockquote><p>If you see an error proceed to the next bootloader, otherwise reboot (sudo reboot).</p><h4>Changing the syslinux bootloader for the LTS linux kernels</h4><p>Edit the config file:</p><blockquote><em>sudo nano /boot/syslinux/syslinux.cfg</em></blockquote><p>Simply add “-lts” to the vmlinuz-linux.img and initramfs-linux.img, such that they are vmlinuz-linux-lts.img and initramfs-linux-lts.img</p><h4>Changing the systemd-boot bootloader for the LTS linux kernels</h4><p>If you are coming from my Arch installation guide, this is your bootloader.</p><p>Edit the config file:</p><blockquote><em>sudo nano /boot/loader/entries/arch.conf</em></blockquote><p>Simply add “-lts” to the vmlinuz-linux.img and initramfs-linux.img, such that they are vmlinuz-linux-lts.img and initramfs-linux-lts.img</p><h3>4. Installing yay, an easy way to install AUR packages</h3><p>You should prefer to use the ultra-fast pacman to install most packages, but an amazing thing about Arch is that users create millions of custom packages that are super easy to install. You can basically find any program you can think of in this repo.</p><p>Install git SVC</p><blockquote><em>sudo pacman -S git<br>mkdir ~/tmp<br>git clone </em><a href="https://aur.archlinux.org/yay-git.git"><em>https://aur.archlinux.org/yay-git.git</em></a><em> ~/tmp/yay<br>cd ~/tmp/yay<br>makepkg -si</em></blockquote><p>Now you can browse all the nice AUR packages in <a href="https://aur.archlinux.org/packages/">https://aur.archlinux.org/packages/</a> or just go for it and type:</p><blockquote><em>yay -S [PACKAGE]</em></blockquote><p>To install it.</p><h3>5. Finally, the real cuda, cudnn, anaconda installation running both TensorFlow 1.15 and 2.0</h3><p>Install Nvidia drivers, cuda, cudnn with a simple command</p><blockquote><em>sudo pacman -S nvidia nvidia-utils cuda cudnn</em></blockquote><p>This takes some time, so grab a coffee or proceed with the next steps</p><p>Download Anaconda, I like Miniconda:</p><blockquote><em>wget </em><a href="https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh"><em>https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh</em></a><em> ~/</em></blockquote><p>Make it executable and install</p><blockquote><em>cd ~/<br>chmod +x ./Miniconda*.sh<br>./Miniconda*.sh</em></blockquote><p>Just leave everything as default.</p><blockquote><em>source ./bash_profile</em></blockquote><p>Reboot your system</p><blockquote><em>sudo reboot</em></blockquote><p>Install tensorflow</p><p>Now is the time to decide between TensorFlow for CPU or GPU. I will continue with the GPU option, but if you want to run the CPU version just remove the “-gpu” from the package name.</p><p>Create an anaconda environment for Tensorflow 2.0</p><blockquote><em>conda create –name tf2.0<br>conda activate tf2.0<br>conda install pip<br>conda install tensorflow-gpu pandas numpy</em></blockquote><p>Done! Now check the result with:</p><blockquote><em>python<br>from tensorflow.python.client import device_lib<br>device_lib.list_local_devices()</em></blockquote><p>If the result shows a device name like this you are done!</p><p>2018–05–01 05:25:25.929575: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1356] Found device 0 with properties: name: GeForce GTX 3080 10GB major: …</p><p>Create an anaconda environment for Tensorflow 1.15</p><blockquote><em>conda deactivate<br>conda create –name tf1.15<br>conda activate tf1.15<br>conda install pip python==3.7<br>conda install tensorflow-gpu==1.15</em></blockquote><p>And again check if everything works and your gpu is recognized:</p><blockquote><em>python<br>from tensorflow.python.client import device_lib<br>device_lib.list_local_devices()</em></blockquote><h3>6. Switching between TensorFlow 1.15 and TensorFlow 2.0 on one device!</h3><p>Just a dream coming true in my opinion, just select the 1.15 version with</p><blockquote><em>conda activate tf1.15</em></blockquote><p>And the TensorFlow 2.0 version with</p><blockquote><em>conda activate tf2.0</em></blockquote><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=c7ffae120f32" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/how-to-ditching-ubuntu-in-favor-of-arch-linux-for-a-deep-learning-workstation-c7ffae120f32">How To — Ditching Ubuntu in favor of Arch Linux for a Deep Learning Workstation</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Installing Arch Linux the easy way with encrypted drives for Deep Learning]]></title>
            <link>https://guese-justin.medium.com/installing-arch-linux-the-easy-way-with-encrypted-drives-for-deep-learning-83fd55035ff7?source=rss-5ceff1a6b0a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/83fd55035ff7</guid>
            <category><![CDATA[linux]]></category>
            <category><![CDATA[encryption]]></category>
            <category><![CDATA[arch-linux]]></category>
            <category><![CDATA[technology]]></category>
            <category><![CDATA[operating-systems]]></category>
            <dc:creator><![CDATA[Justin Güse]]></dc:creator>
            <pubDate>Tue, 29 Sep 2020 12:17:46 GMT</pubDate>
            <atom:updated>2020-09-29T12:17:46.256Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*p4abLfbt9kuL0FhlEMGiew.png" /></figure><h3>The benefits of Arch Linux</h3><p>In my other post, <a href="https://www.datafortress.cloud/blog/howto-arch-linux-deeplearning-workstation/">“How To: Ditching Ubuntu in favor of Arch Linux for a Deep Learning Workstation”</a>, I have been explaining why I switched from Ubuntu to Arch Linux for my Machine Learning workstation. Summarized it is mostly because of speed, because Arch is way closer to the hardware and therefore way faster, less bloatware and therefore less RAM usage which I need to machine learning, and the amazing pacman and AUR packages which are fast and easy to install.</p><p>In Ubuntu, for example, it is quite hard to get TensorFlow and CUDA working for Deep Learning, as the Debian package structure is different and the installation slower compared to Arch. In Arch, dependencies are handled nicely, and packages are not that abstracted as Arch is basically the “pure” Linux kernel itself.</p><p>Additionally packages are coming out way faster, and the ArchWiki is said to be the most extensive documentation there is for an operating system.</p><p>The motto of Arch can be described as “Customize everything, and be as fast as possible”, whilst Ubuntu’s motto would be “Make the installation and usage as easy as possible for new users”. Now, I am not saying that Ubuntu is bad, because the community is huge, and it is the perfect Distro to start your journey into the world of Linux, but once you are more used to Linux and need more speed, Arch might be the best next step.</p><p>I am at least happy with Arch, as it is way faster than everything I have seen before, and I can customize everything.</p><p>I have been thinking about providing an iso or img for my Deep Learning build for an easy install. If you are interested please comment or send me a message.</p><p>Additionally, if you are fine with a slight abstraction layer <a href="https://manjaro.org/">Manjaro Arch</a> might be a good choice for you. It is basically Arch, but made easy and user-friendly. I loved it, but moved to Arch as Manjaro uses their own packages, which are still good, but not the same as the “pure” Arch packages. If you are not a power-user this might be the choice for you.</p><h3>How to Install Arch Linux the easy way</h3><p>Whilst the <a href="https://wiki.archlinux.org/index.php/installation_guide">introductory guide</a> provided by arch is quite good already, it can be confusing for newcomers. That is why I have decided to write my own tutorial.</p><p>My goal has been to install Arch with encrypted drives, as I need to secure the data at rest for my clients.</p><h3>1. Downloading the Arch Linux Image</h3><p>Head over to <a href="https://www.archlinux.org/download/">https://www.archlinux.org/download/</a> and download the image.</p><p>Write it to an USB stick. E.g. (but you can use your own program) for:</p><p>Linux:</p><blockquote><em>dd if=ISOFILE of=/dev/sdX status=progress</em></blockquote><p>Where the sdX is your usb stick. Check it with “lsblk”</p><p>Windows / Mac: I liked to use Etcher <a href="https://www.balena.io/etcher/">https://www.balena.io/etcher/</a></p><p>Boot into the USB stick using your Boot menu (F11, F12 in most cases)</p><h3>2. First setup in Live ISO</h3><p>Once you are in the Live ISO of Arch, you should see the basic command line of Arch.</p><p><strong>Not an US keyboard</strong>? Load your keys with:</p><blockquote><em>loadkeys KEYMAP</em></blockquote><p>Where Keymap is your locale. Get your locale with</p><blockquote><em>localectl list-keymaps | grep -i SEARCHTERM</em></blockquote><p>E.g. using the above command with SEARCHTERM = Germany returns de-latin1, for which the loadkeys command is:</p><blockquote><em>loadkeys de-latin1</em></blockquote><p>(Optional) If you do not have ethernet you might have to connect to your wifi with the following commands. Replace WIFINETWORK and WIFIPASSWORD with your wifi name and password.</p><blockquote><em>wpa_passphrase ‘WIFINETWORK’ ‘WIFIPASSWORD’ » /etc/wpa_supplicant/wpa_supplicant.conf<br>wpa_supplicant -Bc /etc/wpa_supplicant/wpa_supplicant.conf -i ‘wlan0’<br>If the second command does not work your wifi device name might be different. You can check it with “ifconfig” dhclient</em></blockquote><p>Check your connection with curling a website. If the response looks like HTML you are connected, if it says something along the lines of “404”, “timed out” or “no connection” repeat the above steps and watch for typos.</p><blockquote><em>curl </em><a href="http://www.datafortress.cloud/"><em>www.datafortress.cloud</em></a></blockquote><h3>2. Partitioning the drives</h3><p>This is the most difficult step, so once you are done with it you are almost done! I chose the more advanced version where the drive is completely encrypted for advanced security, but feel free to use the “<a href="https://wiki.archlinux.org/index.php/Partitioning">standard method</a>” instead.</p><p>This guide will assume you are planning to wipe your whole drive. So be sure that all data is backed up. Of course, a dual-boot with windows or so is possible as well, but I will explain that in another guide.</p><p>The plan is to have two partitions: First partition: EFI boot drive — which tells your PC how to boot into Arch Second partition: Completely encrypted drive with “sub”-drives containing the swap file, the root folder and a separate /home drive for security. Why three partitions on the second drive? Let’s say you mess up your root installation. In that case your home folder with your documents, settings and so on will be on a separate partition and can easily be reused in a new install. Meaning you could install a fresh new Arch install in the root partition, and just “reuse” your home folder. Nice.</p><h4>Step 1: Get the name of your Drive</h4><blockquote><em>lbslk -lh</em></blockquote><p>This shows you all connected drives. Usually your main drive should be called /dev/sda and your USB stick /dev/sdb.</p><p><strong>Check if sda is really your main drive according to the GB storage capacity and everything. If you simply copy and paste the following commands with sda you might wipe the wrong drive!!</strong> That is why I will name the device sdX in the following paragraphs, but replace the X with your device number (a,b,c, …). Some drives are having different names, just compare and replace accordingly.</p><h4>Step 2: Partition</h4><p>Start gdisk</p><blockquote><em>gdisk /dev/sdX</em></blockquote><p>Boot EFI Drive Setup</p><ol><li>Type ‘o’ to create a partition table</li><li>Type ’n’ for a new partition</li><li>Enter</li><li>Enter</li><li>+256M</li><li>EF00 This will create a 256 mb partition with the EFI format that we need for booting</li></ol><p>Encrypted Arch Drive setup</p><ol><li>Type ’n’ to create a new partition</li><li>Enter</li><li>Enter</li><li>Enter</li><li>8309</li></ol><p>Check if everything looks right with pressing ‘p’. It should look like this:</p><blockquote><em>Number Start (sector) End (sector) Size Code Name<br>1 2048 1050623 256.0 MiB EF00 EFI System<br>2 1050624 242187466 115.0 GiB 8309 Linux LUKS</em></blockquote><p>Looks good? Press ‘w’ to write changes to disk.</p><h3>3. Create the encrypted filesystems</h3><p>The partitions are empty right now. Next we will create the filesystems and encrypted “sub”-partitions. Remember to replace the X with your drive number (usually a).</p><blockquote><em>cryptsetup luksFormat /dev/sdX2<br>cryptsetup open /dev/sdX2 cryptlvm<br>pvcreate /dev/mapper/cryptlvm<br>vgcreate datafortress /dev/mapper/cryptlvm</em></blockquote><p>Feel free to replace the following disk sizes in front of the “G” for Gigabyte with your desired size. For Deep Learning Swap should be 32 Gb, for other cases around the size of your RAM. I recommend the root system to be 40Gb, but the minimum should be 10Gb. You can replace datafortress with anything, or leave it as is to be the cool kid in town.</p><blockquote><em>lvcreate -L 16G datafortress -n swap<br>lvcreate -L 40G datafortress -n root<br>lvcreate -l +100%FREE datafortress -n home</em></blockquote><p>Create filesystems</p><blockquote><em>mkfs.vat -F32 /dev/sdX1<br>mkfs.ext4 /dev/mapper/datafortress-root<br>mkfs.ext4 /dev/mapper/datafortress-home<br>mkswap /dev/mapper/datafortress-swap</em></blockquote><p>Mount them</p><blockquote><em>mount /dev/mapper/datafortress-root /mnt<br>mkdir /mnt/home<br>mkdir /mnt/boot<br>mount /dev/mapper/datafortress-home /mnt/home<br>mount /dev/sdX1 /mnt/boot<br>swapon /dev/mapper/datafortress-swap</em></blockquote><h3>4. Install base Linux</h3><p>Before installing it is recommended to update your mirrorlist such that packages are downloaded from the closest mirrors. <a href="https://www.datafortress.cloud/blog/howto-arch-linux-deeplearning-workstation/">Head over to my other article and search for “reflector” for an instruction on how to do that</a>.</p><blockquote><em>pacstrap /mnt base base-devel linux linux-firmware nano</em></blockquote><h3>5. Install Fstab</h3><p>Fstab is basically an instruction how to mount your drives. Just copy paste it for now.</p><blockquote><em>genfstab -U /mnt » /mnt/etc/fstab</em></blockquote><h3>6. Last polishing touches</h3><p>We are going to chroot into our mount, which is basically like running sudo in our system.</p><blockquote><em>arch-chroot /mnt<br>ln -s /usr/share/zoneinfo/REGION/CITY /etc/localtime</em></blockquote><p>Where Region is your region (just use tab to see the options if you types the /usr/…./zoneinfo/, like Europe, and CITY your city, like Vienna.</p><blockquote><em>hwclock –systohc</em></blockquote><p>Set your hostname. The funny part when installing Arch is you get to do things Ubuntu and others do for you. Set the hostname for your device, e.g. if you are going to ssh into your device later you do not have to type an IP, but can use this name instead. E.g. “ssh user@hunneybunney” could be way nicer to remember than 192.168.0.231.</p><blockquote><em>echo NAME &gt; /etc/hostname</em></blockquote><p>Next we are setting our locale, basically the system language. Open the file in nano and search for your language code. Use the UTF-8 version if possible. Searching with nano is possible using Strg + W. <a href="https://www.howtogeek.com/howto/42980/the-beginners-guide-to-nano-the-linux-command-line-text-editor/">Nano tutorial</a>. E.g. I wanted to have German and English (US), that is why I uncommented en_US.UTF-8 UTF-8 and de_DE.UTF-8 UTF-8</p><blockquote><em>nano /etc/locale.gen</em></blockquote><p>Generate locales</p><blockquote><em>locale-gen</em></blockquote><p>(Optional) Now if you are using a non-US Keyboard add your keymap:</p><blockquote><em>echo KEYMAP &gt; /etc/vconsole.conf</em></blockquote><p>Use the Keymap from the beginning, like de-latin1</p><p>Next install some software. If you already know pacman packages you want to install, do it now. I recommend just using the minimum first and then continue installation of packages if everything works fine.</p><p><strong>Minimum</strong></p><blockquote><em>pacman -Syu<br>pacman -S wpa_supplicant dhclient lvm2 dialog</em></blockquote><p><strong>Installing a desktop environment</strong> Feel free to install a desktop environment already, which allows you to have a GUI of everything.</p><p>Install a display manager</p><blockquote><em>pacman -S lightdm</em></blockquote><p>Now the benefits of Arch are choosing your own Desktop environment. For the looks choose Gnome, but it will be RAM hungry, for efficiency choose Xfce.</p><p><strong>Only install ONE of the following</strong></p><p>To install GNOME:</p><blockquote><em>pacman -S gnome gnome-extra</em></blockquote><p>To install Cinnamon:</p><blockquote><em>pacman -S cinnamon nemo-fileroller</em></blockquote><p>To install XFCE:</p><blockquote><em>pacman -S xfce4 xfce4-goodies</em></blockquote><p>To install KDE:</p><blockquote><em>pacman -S plasma</em></blockquote><p>To install MATE:</p><blockquote><em>pacman -S mate mate-extra</em></blockquote><h3>7. Adding users</h3><p>I recommend using different passwords for the root and default user.</p><blockquote><em>useradd -m -G wheel ‘USERNAME’<br>passwd ‘USERNAME’</em></blockquote><p>Create a password for the root user</p><blockquote><em>passwd</em></blockquote><p>You will login with your default user. If you need to do administrative stuff you can switch users with “su — root”</p><h3>8. Boot configuration</h3><p>Now there are a lot of boot managers, but I liked the systemd-boot loader the most. <a href="https://wiki.archlinux.org/index.php/Category:Boot_loaders">Feel free to choose another one</a>.</p><blockquote><em>nano /etc/mkinitcpio.conf</em></blockquote><p>Search for the HOOKS line and edit it accordingly. Order matters!</p><blockquote><em>HOOKS=(base udev autodetect keyboard keymap consolefont modconf block encrypt lvm2 filesystems resume fsck)</em></blockquote><p>Close and Save the file (Strg + X, y)</p><blockquote><em>mkinitcpio -p linux<br>bootctl install</em></blockquote><p>Edit your boot entries</p><p>You will need the partition id of the encrypted drive. Usually this will be /dev/sda2, but can be different. It will not be the first 256mb drive, as this is the boot partition.</p><p>Check for the name and note it down, most likely it is /dev/sda2</p><blockquote><em>lsblk -lh</em></blockquote><p>Then look for the UUID string in the output of following command at the device id /dev/sdX2</p><blockquote><em>blkid</em></blockquote><p>You can try the same command with a filter as well:</p><blockquote><em>blkid | grep UUID=</em></blockquote><p>Example UUID: 727cac18–044b-4504–87f1-a5aefa774bda</p><blockquote><em>nano /boot/loader/entries/arch.conf Add the following:</em></blockquote><blockquote><em>title ArchLinux<br>linux /vmlinuz-linux<br>initrd /initramfs-linux.img<br>options cryptdevice=UUID=:lvm:allow-discards resume=/dev/mapper/datafortress-swap root=/dev/mapper/datafortress-root rw quiet</em></blockquote><p>Remember to replace datafortress with the id you chose if you changed it. Otherwise leave as is.</p><h3>9. Locking up</h3><p>DONE! Now reboot, remove the USB stick and hope everything worked.</p><blockquote><em>exit<br>umount -R /mnt<br>reboot</em></blockquote><p><a href="https://www.datafortress.cloud/blog/howto-arch-linux-deeplearning-workstation/"><strong>I recommend setting up LTS kernels, as they are more stable, and update the mirrorlist. Check my guide on “creating an Arch Linux Deep Learning station” for more information on how to do that</strong></a><strong>.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=83fd55035ff7" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Going from Wordpress to serverless, unhackable high-speed static websites]]></title>
            <link>https://guese-justin.medium.com/going-from-wordpress-to-serverless-unhackable-high-speed-static-websites-28480630914b?source=rss-5ceff1a6b0a7------2</link>
            <guid isPermaLink="false">https://medium.com/p/28480630914b</guid>
            <category><![CDATA[serverless]]></category>
            <category><![CDATA[web-development]]></category>
            <category><![CDATA[hosting]]></category>
            <category><![CDATA[static-site]]></category>
            <category><![CDATA[website-security]]></category>
            <dc:creator><![CDATA[Justin Güse]]></dc:creator>
            <pubDate>Thu, 17 Sep 2020 20:15:12 GMT</pubDate>
            <atom:updated>2020-09-17T20:15:12.375Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/500/0*cKwuNpDo65-b0xEo.png" /><figcaption>Sourcecode of static website built with Hugo</figcaption></figure><p>I have been building websites in the past, but have always been struggling with the slow performance of WordPress. If it is loaded with plugins, it will need quite some resources and can be a pain if you are just developing an idea on a small server.</p><p>Additionally, safety is of concern as well, and as a system that <a href="https://en.wikipedia.org/wiki/WordPress">33,6% of websites are using</a>, it is quite attractive for hackers to find exploits and other issues in it. But again, as it is massively popular, there is almost every time a plugin for the issues you are having, which makes it easy to use, and a great “all-in-one” tool.</p><h3>Idea 1: Improving WordPress Development</h3><p>One of the first things I did in the past has been to develop WordPress locally (e.g. <a href="https://www.smashingmagazine.com/2018/04/wordpress-local-development-beginners-setup-deployment/">see this AWS post</a>), and then just publish the result on a server. Programming and writing speeds increased enormously, but the uploading part proved to be a problem, as WordPress links are usually “hardwired” into the SQL database it uses. Meaning all my links were referring to “localhost” (My Computer) instead of the target domain. There are ways to solve this, like rewriting your URLs in SQL, or using rewriting htaccess rules to refer “old” URLs to the “new” ones, but still, it was a lot of struggle to get going.</p><h3>Idea 2: Online Development with Offloaded Media files</h3><p>This URL rewriting issue got on my nerves real quick, and local development is bad for multiple developers. That is why I decided to “go online” again, and work “with the cloud”. The architecture I followed has been to deploy one development server, that is only accessible to developers, and to upload media files to shared storage (AWS S3) from which end users are pulling the media files. As media files (pictures, videos, …) are the most demanding parts of WordPress, speed increased drastically, and additionally, it has been easy to set up a CDN on top of it, which basically means the media files are deployed all over the world at unlimited capacity (basically serverless). This means, that one user in e.g. Puerto Rico does not need to access my server in Frankfurt, but has a “local” copy close to him. Additionally, as the “heavy” part of WordPress has been “outsourced”, only “small” servers were needed to handle PHP requests and the “back-office” parts of WordPress. Feel free to ask me more about it in the comments or a direct message, or <a href="https://devops.com/hosting-scalable-wordpress-on-aws/">check out a similar approach by AWS</a>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/0*ME2k-C5UsgX2usDC.png" /><figcaption>An example of how WordPress can be set up in AWS</figcaption></figure><p>Together with Autoscaling, this seemed to be the most ideal setup for WordPress, and it proved great. BUT…</p><p>You still had to check for Plugin Updates, Security, and Monitoring in general. Even though AWS helps a lot to make this architecture quite resilient and fast, there is still a high operational demand. Additionally, running a separate development, database, load balancing, and so on the server can be quite cost expensive, especially for a website that does not have many users. And what did Werner Vogels say at re:invent 2015?</p><blockquote><strong><em>“No server is easier to manage than no server”</em></strong></blockquote><blockquote><em>Werner Vogels at re:invent 2015</em></blockquote><h3>Excursion: A brief history of the Code of web</h3><p>WordPress is great for writers and editors, but form the perspective of a solution architect it is not great. Why? Even though everything is clickable, easy to handle, and so on, all resources and information are pulled from a database in the background, even though if it is pulled for the 100000th time this day. There are methods to reduce the query load on SQL databases, like Redis and Memcached, but why should I “calculate” the same webpage for every single user? “Back in the day”, websites did just load in seconds (except someone has been on the phone) and they were super small — what has changed? Together with new design demands, today’s websites are full of effects and designs that are heavy on computational resources. Even though this is definitely an improvement to the 90s black and white style, loading times of websites increased dramatically — especially as the world’s connection standard is still mobile network.</p><p>To render all the effects, PHP code is used in the background, which is code executed on the server itself. Meaning every time a user connects to a website, the server is calculating the website it is going to show to the user. The 90s version of websites featured just plain HTML code, which are basically simple instructions to the browser on how to handle things. Like the tag tells the browser this is a heading, and is a paragraph. No (sorry for reducing complexity!) calculations are needed.</p><p>Additionally, Javascript and CSS go a similar path as CSS describes design in a similar approach to HTML, and Javascript is executed not on the server, but on the client-side. Meaning the server does not calculate itself, but “sends instructions” to the client’s browser (e.g. your phone).</p><p>So why don’t we use just HTML, Javascript, and CSS? PHP enables us to do a lot of things and allows for content creation frameworks like WordPress to make our lives easier. But the efficient way to produce websites would be to generate them once and then just distribute them already rendered to the masses.</p><h3>Idea 4: Going back to the roots</h3><p>So am I saying we should go back to the black and white HTML pages of the 90s? Of course not, but the combination of HTML and CSS can produce great results, and Javascript becomes more and more capable to handle processes only PHP could handle in the past. And if calculations are needed, there are great new serverless possibilities available like AWS Lambda (check out my blog for some applications of Lambda).</p><p>Going back to the main topic, I have decided to write my blog and any future websites in plain HTML, CSS, and JS, as I do not need to</p><ol><li><strong>manage a server</strong> — I can just host it for almost free on Github or AWS S3</li><li><strong>worry about high-demands</strong> — S3 and Github scale automatically, meaning if thousands of visitors arrive on my website it will not crash my server pay much — as I do not need as many calculations as with WordPress, running this blog is completely free</li><li><strong>do not need to worry about security issues</strong> — my blog is basically unhackable</li></ol><p>Additionally, the website is blazingly fast, with a Google Pagespeed score of 100%, which has a great effect on page ranking as well, as Google favors high-speed websites. The only reason why the current score is down to 90%, is that I decided to include CRM and tracking tools on my blog. When was the last time you saw a free website achieving this score?</p><p>All in all, it is just great, but am I writing all the HTML code by myself?</p><h3>Introducing: Static Website Generators</h3><p>Of course not, and luckily there are great tools to handle this for me. Static website builders like <a href="https://jekyllrb.com/">Jekyll</a> or <a href="https://gohugo.io/">Hugo</a> help a lot to let you basically just type in Markdown language (basically a simple txt file) and convert your texts into HTML and a nice website. The code is just calculated once and can be uploaded to a server, or Github pages and AWS S3 right away to be completely serverless. <a href="https://www.datafortress.cloud/project/serverless-static-website/">How does it work? Check out my case studies on my blog for an in-depth explanation</a>.</p><h3>Summary</h3><p>It feels great to not worry about uptime, scaling, and security anymore. Is it harder than WordPress? It depends. As this technology just develops, there is a rethinking to be done if you worked with WordPress and others in the past, but again there are many great tools that make static website builders similar to the “known” WordPress environment, like forestry.io for example. How? <a href="https://www.datafortress.cloud/project/serverless-static-website/">Check out my blog at www.datafortress.cloud to see the in-depth explanation</a>. For now, I would be interested if you ever tried to go serverless, or what your experiences with WordPress are. <a href="https://www.datafortress.cloud/contact/">Shoot me a message, or write a comment below</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=28480630914b" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>