<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Gürkan Soykan on Medium]]></title>
        <description><![CDATA[Stories by Gürkan Soykan on Medium]]></description>
        <link>https://medium.com/@gsoykan?source=rss-eb44eb87d7fc------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*7Vss1EgtcEFqpa4xV1qkFA.jpeg</url>
            <title>Stories by Gürkan Soykan on Medium</title>
            <link>https://medium.com/@gsoykan?source=rss-eb44eb87d7fc------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Fri, 20 Nov 2026 18:39:55 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@gsoykan/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Training word2vec Skip-gram with Hierarchical Softmax in PyTorch]]></title>
            <link>https://medium.com/@gsoykan/training-word2vec-skip-gram-with-hierarchical-softmax-in-pytorch-bbd3363bcff9?source=rss-eb44eb87d7fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/bbd3363bcff9</guid>
            <category><![CDATA[hierarchical-softmax]]></category>
            <category><![CDATA[word2vec]]></category>
            <category><![CDATA[pytorch]]></category>
            <category><![CDATA[skip-gram]]></category>
            <dc:creator><![CDATA[Gürkan Soykan]]></dc:creator>
            <pubDate>Fri, 22 Mar 2024 19:47:02 GMT</pubDate>
            <atom:updated>2024-03-22T19:47:02.132Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="Huffman Tree for Hierarchical Softmax" src="https://cdn-images-1.medium.com/max/602/1*TKK_vvjRonjGXvwb5aOWXQ.png" /><figcaption>image source: <a href="https://paperswithcode.com/method/hierarchical-softmax">https://paperswithcode.com/method/hierarchical-softmax</a></figcaption></figure><p>In our recent lab session, we focused on the Skip-Gram Word2Vec approach and hierarchical softmax. Although there are numerous resources available on the Skip-Gram algorithm and training with PyTorch, I found the existing resources for hierarchical softmax to be lacking, especially in terms of actual implementations in PyTorch. Furthermore, the examples or tutorials I encountered were often intertwined with negative sampling, which led me to write an implementation of Skip-Gram using purely hierarchical softmax.</p><p>Another issue I noticed was that for Skip-Gram with softmax, the existing examples were typically trained with CrossEntropyLoss in PyTorch. This implied that the target and context words were paired separately as tuples, resulting in a dataset size of context_size * 2 per target word. To address these challenges, in this blog post, I will cover:</p><ol><li>An alternative approach for Skip-Gram with softmax that processes targets and contexts together.</li><li>A vanilla Skip-Gram model with hierarchical softmax, demonstrating how binary cross-entropy loss can be effectively utilized.</li></ol><h3>1) Alternative Forward Pass for Skip-Gram with Softmax</h3><p>Initially, it is important to discuss the __getitem__ function of the dataset and its output:</p><pre>def __getitem__(self, idx) -&gt; Tuple[int, np.ndarray]:<br>    tokens: List[str] = self.instances[idx]<br>    ids = self.tokenizer.encode(tokens)<br>    context = ids[:self.context_size] + ids[self.context_size + 1:]<br>    target = ids[self.context_size]<br>    assert target != self.tokenizer.unk_token_idx, &quot;target is unk&quot;<br>    return target, np.array(context)</pre><p>In this post, I will highlight the key aspects, as the focus is on sharing crucial insights. The __getitem__ function is designed to manage a sliding window in the self.instances list, which contains the text instances.</p><p>The tokenizer’s encode function is called to convert tokens into their corresponding ID values. Subsequently, the context and target tokens are separated: the target is a single integer value (index), and the context is a list of IDs returned in the np.ndarray format.</p><p>As a result, the dataloader will return tensors of size [B] for the target and [B, context_size * 2] for the context.</p><p>The target tensor, with size [B], is processed using the following model:</p><pre>class SkipGramModel(nn.Module):<br>    def __init__(self,<br>                 vocab_size: int,<br>                 embedding_dim: int = 300,<br>                 hidden_dim: int = 2048,<br>                 padding_idx: Optional[int] = None,<br>                 embed_max_norm: float = 1,<br>                 ):<br>        super(SkipGramModel, self).__init__()<br>        self.embeddings = Embedding(<br>            num_embeddings=vocab_size,<br>            embedding_dim=embedding_dim,<br>            padding_idx=padding_idx,<br>            max_norm=embed_max_norm<br>        )<br>        self.linear = nn.Linear(<br>            in_features=embedding_dim,<br>            out_features=hidden_dim,<br>        )<br>        self.output = nn.Linear(<br>            in_features=hidden_dim,<br>            out_features=vocab_size,<br>        )<br><br>    def forward(self, inputs: Tensor):<br>        x = self.embeddings(inputs) # [B, E]<br>        x = self.linear(x) # [B, H]<br>        x = F.relu(x) # [B, H]<br>        x = self.output(x) # [B, V]<br>        return x</pre><p>After the forward pass, the model outputs logits with dimensions [B, V], ready to be passed through a softmax layer.</p><p>However, in our training loop, we encounter an issue: since the ground truth data (contexts) is in [B, context_size * 2], the problem shifts from multiclass classification to multiclass multilabel classification. This shift necessitates changing our loss function from nn.CrossEntropyLoss() to nn.BCEWithLogitsLoss().</p><p>This change also means we need to convert our context IDs into one-hot vectors to match the vocabulary size (V). Here’s how the conversion function might look:</p><pre>def convert_to_onehot(indices: Tensor):<br>    # Flatten the indices tensor to 1D<br>    indices_flat = indices.flatten()<br>    # Convert to one-hot encoding<br>    one_hot_flat = torch.nn.functional.one_hot(indices_flat, num_classes=VOCAB_SIZE)<br>    # Reshape back to [B, ctx*2, V]<br>    one_hot = one_hot_flat.reshape(BATCH_SIZE, -1, VOCAB_SIZE)<br>    # Reshape back to [B, V]<br>    one_hot = one_hot.sum(dim=1)<br>    one_hot = torch.clamp(one_hot, min=0, max=1).float()<br>    return one_hot</pre><p>Then, in the training loop, we should convert the context to one-hot encoding and compute the loss as follows:</p><pre>onehot_context = convert_to_onehot(context)<br>loss = criterion(y_pred, onehot_context) # nn.BCEWithLogitsLoss()</pre><h3>2) Skip-Gram Model with Hierarchical Softmax</h3><p>Hierarchical softmax, introduced by Morin and Bengio in 2005, serves as an efficient alternative to the traditional softmax function by employing a hierarchical layer derived from binary Huffman tree techniques. This approach reduces the complexity of probability calculations from the entire vocabulary V to log_2(V). The Huffman tree is a binary structure that organizes vocabulary words based on their frequency in the document.</p><p>For this discussion, we won’t dive into building a Huffman Tree from scratch. Instead, I recommend exploring resources such as the insightful blog post on <a href="http://neptune.ai/">neptune.ai</a> about word embeddings for implementation details.</p><p>In this section, I will follow a similar structure to the first part, discussing changes in the dataset, model adjustments, and loss calculation. Finally, I will share a simple evaluation based on word similarity in the representation space.</p><p>Like the Skip-Gram with softmax, acquiring the context and target remains unchanged. However, hierarchical softmax necessitates two additional pieces of data:</p><ol><li><strong>Paths of Contexts:</strong> These are sequences of node IDs (or token IDs) that lead to each context word node in the Huffman tree, essential for leveraging the hierarchical softmax’s reliance on dot product calculations between target word embeddings and node embeddings.</li><li><strong>Codes of Contexts:</strong> This binary list indicates the direction (left or right) at each node along the path, serving as the ground truth for our loss function, which will be binary cross entropy.</li></ol><p>It’s worth noting that with the tree structure in place, we can determine the longest path length and appropriately pad paths and codes. Paths are padded with the padding token index, while codes use 0 (the actual value is inconsequential as we apply masking to the computed loss).</p><pre>def __getitem__(self, idx):<br>    tokens: List[str] = self.instances[idx]<br>    ids = self.tokenizer.encode(tokens)<br>    context = ids[:self.context_size] + ids[self.context_size + 1:]<br>    target = ids[self.context_size]<br>    assert target != self.tokenizer.unk_token_idx, &quot;target is unk&quot;<br><br>    context_paths = self.huffman_tree.get_paths_for_tokens(context)<br>    context_codes = self.huffman_tree.get_codes_for_tokens(context)<br><br>    pad_idx = self.tokenizer.pad_token_idx<br>    context_paths = [path + (self.huffman_tree.longest_path_size - len(path)) * [pad_idx] for path in context_paths]<br>    context_codes = [code + (self.huffman_tree.longest_path_size - len(code)) * [0] for code in context_codes]<br><br>    return target, np.array(context), np.array(context_paths), np.array(context_codes)</pre><p>In terms of tensor dimensions within the dataset and dataloader, we observe the following structure:</p><ul><li>target: [B]</li><li>context: [B, context_size * 2]</li><li>paths: [B, context_size * 2, L]</li><li>codes: [B, context_size * 2, L]</li></ul><p>Ok, so how do we process them?</p><p>This question was the catalyst for writing this post, as I struggled to find an easy-to-follow resource on the topic.</p><p>Several modifications are needed for the SkipGramModel. First, the embeddings are used sparsely (refer to the provided links for an in-depth explanation). This sparse usage aligns well with our tree data structure, where not all nodes are used simultaneously. In Huffman trees, the most frequent words are located on the left and have the shortest paths, meaning the majority of the internal nodes are reserved for less frequent words.</p><p>Interestingly, the model doesn’t necessarily need to contain layers other than the embedding layer, which, while minimal, suffices for our purposes.</p><p>It’s also crucial to update the vocabulary size, accounting not only for the actual words (or subwords) but also for the non-leaf nodes in the Huffman tree, totaling (V — 1) nodes. Therefore, the embedding layer should accommodate (2 * vocab_size — 1) entries.</p><p>Let’s look into the forward method:</p><p>The inputs are the center word IDs and the paths for the contexts. Once we obtain the embeddings for these IDs, we calculate their dot product, excluding those with the padding index.</p><p>The outcome of this operation is the step scores, which represent the scores for each node. These scores are then transformed into probabilities by sigmoid, ranging from 0 to 1, indicating the model’s inclination towards a particular branch. Along with the probabilities, I return the mask, which will be utilized for loss calculation. You might prefer to regenerate this mask in the training loop or pass it directly to the forward method as an alternative.</p><pre>class SkipGramHSModel(nn.Module):<br>    def __init__(self,<br>                 vocab_size: int,<br>                 padding_idx: int,<br>                 embedding_dim: int = 300,<br>                 embed_max_norm: float = 1,<br>                 ):<br>        super(SkipGramHSModel, self).__init__()<br>        self.num_nodes_in_huffman_tree = 2 * vocab_size - 1<br>        self.padding_idx = padding_idx<br>        # for sparse embeddings =&gt; &lt;https://medium.com/nvidia-merlin/training-larger-and-faster-recommender-systems-with-pytorch-sparse-embeddings-53348a2cde3f&gt;<br>        # &lt;https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html&gt;<br>        self.embeddings = Embedding(<br>            num_embeddings=self.num_nodes_in_huffman_tree,<br>            embedding_dim=embedding_dim,<br>            padding_idx=padding_idx,<br>            max_norm=embed_max_norm,<br>            sparse=True<br>        )<br><br>    def forward(self,<br>                center_word_ids: Tensor,<br>                paths: Tensor) -&gt; Tuple[Tensor, Tensor]:<br>        &quot;&quot;&quot;<br>        # center_word_ids: batch of center words, shape [batch_size]<br>        # paths: batch of paths, shape [batch_size, context_size*2, max_path_length]<br>        # codes: batch of binary codes, shape [batch_size, context_size, max_path_length]<br><br>        Loss calculation<br>        # Compute binary cross-entropy loss<br>        loss = F.binary_cross_entropy(probs, codes.float(), weight=mask.float(), reduction=&#39;sum&#39;) / mask.sum()<br><br>        &quot;&quot;&quot;<br>        center_word_embeddings = self.embeddings(center_word_ids)  # [batch_size, embedding_dim]<br>        path_node_embeddings = self.embeddings(paths)  # [batch_size, ctx, max_path_length, embedding_dim]<br><br>        mask = paths.ne(self.padding_idx)  # [batch_size, ctx, max_path_length]<br><br>        # Compute dot products<br>        step_scores = torch.sum(path_node_embeddings * center_word_embeddings.view(center_word_ids.size(0), 1, 1, -1),<br>                                dim=3) * mask<br><br>        # Compute probabilities using sigmoid<br>        probs = torch.sigmoid(step_scores)  # [batch_size, ctx, max_path_length]<br><br>        return probs, mask</pre><p>Now that we have computed the probabilities for each context and their respective nodes in the batch [B, context_size*2, L], it’s time to define our loss function, which deviates slightly from the default behavior. Additionally, our choice of optimizer differs due to the use of sparse embeddings.</p><pre>from functools import partial<br>import torch.nn.functional as F<br>import torch.optim as optim<br><br>criterion = partial(F.binary_cross_entropy, reduction=&#39;sum&#39;)<br>optimizer = optim.SparseAdam(model.parameters(), lr=LEARNING_RATE)</pre><p>We employ binary cross-entropy loss with a ‘sum’ reduction to accommodate the presence of padding indices, which should be ignored. These unwanted indices are neutralized by assigning them a weight of 0 through the mask, preventing them from influencing gradient updates. To compute the mean loss, we use the sum of the weights of the active indices, facilitated by the mask.</p><pre>probs, mask = model(center_word, context_path)<br>loss = criterion(probs, context_code.float(), weight=mask.float()) / mask.sum()</pre><p>For a qualitative assessment, I used a small sample of English Wikipedia data from the Hugging Face datasets:</p><pre>from datasets import load_dataset<br><br>wikipedia_simple_subset = load_dataset(&quot;wikipedia&quot;,<br>                                       &quot;20220301.simple&quot;,<br>                                       split=&#39;train[:2%]&#39;,<br>                                       trust_remote_code=True)</pre><p>The training was configured with the following hyperparameters:</p><pre>NUM_EPOCHS = 3<br>BATCH_SIZE = 4096<br>LEARNING_RATE = 0.01<br>VOCAB_SIZE = 20000</pre><p>The training process took approximately 5-10 minutes. Afterward, I obtained the top-5 similar words for &quot;queen,&quot; yielding the following results:</p><pre>index = tokenizer.vocab[&#39;queen&#39;]<br>topk_indices, topk_values = get_topk_similar(similarity_matrix, index, k=5)<br><br>print(&quot;Top-k similar indices:&quot;, tokenizer.decode(topk_indices))<br>print(&quot;Similarity values:&quot;, topk_values)</pre><p>The results were:</p><pre>Top-k similar indices: [&#39;england&#39;, &#39;crown&#39;, &#39;france&#39;, &#39;elected&#39;, &#39;ethiopia&#39;]<br>Similarity values: tensor([0.9817, 0.9814, 0.9791, 0.9778, 0.9777], device=&#39;cuda:0&#39;)</pre><p>Which seems ok <a href="https://emojipedia.org/princess">👸</a> (of course more robust evaluation is needed — you can write your your results as comments :) and maybe compare it with the vanilla softmax )</p><p>Finally, to conclude, utilizing hierarchical softmax significantly enhances resource efficiency compared to vanilla softmax. This efficiency allows for larger batch sizes, facilitating experiments even on less powerful machines like laptops.</p><p>I hope this post was helpful. Thank you for reading!</p><h3>Sources</h3><p>- &lt;https://github.com/weberrr/pytorch_word2vec/blob/master/skip_gram_on_hierarchical_softmax/input_data.py&gt;<br>- &lt;https://neptune.ai/blog/word-embeddings-guide&gt;<br>- &lt;https://adoni.github.io/2017/11/08/word2vec-pytorch/&gt;<br>- &lt;https://colab.research.google.com/github/d2l-ai/d2l-en-colab/blob/master/chapter_natural-language-processing-pretraining/approx-training.ipynb&gt;<br>- &lt;https://d2l.ai/chapter_natural-language-processing-pretraining/approx-training.html&gt;<br>- &lt;https://youtu.be/pzyIWCelt_E?si=MZibI1riTDkWBU7b&gt; ( Q&amp;A - Hierarchical Softmax in word2vec )</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=bbd3363bcff9" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Intro. to Linux Command line & Shell Scripting]]></title>
            <link>https://medium.com/@gsoykan/intro-to-linux-command-line-shell-scripting-465a3c7c2601?source=rss-eb44eb87d7fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/465a3c7c2601</guid>
            <category><![CDATA[shellscripting]]></category>
            <category><![CDATA[shell]]></category>
            <category><![CDATA[linux-command-line]]></category>
            <category><![CDATA[shell-script]]></category>
            <category><![CDATA[command-line]]></category>
            <dc:creator><![CDATA[Gürkan Soykan]]></dc:creator>
            <pubDate>Tue, 20 Feb 2024 19:09:27 GMT</pubDate>
            <atom:updated>2024-02-20T19:09:27.334Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/0*T1164HvFvIyXc5ie" /><figcaption>Photo by <a href="https://unsplash.com/@6heinz3r?utm_source=medium&amp;utm_medium=referral">Gabriel Heinzer</a> on <a href="https://unsplash.com/?utm_source=medium&amp;utm_medium=referral">Unsplash</a></figcaption></figure><p>This semester (Spring ‘24), I will be serving as a Teaching Assistant for the NLP course (COMP 442/542) at Koç University. As part of the course, we will conduct lab sessions, starting with the basics of the Linux command line, shell, and PyTorch, before progressing to more advanced topics such as state-of-the-art Language Models, multilinguality, Prompt Engineering and Fine-Tuning Techniques, among others. I’ve decided to share my lab notes in the form of stories on Medium. This approach, hopefully, not only aids others but also serves as a personal reference for me and can be thought as a personal warm-up for “collaborative” or “social learning”. As you might expect, the first week features a gentle introduction to the Linux Command Line and Shell Scripting. By the end, I’ll present several examples from my own research that include practical shell and Python scripts 🍭🍬</p><h3>Linux Command line</h3><p>The Linux command line, known by various names such as the shell, terminal, console, or prompt, serves as a text-based interface to your computer. Despite its initial appearance of complexity and potential for confusion, it offers a really powerful means of interaction with your system. Basically, you can do a LOT with it :)</p><p><strong>Brief History</strong><br>Unix, one of the early operating systems, utilized a method of remote connection through “Terminals” to mainframe computers — a common practice in the good old days :) An advantage for text-based interaction was its low memory overhead; since text requires significantly less memory to transfer compared to images, making it an efficient choice for then. In the 1970s, users were capable of performing all computing tasks using solely this textual interface, indicating that many of your computer science heroes operated in a text-only environment. So, no need to panic when using the terminal today :)</p><p>Each task within this environment required a specific “command” or “program” to execute. The “shell” program functioned as the master program, facilitating user interaction with the operating system. To streamline and automate repetitive tasks, “shell scripts” were developed, allowing for the execution of long series of shell commands with a single instruction.</p><p>In the context of a modern Linux system, “bash” (Bourne Again SHell) is the predominantly used shell. It continues the tradition of providing a powerful, scriptable interface for users to perform a wide range of tasks. We will see some basic examples real quick…</p><h3>Let’s use the Terminal</h3><p>We should start with “pwd” and “cd” — you might have already used these, but let’s start simple and then we will gradually progress to slightly more complicated examples.</p><pre># Pwd stands for &#39;print working directory&#39;.<br># It shows the current directory you are in.<br>pwd<br># Output: /home/username (This will vary based on your current directory)</pre><p>You should also note that commands are often case sensitive, so if you had typed “PWD,” then you would not be able to see your working directory — you can try it and see the result for yourself :)</p><p>The other command is “cd,” which stands for “Change Directory.” To navigate to a different directory, use ‘cd’ followed by the directory name.</p><pre>cd /path/to/directory<br># No output is shown, but the current directory changes to /path/to/directory.<br># and prompt is ready for the next command :)</pre><p>When changing directories, be aware of relative and absolute paths. For instance, “/” refers to the root directory of the file system, and “~” refers to the home directory of the current user. Speaking of “root,” you should not confuse the root directory with “root” as the superuser. The root directory is the top-level directory in the filesystem, denoted by “/”. “Root” as the superuser is the administrative user with full access to the system.</p><p>Another basic command is “ls,” which stands for “list.” It lists directory contents — try it and see the outcome :)</p><h3>Creating Folders and Files</h3><p>Several commands make navigating and manipulating the filesystem intuitive and powerful. One such command is <strong>`mkdir`</strong>, short for ‘make directory’. This command is used to create new directories within the filesystem.</p><p>For example, <strong>`mkdir -p dir4/dir5/dir6`</strong> creates a new directory path, and the `-p` option (or switch) instructs the command to create the parent directories as well if they do not exist. <strong>Options</strong> like `-p` are used to modify how a command operates, offering versatility in their behavior.</p><p>When dealing with commands, the concept of “escaping” is crucial. Namely if you try to make a new file called “sample dir” you can’t use “mkdir sample dir” — in fact you can but the output might be unexpected :) Instead you can use “mkdir “sample dir”” or “rmdir sample\ dir/”. So basically the <strong>escaping</strong> is a computing term that refers to using special codes or syntax to tell the computer to treat particular characters differently than normal.</p><h3>The redirection operator `&gt;`</h3><p>is another powerful feature, used to direct the output of a command to a file or another command. For example,</p><pre>ls &gt; output.txt </pre><p>redirects the output of the ls command to a file named output.txt, effectively saving the list of files and directories to the file.</p><p>If you want to see the inputs of this “output.txt” then you can use</p><pre>cat output.txt<br># Desktop, Documents, Downloads...</pre><p>the `cat` command, short for concatenate, is a standard Unix utility used to read files sequentially, writing them to standard output. It’s often used to display the contents of files.</p><h3>Text Editors for Command line</h3><p>now that we started to talk about files we should mention text editors that you can use from command line.</p><p><strong>Vim (Vi Improved)</strong> stands out as a highly configurable, cross-platform text editor renowned for its efficiency. Originating as an extension of Vi, Vim has evolved to offer multiple modes for editing. Vim’s versatility makes it my favorite ¯\_(ツ)_/¯. Further resources and guides are available at <a href="https://web.stanford.edu/class/cs107/resources/vim">Stanford’s CS107 Vim Resources</a> and <a href="https://www.linuxfoundation.org/blog/blog/classic-sysadmin-vim-101-a-beginners-guide-to-vim">Linux Foundation’s Vim 101</a>.</p><p><strong>Emacs</strong> is another powerful text editor that is cross-platform and available in both command-line and graphical interfaces. Its standout feature is extensibility; Emacs can be customized and extended with various modules, transforming it far beyond a simple text editor.<a href="https://www.emacswiki.org/emacs/OnlineResources"> I mean really…</a></p><p><strong>Nano: </strong>For those seeking simplicity in text editing, Nano presents an accessible and straightforward option. It is really good at <a href="https://help.ubuntu.com/community/Nano">basic text editing tasks</a> without the steep learning curve of Vim or Emacs.</p><h3>Moving &amp; Manipulating Files</h3><p>Let’s continue from terminal and speed up a little :)</p><pre># Moves a file to a specified directory.<br>mv &lt;file&gt; &lt;dir&gt;<br># Creates a copy of a file.<br>cp &lt;file&gt; &lt;copy_of_file&gt;`<br># Permanently deletes a file.<br>rm &lt;file&gt;<br># Removes an empty directory.<br>rmdir &lt;folder&gt;<br># Recursively deletes a directory and all its contents.<br>rm -r &lt;folder&gt;</pre><p>Remember, `rm` deletes files permanently without moving them to a “trash” bin, making it crucial to use with caution, especially with wildcards or when using the `-r` option. For safer deletion, use `rm -i` to confirm each file’s deletion interactively 😉</p><h3><strong>Combinations</strong> 🦵💥🦶 👊</h3><p>Combo’s are possible with piping (|) which connects the output of one command directly to the input of another, operating entirely in memory for efficient data processing. Also, you can use less command for paginated viewing, and uniq filters out repeated lines. Below are possible with these and what we’ve learned so far</p><pre># Counts the number of lines in a file.<br>wc -l &lt;file&gt;<br><br># Counts the files in the home directory directly using a pipe, without creating a file.<br>ls ~ | wc -l<br><br># Displays the contents of the /etc directory one page at a time.<br>ls /etc | less<br><br># Counts the unique lines in a file.<br>cat &lt;file&gt; | uniq | wc -l<br><br># Sorts the contents of a file and displays it one page at a time.<br>sort &lt;file&gt; | less</pre><p>if you want to know more about a command you can always use the MANUAL :)</p><pre># Displays the manual page for a command.<br>man &lt;command&gt; # ls<br><br>### output<br>NAME<br>       ls - list directory contents<br><br>SYNOPSIS<br>       ls [OPTION]... [FILE]...<br><br>DESCRIPTION<br>       List  information  about ....<br>###</pre><h3>Superuser</h3><p>“superuser” in Unix, traditionally uses the username “root”. This, root, account has full system access, capable of executing any command. The su command, short for &#39;superuser&#39; or &#39;switch user&#39;, enables switching between users without logging out. Meanwhile, sudo (switch user and do) allows executing commands with superuser privileges.</p><h3>Some more tricks</h3><pre># Outputs the end of a file, useful for checking new data additions.<br>tail<br><br># Displays Linux processes in real-time.<br>top<br><br># An interactive monitor for system resources or processes, an improved version of top.<br>htop<br><br># Searches for a file named &quot;thisfile.txt&quot; in the current and sub-directories.<br>find . -name thisfile.txt<br><br># Searches for text patterns within files based on regular expressions.<br>grep<br><br># Searches for &quot;Gürkan&quot; in test.sh, ignoring case.<br>grep -i &quot;Gürkan&quot; test.sh<br><br># Lists filenames containing &quot;gsoykan&quot; in the current directory and subdirectories.<br>grep -l &quot;gsoykan&quot; *<br><br># Reverse-i search: searches through command history for a specific string.<br>ctrl+R</pre><h3>Shell Scripting Intro.</h3><p>At its most basic, a shell script is just a file filled with commands. When the shell reads this file, it follows the commands as if someone had typed them in manually. The shell is special because it acts as both a way to type commands into the computer and a program that can run these command scripts. As we’ll learn, almost anything you can do by typing commands can also be done in a script, and vice versa. The below samples will be coming from <a href="http://linuxcommand.org/tlcl.php">The Linux Command Line</a> by <strong><em>William Shotts, </em></strong>you can the book for more details and samples.</p><p>To create a shell script, follow these steps:</p><ol><li>Write your script.</li><li>Make your script executable.</li><li>Place your script in a location where the shell can access it.</li></ol><pre>#!/bin/bash<br># This is our first script. echo =&gt; is like print :)<br>echo &#39;Hello World!&#39;</pre><p>The “#!” shebang is a way to inform the kernel about which interpreter to use for executing the commands in the file.</p><h3>Executable Permissions</h3><p>Scripts typically have two main types of permission settings:</p><ul><li>755 for scripts that anyone can run.</li><li>700 for scripts that only the script&#39;s owner can run.</li></ul><p>Remember, for a script to be run, it must also be readable.</p><pre>ls -l hello_world<br># -rw-r--r-- 1 me me 63 2009-03-07 10:10 hello_world<br>chmod 755 hello_world<br>ls -l hello_world<br># -rwxr-xr-x 1 me me 63 2009-03-07 10:10 hello_world</pre><h3>Variables &amp; Constants</h3><pre>a=z # Assign the string &quot;z&quot; to variable a.<br>b=&quot;a string&quot; # Embedded spaces must be within quotes.<br>c=&quot;a string and $b&quot; # Other expansions such as variables can be<br># expanded into the assignment.<br>d=&quot;$(ls -l foo.txt)&quot; # Results of a command.<br>e=$((5 * 7)) # Arithmetic expansion.<br>f=&quot;\t\ta string\n&quot; # Escape sequences such as tabs and newlines.</pre><h3>Here Documents</h3><p>This technique is another type of I/O (input/output) (like echo but more) redirection that allows us to include a block of text directly in our script and then pass it as input to a command. You can use it in the following format — create a file like this and run it :)</p><pre>#!/bin/bash<br># Program to output a system information page<br># command &lt;&lt; token<br># text<br># token<br><br>TITLE=&quot;System Information Report For $HOSTNAME&quot;<br>CURRENT_TIME=&quot;$(date +&quot;%x %r %Z&quot;)&quot;<br>TIMESTAMP=&quot;Generated $CURRENT_TIME, by $USER&quot;<br>cat &lt;&lt; _EOF_<br>&lt;html&gt;<br>&lt;head&gt;<br>  &lt;title&gt;$TITLE&lt;/title&gt;<br>&lt;/head&gt;<br>&lt;body&gt;<br>  &lt;p&gt;$TIMESTAMP&lt;/p&gt;<br>&lt;/body&gt;<br>&lt;/html&gt;<br>_EOF_</pre><h3>Shell Functions</h3><p>Shell functions can be thought of as “mini-scripts” nested within larger scripts, functioning like independent programs. There are two ways to write shell functions:</p><pre>function name {<br>commands<br>return<br>}<br># Here is a simpler form:<br>name () {<br>commands<br>return<br>}</pre><h3>if statements</h3><pre><br># sytax <br>if commands; then<br>  commands<br>[elif commands; then<br>  commands...]<br>[else<br>  commands]<br>fi<br><br># sample script<br>x=5<br><br>if [ &quot;$x&quot; -eq 5 ]; then<br>  echo &quot;x equals 5.&quot;<br>else<br>  echo &quot;x does not equal 5.&quot;<br>fi</pre><h3>String Expressions</h3><figure><img alt="string expressions from table 27–2 (The Linux Commandline)" src="https://cdn-images-1.medium.com/max/733/1*2eBlA3aMGu6af5x4Jm8JDg.png" /><figcaption>string expressions from table 27–2 (The Linux Command Line)</figcaption></figure><h3>Integer Expressions</h3><figure><img alt="integer expressions from table 27–3 (The Linux Command Line)" src="https://cdn-images-1.medium.com/max/740/1*vmBQAGh7QRmoBs2Qm-Ev7Q.png" /><figcaption>integer expressions from table 27–3 (The Linux Command Line)</figcaption></figure><h3>Test Operators &amp; Compounds</h3><p>These can be bit confusing. I suggest you to check <a href="https://stackoverflow.com/a/39076393/8265079">this stackoverflow post</a> for clarification :)</p><h3>For Loops and Arrays</h3><p>I am adding a mini sample here (Sorting an Array) for you to remember the syntax — if you want to dive deeper you can always visit the reference book :)</p><pre>#!/bin/bash<br># array-sort: Sort an array<br><br>a=(f e d c b a)<br><br>echo &quot;Original array: ${a[@]}&quot;<br>a_sorted=($(for i in &quot;${a[@]}&quot;; do echo $i; done | sort))<br>echo &quot;Sorted array: ${a_sorted[@]}&quot;<br><br># the output should be:<br>#   Sorted array: a b c d e f</pre><h3>A Use Case</h3><p>Before we conclude let me share my use case and script. Recently, I’ve been working on multilingual Large Language Models (LLMs), and you can see that I use the BLOOM model as the base model in the example below. I fine-tuned the LLM with LoRA according to various configurations (referred to as peft_aliases). For each configuration that I fine-tuned, I attempted to generate responses to instructions (questions) in various languages (referred to as languages). However, doing this one by one would be very time-consuming and tedious, so I decided to automate the process with a script like the one below. At the end of the day, what I want to achieve is to call the get_peft_model_answer.py script with different settings based on the user.</p><pre>#!/bin/bash<br><br>current_user=$(whoami)<br>if [ &quot;$current_user&quot; = &quot;gsoykan1&quot; ]; then<br>    export PYTHONPATH=$PYTHONPATH:/gsoykan1/projects/multilingual-instruction-tuning/<br>elif [ &quot;$current_user&quot; = &quot;gsoykan2&quot; ]; then<br>export PYTHONPATH=$PYTHONPATH:/gsoykan2/projects/multilingual-instruction-tuning/<br>else<br>    echo &quot;Unknown user: $current_user&quot;<br>    exit 1<br>fi<br><br># Base directory paths<br>BASE_DIR=&quot;...&quot;<br>TEMPLATE_DIR=&quot;${BASE_DIR}/templates&quot;<br><br># Base model and other static configurations<br>BASE_MODEL=&quot;bigscience/bloom-7b1&quot;<br>BASE_MODEL_ALIAS=&quot;bloom-7b1&quot;<br>LOAD_8BIT=&quot;True&quot;<br>PROMPT_TEMPLATE_NAME=&quot;template&quot;<br>NUM_GPUS=1<br><br># Available PEFT aliases<br>peft_aliases=(&quot;family&quot; &quot;geo&quot; &quot;learned&quot; &quot;random&quot; &quot;typology&quot; &quot;semantic&quot; &quot;vi&quot;)<br><br># Available languages<br>languages=(&quot;ar&quot; &quot;en&quot; &quot;es&quot; &quot;ru&quot; &quot;tr&quot; &quot;vi&quot;)<br><br>for alias in &quot;${peft_aliases[@]}&quot;; do<br>  for language in &quot;${languages[@]}&quot;; do<br>        # Construct dynamic file paths<br>        PEFT_WEIGHTS_PATH=&quot;${BASE_DIR}/output/${BASE_MODEL_ALIAS}-ls_${alias}-lora&quot;<br>        QUESTION_FILE_PATH=&quot;${BASE_DIR}/data/q_${language}.jsonl&quot;<br>        ANSWER_FILE_PATH=&quot;${BASE_DIR}/output/a_${alias}_${language}.jsonl&quot;<br><br>        # Execute the Python script with dynamic and static arguments<br>        python get_peft_model_answer.py --base_model &quot;$BASE_MODEL&quot; \<br>                                        --load_8bit &quot;$LOAD_8BIT&quot; \<br>                                        --peft_weights &quot;$PEFT_WEIGHTS_PATH&quot; \<br>                                        --template_dir &quot;$TEMPLATE_DIR&quot; \<br>                                        --prompt_template_name &quot;$PROMPT_TEMPLATE_NAME&quot; \<br>                                        --question_file &quot;$QUESTION_FILE_PATH&quot; \<br>                                        --answer_file &quot;$ANSWER_FILE_PATH&quot; \<br>                                        --num_gpus &quot;$NUM_GPUS&quot;<br><br>        # Optional: Echo command or message to indicate progress<br>        echo &quot;Processed ${alias} model for ${language} language.&quot;<br>    done<br>done</pre><p><strong><em>Note for KU Students:</em></strong> You may need to use Koç University’s HPC Cluster, and this guide is aimed to help you navigate it with ease. You can find more details at <a href="https://ai.ku.edu.tr/kuacc-hpc-cluster/">https://ai.ku.edu.tr/kuacc-hpc-cluster/</a> about how to access and use it.</p><h3>Sources and Further Reading</h3><h4>Introduction to the Linux Command Line</h4><ul><li><strong>Ubuntu Tutorials — Command Line for Beginners</strong>: A comprehensive guide for those new to the Linux command line, covering basic commands and concepts. <a href="https://ubuntu.com/tutorials/command-line-for-beginners">Read more</a></li></ul><h4><a href="http://linuxcommand.org/tlcl.php">Deep Dive into Command Line Tools</a></h4><ul><li><strong>The Linux Command Line</strong> by William Shotts: An in-depth book offering insights into the Linux command line, from basic to advanced topics.</li></ul><h4>Comparison of Text Editors and Commands</h4><ul><li><strong>GeeksforGeeks — Difference Between VI Editor and CAT Command</strong>: An article comparing the functionalities of the VI editor with the CAT command, useful for understanding text editing in Linux. <a href="https://www.geeksforgeeks.org/difference-between-vi-editor-and-cat-command/">Read more</a></li></ul><h4>Understanding Shell Scripts</h4><ul><li><strong>Dev.to — What the Shebang Really Does and Why It’s So Important in Your Shell Scripts</strong>: An informative piece explaining the significance of the shebang line in shell scripts. <a href="https://dev.to/meleu/what-the-shebang-really-does-and-why-it-s-so-important-in-your-shell-scripts-2755">Read more</a></li></ul><h4>Overview of Command Line Text Editors</h4><ul><li><strong>It’s FOSS — Command Line Text Editors in Linux</strong>: A brief overview of various text editors available through the Linux command line, helping readers choose the right tool for their needs. <a href="https://itsfoss.com/command-line-text-editors-linux/">Read more</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=465a3c7c2601" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Turkish Coffee Cup Binary Classifier]]></title>
            <link>https://medium.com/naylalabs/turkish-coffee-cup-binary-classifier-90493de31c09?source=rss-eb44eb87d7fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/90493de31c09</guid>
            <category><![CDATA[coffee-cup-reading]]></category>
            <category><![CDATA[deep-learning]]></category>
            <category><![CDATA[streamlit]]></category>
            <category><![CDATA[tflite]]></category>
            <category><![CDATA[machine-learning]]></category>
            <dc:creator><![CDATA[Gürkan Soykan]]></dc:creator>
            <pubDate>Wed, 06 Apr 2022 14:26:38 GMT</pubDate>
            <atom:updated>2022-04-07T14:29:16.423Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*862EwQEND5e94vdEgMyiEw.png" /></figure><h4>End-to-End ML Model Training and Deployment to an Android App</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/180/1*Ka4lSO3tY-SXpy7GH0NFlw.png" /><figcaption>Orta Şeker — Logo</figcaption></figure><h3>Motivation &amp; Problem</h3><p>As a creative development agency, we deal with interesting projects of all kinds. Some of them are projects and business ideas brought to us by clients, and some are our own in-house projects.</p><p>One of our in-house projects is called Orta Şeker, which is a fortune-telling app that uses photos of Turkish coffee cups uploaded by the users to tell their fortune. Coffee reading (tasseomancy) is a fortune-telling technique in Turkish culture and mysticism, and it is still being practiced casually. In our project, we want to use a classifier to ensure that we only receive photos of real Turkish coffee cups and not anything else.</p><p><a href="https://play.google.com/store/apps/details?id=com.naylalabs.ortaseker&amp;hl=en&amp;gl=US">Orta Şeker — Gerçek Kahve Falı — Apps on Google Play</a></p><p>Although we check the images coming from our users before sending out fortune readings, it is a tedious job to do manually. So, we wanted it to be automated by a deep learning model!</p><p>Since our app has been out there for several years, we have more than enough images of Turkish coffee cups at our disposal. So all we need is to feed this data to a good and relatively small pretrained deep learning model and fine-tune it! After that, we can use the model on the device to predict whether our users are sending images enabling us to see their future 😇</p><p>Overall in this writing, we will see</p><ul><li>Dataset Creation</li><li>Model Implementation</li><li>Experiment results and interpretation</li><li>Deployment to Streamlit with basic front-end</li><li>Preparing our model to be used for our Android app</li></ul><h3>Dataset Creation</h3><p>We have been using Amazon S3 object storage to save our data on the cloud. First, we need to get some partition of it and then clean it somehow. Because it includes all sorts of images uploaded to our app.</p><h3>Getting “Turkish coffee cup” images from S3</h3><pre>aws configure --profile &lt;your-profile&gt;<br># AWS Config<br>acces key &lt;your-key&gt;<br>secret acces key &lt;your-key&gt;<br>region &lt;your-region&gt;<br>output json<br>cd &lt;your-location&gt;<br>aws s3 cp s3://&lt;s3-name&gt; . --recursive --page-size 10000 --profile &lt;your-profile&gt;</pre><p>We fetched approximately 6k images from our storage. However, as stated some of them needs to be removed. Below you can find some of those ‘noise’ images.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZZEwUAN7haNllYU1UKqPvA.png" /><figcaption>Noisy Images</figcaption></figure><p>We deleted 2k of them by hand. Also, we noticed that some of our users have been uploading their own photos, like selfies, which we needed to delete too. So at the end of this process, we are left with 4004 images 🙂</p><h3>Getting Non-Coffee Cup Images</h3><p>Since we are building a binary classifier we need the ‘non Turkish coffee cup’ class of images. Instances of this class can be anything but Turkish coffee cups. Since we have started with 4k of positive samples we need a dataset around that number. In the end, we decided to download <a href="https://cocodataset.org/#download">COCO 2017 validation dataset</a> having 5k images from 80 object classes. Since we want to have a balanced dataset we decided to use 4k of them.</p><p>Here is our train/val/test split counts: 6000, 1008, 1000</p><h3>Implementation</h3><p>Since we do not want to invent the wheel we are using <a href="https://github.com/ashleve/lightning-hydra-template">this amazing template</a> for our PyTorch projects. This is our base for this project. Since we want a small but effective pretrained model we decided to use <a href="https://ai.googleblog.com/2019/05/efficientnet-improving-accuracy-and.html">EfficientNet-b0</a> with a binary classification head.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/640/1*soSe3Shc3lO2rs7SEruCvw.png" /><figcaption>source: <a href="https://ai.googleblog.com/2019/05/efficientnet-improving-accuracy-and.html">https://ai.googleblog.com/2019/05/efficientnet-improving-accuracy-and.html</a></figcaption></figure><p>Before directly feeding images to the model we applied the below augmentations for training steps. For the augmentations, we chose our favorite package, <a href="https://albumentations.ai/">Albumentations</a>.</p><pre>N = 224<br>N_margin = 30<br>margined_n = N + N_margin<br>self.train_transforms = A.Compose([<br>            A.LongestMaxSize(max_size=margined_n, interpolation=1),<br>            A.PadIfNeeded(min_height=margined_n, min_width=margined_n, border_mode=0, value=(0, 0, 0)),<br>            A.RandomCrop(N, N),<br>            A.RandomRotate90(),<br>            A.Flip(),<br>            A.Transpose(),<br>            A.OneOf([<br>                A.GaussNoise(),<br>            ], p=0.2),<br>            A.OneOf([<br>                A.MotionBlur(p=.2),<br>                A.MedianBlur(blur_limit=3, p=0.1),<br>                A.Blur(blur_limit=3, p=0.1),<br>            ], p=0.2),<br>            A.ShiftScaleRotate(shift_limit=0.0625, scale_limit=0.2, rotate_limit=45, p=0.2),<br>            A.OneOf([<br>                A.OpticalDistortion(p=0.3),<br>                A.GridDistortion(p=.1),<br>                A.PiecewiseAffine(p=0.3),<br>            ], p=0.2),<br>            A.OneOf([<br>                A.CLAHE(clip_limit=2),<br>                A.Sharpen(),<br>                A.Emboss(),<br>                A.RandomBrightnessContrast(),<br>            ], p=0.3),<br>            A.HueSaturationValue(p=0.3),<br>            A.Normalize(<br>                mean=[0.485, 0.456, 0.406],<br>                std=[0.229, 0.224, 0.225],<br>            ),<br>            ToTensorV2()<br>        ])</pre><p>Since this is a classification task Cross-Entropy Loss is being used.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/700/0*qc-f30K3Kz8BHX1Z.png" /><figcaption>source: <a href="https://towardsdatascience.com/cross-entropy-loss-function-f38c4ec8643e">https://towardsdatascience.com/cross-entropy-loss-function-f38c4ec8643e</a></figcaption></figure><h3>Experiments Results &amp; Interpretation</h3><p>Training lasted for 4 epochs because we have early stopping if val accuracy does not improve for 2 epochs. Below you can find loss and accuracy values for training and validation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Xn8oyyVK7bpvMy6IARp1GQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UL70acnduOKn181Ep6UfNA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZOvJD8BxKDbUizCj6jbTcA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0rzfIGngasJsVuWZcROU2A.png" /></figure><p>With our best model in terms of accuracy, we achieved loss: 0.01319 and ACC: 0.996</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7XdLlR4EEMEYDg1K6HJYAw.jpeg" /><figcaption>Some samples with their ground truth labels and predictions.</figcaption></figure><p>We used <a href="https://wandb.ai/site">‘Weights and Biases’</a> as our main logger and all above graphs and visuals were logged automagically with W&amp;B.</p><p>In the end, we ended up training a highly successful model in both validation and test set with several thousand images for this task in just a few epochs of fine-tuning.</p><h3>Deployment</h3><p>We use <a href="https://docs.streamlit.io/streamlit-cloud/get-started/deploy-an-app">Streamlit for deployment</a> and followed below for creating a simple UI for our model.</p><p><a href="https://discuss.streamlit.io/t/how-to-build-interactive-image-classification-app-using-streamlit/6031">How to build Interactive Image Classification App using Streamlit</a></p><p>We already had a Streamlit account but if you don’t registration might take a couple of days to be approved.</p><p>Finally, we deployed our app to <a href="https://share.streamlit.io/gsoykan/coffee-cup-binary-classifier/main">this</a>!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*q_JW6qKgdfvCuuT5CCXunQ.png" /><figcaption>Our interface for testing and sharing with friends 😊</figcaption></figure><h3>Using in Our Android App</h3><p>Now that we have our PyTorch model in place that works really well the next step is to use it with our own app for a real use case.</p><p>However, we can’t deploy our model to the app directly because it is a native android app. Although there are several options for us to deploy our model we choose to do it with Firebase.</p><p><a href="https://firebase.google.com/docs/ml/android/use-custom-models">Use a custom TensorFlow Lite model on Android | Firebase Documentation</a></p><p>Because we do not want to increase our app size and want to update the model without any need to release a new update for the app itself.</p><p>Ok, It is all good but we have a problem!</p><p>Firebase provides support for TFLite and Keras models.</p><p>So we have to convert our model to TFLite model somehow.</p><p>This requires the followings steps:</p><ol><li>We need to convert the PyTorch model to ONNX (Open Neural Network Exchange Format) which is a format designed to represent any type of Machine Learning and Deep Learning model.</li><li>Conversion from ONNX format to TF format.</li><li>Conversion from TF format to TF Lite format so that we can run our model on the device.</li></ol><p>We followed instructions from <a href="https://github.com/sithu31296/PyTorch-ONNX-TFLite">this</a> repository.</p><p>All of those steps take some time so we needed to be patient 😇</p><p>We had a single problem just before the first step.</p><p>That was related to our model having a type of layer, Swish Layer, not compatible with ONNX format.</p><p>But hopefully, it was solved by a feature that model owners provided.</p><pre># required for conversion: &lt;https://github.com/lukemelas/EfficientNet-PyTorch/issues/91&gt;<br>model.set_swish(memory_efficient=False)</pre><p>Now that the conversion is complete we need to upload it to Firebase!</p><p>Detailed instructions for this can be found <a href="https://firebase.google.com/docs/ml/manage-hosted-models">here</a> and here is the whole script for doing that!</p><pre>import firebase_admin<br>from firebase_admin import ml<br>from firebase_admin import credentials</pre><pre># initialize fb admin with bucket<br>firebase_admin.initialize_app(<br>  credentials.Certificate(service_account_path),<br>  options={<br>      &#39;storageBucket&#39;: storage_bucket,<br>  })</pre><pre># Load a tflite file and upload it to Cloud Storage<br>source = ml.TFLiteGCSModelSource.from_tflite_model_file(TF_LITE_MODEL_PATH)</pre><pre># Create the model object<br>tflite_format = ml.TFLiteFormat(model_source=source)<br>model = ml.Model(<br>    display_name=&quot;turkish_coffee_cup_classifier&quot;,  # This is the name you use from your app to load the model.<br>    tags=[&quot;turkish_coffee_cup_classifier&quot;, &#39;classification&#39;, &#39;ortaseker&#39;],             # Optional tags for easier management.<br>    model_format=tflite_format)</pre><pre># Add the model to your Firebase project and publish it<br>new_model = ml.create_model(model)<br>ml.publish_model(new_model.model_id)</pre><p>Now we can check whether it is uploaded or not from Firebase storage.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/940/1*VtOEpQgtvOBe28jgkRE11A.jpeg" /><figcaption>Firebase Storage Interface to check our TFLite Model</figcaption></figure><p>The final step is to use the TFLite model in our app.</p><p>This <a href="https://firebase.google.com/docs/ml/android/use-custom-models#kotlin+ktx_1">link</a> has the steps to fetch the model to the device. However, one needs to be very careful when it comes to handling inputs for ML models in general and our case is no exemption.</p><p>Our model needs images to be at 224x224 and they should be normalized with mean 0.485, 0.456, 0.406 and standard deviation 0.229, 0.224, 0.225</p><p>The following sources are helpful for that!</p><p><a href="https://www.tensorflow.org/lite/inference_with_metadata/lite_support#imageprocessor_architecture">Process input and output data with the TensorFlow Lite Support Library</a></p><p><a href="https://www.tensorflow.org/lite/api_docs/java/org/tensorflow/lite/support/common/ops/NormalizeOp">NormalizeOp | TensorFlow Lite</a></p><h3>Conclusion</h3><p>In this post, we have trained a model for our binary classification purposes and then played with it through Streamlit. After we were convinced that it is good enough to use in our app we explored ways to deploy to our own Android mobile application and did that by Firebase ML!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=90493de31c09" width="1" height="1" alt=""><hr><p><a href="https://medium.com/naylalabs/turkish-coffee-cup-binary-classifier-90493de31c09">Turkish Coffee Cup Binary Classifier</a> was originally published in <a href="https://medium.com/naylalabs">Naylalabs</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Crush Privacy Policy]]></title>
            <link>https://medium.com/@gsoykan/crush-privacy-policy-2c2571d7dc8d?source=rss-eb44eb87d7fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/2c2571d7dc8d</guid>
            <dc:creator><![CDATA[Gürkan Soykan]]></dc:creator>
            <pubDate>Thu, 28 May 2020 18:35:30 GMT</pubDate>
            <atom:updated>2026-05-24T19:52:12.367Z</atom:updated>
            <content:encoded><![CDATA[<h3>Crush’d Privacy Policy and Terms &amp; Conditions</h3><p>Gurkan Soykan built the Crush’d app as a Free app. This SERVICE is provided by Gurkan Soykan at no cost and is intended for use as is. However, if you would like to enhance your experience you might like to buy some “arrow packs” to create more opportunities to meet with your crush!</p><p>This page is used to inform visitors regarding my policies with the collection, use, and disclosure of Personal Information if anyone decided to use my Service.</p><p>If you choose to use my Service, then you agree to the collection and use of information in relation to this policy. The Personal Information that I collect is used for providing and improving the Service. I will not use or share your information with anyone except as described in this Privacy Policy.</p><p>The terms used in this Privacy Policy have the same meanings as in our Terms and Conditions, which is accessible at Crush’d unless otherwise defined in this Privacy Policy.</p><p>**Information Collection and Use**</p><p>For a better experience, while using our Service, I may require you to provide us with certain personally identifiable information. The information that I request will be retained on your device and is not collected by me in any way.</p><p>The app does use third party services that may collect information used to identify you.</p><p>Link to privacy policy of third party service providers used by the app</p><p>* [Google Analytics for Firebase](https://firebase.google.com/policies/analytics)</p><p>* [Firebase Crashlytics](https://firebase.google.com/support/privacy/)</p><p>**Log Data**</p><p>I want to inform you that whenever you use my Service, in a case of an error in the app I collect data and information (through third party products) on your phone called Log Data. This Log Data may include information such as your device Internet Protocol (“IP”) address, device name, operating system version, the configuration of the app when utilizing my Service, the time and date of your use of the Service, and other statistics.</p><p>**Cookies**</p><p>Cookies are files with a small amount of data that are commonly used as anonymous unique identifiers. These are sent to your browser from the websites that you visit and are stored on your device’s internal memory.</p><p>This Service does not use these “cookies” explicitly. However, the app may use third party code and libraries that use “cookies” to collect information and improve their services. You have the option to either accept or refuse these cookies and know when a cookie is being sent to your device. If you choose to refuse our cookies, you may not be able to use some portions of this Service.</p><p>**Service Providers**</p><p>I may employ third-party companies and individuals due to the following reasons:</p><p>* To facilitate our Service;</p><p>* To provide the Service on our behalf;</p><p>* To perform Service-related services; or</p><p>* To assist us in analyzing how our Service is used.</p><p>I want to inform users of this Service that these third parties have access to your Personal Information. The reason is to perform the tasks assigned to them on our behalf. However, they are obligated not to disclose or use the information for any other purpose.</p><p>**Security**</p><p>I value your trust in providing us your Personal Information, thus we are striving to use commercially acceptable means of protecting it. But remember that no method of transmission over the internet, or method of electronic storage is 100% secure and reliable, and I cannot guarantee its absolute security.</p><p>**Links to Other Sites**</p><p>This Service may contain links to other sites. If you click on a third-party link, you will be directed to that site. Note that these external sites are not operated by me. Therefore, I strongly advise you to review the Privacy Policy of these websites. I have no control over and assume no responsibility for the content, privacy policies, or practices of any third-party sites or services.</p><p>**Eligible Users**</p><p>You must be 1) at least 18 years of age; or 2) at least 13 years of age and have parent or legal guardian consent and supervision to use this Application. By using this Application you are agreeing to the terms and conditions of the Agreement and warrant and represent that You are at least 18* years of age or are at least 13 and have consent from and are under supervision in use of a parent or legal guardian. You may not use the Application 1) if You do not agree to the terms and conditions of the Agreement; or 2) if you are not at least 18* or at least 13 years of age and have parent or legal guardian consent and supervision. The Application content is NOT appropriate for anyone under 13. The Application’s intended use is for anyone interested in dating with strangers. By using the Application, You also represent and warrant that You have the right, authority and capacity to enter into this Agreement. Be advised that persons under that Age of Majority in the State they reside or place of residence typically cannot enter into legal agreements. The Application is available to adults in the United States and all other countries. Any misstatements and/or misrepresentations regarding the age, background, felonies, criminal offenses, experience and/or eligibility for employment in the jurisdiction of the United States or any other country of any user of this Application are not the responsibility of Crush’d.</p><p>**Children’s Privacy**</p><p>These Services do not address anyone under the age of 13. I do not knowingly collect personally identifiable information from children under 13\. In the case I discover that a child under 13 has provided me with personal information, I immediately delete this from our servers. If you are a parent or guardian and you are aware that your child has provided us with personal information, please contact me so that I will be able to do necessary actions.</p><p><strong>** </strong>Reporting / Blocking <strong>**</strong><br>If you encounter any violation of Crush’d Terms &amp; Conditions and/or abusive and hateful content in a crush record or request, you may report it by reporting functionality in the App. Reasons to report user behavior may include (but are not limited to) Reasons to report user behavior include (but are not limited to):</p><ul><li>Harassment or bullying</li><li>Hate speech</li><li>Sexual exploitation</li><li>Child endangerment</li><li>Child Sexual Abuse Material (CSAM)</li><li>Grooming behavior</li><li>Violent or abusive conduct</li><li>Spam or fraudulent activity</li></ul><p>Reported inappropriate content will be reviewed and removed when possible, and actions may be taken against the offending user. If you do not wish to see new records from another user you can simply block that user via their records or requests.</p><p>**Changes to This Privacy Policy**</p><p>I may update our Privacy Policy from time to time. Thus, you are advised to review this page periodically for any changes. I will notify you of any changes by posting the new Privacy Policy on this page.</p><p>This policy is effective as of 2020–05–28</p><h3>Child Safety Standards and CSAE Policy</h3><p>Crush’d by Basement has zero tolerance for any form of Child Sexual Abuse and Exploitation (CSAE), Child Sexual Abuse Material (CSAM), grooming, sexualization of minors, or any exploitative behavior involving children.</p><p>Users are strictly prohibited from:</p><ul><li>Sharing, creating, uploading, or distributing Child Sexual Abuse Material (CSAM)</li><li>Attempting to exploit or groom minors</li><li>Soliciting sexual content involving minors</li><li>Using the platform in any way that endangers children</li></ul><p>Any content or behavior suspected to involve CSAE/CSAM will result in immediate action, including:</p><ul><li>Content removal</li><li>Permanent account suspension or banning</li><li>Reporting to appropriate law enforcement authorities and child protection organizations where required</li></ul><p>Crush’d by Basement is committed to complying with all applicable child safety laws and regulations and cooperates with relevant legal authorities regarding child safety matters.</p><p>If you believe a user or piece of content violates these standards, please report it immediately through the in-app reporting tools or contact us directly.</p><p>Child Safety Contact:<br><a href="/service/mailto:gsoykan@sabanciuniv.edu">gsoykan@sabanciuniv.edu</a></p><p>**Contact Us**</p><p>If you have any questions or suggestions about my Privacy Policy, do not hesitate to contact me at gsoykan@sabanciuniv.edu.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=2c2571d7dc8d" width="1" height="1" alt="">]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[To Code or Not to Code?]]></title>
            <link>https://medium.com/re-coded/to-code-or-not-to-code-5fe62b45ff72?source=rss-eb44eb87d7fc------2</link>
            <guid isPermaLink="false">https://medium.com/p/5fe62b45ff72</guid>
            <category><![CDATA[bootcamp]]></category>
            <category><![CDATA[code]]></category>
            <category><![CDATA[programming]]></category>
            <category><![CDATA[tech-for-good]]></category>
            <category><![CDATA[mobile-app-development]]></category>
            <dc:creator><![CDATA[Gürkan Soykan]]></dc:creator>
            <pubDate>Sat, 12 May 2018 12:01:00 GMT</pubDate>
            <atom:updated>2018-05-12T12:01:00.917Z</atom:updated>
            <content:encoded><![CDATA[<p>Hi my name is Gürkan, and I’m a Re:Coded Fellow in Istanbul. I’m almost at the end of my undergraduate study and feeling excited to move forward, as always. I have been living in Istanbul for almost 6 years as of now and learned many many things. Initially, I chose to be here to study in the university, before that I lived in Ankara, Düzce, Samsun and so on. But one thing is crucial here and that is there is no end of opportunities here. We can elaborate that further but that is another issue by itself :) My current and biggest example for that is Re:Coded.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/590/1*j7ZMxfiegNeM9l-U-udu6Q.jpeg" /></figure><p>Before diving into what is Re:Coded and what we are doing within the scope of it, I would like to share how I ended up here. Not long ago I was studying Materials Science and Nano-Engineering in Sabanci University. But at some point I heard a voice in my head. It was inviting me to learn code and develop software. In retrospect, I think that was because of the following reasons:</p><ul><li>Learning and writing code requires problem solving. I like to solve problems on any scale. Because one has to dissect every bit of what s/he has in mind and even think for future, let’s say for code readability and other purposes.</li><li>I believe coding is a very creative process. Although, there are best practices to write code, there is no one “true” way to do it.</li><li>Product development is extremely fast compared to other areas of expertise. One can simply develop an MVP in a relatively short time period.</li><li>There is also somewhat ethical side to code which is helping others to build or develop their ideas. That can be coding for open source projects, helping NGO’s and many things. Because, you can do without physical barriers by simply be willing and competent enough.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/400/1*nvu0nUirPRL90MqkFLBi1w.png" /></figure><p>Although coding has been rewarding, it hasn’t been easy for me — it still is a challenge to this day. Because, the very things that make coding enjoyable are the things that make it hard and frustrating.</p><ul><li>Imagine you are dealing with a complex problem, and you are aware of the fact that what you are doing now might be actually not working at the end of the day. So you need clarity of what you are dealing with. That might be easy once you are an expert. However, if you are a junior or beginner that might be extremely stressful and paralyze you. I’ve experienced this for years.</li><li>Since it is creative process and there is no “true” or “rightful way” you need to come up with your own solutions. I believe this could make you feel like an artist once you got necessary skill set but in the beginning I found myself looking at the blank screen without typing anything. It was like a blank canvas waiting to be filled.</li><li>Product development is fast and you give yourself let’s say 3 months to finish a project. On the 6th month if that is not finished that could lead you to the quarter life crisis.</li></ul><p>Now we admit that this process is not easy, there are challenges.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/970/1*N_1xWc_q9w-rqBeOH-I5qw.jpeg" /></figure><p>But still need to repeat these issues because it is sort of double edged and what creates the difference is our attitude.</p><p>Overall, it’s been a rewarding journey for myself and others around me by being part of the change I want.</p><p>Being a fellow in Re:Coded has been a milestone for me. You can check their website for the details but I’d like to give “insider’s knowledge.”</p><p>At Re:Coded, we’re learning from the best. Our instructors are extremely talented and smart people. I could say that the worst answer I got from them was something like, when I am asking the question, their answer is more than enough to satisfy my curiosity and they say in the end “But I can not give a lecture about that specific topic”. The takeaway is that they can give lectures on more topics than we have time for. Apart from the lectures, having Udacity by our side is priceless. Because of the projects that we are dealing with we have the opportunity to hone our skills and what we are learning in the lectures.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Q5HVB6DgO1ULiSo-O-Bl9w.png" /></figure><p>The coordinators are extremely helpful and it can be said that they are doing their bests for all 34 of us. As a bonus we have monthly speaker sessions. The last guest was Ömer Erkmen. During the session he gave us many insights about the startups and technology. He answered our questions as well and I believe we have our own worries about the future. Most of us have different backgrounds and yet we still strive to learn android development. Ömer Abi shared his wisdom with us. As for me, I have various different paths for the future ( can not ignore possibilities), each of them independent from the other. I am still in the process of deciding what to do next, I guess that will always be THE question :). Anyways, currently, I am struggling to finish my undergraduate study and spend most of my time in Re:Coded. However, there is also one iOS application that I am working on, Gokceada, you can find it on App Store. My plan for the following year is to be able to do freelance projects and gain insights on startups so that eventually I would be able to start my own company.</p><p>I hope that our paths can cross again.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=5fe62b45ff72" width="1" height="1" alt=""><hr><p><a href="https://medium.com/re-coded/to-code-or-not-to-code-5fe62b45ff72">To Code or Not to Code?</a> was originally published in <a href="https://medium.com/re-coded">Re:Coded</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>