<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Vladimir Palmin on Medium]]></title>
        <description><![CDATA[Stories by Vladimir Palmin on Medium]]></description>
        <link>https://medium.com/@palminv?source=rss-c54ca82a2ab8------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/0*TjvoytYeHu4OmCRf.jpg</url>
            <title>Stories by Vladimir Palmin on Medium</title>
            <link>https://medium.com/@palminv?source=rss-c54ca82a2ab8------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 11:33:29 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@palminv/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[How to create not another Data Analysis course]]></title>
            <link>https://medium.com/@palminv/how-to-create-not-another-data-analysis-course-296dfe791733?source=rss-c54ca82a2ab8------2</link>
            <guid isPermaLink="false">https://medium.com/p/296dfe791733</guid>
            <category><![CDATA[statistics]]></category>
            <category><![CDATA[data-analysis]]></category>
            <category><![CDATA[education]]></category>
            <dc:creator><![CDATA[Vladimir Palmin]]></dc:creator>
            <pubDate>Mon, 04 Jul 2022 19:18:46 GMT</pubDate>
            <atom:updated>2022-07-04T19:25:34.768Z</atom:updated>
            <content:encoded><![CDATA[<p>There are a lot of trash courses. Some of them even want you to pay for them. Let’s try to figure out what a good course could be.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/492/1*JvFK7y1kQyFZ3P2Qd4R0Ew.jpeg" /><figcaption>source: <a href="https://web.stat.tamu.edu/~scottcrawford/teachingmemes.html">web.stat.tamu.edu/~scottcrawford/teachingmemes.html</a></figcaption></figure><h3>Goals</h3><p>There are numerous courses about data analysis, data science, and other data-related topics. Some of them are enjoyable. But there are many garbage courses that teach students poorly if their goal is to master the art of data analysis.</p><p>The reasons for such trash courses come from opposite fields. Mathematicians prefer to focus on building algebras; they despise forgetting some necessary property without which a theorem cannot be proved. And it is really cool if you want to be a mathematician. However, it is not very important for real-world applications even if one creates a new method to provide a new type of analysis. On the other hand, some courses tend to consider only specific methods. As a result, students may know 100 different ways to estimate some sh*t, but they may not understand how those methods relate to one another or what main ideas they contain. In fact, such “real-world-application” courses are more theoretical than practical.</p><p>Thus, to determine the ideal course, we need to understand how to combine theoretical and practical aspects. We need to provide deep knowledge of the basic ideas to create intuition that will help to understand specific methods. We also need to consider some methods by describing how one can obtain them from the basic principles as well as how and when to use them. It might help to create a comprehension so a student may explore a new field without too much trouble. Understanding how methods are realized as algorithms is also beneficial. Some moments also need to be considered, otherwise students will see the world as a black box. However, it is rather related to a numerical methods course.</p><p>Of course, we can try to create an advanced course that requires students to have prior deep knowledge. However, it is more important to build a good basic course. It does not mean that good advanced courses are not required. It means that good advanced courses are not required without good basic ones because a deep understanding of the fundamentals makes it much easier to delve deeper into advanced methods. So, I can describe the students of our course as someone who may already have some experience in data analysis or at least have passed an aprobability theory course and who wants to gain the necessary knowledge to progress further along the path of the data analysis master.</p><p>In the following sections, I will describe how we attempted to achieve this goal in our course.</p><h3>Syllabus</h3><p>Our course is in two parts. The first one aims to provide the required knowledge. It will be discussed in these three sections. The second part is dedicated to projects and described in the last section. The idea is that students obtain the necessary knowledge in the first part and then practice it in the second one to gain experience. Each part lasts one semester.</p><p>A typical course includes lectures, seminars, and homework. And that homework plays a key role. A student is likely to feel that the material is clear after a lecture. However, when this student faces problems, he will realize that there are still many questions to be answered. In addition, when planning a course, keep in mind that there are numerous perfect and open lectures and articles. As a result, live communication with students and the structure that lectures form are more important than lectures themselves. So, the lectures might be seen more as a tool that helps get into a topic. It is essential to have strong oral skills in order to provide excellent lectures. And there are a lot of books on this subject. But what is said in lectures is also important. The following is an example of a syllabus:</p><ol><li><em>Statistical Decision Theory and Probability Theory Fundamentals</em>. We recall some basic concepts, like the definitions of probability and how they are connected. The statistical decision theory is then iteratively developed by solving various decision-making problems. We consider this lecture fundamental, so we refer to it throughout the course.</li><li><em>Distributions’ Properties</em>. Since data distribution is often assumed to make a method work correctly and efficiently, it is critical to understand the properties of the most common distributions as well as in which cases to use them. We also discuss convolution, an important operator for distributions, and explain why it is hard to evaluate efficiently and how to solve it.</li><li><em>Parameter Estimates</em>. We use the maximum likelihood estimator to develop the chi2 and least squares methods to illustrate their assumptions. We also pay attention to the properties of estimators, especially their efficacy connected to the Rao-Cramer bound. We also describe the link between point and interval estimations.</li><li><em>Errors</em>. A good course should have an “exotic” topic. Since we have experience in the analysis and design of a fundamental physics experiment, we describe approaches to evaluating errors, their types, and how to deal with errors. Although I called it “exotic”, error analysis is essential for estimating method limits or understanding analysis results. However, only few courses cover this topic. That is why it is referred to as “exotic”.</li><li><em>Statistical Hypothesis Testing</em>. There are numerous methods to provide testing for a specific problem. However, we focus on fundamental concepts that are shared by most methods. We also connect this topic to the previous ones.</li><li><em>Optimization. </em>Optimization is obviously a vast subject. We provide it in the form of an overview, taking into account differences and similarities, as well as advantages and disadvantages.</li></ol><p>There may also be minor changes in the syllabus due to students’ interests. As our experience has shown, this syllabus is sufficiently balanced for students to gain sufficient knowledge and skills to self-educate in the following required subjects. Some more advanced topics can be studied as a part of a project during the second part. Furthermore, adding any special material will add skew to one field or another or will make this part overloaded.</p><h3>Homework</h3><p>The next difficult and crucial task is to create homework. One first has to understand what skills are required to train. I classified possible tasks into three groups:</p><ul><li><strong><em>Statistics at home</em></strong>. Every day, people process massive amounts of data. However, it is common for people to make similar mistakes during this process. The reason for this is a lack of probability intuition. So it’s nice to have some tasks that at least demonstrate this issue. There are some excellent books from which one can get inspired to create such problems. Examples are “<a href="https://www.amazon.co.uk/Thinking-Fast-Slow-Daniel-Kahneman/dp/0141033576">Thinking, Fast and Slow</a>” and “<a href="https://www.amazon.co.uk/How-Lie-Statistics-Penguin-Business/dp/0140136290">How to Lie with Statistics</a>”. <em>An </em>e<em>xample of tasks</em>:</li></ul><blockquote>The prosecutor indicated that O. J. Simpson had beaten his wife. The lawyer replied: “Only one in 2,500 women who have experienced domestic violence is killed, so it’s not relevant.” The court agreed with counsel; is this argument correct?</blockquote><ul><li><strong><em>Basics of Statistics</em></strong>. Many people who have taken statistics courses still don’t understand the basics, such as working with distributions, evaluating errors, or using ready-made solutions. From common sense, two ideas about how to solve this issue can be seen. Such tasks shouldn’t be too simple or similar because they are boring and do not help with learning. Otherwise, they shouldn’t be overly complicated. As a result, ideal tasks are middle-level problems related to issues people often have difficulty. We used our previous experience to accomplish such tasks. Good interview questions from top companies can also be considered. <em>An </em>e<em>xample of tasks:</em></li></ul><blockquote>You flip a fair coin 576 times. Without a calculator, estimate the probability of getting at least 312 heads.</blockquote><ul><li><strong><em>Real Problems</em></strong>. Real problems differ from others in that they use real-world data and are harder to solve. In fact, these problems are projects that have been solved. We used data from various fields. Some come from international experiments, others from cryptocurrency exchanges or atmospheric physics. <em>An </em>e<em>xample of tasks:</em></li></ul><blockquote>Passing through matter, the proton leaves ionization losses, which have some random character. Use simulations of these losses and the recorded proton energy loss to determine the detector error. Link to the required data.</blockquote><p>Our homework contains six assignments related to the sybbalus. Each assignment has two parts. “Questions” does not require students to calculate something, only to use logic and knowledge to answer questions. And “Exersises” where students have to implement and use methods.</p><p>Students also have the option of doing only a portion of the tasks. As a result, students can choose what is more satisfying or impressive to them. However, the number of people who can complete this single task is limited. It fosters competition among students and prevents them from doing only the same problems.</p><h3>Duplicate Search Program</h3><p>Some students prefer to copy the work of other students. In fact, it is a big problem. The plagiralis problem among students has different forms. Some issues related to this are described in the following articles: <a href="https://edintegrity.biomedcentral.com/articles/10.1007/s40979-017-0021-6">Detecting contract cheating in essay and report submissions: process, patterns, clues and conversations</a>; <a href="https://edintegrity.biomedcentral.com/articles/10.1007/s40979-022-00104-1">Using digital forensics in higher education to detect academic misconduct</a>. We consider the case when a student tries to submit not their own work but that of other students. And I should note that such programs can only assist a teacher, not make their own decisions. And still, some ethical issues may arise, like the one described in the arctile “<a href="https://www.nytimes.com/2022/05/27/technology/college-students-cheating-software-honorlock.html">Accused of Cheating by an Algorithm, and a Professor She Had Never Met</a>”.</p><p>There are several solutions. But most of them are paid and compare the file with their huge databases. It is convenient for essays but not for the homework of a statistics course that is submitted as a Jupyter notebook. And our needs require a comparison with other students’ work. In other words, we need our own database. So we created a simple find-clones program to automate the checking process. The code is open and provided on <a href="https://github.com/VladimirPalmin/DuplicateSearch">Github</a>.</p><p>To develop an algorithm, we first have to determine the type of copying we want to detect. We want to find a completely copied section with minor changes because it indicates that the student did nothing. Hence, the algorithm does not really understand the work it is analyzing. Student work is a Jupyter notebook, so it contains both text and code. We will consider them separately.</p><p>The steps in the duplicate-search algorithm in texts are as follows:</p><ol><li>Data cleaning entails deleting characters, changing case, and so on</li><li>Tokenization is a text-splitting technique (shingles)</li><li>Stop Words Removal: removing commonly used words</li><li>Stemming is the process of reducing words to their root forms (stems)</li><li>Statistical analysis to detect duplicates</li></ol><p>Hence, we can use only language rules. So, we used the <a href="https://www.nltk.org/">NLTK library</a>, which has stop words and stemming for various languages.</p><p>Duplicate search algorithms for code may be more complex since some of them attempt to determine a syntactic structure as well. However, because we only need to solve a simple problem, we can use text methods such as the fingerprint method, which is very similar to the one described above.</p><p>A database of students’ work is an essential component of the entire checking process. In order to make it more convenient, we tried Google Drive. It is convenient to load work, but downloading is not fast. So this approach is actually not suitable for large-audience courses. Nevertheless, the program works. And we tested that it helps to detect cheating. However, among motivated students, there is almost no cheating. So, one way to fight cheating is to motivate students to learn, not to pass a course.</p><h3>Projects’ Part</h3><p>In the first part of the course, students are expected to learn the fundamentals to obtain sufficient skills. And they can use the knowledge to gain experience in the second part by solving real projects. It is even possible to publish a scientific article based on the results. The list of projects will change from year to year. Here are a couple of examples:</p><ol><li><em>Analysis of NMR DOSY data</em>. The project is a mixture of chemistry and data analysis. The data is the sum of exponents. So, the problem is to estimate the optimal number of exponents and their parameters. While fitting might be implemented only with the skills obtained from the course, optimal number estimation requires additional information from the domain. As a result, students have to interact with experimentators like clients that thains their softskills.</li><li><em>Fisher Information for Optimal Experiment Design. </em>The project is based on theotetical statistics combined with advanced physics. The problem here is to optimize the measurement strategy of the neutrino mass estimation experiment. It requires a deep understanding of the physics and statistical background of the optimization method.</li><li><em>Ship Trajectory Optimization</em>. The problem is to optimize a path, taking into account the physical characteristics of a ship. This project is closer to the industry, so it has additional standards for the code development.</li></ol><p>So, projects might provide not only experience in data analysis, but also additional necessary skills.</p><p>There are also lectures on some advanced topics to make the semester more interesting. The purpose of lectures, as opposed to the first part, is to broaden one’s horizons. Here are some ideas for topics:</p><ol><li>Bayesian Deep Learning</li><li>Fisher Information and Active Learning</li><li>Machine learning in Kotlin, KotlinDL</li><li>Deep Learning in Cinema</li><li>Bayesian Optimization</li><li>MCMC on Julia</li></ol><p>So I tried to describe ideas to consider when planning a course and shared our experience. I’m hoping it can help someone.</p><p>The course I described is a part of the <a href="https://sciprog.center/">Scientific Programming Centre</a>.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=296dfe791733" width="1" height="1" alt="">]]></content:encoded>
        </item>
    </channel>
</rss>