<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0" xmlns:cc="http://cyber.law.harvard.edu/rss/creativeCommonsRssModule.html">
    <channel>
        <title><![CDATA[Stories by Ingo Nowitzky on Medium]]></title>
        <description><![CDATA[Stories by Ingo Nowitzky on Medium]]></description>
        <link>https://medium.com/@ingo.nowitzky?source=rss-2628a598423a------2</link>
        <image>
            <url>https://cdn-images-1.medium.com/fit/c/150/150/1*ZY-LyKIBI1TcixIzsuRMDA@2x.jpeg</url>
            <title>Stories by Ingo Nowitzky on Medium</title>
            <link>https://medium.com/@ingo.nowitzky?source=rss-2628a598423a------2</link>
        </image>
        <generator>Medium</generator>
        <lastBuildDate>Tue, 17 Nov 2026 13:33:14 GMT</lastBuildDate>
        <atom:link href="https://medium.com/@ingo.nowitzky/feed" rel="self" type="application/rss+xml"/>
        <webMaster><![CDATA[yourfriends@medium.com]]></webMaster>
        <atom:link href="http://medium.superfeedr.com/" rel="hub"/>
        <item>
            <title><![CDATA[Why my enterprise AI system needed a deterministic analysis layer]]></title>
            <link>https://medium.com/data-science-collective/why-my-enterprise-ai-system-needed-a-deterministic-analysis-layer-cfc3419d3039?source=rss-2628a598423a------2</link>
            <guid isPermaLink="false">https://medium.com/p/cfc3419d3039</guid>
            <category><![CDATA[data-science]]></category>
            <category><![CDATA[generative-ai]]></category>
            <category><![CDATA[artificial-intelligence]]></category>
            <category><![CDATA[ai-agent]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Ingo Nowitzky]]></dc:creator>
            <pubDate>Thu, 04 Jun 2026 14:55:06 GMT</pubDate>
            <atom:updated>2026-06-06T07:33:26.679Z</atom:updated>
            <content:encoded><![CDATA[<figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4vDL0Q8ONB7RSAA2Ob8A3w.png" /><figcaption>Feature image generated by author using ChatGPT</figcaption></figure><h3>Introduction</h3><p>I tried to build an agentic AI network for my company that advises manufacturing plants on how to mature their operations. The system was designed to be data-driven, allowing users to upload assessment data directly through the chat interface. The first working prototype was finished surprisingly quickly, and at first glance the results looked promising.</p><p>There was only one problem: Most of the results were wrong!</p><p>Even worse, the AI quickly learned which numerical ranges looked plausible and began generating convincing — but fabricated — outputs. Combined with the eloquent language generation of the LLM, these results could easily be mistaken for truth. And this behavior was not limited to a single model. Similar patterns appeared across all tested systems: ChatGPT, Gemini Enterprise, DIA Brain, and Microsoft Copilot.</p><p><strong>But plausible analytics are not enough. Enterprise AI systems require reliable analytical foundations.</strong></p><p>At first, I assumed these errors were caused by prompt quality and could be reduced through better instructions. However, repeated testing revealed a different, recurring failure mode. Even with “Code Interpreter” enabled, the systems:</p><ul><li>skipped rows or columns,</li><li>applied incorrect filters,</li><li>returned identical results for different inputs,</li><li>silently mixed parts of the dataset,</li><li>or simply collapsed under more complex analytical tasks.</li></ul><p>This led to a crucial realization:</p><p><strong>Probabilistic reasoning is extremely powerful for interpretation and interaction — but foundational data analysis requires deterministic execution.</strong></p><h3>Table of contents</h3><p>1 The Use Case<br>2 The Solution Architecture<br>3 The Analysis Planner<br>4 The Analysis Engine<br>5 An end-to-end example<br>6 Why AI Architecture Matters<br>Resources</p><h3>1 The Use Case</h3><p>Although the specific use case is of secondary importance, it is briefly outlined here to support the practical understanding of the underlying architectural challenge.</p><p>The primary task of our agent is to advise manufacturing plants and value streams on how to improve their operational performance and maturity: optimizing processes, improving productivity, reducing inventory levels, and ultimately lowering operational costs. To achieve this, the consultation agent operates in two modes:</p><ol><li>It provides generic recommendations for improving specific operational topics based on the retrieval of specialized “how-to” documentation and assessment questionnaires.</li><li>The agent is intended to analyze the current situation of a plant or value stream based on assessment results and assessors’ written recommendations. Based on this analysis, it is expected to provide highly specific recommendations for the next improvement steps.</li></ol><p>In both modes — as with most LLM-based AI models — the user can interactively discuss ideas and recommendations with the agent in order to derive the most suitable action plan.</p><p>For the second operation mode, it is essential that the agent can reliably process and analyze assessment data. In our case, this data is provided as an Excel export from a central database. Ideally, the agent should be able to process the file without any prior manual preparation.</p><p>The structure of the file, however, is challenging. Since all assessment results, intermediate calculations, metadata, and detailed assessment questions are stored in separate columns, the worksheet contains more than 800 columns. The number of rows corresponds to the number of assessments in the database and can range from one to several hundreds (Fig. 1). Assessment ratings are represented as integers from 0 to 4. In addition, the file contains more than 160 free-text fields with qualitative observations, strengths, weaknesses, and recommendations from the assessors.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*n1N7KSjYQinG8_wl6tOluQ.png" /><figcaption>Figure 1: Assessment data structure | image by author</figcaption></figure><p>The analytical tasks of the agent include filtering relevant rows and columns for a specific request, calculating averages, aggregating maturity scores, summarizing textual recommendations, and deriving meaningful improvement suggestions from the results.</p><p>Initially, these tasks appeared to be well within the capabilities of modern LLM-based AI systems, especially with “Code Interpreter” mode enabled. As already mentioned in the introduction, this assumption quickly turned out to be a misconception.</p><h3>2 The Solution Architecture</h3><p>The core idea for overcoming the analytical challenge was to clearly separate deterministic data analysis from LLM-based reasoning and interpretation. Fig. 2 shows the selected system architecture after several improvement iterations. The system was implemented in Microsoft Copilot Studio because the platform allows deterministic workflow elements, such as topics and flows, to be combined with LLM-based reasoning components.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*eRgrTvk6zo1nx7NkBNGBBw.png" /><figcaption>Figure 2: System architecture of the consultation agent with integrated analytics module | image by author</figcaption></figure><p>The parent agent handles all communication with the user. It orchestrates the sub agents and the analytics module, delegates tasks to them, receives their responses, and composes the final answer.</p><p>The sub agents are specialized LLM-based modules with access to specific knowledge sources. These include descriptions of maturity-level expectations for the value streams, questionnaires with detailed assessment questions, and more general guidelines for operational excellence. The sub agents are called by the parent agent according to their specific capabilities and respond to the parent agent rather than directly to the user.</p><p>The analytics module is the main focus of this article. It performs the deterministic data analysis and is designed to provide reproducible and reliable analytical results. It receives an analysis instruction in natural language from the parent agent, referred to as Parent_Instruction. The analytics module itself consists of topics, flows, and AI modules, which are called “prompts” in Copilot Studio.</p><p>The topic T_receive_Excel_File handles the upload and storage of assessment files. It is triggered when a file is uploaded in the chat window, indicated by the variable System.Activity.Attachments having a value. The topic checks whether the uploaded file is an Excel file and, if so, stores it in the global variable Assessment_File.</p><p>The topic T_analyze_assessments is actively called by the parent agent if it has an analytics task to conduct and receives Parent_Instruction as input. A second input is the assessment data stored in the global variable Assessment_File. The topic contains the two core analytics components: Analysis_Planner and Analysis_Engine. Both are embedded in agentic flows, F_Call_Analysis_Planner and F_Call_Analysis_Engine. These flows serve as connectors between the topic T_analyze_assessments and the AI prompts P_Analysis_Planner and P_Analysis_Engine.</p><p>F_Call_Analysis_Planner receives only one input, Parent_Instruction, and forwards it to P_Analysis_Planner. This component generates the Selection_Rule, the core analysis instruction to be executed by P_Analysis_Engine. The inner workings of P_Analysis_Planner are discussed in Chapter 3.</p><p>F_Call_Analysis_Engine receives three inputs: the Selection_Rule from Analysis_Planner, a Mapping_File provided from SharePoint, and the Assessment_File. All three inputs are forwarded to the AI prompt P_Analysis_Engine, which conducts the data analysis as specified by Analysis_Planner. The P_Analysis_Engine is discussed in detail in Chapter 4.</p><h3>3 The Analysis Planner</h3><p>The P_Analysis_Planner is the intelligent part of the data analysis pipeline and generates the analysis instruction, called Selection_Rule. This instruction is a translation of the natural language Parent_Instruction and is generally unique for each request. In order to minimize probabilistic variation, the translation process is constrained by strict rules.</p><p><strong>The Analysis_Planner does not analyze the assessment data itself. Its sole responsibility is to translate the probabilistic </strong>Parent_Instruction<strong> into a deterministic analysis specification.</strong></p><p>In the following, we will examine selected parts of the instruction in more detail. You can download the full instruction <a href="https://www.dropbox.com/scl/fi/ksbs011eslbo10xlrt7lm/Instruction_Analysis_Planner.txt?rlkey=fyp8y9824fb0kdm55dmlvdv1r&amp;st=fg8qdxah&amp;dl=0">here</a>.</p><pre>You are Analysis_Planner, an expert assistant for translating natural-language assessment analysis requests into structured Selection_Rules.<br>Your task is to create a Selection_Rule JSON object for the Analysis_Engine.<br><br>You receive only one input:<br><br>1. Parent_Instruction :<br>A natural-language analysis request from the parent agent (orchestrator).<br><br>You must analyze Parent_Instruction and determine:<br>- which type of analysis is required,<br>- which assessment content categories are relevant,<br>- whether concept or execution maturity/findings are requested,<br>- whether specific chapters are requested,<br>- and whether row filters are required.<br><br>The Selection_Rule you generate will later be used by the Analysis_Engine together with:<br>- the real assessment data file,<br>- and the Mapping_File<br>to execute the analysis deterministically.</pre><p>The code box above shows the initial instruction for P_Analysis_Planner. It clearly defines purpose and scope and explicitly separates planning from execution. The planner translates the request, while the actual execution is delegated to the P_Analysis_Engine.</p><p>Next in the instruction follows a longer section describing the semantics of the assessment data. Of course, this part is highly specific to the individual use case and dataset. It defines semantic categories used for row filtering and categories used to select the actual analysis targets (TARGET CONTENT CATEGORIES and TARGET SELECTION ATTRIBUTES).</p><pre>ASSESSMENT DATA SEMANTICS<br><br>The assessment data can be addressed through the following semantic categories.<br><br>ROW FILTER CATEGORIES<br><br>Use these categories only for row_filters:<br><br>- VS_Nr:<br>    Unique identifier of the value stream.<br>    Use when filtering by value stream number.<br><br>- Value Stream:<br>    Name of the value stream.<br>    Use when filtering by value stream name.<br><br>- ...<br><br><br>TARGET CONTENT CATEGORIES<br><br>Use these categories only in target_selection_rules.data_category:<br><br>- chapter_score:<br>    Numeric maturity score.<br>    Use for maturity calculations, score analysis, and average maturity analysis.<br><br>- strength:<br>    Assessor statements describing strengths.<br><br>- ...<br><br><br>TARGET SELECTION ATTRIBUTES<br><br>Use these attributes only inside target_selection_rules:<br><br>- data_category:<br>    Defines which target content category is needed.<br><br>- aggregation_allowed:<br>    Use:<br>        - mean for numeric maturity averages<br>        - summary for textual summaries<br><br>- ...</pre><p>The planner never interacts directly with physical dataset columns. Instead, it operates on a semantic abstraction layer that decouples natural language from the underlying dataset structure.</p><p>This separation is important because the assessment dataset contains more than 800 columns, including:</p><ul><li>maturity ratings,</li><li>textual assessor findings,</li><li>metadata,</li><li>organizational mappings,</li><li>questionnaire variants,</li><li>and concept/execution distinctions.</li></ul><p>Selecting the correct target columns therefore becomes a critical part of the analysis process.</p><p>Restricting the allowed analysis types is equally important. The planner is intentionally prevented from inventing arbitrary analytical operations. The section ANALYSIS TYPES therefore defines the only valid analysis types — currently just two. This significantly improves the predictability and robustness of downstream execution. Of course, the list can easily be extended for individual use cases.</p><pre>ANALYSIS TYPES<br><br>Use exactly one of these analysis_type values:<br><br>- numeric_mean<br>    Use for:<br>    - average maturity<br>    - mean maturity<br>    - ...<br><br>- text_summary<br>    Use for:<br>    - strengths<br>    - improvement potentials<br>    - ...</pre><p>The next section defines how the planner selects the relevant target columns in an abstract and deterministic way. The rules distinguish between the two predefined analysis types numeric_mean and text_summary and finally determine which dataset columns are selected for a specific request.</p><pre>RULES FOR target_selection_rules<br><br>NUMERIC MATURITY ANALYSIS<br><br>For numeric maturity analysis:<br>- analysis_type must be:<br>    &quot;numeric_mean&quot;<br>- data_category must be:<br>    [&quot;chapter_score&quot;]<br>- ...<br><br><br>TEXT SUMMARY ANALYSIS<br><br>For textual summary analysis:<br>- analysis_type must be:<br>    &quot;text_summary&quot;<br>- data_category:<br>    include only requested categories:<br>        - &quot;strength&quot;<br>        - &quot;potential&quot;<br>        - &quot;recommendation&quot;<br>        - &quot;remark&quot;<br>- ...</pre><p>A similar logic applies to the row filtering process.</p><pre>RULES FOR row_filters<br><br>Use row_filters only for filtering rows in the assessment dataset.<br><br>Allowed row filter keys are:<br>- VS_Nr<br>- Value Stream<br>- ...<br><br>Do NOT use row_filters for:<br>- chapter_id<br>- ...<br><br>These belong only to target_selection_rules.</pre><p>Finally, the instruction defines the required output structure together with several strict “do-not rules”. This section is particularly important because the generated output is directly forwarded to the P_Analysis_Engine and therefore must follow a clearly defined and machine-readable structure.</p><pre>OUTPUT FORMAT<br><br>Return only valid JSON.<br>Do not return markdown.<br>Do not return Python code.<br>...<br><br><br>Use exactly this structure:<br><br>{<br>  &quot;status&quot;: &quot;success&quot;,<br>  &quot;parent_instruction_summary&quot;: &quot;&quot;,<br>  &quot;selection_rule&quot;: {<br>    &quot;analysis_type&quot;: &quot;&quot;,<br>    &quot;target_selection_rules&quot;: {<br>      &quot;data_category&quot;: [],<br>      &quot;aggregation_allowed&quot;: [],<br>      &quot;concept_execution&quot;: null,<br>      &quot;chapter_id&quot;: null<br>    },<br>    &quot;row_filters&quot;: {}<br>  },<br>  &quot;warnings&quot;: []<br>}</pre><p>If the request is unclear, the planner must explicitly return an error structure instead of “guessing” a potentially wrong analysis instruction.</p><pre>If the task is unclear, return:<br><br>{<br>  &quot;status&quot;: &quot;error&quot;,<br>  &quot;parent_instruction_summary&quot;: &quot;&quot;,<br>  &quot;selection_rule&quot;: {<br>    &quot;analysis_type&quot;: null,<br>    &quot;target_selection_rules&quot;: {<br>      &quot;data_category&quot;: [],<br>      &quot;aggregation_allowed&quot;: [],<br>      &quot;concept_execution&quot;: null,<br>      &quot;chapter_id&quot;: null<br>    },<br>    &quot;row_filters&quot;: {}<br>  },<br>  &quot;warnings&quot;: [<br>    &quot;The analysis task is not clearly understood.&quot;<br>  ]<br>}</pre><p>At this point, the planner has transformed ambiguous natural language into a deterministic analysis specification. However, the actual data execution still has not happened.</p><p>In chapter 5, we will follow a real user request through the complete pipeline and examine how P_Analysis_Planner generates the Selection_Rule and how P_Analysis_Engine executes it on the assessment dataset.</p><h3>4 The Analysis Engine</h3><p>Unlike the P_Analysis_Planner, the P_Analysis_Engine does not reason about the task. It only executes the analysis specification generated by P_Analysis_Planner.</p><p>As in chapter 3, we will focus only on the most relevant parts of the instruction. The full specification can be downloaded <a href="https://www.dropbox.com/scl/fi/6rsqjcokgdwj6bczxax7u/Instruction_Analysis_Engine.txt?rlkey=k0wiqgzusrafx45iv8nr5n06p&amp;st=4kqtw3le&amp;dl=0">here</a>.</p><p>The instruction of P_Analysis_Engine starts with the basic task definition. In essence, the AI prompt is used as a controlled Python execution environment. The code is predefined in the prompt instruction and must only be executed, not modified.</p><pre>You are Analysis_Engine, a deterministic pandas-based analysis executor.<br><br>Your task is to analyze an Excel assessment dataset using Code Interpreter.<br><br>You receive three inputs:<br><br>1. document <br>   The Excel file containing the assessment data.<br><br>2. Mapping_File <br>   The Excel file describing the columns of document.<br><br>3. Selection_Rule <br>   A JSON object that defines:<br>   - which columns to select from Mapping_File<br>   - which row filters to apply to document<br>   - which type of analysis to perform<br><br>You must not reinterpret the original user request.<br>You must not infer additional columns.<br>You must not change Selection_Rule.<br>You must not generate a new analysis approach.<br>You must only execute the deterministic Python script below.<br><br>Use Code Interpreter to execute the Python script.<br>Return only the JSON result printed by the script.<br>Do not return markdown.<br>Do not explain the code.<br>Do not add text before or after the JSON result.</pre><p>P_Analysis_Engine receives three input files:</p><ol><li>The Assessment_File uploaded from the user in the chat interface. It is stored in the prompt-internal variable document.</li><li>A Mapping_File which the flow F_Call_Analysis_Engine loads from SharePoint in preparation of the execution.</li><li>The Selection_Rule generated by P_Analysis_Planner (see chapter 3).</li></ol><p>The Mapping_File plays a crucial role in defining the semantics of the many columns in Assessment_File on a higher level of abstraction. With this abstraction layer, the Selection_Rule only needs to specify which type of information is required, while the P_Analysis_Engine selects the corresponding dataset columns during execution.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gzVdtR13l-jb5XQg_sbnug.png" /><figcaption>Figure 3: Structure of Mapping_File | image by author</figcaption></figure><p>Fig. 3 shows the structure of Mapping_File. It contains a row for each column of Assessment_File, that is potentially relevant for the data analysis. Data columns that are clearly irrelevant are not represented in Mapping_File and are therefore invisible to P_Analysis_Engine. For each row the file specifies the selection criteria:</p><ul><li>data_category: <br>Functional meaning of the column, e.g. maturity score, strength, plant name, region, or season.</li><li>chapter_id:<br>Unique identifier of the assessment chapter.</li><li>chapter_name:<br>Human-readable name of the assessment chapter.</li><li>concept_execution:<br>Indicates whether the column belongs to concept or execution maturity.</li><li>aggregation_allowed:<br>Defines which type of aggregation is valid for the column, e.g. mean for numeric maturity scores or summary for textual findings.</li></ul><p>Next in P_Analysis_Engine’s instruction comes a paragraph about how to interpret the Selection_Rule.</p><pre>Rules for Selection_Rule:<br><br>- analysis_type = &quot;numeric_mean&quot;:<br>  Calculate arithmetic means for all selected numeric target columns.<br><br>- analysis_type = &quot;text_summary&quot;:<br>  Collect non-empty text entries from all selected text target columns.<br><br>- target_selection_rules:<br>  Select target columns by matching Mapping_File attributes.<br>  A rule value of null means: do not filter by this attribute.<br>  A list means: keep rows where the Mapping_File attribute is in the list.<br><br>- row_filters:<br>  Apply row filters to document.<br>  Keys are data_category values from Mapping_File, such as &quot;Plant&quot;, &quot;Region&quot;, &quot;Production Principle&quot;, &quot;Season&quot;.<br>  Values are lists of accepted values.</pre><p>This section specifies:</p><ul><li>which analysis operation must be executed (analysis_type),</li><li>how relevant target columns are selected from the Mapping_File (target_selection_rules),</li><li>and how the assessment dataset is filtered before the analysis is performed (row_filters).</li></ul><p>This instruction is intentionally deterministic. The P_Analysis_Engine is not allowed to reinterpret the original user request or invent additional analytical operations.</p><p>After the instruction block, the P_Analysis_Engine receives the actual Python script. The full script contains more than 300 lines of code and is part of the AI prompt instruction. It is linked at the top of this chapter or below in the resources and can be downloaded. Many of the code lines are not conceptually important for the architecture. They handle practical robustness: cleaning column names, normalizing input values, handling missing columns, converting Copilot wrapper objects, and returning structured error messages.</p><p>For the article, I will focus only on the central logic.</p><p>The first important step is that the engine loads the uploaded assessment data (available in document) and the Mapping_File. From this point on, the LLM is no longer interpreting the user request. It only executes the deterministic script based on the Selection_Rule.</p><pre>mapping_df = pd.read_excel(Mapping_File)<br>data_df = pd.read_excel(document)<br><br>mapping_df = strip_column_names(mapping_df)<br>data_df = strip_column_names(data_df)</pre><p>The key architectural element is the selection of target columns. The P_Analysis_Engine never guesses which Excel columns may be relevant. Instead, it filters the Mapping_File according to the attributes defined in target_selection_rules.</p><pre>target_mapping = mapping_df.copy()<br><br>for attr, rule_value in target_selection_rules.items():<br><br>    values = normalize_rule_value(rule_value)<br>    values = normalize_list_for_matching(values)<br><br>    if values is None:<br>        continue<br><br>    target_mapping = target_mapping[<br>        target_mapping[attr]<br>        .apply(normalize_for_matching)<br>        .isin(values)<br>    ]<br><br>selected_target_columns = (<br>    target_mapping[&quot;source_column_name&quot;]<br>    .dropna()<br>    .tolist()<br>)</pre><p>This is the point where the abstract analysis instruction becomes concrete. For example, a rule such as chapter_id = [&quot;3.5&quot;], data_category = [&quot;chapter_score&quot;], and aggregation_allowed = [&quot;mean&quot;] is translated into the actual Excel columns containing the Concept and Execution maturity scores for chapter 3.5.</p><p>The same principle is applied to row filters. Again, the engine does not infer anything from natural language. It only applies the filters explicitly provided in the Selection_Rule.</p><pre>filtered_df = data_df.copy()<br><br>for filter_category, filter_values in row_filters.items():<br><br>    filter_mapping = mapping_df[<br>        mapping_df[&quot;data_category&quot;]<br>        .apply(normalize_for_matching)<br>        == normalize_for_matching(filter_category)<br>    ]<br><br>    filter_col = filter_mapping[&quot;source_column_name&quot;].iloc[0]<br><br>    filtered_df = filtered_df[<br>        filtered_df[filter_col]<br>        .apply(normalize_for_matching)<br>        .isin(values)<br>    ]</pre><p>After column selection and row filtering, the actual analysis logic becomes intentionally straightforward. For numeric maturity analysis, the engine calculates arithmetic means for all selected numeric target columns.</p><pre>if analysis_type == &quot;numeric_mean&quot;:<br><br>    numeric_result = {}<br><br>    for col in available_target_columns:<br><br>        series = pd.to_numeric(filtered_df[col], errors=&quot;coerce&quot;)<br>        valid_count = int(series.notna().sum())<br><br>        numeric_result[col] = {<br>            &quot;mean&quot;: float(series.mean()) if valid_count &gt; 0 else None,<br>            &quot;valid_count&quot;: valid_count<br>        }<br><br>    result[&quot;result&quot;] = numeric_result</pre><p>For textual analysis, the engine collects non-empty assessor statements instead of calculating values.</p><pre>elif analysis_type == &quot;text_summary&quot;:<br><br>    text_result = {}<br><br>    for col in available_target_columns:<br><br>        values = [<br>            clean_text_value(v)<br>            for v in filtered_df[col].tolist()<br>        ]<br><br>        values = [v for v in values if v is not None]<br><br>        text_result[col] = {<br>            &quot;entries&quot;: values,<br>            &quot;entry_count&quot;: len(values)<br>        }<br><br>    result[&quot;result&quot;] = text_result</pre><p>Finally, the result is returned as JSON. This is important because the output is not yet the final user-facing answer. It is the reliable analytical foundation for the next LLM step: interpretation by the parent agent.</p><pre>print(json.dumps(result, indent=2, ensure_ascii=False))</pre><p>This design deliberately keeps the P_Analysis_Engine “boring”. It does not reason, it does not explain, and it does not improve the analysis. It only executes. And that is exactly the point. The more deterministic this layer is, the more trust can be placed in the later LLM-generated interpretation.</p><h3>5 An end-to-end Example</h3><p>To illustrate the complete workflow, let us follow a realistic example through the full pipeline.</p><p>Triggered by the user interaction, the parent agent might raise the following Parent_Instruction to the analytics module:</p><p><strong>“Summarize the main improvement potentials for chapter 1.4 Failure Prevention System in plant AbcP.”</strong></p><p>The request looks simple for a human reader, but it already contains multiple semantic tasks:</p><ul><li>identify the requested assessment chapters,</li><li>detect the requested content type,</li><li>apply a row filter,</li><li>retrieve the correct text columns,</li><li>aggregate textual findings,</li><li>and finally generate a meaningful interpretation ( → parent agent).</li></ul><p>This is exactly the type of task where a pure LLM-based analysis becomes unreliable. The system therefore separates the workflow into deterministic execution and probabilistic interpretation steps.</p><h4>5.1 Translation from Analysis Planner</h4><p>The first step is performed by P_Analysis_Planner.<br> It translates the natural language request into a deterministic Selection_Rule.</p><pre>{<br>  &quot;status&quot;: &quot;success&quot;,<br>  &quot;parent_instruction_summary&quot;: &quot;Summarize improvement potentials for chapter 1.4 Failure Prevention System in plant AbcP.&quot;,<br>  &quot;selection_rule&quot;: {<br>    &quot;analysis_type&quot;: &quot;text_summary&quot;,<br>    &quot;target_selection_rules&quot;: {<br>      &quot;data_category&quot;: [&quot;potential&quot;],<br>      &quot;aggregation_allowed&quot;: [&quot;summary&quot;],<br>      &quot;concept_execution&quot;: null,<br>      &quot;chapter_id&quot;: [&quot;1.4&quot;]<br>    },<br>    &quot;row_filters&quot;: {<br>      &quot;Plant&quot;: [&quot;AbcP&quot;]<br>    }<br>  },<br>  &quot;warnings&quot;: []<br>}</pre><p>The Selection_Rule already contains the complete deterministic analysis specification:</p><ul><li>analysis_type = &quot;text_summary&quot;<br>indicates that textual assessor findings must be collected instead of numeric calculations.</li><li>data_category = [&quot;potential&quot;]<br>restricts the analysis to improvement potentials.</li><li>chapter_id = [&quot;1.4&quot;]<br>limits the analysis to the Failure Prevention System chapter.</li><li>row_filters = {&quot;Plant&quot;: [&quot;AbcP&quot;]}<br>restricts the dataset to the requested plant.</li></ul><p>At this stage, no data analysis has happened yet. The result is only an execution instruction for the next step.</p><h4>5.2 Execution from Analysis Engine</h4><p>This Selection_Rule is handed over to P_Analysis_Engine for execution. First, the engine selects all matching target columns from the Mapping_File.</p><pre>target_mapping = target_mapping[<br>    target_mapping[attr]<br>    .apply(normalize_for_matching)<br>    .isin(values)<br>]</pre><p>This translates the abstract selection criteria into real dataset columns, for example:</p><pre>selected_target_columns = [<br>    &quot;1.4 CON L2 Improvement potentials&quot;,<br>    &quot;1.4 CON L3 Improvement potentials&quot;,<br>    &quot;1.4 EXE L2 Improvement potentials&quot;,<br>    &quot;1.4 EXE L3 Improvement potentials&quot;<br>]</pre><p>Next, the row filters are applied:</p><pre>filtered_df = filtered_df[<br>    filtered_df[filter_col]<br>    .apply(normalize_for_matching)<br>    .isin(values)<br>]</pre><p>In this example, the dataset is reduced to assessment rows belonging to plant AbcP.</p><p>Finally, the engine collects all non-empty text entries from the selected columns.</p><pre>values = [<br>    clean_text_value(v)<br>    for v in filtered_df[col].tolist()<br>]<br><br>values = [v for v in values if v is not None]</pre><p>As we can see, the engine does not interpret the findings. It only retrieves and structures them according to the Python script.</p><p>The engine’s output is a collection of assessors’ written statements about the value stream’s improvement potentials as a JSON object.</p><pre>{<br>  &quot;entry_count&quot;: 6,<br>  &quot;entries&quot;: [<br>    &quot;Root causes are not systematically tracked.&quot;,<br>    &quot;Escalation rules for recurring failures are unclear.&quot;,<br>    &quot;Lessons learned are not transferred between shifts.&quot;,<br>    &quot;Preventive maintenance findings are not integrated into CIP activities.&quot;,<br>    &quot;Failure trends are visualized inconsistently.&quot;,<br>    &quot;Problem-solving activities focus mainly on symptoms instead of root causes.&quot;<br>  ]<br>}</pre><p>At this point, the system has still not generated any recommendations. It has only produced a reliable collection of relevant assessment findings. <strong>This JSON object is returned to the parent agent for interpretation and generation of the final response to the user.</strong></p><h4>5.3 Interpretation from Parent Agent</h4><p>In the final step, the parent agent collects all responses (potentially additional responses from the sub agents) and generates the final output.</p><pre>The collected findings indicate that the Failure Prevention System is<br>currently more reactive than preventive. Most gaps are related to missing<br>systematic root-cause management and weak organizational learning across<br>shifts and teams. The highest leverage improvements would likely come from<br>strengthening escalation routines, integrating preventive maintenance findings<br>into CIP activities, and establishing consistent cross-shift learning<br>mechanisms.</pre><p>To summarize the central architectural idea of the system:</p><p>The LLM no longer creates the analytical foundation itself. Instead, it interprets a deterministic set of already validated findings.</p><p>The probabilistic reasoning capability of the LLM is used where it creates value: interpretation, prioritization, explanation, and communication — not data processing itself.</p><h3>6 Why AI Architecture Matters</h3><p>Large Language Models are naturally strong at interpretation, reasoning, and language generation, but still weak at reliable numerical analytics. Their optimization target is plausibility, not deterministic reproducibility. Even with extensions such as “Code Interpreter”, this limitation remains visible in more complex analytical scenarios.</p><p>The good news is that this weakness can largely be compensated through system architecture. In the presented approach, deterministic execution layers provide the analytical foundation, while the LLM focuses on interpretation, prioritization, explanation, and communication.</p><p>Looking back, the most important lesson was not about model capability but about system design. The challenge was not to make the LLM smarter. It was to decide which tasks could rely on probabilistic reasoning and which required deterministic execution.</p><p>I also needed a practical way to integrate deterministic analytics into a modern agentic system. Interestingly, this turned out to be possible by combining carefully designed prompts with embedded Python code executed under strict control.</p><p>Reliable enterprise AI systems will therefore not emerge from increasingly powerful models alone. They will emerge from architectures that assign each task to the technology best suited to perform it.</p><h3>Resources</h3><ul><li>Full instruction used in Analysis_Planner LLM: <a href="https://www.dropbox.com/scl/fi/ksbs011eslbo10xlrt7lm/Instruction_Analysis_Planner.txt?rlkey=fyp8y9824fb0kdm55dmlvdv1r&amp;st=fg8qdxah&amp;dl=0">LINK</a></li><li>Full instruction used in Analysis_Engine LLM: <a href="https://www.dropbox.com/scl/fi/6rsqjcokgdwj6bczxax7u/Instruction_Analysis_Engine.txt?rlkey=k0wiqgzusrafx45iv8nr5n06p&amp;st=4kqtw3le&amp;dl=0">LINK</a></li><li>Original article published on TDS: <a href="https://towardsdatascience.com/hybrid-ai-combining-deterministic-analytics-with-llm-reasoning/">LINK</a></li></ul><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=cfc3419d3039" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science-collective/why-my-enterprise-ai-system-needed-a-deterministic-analysis-layer-cfc3419d3039">Why my enterprise AI system needed a deterministic analysis layer</a> was originally published in <a href="https://medium.com/data-science-collective">Data Science Collective</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[From Concept to Code: Unveiling the ChatGPT Algorithm]]></title>
            <link>https://pub.towardsai.net/from-concept-to-code-unveiling-the-chatgpt-algorithm-77e44e19f466?source=rss-2628a598423a------2</link>
            <guid isPermaLink="false">https://medium.com/p/77e44e19f466</guid>
            <category><![CDATA[chatgpt]]></category>
            <category><![CDATA[transformers]]></category>
            <category><![CDATA[attention]]></category>
            <category><![CDATA[large-language-models]]></category>
            <category><![CDATA[llm]]></category>
            <dc:creator><![CDATA[Ingo Nowitzky]]></dc:creator>
            <pubDate>Tue, 21 Jan 2025 18:03:04 GMT</pubDate>
            <atom:updated>2025-01-21T18:03:04.971Z</atom:updated>
            <content:encoded><![CDATA[<h4>How Transformers work, explained step-by-step with a simplified code example</h4><p>For the past two years, ChatGPT and Large Language Models (LLMs) in general have been the big thing in artificial intelligence. Many articles about how-to-use, prompt engineering and the logic behind have been published. Nevertheless, when I started familiarizing myself with the algorithm of LLMs — the so-called transformer — I had to go through many different sources to feel like I really understood the topic.<br>In this article, I want to summarize my understanding of Large Language Models. I will explain conceptually how LLMs calculate their responses step-by-step, go deep into the attention mechanism, and demonstrate the inner workings in a code example.<br>So, let’s get started!</p><h3>Table of contents</h3><h4>Part 1: Concept of transformers</h4><p>1.1 Introduction to Transformers<br>1.2 Tokenization<br>1.3 Word Embedding<br>1.4 Positional Encoding<br>1.5 Attention Mechanism<br>1.6 Layer Norm<br>1.7 Feed Forward<br>1.8 Softmax<br>1.9 Multinomial</p><h4>Part 2: Implementation in code</h4><p>2.1 Data Preparation<br>2.2 Tokenization<br>2.3 Data Feeder Function<br>2.4 Attention Head<br>2.5 Multi-head Attention<br>2.6 Feed Forward of Attention Block<br>2.7 Attention Block<br>2.8 Transformer Class<br>2.9 Instantiate the Transformer<br>2.10 Model Training<br>2.11 Generate new Tokens</p><h3>Part 1: Concept of Transformers</h3><h4>1.1 Introduction to Transformers</h4><p>We cannot discuss the topic of Large Language Models without citing the famous paper <a href="https://arxiv.org/pdf/1706.03762">“Attention Is All You Need”</a> published by Vaswani et al. in 2017. In this article, the group of researchers introduced the attention mechanism and the transformer architecture that sparked the revolution in generative AI we experience today. Originally, the paper referred to machine language translation and introduced an encoder-decoder structure.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*JKLdA7_YExo9nf9jufKcBg.png" /><figcaption>Fig. 1.1.1: Transformer architecture introduced by <a href="https://arxiv.org/pdf/1706.03762">Vaswani et al.</a> | left original, right with explanation by author</figcaption></figure><p>Fig. 1.1.1 on the left, shows the transformer as published in the paper, and on the right I marked the encoder and the decoder part. In machine language translation, the initial language is encoded by the encoder and decoded into the target language by the decoder.<br>In contrast, ChatGPT has a decoder-only architecture. Therefore, in the following, we will ignore the left side and fully concentrate on the decoder.</p><p>Before I start explaining the transformer, we need to recall that ChatGPT generates its output in a loop, one token after the other. Let’s assume we input the words “ChatGPT writes…” (yes, I know this context is unrealistically short). ChatGPT might output the token “…one” in the first cycle. The initial words plus the first output build the context for the second generation cycle, so the input is “ChatGPT writes one…”. Now, ChatGPT might output “…word”, which is concatenated to the existing context and inputted again. This loop goes on until the generated output is a stop token, which indicates that the response has reached its end and the generation loop is finished until the next user interaction.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uw2S1zaGB0-Ai6VjtjGWpA.png" /><figcaption>Fig. 1.1.2: ChatGPT generates its output in a loop one token after the other | image by author</figcaption></figure><p>Now, the big question is: what happens inside the magic box denoted “ChatGPT” in Fig. 1.1.2? How does the algorithm conclude which token to output next? This is exactly the question we will answer in this article.</p><p>Fig. 1.1.3 shows the processing steps of the transformer in a sequence and is an alternative illustration to that in the original paper (Fig. 1.1.1). I prefer using this image because it allows me to better structure the explanation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*BTiqZJSDlJV3EbO2X3QvBA.png" /><figcaption>Fig. 1.1.3: Transformer architecture as used in ChatGPT | image by author</figcaption></figure><p>In Fig. 1.1.3, we see the input into the transformer on the bottom left – the token sequence “ChatGPT writes…” – and the output of the transformer on the top right, which is “…one”. What happens between input and output?</p><ul><li>On the left side of Fig. 1.1.3, we find some preprocessing steps: tokenization, word embedding, and positional encoding. We will study these steps right after this introduction.</li><li>In the middle part, we see the so-called attention block. This is where the context of the words and sentences is processed. The attention block is the magic of ChatGPT and the reason why the outputs of the bot are so convincing.</li><li>On the right side of Fig. 1.1.3, we see that the output of the attention block is normalized (“Layer Norm”), fed into a neural network (“Feed forward”), softmaxed, and finally runs through a multinomial distribution. Later, we will see that, with those four steps, we calculate the probabilities for all tokens in our vocabulary to be the next output, and sample the actual output from the multinomial distribution according to those probabilities. But be patient — we will study this in the required details later in the article. For now, we accept that the output of this process is the token “one“.</li></ul><p>With this overview in mind, let us go through the processing steps one by one in the next chapters.</p><h4>1.2 Tokenization</h4><p>Tokens are the basic building blocks for text processing in Large Language Models. The process of splitting text into tokens is called tokenization. Depending on the tokenization model, the received tokens can look quite different. Some models split text into words, others into subwords or characters. Independent of the granularity, tokenization models also include punctuation marks and special tokens like &lt;start&gt; and &lt;stop&gt; for controlling the LLM’s output to a user interaction.<br>The basic idea of tokenization is to split the processed text into a potentially large but <strong>limited number</strong> of tokens the LLM knows.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QfyxnUcxlVS0Ii0oV1wnHg.png" /><figcaption>Fig. 1.2.1: Tokenization | image by author</figcaption></figure><p>Fig. 1.2.1 shows a simple example. The context “Let’s go in the garden” is split into the seven tokens “let”, “ ‘ “, “s”, “go”, “in”, “the”, “garden”. These tokens are known to the LLM and will be represented by an internal number for further processing. Vice versa, when the LLM processes its output, it determines the next token through probabilities and composes the outputted sentences from the tokens of several generation cycles.</p><h4>1.3 Word Embedding</h4><p>So far, we have seen that the tokenizer splits the input sentences into tokens. Next, word embedding translates tokens into large vectors with usually several hundred or several thousand of dimensions, depending on the chosen model. In general, the higher the embedding depth (meaning larger vectors), the more information the embedding can capture.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*vkSbOLLmuJuUXnP_S0BrXg.png" /><figcaption>Fig. 1.3.1: Word embedding of our example sentence | image by author</figcaption></figure><p>In Fig. 1.3.1, we continue with our example: The tokenizer has split the sentence “Let’s go in the garden” into the seven tokens: “let”, “ ’ ”, “s”,…<br>The word embedding translates the tokens into the shown vectors. In Fig. 1.3.1, I denoted the number of dimensions with <em>n</em>, but in fact, I used 100 — a relatively small number!</p><p>The interesting point about word embedding is that it captures features of the tokens. One of the implications is that words with similar meanings get similar word embeddings and, therefore, are located nearby in the embedding space. This characteristic of tokens is automatically captured in the embedding process because words with similar meanings are used in similar contexts.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*affPc4Z4sLYHp7etTliBdA.png" /><figcaption>Fig. 1.3.2: Word embedding of example words with original depth n=100, reduced to n_pca=2 with principal component analysis for demonstration purpose | image by author</figcaption></figure><p>In Fig. 1.3.2, you see example words with an original embedding depth of 100. To plot the embeddings in the plane, I reduced the embedding vectors to dimension 2 with the principal component analysis (PCA). We see that words describing animals build one cluster, fruits build another cluster, and the same is true for tools, vehicles and sports. Thus, words with similar meanings are located nearby because they have similar embeddings.</p><p>But word embedding does not only capture similarities but also relations between words. As an example, in Fig. 1.3.3, you see a couple of adult-baby relations. Again, the original embeddings have a depth of 100 and were reduced to 2 using PCA.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*W1-T2OlSLsNUrLQ8cxaA6g.png" /><figcaption>Fig. 1.3.3: Word embedding of adult-baby relations between words; original embedding depth of n=100 reduced to n_pca=2 with PCA | image by author</figcaption></figure><p>We see that “puppy” relates to “dog” like “kitten” to “cat”, “toddler” to “human” or “calf” to “cow”. In each case, the baby word is in the top left and the adult word in the bottom right.</p><p>In this article, we take word embeddings as given because we have plenty of ready-to-use models (e.g., Word2Vec, GloVe, fastText). Nevertheless, I’d like to give you some high-level intuition about how word embeddings are calculated. Be aware that word embedding methods vary. The way sketched here is only one approach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*SWUHpD5I6aC7Dm5h2q81Rw.png" /><figcaption>Fig. 1.3.4: Calculation of word embeddings in an encoder-decoder neural network | image by author</figcaption></figure><p>Word embeddings come from Encoder-Decoder neural networks. Typical for this architecture is that the inputs are compressed into a lower dimension, and from there we try to reconstruct information to a higher dimension. In Fig. 1.3.4, we have one node for each token in the vocabulary in the first and the last layer. We feed in a word or a couple of words into the encoder with a “1” in the respective node and “0” in all other nodes (one-hot encoding). The network compresses the information to a lower dimension — which equals the embedding depth — and from there tries to fulfill a task, e.g. predicting the next word in a sequence. If the calculated output is wrong, the weights of the model are updated via backpropagation. Once we achieve a sufficient accuracy, we take the lower dimensions’ weights as our word embeddings. Fig. 1.3.5 shows the embedding of “abacus” as an example. The embedding equals the weights behind the relations marked in bold.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Td9xAMstsuc0Ykzr1MJIZw.png" /><figcaption>Fig. 1.3.5: The word embedding for “abacus” | image by author</figcaption></figure><h4>1.4 Positional Encoding</h4><p>Word embeddings do not keep track of the word order. The sentence “Today, I’m not happy, I am sad!” gets the exact same embedding vectors as the sentence “Today, I’m not sad, I am happy!” Of course, as humans we know that both statements have the opposite meaning, but the transformer does not!<br>This is where positional encoding comes into play. The main idea is to add a vector of the same size as the word embedding to each embedding vector. The positional vector specifies the position of the token in the context.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OJv6bw2Xl9jf5gyiVrNZiQ.png" /><figcaption>Fig. 1.4.1: Positional encoding | image by author</figcaption></figure><p>Fig. 1.4.1 continues our example of “Let’s go in the garden”. We see that for each of the seven tokens, a positional vector of equal size <em>n</em> is added. The positional vectors encode the 1st, 2nd, 3rd,… position in the given context. Therefore, the sum of the embedding vector and the positional vector holds information about the token and its position and is no longer independent of the word order. With positional encoding, the embedding vectors for “I’m not happy, I am sad!” and “I’m not sad, I am happy!” are no longer equal!</p><p>Now, how do we determine the positional vectors? In fact, there are different methods, and I will present only two of them:</p><ul><li>The method described in “Attention is all you need” by Vaswani et al. and</li><li>A simplified method we use in our own coding.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*L3bN_r1DGyYkwVHcuoWVkg.png" /><figcaption>Fig. 1.4.2: Two methods to determine positional encoding vectors | image by author</figcaption></figure><p>Vaswani et al. propose to use sine and cosine functions to determine the positional vectors (Fig. 1.4.2 left). “<em>pos</em>” stands for the <strong>token</strong> position in the context: 1st token, 2nd token, 3rd token… and <em>2i</em> and <em>2i+1</em> represent the odd and even <strong>embedding</strong> positions (the position within the positional vector). <em>i</em> runs from 0 to <em>dmodel/2</em>. For even embedding positions, we use the sine function, and for odd embedding positions the cosine function.</p><p>Fig. 1.4.2 right, shows the approach we use in our own coding (part 2). We make use of the PyTorch embedding module, which initializes to standard normal distributed values. This means we determine only the size of the positional vectors, but the vector values are random. Of course, once determined, we keep the positional vectors for the 1st, 2nd, 3rd,… position in the context fixed for the entire life of the LLM.</p><p>My personal impression is that the way positional vectors are calculated is less important for the performance of a transformer. However, it is crucial to use positional encoding, no matter how the vectors have been calculated.</p><h4>1.5 Attention Mechanism</h4><p>The attention mechanism is the heart of the transformer. It is the main reason why ChatGPT is so good at language processing. The key word in everything we discuss next is “context”.</p><p>The explanations in this chapter largely follow the ideas of <a href="https://medium.com/u/ff11baf3f5fe">Luis Serrano</a>. I recommend his video series “<a href="https://www.youtube.com/watch?v=UPtG_38Oq8o&amp;list=PLs8w1Cdi-zvYskDS2icIItfZgxclApVLv&amp;index=2">Attention Mechanisms in LLMs</a>” on YouTube to anyone who is interested.</p><h4>1.5.1 Adjusting Word Embeddings to the Context</h4><p>In human language, the context of words and sentences is very important to understand their meaning. Words can have different meanings in different contexts.<br>We have learned that word embeddings already take into account which words appear in connection with others more often. But word embeddings do not consider the context in a specific situation. We could say word embeddings have a <strong>static context</strong>, while we need a <strong>dynamic context </strong>that considers a specific sentence in its specific logical setting.<br>What a difference this makes can be studied with our mobile phones: The auto-complete function knows which word is most likely to follow the previous one. But without “understanding” the context, it starts outputting pure nonsense after three or four auto-completes (Fig. 1.5.1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*-4z8uwgGo9leBM7C31jNzA.png" /><figcaption>Fig. 1.5.1: Auto-complete function of mobile phones as an example for “static” context | image by author</figcaption></figure><p>I guess we all agree: this is not the kind of response we expect from ChatGPT and other LLMs, right?</p><p>As mentioned briefly before, words can have different meanings in different contexts. As a demonstration, I’d like to quote a nice example from Luis Serrano.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YeTNroUMneBiEPUWvxTPlA.png" /><figcaption>Fig. 1.5.2: “Static” word embedding of “apple”, “orange” and “phone” | image based on <a href="https://serrano.academy/">Serrano Academy</a></figcaption></figure><p>Fig. 1.5.2 shows the word embedding as we get it from GloVe. As you can see, the word “apple” is neither near to the word “phone” nor to the word “orange”.<br>Now, imagine that we want to embed the sentence “apple unveiled a new phone”. As humans, we immediately know that the word “apple” stands for the tech company. How do we know? The specific context tells us, in particular the word “phone” (Fig. 1.5.3 left).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*PJ_k8-j8jJaIhf8cDm_b_A.png" /><figcaption>Fig. 1.5.3: Two meanings of “apple” in different contexts | image based on <a href="https://serrano.academy/">Serrano Academy</a></figcaption></figure><p>Next, imagine we want to embed the sentence “bring me an apple and an orange”. Again, as humans, we immediately know that this time we are talking about the fruit. In particular, the word “orange” is very helpful (Fig. 1.5.3 right).</p><p>How can we teach the computer to understand the context? This is what the attention mechanism is about! The core idea of attention is to modify the word embeddings in the very specific context. Therefore, we move words closer to those words in the embedding space that have a contextual relation.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*wcx0eQknbOpaC8PikIKssw.png" /><figcaption>Fig. 1.5.4: Core idea of attention is to move related tokens closer in the embedding space | image by author</figcaption></figure><p>Fig. 1.5.4 illustrates this core idea. In cases where we talk about the fruit, we move the word embedding of “apple” closer to “orange” while we shift it to “phone” in cases where we talk about the tech company.</p><p>How can we teach the computer to understand that “apple” is in relation either with “phone” or with “orange”? We need to calculate the word affinities between “apple” and all the other words in the context — including itself. This tells us where there is a strong affinity and where there is a weaker affinity. In fact, we calculate the word affinities between all combinations of tokens in the context (Fig. 1.5.5).<br>Finally, according to the affinities, we modify the word embeddings and move tokens with a stronger context closer.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*roph5yvMm8xfUzmSMRNVoQ.png" /><figcaption>Fig. 1.5.5: Word affinities and modification of embeddings | image by author</figcaption></figure><p>OK, at this point, we know what we want to do once we know the word affinities. But how can we calculate them?<br>There are different methods for calculating the affinities — often called similarities. Here, we use the <strong>scaled dot product</strong>, which is also proposed in the paper “Attention Is All You Need”.</p><p>The scaled dot product is the dot product between the word embedding vectors <em>a</em> and <em>b</em> of the two tokens we evaluate, divided by the square root of the number of dimensions <em>d</em> of the word embedding vectors.</p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fmath.embed.fun%2F2Y4mcpjzbQ1x2dztrMyXa3&amp;type=text%2Fhtml&amp;schema=embed&amp;display_name=Embed+Fun&amp;src=https%3A%2F%2Fmath.embed.fun%2Fembed%2F2Y4mcpjzbQ1x2dztrMyXa3" width="278" height="94" frameborder="0" scrolling="no"><a href="https://medium.com/media/d7c4868ee693aeb1e5c6d3acd2976270/href">https://medium.com/media/d7c4868ee693aeb1e5c6d3acd2976270/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bwc80HqEdNONbGVFOy9QKg.png" /><figcaption>Fig. 1.5.6: Scaled dot product affinities in the context “bring me …” | image by author</figcaption></figure><p>Fig. 1.5.6 shows the scaled dot product affinities for all token combinations in the context “bring me an apple and an orange”. As we see in the last column of the table, the sums of the rows add up to values between approximately 10 and 18. For the calculation of new word embeddings, it is more convenient to have sums of exactly 1. To achieve this, we use the softmax function:</p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fmath.embed.fun%2FnxQ8uiPMWkM2DDXypvdtVj&amp;type=text%2Fhtml&amp;schema=embed&amp;display_name=Embed+Fun&amp;src=https%3A%2F%2Fmath.embed.fun%2Fembed%2FnxQ8uiPMWkM2DDXypvdtVj" width="245" height="94" frameborder="0" scrolling="no"><a href="https://medium.com/media/2273e24174599318493091b8fef4d242/href">https://medium.com/media/2273e24174599318493091b8fef4d242/href</a></iframe><p>According to Eq. 1.5.2, softmaxing means taking every value of a row to the e and dividing it by the sum of all values to the e. For our example, after applying softmax, we obtain the values shown in Fig. 1.5.7.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9Cby7dRFV7OMOfN_FgKBEg.png" /><figcaption>Fig. 1.5.7: Scaled dot product affinities after applying softmax | image by author</figcaption></figure><p>The affinities in Fig. 1.5.7 are the coefficients for calculating the context-adjusted „dynamic“ word embedding vectors: the higher the affinity, the greater the impact of a token on the new embedding. Let’s take „apple“ as an example (fourth row in Fig. 1.5.7):</p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fmath.embed.fun%2FcVh8zg84nhxsZ927QbcrAm&amp;type=text%2Fhtml&amp;schema=embed&amp;display_name=Embed+Fun&amp;src=https%3A%2F%2Fmath.embed.fun%2Fembed%2FcVh8zg84nhxsZ927QbcrAm" width="597" height="153" frameborder="0" scrolling="no"><a href="https://medium.com/media/2cc8d23bae8f64de2971650cb368a578/href">https://medium.com/media/2cc8d23bae8f64de2971650cb368a578/href</a></iframe><p>We see that the context-adjusted „dynamic“ embedding of „apple“ is the sum of the products of all tokens in the context multiplied by the affinity of „apple“ to this specific token. In our example, the word embedding for „apple“ remains only 63.1% as it was and is 36.9% modified to better fit to its context.<br>Please remember that „bring“ etc. in Eq. 1.5.3 refer to the embedding vectors. In vector notation, Eq. 1.5.3 looks as follows:</p><iframe src="https://cdn.embedly.com/widgets/media.html?url=https%3A%2F%2Fmath.embed.fun%2FneVsxbLE1GpwFTGMAUhViA&amp;type=text%2Fhtml&amp;schema=embed&amp;display_name=Embed+Fun&amp;src=https%3A%2F%2Fmath.embed.fun%2Fembed%2FneVsxbLE1GpwFTGMAUhViA" width="349" height="201" frameborder="0" scrolling="no"><a href="https://medium.com/media/4562d0629916d226d08886484f736280/href">https://medium.com/media/4562d0629916d226d08886484f736280/href</a></iframe><p>The word embeddings for all other tokens in the context are adjusted accordingly.</p><h4>1.5.2 Queries, Keys and Values Matrices</h4><p>So far, we have learned how to calculate the context-adjusted new word embeddings in principle. If we look into the paper “Attention Is All You Need”, we find a modified approach with so-called queries <em>Q</em>, keys <em>K</em> and values <em>V</em> matrices. Let’s explore the idea behind this approach.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*w7_YKhTbUFHJBZWQOTTZzw.png" /><figcaption>Fig. 1.5.8: Scaled dot product as described in “Attention Is All You Need” | image by <a href="https://arxiv.org/pdf/1706.03762">Vaswani et al.</a></figcaption></figure><p>With the queries <em>Q</em>, keys <em>K</em> and values <em>V</em> matrices, we add an extra piece of <strong>backpropagation learning </strong>to the attention mechanism. The matrices help the model find better word embeddings in the given context. For the moment, let’s concentrate on the queries <em>Q</em> and keys <em>K</em> matrices. The values <em>V</em> come later.</p><p>So far, we have calculated the word affinities with a simple dot product according to Eq. 1.5.1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*m4Oz8wY1qvdsiWARMDAN8A.png" /><figcaption>Fig. 1.5.9: Scaled dot product without Q and K matrices | image by author</figcaption></figure><p>Fig. 1.5.9 shows the calculation of the affinities in vector notation. For the moment, we concentrate on the multiplication and add the scaling with <em>sqrt(d)</em> in a later step. On the right side, we see how to calculate the dot product as a vector product: we take each entry of the apple embedding vector — now written as a row vector — and multiply it with the corresponding entries of the orange column vector, adding it all up. As a result, we receive a scalar.</p><p>If we recall from math, multiplying a vector with a matrix geometrically means a transformation. This transformation either stretches, compresses, rotates, or distorts the vector space.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*HSgfi3Llm0LeMDLRbAhgsQ.png" /><figcaption>Fig. 1.5.10: Q and K matrices transform the vector space | image based on <a href="https://serrano.academy/">Serrano Academy</a></figcaption></figure><p>In the example shown in Fig. 1.5.10, the space is only 2D, but in the attention algorithm, it can have any number of dimensions.<br>Why do we distort the vector space? The idea is to find a space that is ideal for representing the meanings of the tokens in the specific context.<br>In Fig. 1.5.10, we see that the left plane is OK for separating the two meanings of the word “apple”. The middle plane does a terrible job because, even if we move the word “apple” closer to the “phone” or the “orange”, we still have both representations very close. The third plane is clearly the best. It strongly supports separating the two meanings of “apple”.</p><p>With the queries <em>Q</em> and the keys <em>K</em> matrices, we let the transformer network find the optimal parameters for capturing the context of the tokens. Therefore, we add the two matrices to the calculation of the word affinities. Instead of multiplying the two embedding vectors directly, we multiply the product of the first embedding vector and the <em>Q</em> matrix (which is itself a vector) with the (transposed) product of the second embedding vector and the <em>K</em> matrix.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ii1sNR8B4FAtWu44d1aElg.png" /><figcaption>Fig. 1.5.11: Scaled dot product with Q and K matrices | image based on <a href="https://serrano.academy/">Serrano Academy</a></figcaption></figure><p>The product of the <em>Q</em> and <em>K</em> matrices in the middle of Fig. 1.5.11 (dotted box) is again a matrix that determines the linear transformation of the vector space. It is responsible for transforming the given space into the one that is optimal for the word embedding.</p><p>In summary, without the queries <em>Q</em> and keys <em>K</em> matrices, we use the given vector space. With <em>Q</em> and <em>K, </em>we find an optimal vector space and, consequently, a better dynamic word embedding according to the context of the input sentences.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*j71dKQWCcAjDOdYbNO1G5g.png" /><figcaption>Fig. 1.5.12: Comparison of scaled dot product with and without Q and K matrices | image based on <a href="https://serrano.academy/">Serrano Academy</a></figcaption></figure><p>Now, let me introduce the values matrix <em>V</em> and its function.<br>The vector space we found with the queries <em>Q</em> and keys <em>K</em> matrices is optimized for the word affinities. The values matrix <em>V</em> prepares the next step, which is calculating the probabilities of the next token following the given context. Therefore, we calculate a third linear transformation, which gives us a vector space optimized for this task.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QvYm6MrRfXnKgdqhcwu5PA.png" /><figcaption>Fig. 1.5.13: The values matrix V prepares for calculating the token probabilities | image based on <a href="https://serrano.academy/">Serrano Academy</a></figcaption></figure><p>So far, we have considered a toy example with only one token as input. With regard to the code example in part 2, I’d like to demonstrate how the matrix operations look like if we input a whole sentence. From there, it is easy to conclude how real-life matrix operations look.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ulKVY9f79RRK__9G_9VQfQ.png" /><figcaption>Fig. 1.5.14: Scaling up the multiplication Q dot K transposed for an input sentence | image by author</figcaption></figure><p>In Fig. 1.5.14, the red 7x3 matrix <em>X</em> represents the input sentence and its embeddings. Each row corresponds to a token. The three columns are the three dimensions of the word embedding. Remember, real-life word embeddings have hundreds or thousands of dimensions!<br>The green 3x4 matrix is our queries matrix <em>Q</em>, which adds extra learning capabilities to the attention model. The three rows are fixed because they need to correspond to the embedding dimension of <em>X</em>. The four columns are called “head-size” and are a hyperparameter of the transformer model. Theoretically, we can choose any value. The more columns, the greater the learning capabilities.</p><p>The keys matrix <em>K</em> is always the same size as the queries matrix <em>Q</em>. Therefore, it has the dimension 3x4, but in Fig. 1.5.14, it is transposed to 4x3. The same is true for the yellow input matrix <em>X</em>, which corresponds to the red input matrix.<br>The multiplication operation returns a 7x7 matrix with the unscaled affinities for each possible token combination in the input sentence.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*7F0SD0Jhnq3CgYUtRy3xZQ.png" /><figcaption>Fig. 1.5.15: Scaling up the matrix multiplication with the values matrix V | image by author</figcaption></figure><p>Fig. 1.5.15 continues the example from Fig. 1.5.14. The 7x7 matrix of unscaled token affinities is multiplied by the 7x3 input matrix <em>X</em>, which we have used earlier. The next step is to multiply the result by the 3x4 values matrix <em>V</em>. Again, the three rows correspond to the dimensions of the word embedding, and the four columns are the hyperparameter head-size. The multiplication results in a 7x4 matrix, which represents the word embeddings for calculating the probabilities of the next token to follow the input. We will explore in the next chapters how this is continued.</p><p>So far, we have omitted the scaling and softmaxing, but both steps are very simple and nearly identical to what we discussed in chapter 1.5.1.<br>Scaling is applied immediately after calculating the dot product of the queries matrix <em>Q</em> and the keys matrix <em>K</em>. We simply divide each value of the 7x7 matrix of unscaled affinities by the square root of the embedding dimension <em>sqrt(d)</em>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*3beNhvei2qX8LypcMOis-Q.png" /><figcaption>Fig. 1.5.16: Scaling the affinities by sqrt(d) | image by author</figcaption></figure><p>The next step after scaling is softmaxing. As with scaling, softmax is applied element-wise, and each element is divided by the sum of the respective row (as described in chapter 1.5.1).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tMbl_WS7dMj3HY5btZTC_Q.png" /><figcaption>Fig. 1.5.17: Softmaxing the scaled word affinities | image by author</figcaption></figure><p>Finally, let me summarize the steps of the attention matrix operations:</p><ol><li>We calculate the matrix product of <em>X</em> * <em>Q</em> * <em>K_transposed</em> * <em>X_transposed</em> and receive the unscaled affinities as a 7x7 matrix for seven input tokens.</li><li>We divide each element of the 7x7 matrix by <em>sqrt(d)</em>. This gives us the scaled affinities. Scaling is important for the softmax operation, as initial values should be small.</li><li>Masking is an optional step mentioned in the original paper but not applied in ChatGPT. We ignore it.</li><li>We softmax the matrix of scaled affinities. As a result, each row of the 7x7 matrix adds up to 1.</li><li>Finally, we multiply the scaled and softmaxed 7x7 matrix of word affinities with <em>X</em> * <em>V</em> and receive the context-adjusted word embeddings as the final result (Fig. 1.5.18).</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ZZNRBodAxkPzSuDFpaVNww.png" /><figcaption>Fig. 1.5.18: Multiplication of scaled and softmaxed word affinities with values matrix V | image by author</figcaption></figure><h4>1.5.3 Multi-head Attention</h4><p>All we have discussed so far is one single attention operation, called an attention head. But ChatGPT and other transformers have <em>h</em>-many of them. This means we calculate <em>h</em> queries, keys and values matrices in parallel. This gives us <em>h</em> attempts to find a good embedding, resulting in <em>h</em> 7x4 result matrices.<br>What do we do with the <em>h</em> result matrices? We concatenate them along the column axis and define a fully connected neutral network (“linear layer”) on top of the concatenated matrix with one node per column. The task of the linear layer is to weight the information in the concatenated result matrix: the useful information receive higher weights, while the less useful information is weighted less. This way, the transformer learns to pick the cherries.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*NdniecrDlO2ZtEI87eh3Ug.png" /><figcaption>Fig. 1.5.19: Multi-head attention | image by author</figcaption></figure><p>With this, we have discussed all steps of the attention mechanism and can proceed with the next steps of the transformer.</p><h4>1.6 Layer Norm</h4><p>The attention mechanism gives us the “dynamic” embedding vectors — including the token positions and the relations between the tokens. In the next steps, we intend to calculate probability values for all tokens in our dictionary for being the next output of the LLM. Before we do this, we need a more technical step: Layer Norm.<br>Layer Norm is a technique commonly used in deep learning models that improves the model’s <strong>stability</strong> and <strong>convergence</strong> during training. It normalizes the values over a specified dimension — in our case, the embedding vectors — to have a <strong>mean = 0 </strong>and a <strong>standard deviation ≈ 1</strong>.<br>Let’s study a small example. Imagine we have two embedding vectors with an embedding depth of 5. This gives us a tensor x of size (2, 5). For demonstration purposes, we instantiate the tensor with random numbers between 0 and 10.</p><pre>import torch<br>import torch.nn as nn<br><br># Define word embedding vectors<br>x = torch.rand(2, 5)*10<br><br>print(&quot;Original input tensor:&quot;)<br>print(x)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/643/1*zLpBTdVspcxoytEeq2EbLA.png" /></figure><p>Next, we instantiate a Layer Norm and define the input shape as the five values per embedding vector. We run the tensor x through the Layer Norm operation, output the transformed tensor normalized_x and check the new mean and the new standard deviation.</p><pre># Apply layer normalization<br>layer_norm = nn.LayerNorm(x.size()[1])<br>normalized_x = layer_norm(x)<br><br>print(&quot;Normalized input tensor:&quot;)<br>print(normalized_x)<br><br>print(&quot;\nMean:&quot;)<br>print(normalized_x.mean(dim=1))<br><br>print(&quot;\nStandard deviation:&quot;)<br>print(normalized_x.std(dim=1, unbiased=False))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/645/1*1eW5Ti-bJr4sTM-TGjWsJA.png" /></figure><p>We see that the values of normalized_x range from approximately -1.6 to +1.6, have a mean close to 0, and a standard deviation of 1.</p><h4>1.7 Feed Forward</h4><p>The feed forward layer is the computing engine in the transformer. It receives the normalized and context-adjusted (“dynamic”) word embeddings of our input and has the task of calculation probability values for the next token. It has as many input nodes as the embedding vectors have dimensions and one output node for each token in the vocabulary of the LLM. The layer is fully connected, so each embedding dimension has a relation to each token in the vocabulary.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*bP1iGavqZcJPrWCI9PVUSA.png" /><figcaption>Fig. 1.7.1: Feed forward neural network (fully connected) | image by author</figcaption></figure><p>Fig. 1.7.1 continues our example of “ChatGPT writes…”. We see the two input tokens “ChatGPT” and “writes”. Both tokens have embedding vectors of size <em>n</em> that feed their values into the <em>n</em> input nodes of the feed forward network. The task during model training is to learn the best weights that transform the embedding values to probability values for the next token to follow “ChatGPT writes…”.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*9MPkH8os6GoLld0l_KgiSQ.png" /><figcaption>Fig. 1.7.2: Calculation of logits | image by author</figcaption></figure><p>With a given set of weights in the neural network, we can calculate the logits as the output of the feed forward layer. The token with the highest value is the potential winner of the search for the next token. So, theoretically, we could stop at this point! But in fact, this is not how ChatGPT works. If we stopped here, ChatGPT would give the exact same answer to an identical prompt. However, as we know, ChatGPT provides slightly different answers if we input the same prompt. Consequently, we need some randomness in our model. To give the model this characteristic, the transformer architecture has two more steps: softmax and multinomial.</p><h4>1.8 Softmax</h4><p>Recall, we want to give the LLM an indeterministic behavior. The first step in achieving this is to transform the logits from the previous step (feed forward) into probabilities between 0 and 1. In Fig. 1.7.2, we see all logits outputted from the network. With softmax, we take every value to the e and divide it by the sum of all values to the e.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/554/1*7G2GYltk-bxojMAGUVB0ow.png" /><figcaption>Eq. 1.8.1: Softmax function</figcaption></figure><p>What is the effect of softmaxing?</p><ul><li>Taking every value to the e helps us operating with negative logit values. Negative values close to 0 are transformed to values close to 1, while negative values approaching negative infinity are transformed into 0.</li><li>Dividing each value by the sum of all values guarantees that the sum of all probabilities adds up to 1.</li></ul><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*pVKGeE3VsecatBvRZ7olvg.png" /><figcaption>Fig. 1.8.1: Probabilities of tokens coming from softmaxed logits | image by author</figcaption></figure><p>In summary, softmax transforms the logits to proper probabilities with values between 0 and 1, ensuring that the sum of all probabilities equals 1.</p><h4>1.9 Multinomial</h4><p>We have calculated the probabilities for each token in the vocabulary of the LLM. Now, it is time to draw a sample from a multinomial distribution function.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*4rDTa73MQKij8b19YndhOw.png" /><figcaption>Fig. 1.9.1: The next token is sampled from a multinomial distribution function | image by author</figcaption></figure><p>Fig. 1.9.1 illustrates the process of sampling:</p><ul><li>“aardvark” (the first word in the English dictionary) will be chosen with 0.1% likelihood.</li><li>“bottom” with 0.3%.</li><li>Most likely, “one” will be sampled because it has a likelihood of 95.1%.</li><li>“word” has a likelihood of 4.6%.</li><li>And “zygote” (the last word in the English dictionary) is almost impossible with a rounded likelihood of 0.0%</li></ul><p>Here, we assume that “one” will be selected as the output. But we have learned that this is not guaranteed, and the tiny probability of seeing other outputs leads to the intended indeterministic behavior of the LLM!</p><p><strong>Remember:</strong> Everything we discussed is one cycle through the transformer, and we have decided on the one token to output next. We repeat the same processes with the concatenated inputs again and again until the &lt;stop&gt; token is sampled as the next output!</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*YGU3B_KSy6LZG-W_mPv26g.png" /><figcaption>Fig. 1.9.2: The generation process stops when a &lt;stop&gt; token is selected as the next output. | image by author</figcaption></figure><h3>Part 2: Implementation in code</h3><p>So far, we have gone through a lot of theory. In the second part of the article, I want to demonstrate the theory in action and code with you a small GPT, including all components we have discussed before.<br>Please be aware that real-life applications run on ‘supercomputers’ and are trained with terabytes of data on thousands of GPUs for weeks. Here, we have only our PCs or laptops available, with one GPU at best. This means we need to scale down our expectations significantly. Nevertheless, the example will validate the theory and strongly deepen our understanding of the transformer architecture.</p><p>We are going to code a Fairy_Tale_GPT that learns English words from a collection of fairy tales (Brothers Grimm and H.C. Andersen). The data is freely available from the <a href="https://www.gutenberg.org/">Gutenberg Project</a>. In fact, the content of the data is less important than the availability of free-for-use text itself. You can use other data as well.<br>To keep the model relatively small, we will use the characters of the fairy tale texts as our tokens — not the words or word fragments. This keeps the model’s vocabulary small. I tried training the model on the word tokens as well — which works in principle — but the demand for training data is significantly higher than what we have available. Consequently, the learning success was very limited.</p><p>The Fairy_Tale_GPT is based on a project by <a href="https://karpathy.ai/">Andrej Karpathy</a> called nanoGPT. If you want to see Andrej’s original model and explanations, check out his <a href="https://www.youtube.com/watch?v=kCc8FmEb1nY">YouTube video</a>.</p><h4>2.1 Data Preparation</h4><p>First, we need the data. You can download the file ‘Fairy_Tales.txt’ from my <a href="https://www.dropbox.com/scl/fi/p28bf4fjuh0ofandwfgd8/Fairy_Tales.txt?rlkey=khbufbmoz7fl7g8o62scrze75&amp;st=co6w050n&amp;dl=0">Dropbox</a>. Here, I assume that the file is stored in the same directory as the Jupyter Notebook containing the code.</p><pre># Libraries we will use in this notebook<br>import torch<br>import torch.nn as nn<br>from torch.nn import functional as F<br>import matplotlib.pyplot as plt<br>from IPython.display import clear_output<br>torch.manual_seed(1337)<br><br># We load the text file&#39;s content into the variable &#39;text&#39;.<br>with open(&#39;Fairy_Tales.txt&#39;, &#39;r&#39;, encoding=&#39;utf-8&#39;) as f:<br>    text = f.read()</pre><p>Please familiarize yourself with the data. Since it is stored in a simple .txt file, you can even open the file and read the fairy tales.<br>Let us check the file size in characters and print the first 300 characters.</p><pre># Let&#39;s see what we have in &#39;text&#39;. How many characters?<br>print(&quot;Length of dataset in characters: &quot;, format(len(text),&#39;,&#39;))<br>print()<br><br># To get an impression of the file content, let&#39;s print the first 300 characters.<br>print(text[:300])</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*tmrJAOLZaPuv-sw4HjWEVg.png" /></figure><p>In text, we have a string of more than 807,000 characters. The fairy tales are stored one below the other, and the first fairy is THE GOLDEN BIRD.</p><h4>2.2 Tokenization</h4><p>As discussed in chapter 1.2, tokenization is the process of splitting the data into words, word fragments, or characters, finding the unique tokens, and assigning them unique numbers. The idea behind tokenization is to limit the size of the LLM’s vocabulary and prepare the model to process those tokens.<br>The first step of our tokenization process is to find the unique characters in the fairy tales.</p><pre># Find a list of all unique characters in the text.<br># The data type &#39;set&#39; eliminates all doubles.<br><br>chars = sorted(list(set(text)))<br>vocab_size = len(chars)<br><br>print(&#39;&#39;.join(chars))<br>print(&#39;\nSize of vacabulary: &#39;, vocab_size)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/744/1*wvbDOTP4R7mhjj_zSBkIiA.png" /></figure><p>We have 80 unique characters, including punctuation marks and other special characters. These are the base units the LLM needs to learn.</p><p>Next, we need an encoding and a decoding function to translate the tokens — in this case characters, in other applications words or word fragments — into token numbers. As a first step, we define two dictionaries: one for token → number and one for number → token.</p><pre># Dictionary: Characters (c) to numbers (i)<br>ctoi = {c:i for i,c in enumerate(chars)}<br><br># Dictionary: Numbers (i) to characters (c)<br>itoc = {i:c for i,c in enumerate(chars)}</pre><p>Before we proceed with the encoding and decoding functions, we should consider how we present the data to the LLM. Knowing that we have a limited amount of data, we should make the best use of it.<br>Later, we will split the data into a training and a validation dataset. Accepting, for example, the first 90% as training data and the remaining 10% as validation data would mean to validate with different fairy tales than used for training. This does not sound ideal. Therefore, we split our full dataset into paragraphs. This allows us to mix the paragraphs when dividing the data into training and validation datasets.<br>The split_text_into_paragraphs() function does exactly this.</p><pre># Split the full text into paragraphs of min 50 words<br># Return a list of lists<br><br>def split_text_into_paragraphs(text, min_words=50):<br><br>    lines = text.split(&#39;\n&#39;)<br>    current_paragraph = &quot;&quot;<br>    paragraphs = []<br><br>    for line in lines:<br><br>        # Add line to the current paragraph buffer<br>        current_paragraph += line + &quot;\n&quot;<br><br>        # If current paragraph has at least &#39;min_words&#39; words, store it and reset<br>        if len(current_paragraph.split()) &gt;= min_words:<br>            paragraphs.append([current_paragraph])<br>            current_paragraph = &quot;&quot;<br><br>    # Add left-overs<br>    if current_paragraph:<br>        paragraphs.append([current_paragraph])<br><br>    return paragraphs</pre><p>The function accepts a text variable and a minimum number of words min_words. It splits the text by the line-break separator \n and processes all text snippets, concatenating each snippet to the buffer variable current_paragraph. If the concatenated text is at least min_words long, it is considered a paragraph and added as a list. The function’s final output is a list of lists containing the text paragraphs of minimum length.</p><p>Now we can continue encoding and decoding.<br>The encode() function accepts the list of lists with strings as the paragraphs variable and the dictionary ctoi (read “c to i”). It enumerates the paragraphs, takes the texts, and encodes them based on the ctoi dictionary. Finally, the encodings are appended to a list and outputted. The function returns a list of lists of integers.</p><pre># Encode strings to list of integers<br>def encode(paragraphs, ctoi):<br>    &quot;&quot;&quot;<br>    Translates a list of lists with text into a <br>    list of lists with token numbers.<br>    &quot;&quot;&quot;<br>    encoded_paragraphs = []<br><br>    for paragraph in paragraphs:<br><br>        # Get text from inner list<br>        text = paragraph[0]<br><br>        # Translate tokens to integers<br>        encoded = [ctoi[c] for c in text if c in ctoi]<br><br>        # Add the encoded paragraph to the list<br>        encoded_paragraphs.append(encoded)<br><br>    return encoded_paragraphs<br><br><br># Decode list of numbers (li) to string (s)<br>def decode(li, itoc):<br>    <br>    &quot;Translates a list of integers back to a sting.&quot;<br>    <br>    # Translate integers to tokens<br>    tokens = [itoc[i] for i in li]<br>    <br>    # Join tokens to a string<br>    decoded_text = &quot;&quot;.join(tokens)<br><br>    return decoded_text</pre><p>The decode() function is simpler. It takes a list of integers li, and the itoc (“i to c”) dictionary, and decodes the integer token numbers into stings. Finally, the strings are concatenated to decoded_text and returned as output.<br>If you are following the code, please test the functions. You will see that they fulfill their tasks.</p><p>The next step is to encode the whole dataset. This transforms the list of lists of <strong>strings</strong> into a list of lists of <strong>integers</strong>.</p><pre># Encode the full dataset<br>data = encode(split_text_into_paragraphs(text), ctoi)</pre><p>Now, we shuffle the paragraphs (of integer token numbers) and split them into the training dataset train_data_list and the validation dataset val_data_list.</p><pre>import random<br><br># Shuffle data<br>random.shuffle(data)<br><br># Split data into train (90%) and val (10%)<br>n = int(0.9*len(data))<br>train_data_list = data[:n]<br>val_data_list = data[n:]</pre><p>So far, both datasets are Python lists of integer numbers. But for the LLM, we need PyTorch tensors. Since we are not going to further modify the validation dataset, we can transform it into a PyTorch tensor immediately. To do this, we flatten the list of lists of integers into a single list of integers flat_list and load the data into a PyTorch tensor.</p><pre># Flatten the list of lists into a single list<br>flat_list = [token for paragraph in val_data_list for token in paragraph]<br><br># Load validation data into PyTorch tensor<br>val_data = torch.tensor(flat_list, dtype=torch.long)</pre><p>We don’t do the equivalent operations for the training data yet. The reason is that I prefer to shuffle the paragraphs as part of the training loop. This gives the data a little more variance and data augmentation.</p><h4>2.3 Data Feeder Function</h4><p>The data feeder provides the model with the training data and the corresponding labels, or, in the case of validation, with the validation data.</p><p>First, we define three important model parameters.</p><ul><li>batch_size determines how many chunks of data are processed in parallel during one training loop. We set it to 64.</li><li>block_size defines the length of the context the model sees when it calculates the next token. We set it to 128 tokens.</li><li>device is either <em>‘cuda’</em> or <em>‘cpu’</em> and determines whether the model is processed on the CPU or the GPU of your computer.</li></ul><pre>batch_size = 64 # Number of independent sequences we process in parallel<br>block_size = 128 # Length of token sequences as context<br>device = &#39;cuda&#39; if torch.cuda.is_available() else &#39;cpu&#39; # Use GPU instead of CPU, if available<br>print(device)<br><br># Function, that provides the model with a batch of training or validation data<br># and the corresponding labels (the correct next token)<br><br>def get_batch(ValTrain):<br><br>    # Define data source<br>    if ValTrain == &quot;val&quot;:<br>        data = val_data<br>    else:<br>        # Flatten and shuffle the training data (data augmentation)<br>        shuffled_data = [<br>            token for paragraph in random.sample(train_data_list, len(train_data_list))<br>            for token in paragraph<br>            ]<br>        data = torch.tensor(shuffled_data, dtype=torch.long)       <br><br>    # Generate sliding windows over the data<br>    sliding_windows = [<br>        data[i:i + block_size + 1] <br>        for i in range(0, len(data) - block_size, block_size // 2)  # Step size = block_size // 2<br>    ]<br>    <br>    # Select batch_size many windows randomly<br>    selected_windows = random.sample(sliding_windows, batch_size)<br><br>    # Split each window into input (x) and target (y)<br>    x = torch.stack([window[:-1] for window in selected_windows])  # All but the last token<br>    y = torch.stack([window[1:] for window in selected_windows])  # All but the first token<br><br>    # Move tensors to device (CPU/GPU)<br>    x, y = x.to(device), y.to(device)<br><br>    return x, y</pre><p>The get_batch() function expects the variable ValTrain, which should be either <em>‘val’</em> or <em>‘train’</em>. If it is set to <em>‘val’</em>, the function returns validation data and simply loads the the tensor val_data into the data variable. In cases where ValTrain is set to any other value, the function returns training data. Remember, we have not prepared the training data tensor yet and need to do so now. To achieve this, we shuffle and flatten the list of lists of integer token numbers and load it into shuffled_data. Next, we load the data into a PyTorch tensor and save it in the data variable.</p><p>The next step in the get_batch() function is to create sliding windows over the data. Intentionally, the windows overlap by half of the block_size. We define the sliding windows as a list of lists, where each inner list contains block_size many tokens, and store them in sliding_windows. Next, we randomly select batch_size many samples and stack them into two-dimensional tensors with the training data x and the corresponding labels y, <strong>shifted by one token</strong>. This way, the labels y contain the next token following the training data x.</p><p>Both tensors, x and y, are shifted to device and returned as the results of the get_batch() function.</p><p>When we test the get_batch() function, we see that the input_data and the labels are tensors of size (64, 128) and that labels is shifted by one token to the right.</p><pre># Test the feeder function<br>input_data, labels = get_batch(&#39;train&#39;)<br><br>print(&quot;Shape of input_data:&quot;, input_data.shape, &quot;\n&quot;)<br>print(&quot;Input_data:\n&quot;, input_data, &quot;\n&quot;)<br>print(&quot;labels:\n&quot;, labels, &quot;\n&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/641/1*2QB3J0sOKmHH2Vy28yCl7A.png" /></figure><p>To confirm that everything works correctly, we decode the first row of input_data back into plain English.</p><pre># Decode the first row of the input data back into English<br>print(decode(input_data.cpu().numpy()[0],itoc))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/785/1*Sp9CDPtDtKcBVD7mKZGK9Q.png" /></figure><h4>2.4 Attention Head</h4><p>Next, we start coding the attention mechanism, beginning with a single attention head and the corresponding matrix operations as discussed in chapter 1.5.2.<br>The attention head is part of the multi-head attention, which itself is part of the attention block. Fig. 2.4.1 shows the attention heads’ positions in the multi-head attention. We will refer to the figure frequently in the next steps, so keep it in mind.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*UyglJvdR1CScDDORt9IiiA.png" /><figcaption>Fig. 2.4.1: Single attention head as part of multi-head attention | image by author</figcaption></figure><p>We start the coding with two additional hyperparameters:</p><ul><li>n_embd defines the embedding depth for all tokens (chapter 1.3). We set it to 192.</li><li>dropout is the percentage of parameters we randomly set to zero during training. This serves as a measure against overfitting. Since the model tends to memorize the data instead of generalizing the patterns (presumably due to insufficient size of dataset), we set dropout to the relatively high value of 0.4.</li></ul><p>The attention head is implemented as its own class called Head(), with a __init__() and a forward() method. In __init__(), we define the queries <em>Q</em>, keys <em>K,</em> and values <em>V</em> matrices, each as a nn.Linear() layer of size (n_embd, head_size). The nn.Linear() layer manages the weight matrices and processes the matrix multiplications of <em>Q</em>, <em>K</em> and <em>V</em> with the input data x. It is important to understand that the variables self.query, self.key, and self.value represent the matrix products <em>x @ Q</em>, <em>x @ K</em>, and <em>x @ V</em>, and not just the weight matrices (Fig. 1.5.14 and 1.5.15).<br>Additionally, in __init__() we define a register_buffer. This is a kind of method that saves a tensor to the module’s state dictionary but excludes it from training. This means the values are not updated through backpropagation. Here, we use it to store a lower triangular matrix tril of size (block_size, block_size) with values set to 1. We will use it later in the calculations.</p><pre># More hyperparameters<br>n_embd = 192 # The embedding depth for each token<br>dropout = 0.4 # The percentage of weights we set to 0 during training for regularization<br><br># Class for single attention head<br>class Head(nn.Module):<br>    &quot;&quot;&quot; Single head of self-attention &quot;&quot;&quot;<br><br>    def __init__(self, head_size):<br>        super().__init__()<br>        self.key = nn.Linear(n_embd, head_size, bias=False) # (C,H)<br>        self.query = nn.Linear(n_embd, head_size, bias=False) # (C,H)<br>        self.value = nn.Linear(n_embd, head_size, bias=False) # (C,H)<br>        self.register_buffer(&#39;tril&#39;, torch.tril(torch.ones(block_size, block_size))) # Lower triangular matrix<br>        self.dropout = nn.Dropout(dropout) # Ignore a portion of neurons per training loop --&gt; prevent overfitting<br><br>    def forward(self, x):<br>        B,T,C = x.shape   # C=n_embd<br><br>        k = self.key(x)   # x @ key (B,T,C) @ (C,H) --&gt; (B,T,H)<br>        q = self.query(x) # x @ query (B,T,C) @ (C,H) --&gt; (B,T,H)<br>        <br>        # compute attention scores (&quot;affinities&quot;)<br>        wei = q @ k.transpose(-2,-1) * C**-0.5 # (B,T,H) @ (B,H,T) -&gt; (B,T,T)<br>        wei = wei.masked_fill(self.tril[:T, :T] == 0, float(&#39;-inf&#39;)) # (B,T,T) # Fill with &#39;-inf&#39; where template has 0<br>        wei = F.softmax(wei, dim=-1) # (B,T,T)<br>        wei = self.dropout(wei) # (B,T,T)<br>        <br>        # Perform the weighted aggregation of the values<br>        v = self.value(x) # x @ value (B,T,C) @ (C,H) --&gt; (B,T,H)<br>        out = wei @ v # (B,T,T) @ (B,T,H) --&gt; (B,T,H)<br>        return out</pre><p>In the forward() method, we calculate the matrix products <em>x @ K</em> and <em>x @ Q</em> and save them in the variables k and q. Both results have the dimension <em>batch size</em> <em>(B)</em> x <em>number of tokens</em> (block size, <em>T</em>) x <em>head size</em> <em>H</em>, in short <em>(B, T, H)</em>. Please compare Fig. 1.5.14 and 1.5.15 in chapter 1.5.2 for reference.<br>Next, we calculate the scaled word affinities with <em>q @ k_transpose / sqrt(C)</em> and store them in wei (Fig. 1.5.16). This gives us a tensor of size <em>(B, T, T)</em>. Now, we use the lower triangular matrix tril (size (T, T)) as a template for <em>torch.masked_fill()</em> to set each value of wei where tril has the value 0 (above the diagonal) to <em>-inf</em>.<br>Why do we do this? In the next step, we apply softmax to wei, and softmaxing <em>-inf</em> results in 0. This effectively eliminate the word affinities for token combinations that have not been processed from the wei matrix. With this step we exclude knowledge in the training that the model cannot have in production.<br>With self.dropout(wei), we randomly set a specified share of weights to 0 as a measure against overfitting.<br>Finally, we calculate <em>x @ V</em> as v and the context-adjusted word embeddings with out = wei @ v. This is the return value of the function and is of size <em>(B, T, H)</em>.</p><h4>2.5 Multi-head Attention</h4><p>As described in chapter 1.5.3, multi-head attention utilizes <em>n_head</em> attention heads in parallel. Additionally, we apply a linear layer to weight the more beneficial attention head responses higher and the less beneficial lower.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*aq9yUIqVOGBbqCWmuBioAg.png" /><figcaption>Fig. 2.5.1: Process steps of multi-head attention | image by author</figcaption></figure><p>Fig. 2.5.1 shows all process steps of multi-head attention in the sequence of processing.</p><pre># Class that bundles 3 attention heads<br>class MultiHeadAttention(nn.Module):<br>    &quot;&quot;&quot; Three heads of self-attention in parallel &quot;&quot;&quot;<br><br>    def __init__(self, num_heads, head_size):<br>        super().__init__()<br>        self.heads = nn.ModuleList([Head(head_size) for _ in range(num_heads)]) # Just a container<br>        self.weighting = nn.Linear(n_embd, n_embd) # Linear layer to weight the attention heads<br>        self.dropout = nn.Dropout(dropout) # Prevent overfitting<br><br>    def forward(self, x):<br>        out = torch.cat([h(x) for h in self.heads], dim=-1) # Feed parallel attention heads and concatenate results<br>        out = self.dropout(self.weighting(out)) # Weighting the attention head results and dropout<br>        return out</pre><p>We bundle all multi-head activities in the class MultiHeadAttention().<br>In the __init__() method, we define a list of num_heads attention heads and save it in the variable self.heads. The linear layer for weighting the heads’ responses according to their usefulness is stored in self.weighting. Finally, we define another dropout layer in self.dropout.</p><p>In the forward() method, we take the normalized and position-enriched word embeddings of the input context as x and run them independently through the three attention heads. Then, we concatenate the responses along the column axis, resulting in a tensor of <em>3 x </em>head_size columns (which is equivalent to n_embd). This tensor is cached in out and fed into the linear layer self.weighting(), which regulates the weights of the responses. Lastly, the weighted head responses pass through dropout, where 40% of the tensor elements are set to zero to prevent overfitting.</p><h4>2.6 Feed Forward of Attention Block</h4><p>The feed forward layer in the attention block has a general computational purpose and consists of four steps. Therefore, we define it as its own class FeedForward().</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*plU1hLoih2OKixwQqT_pOw.png" /><figcaption>Fig. 2.6.1: Feed forward in the attention block | image by author</figcaption></figure><p>It takes the normalized multi-head responses as input x and passes them through two fully connected neural network layers: the first has a size of (n_embd, <em>4 x</em> n_embd), and the second has a size of (<em>4 x</em> n_embd, n_embd). This means the layers are expanded by factor 4 and compressed back to the original size — simply to add additional learnable weights. Between the layers, we include a ReLU activation function to introduce a non-linear behavior into the network.</p><pre># A linear layer for general calculation purpose<br><br>class FeedForward(nn.Module):<br>    &quot;&quot;&quot; Simple linear layer followed by a non-linearity &quot;&quot;&quot;<br><br>    def __init__(self, n_embd):<br>        super().__init__()<br>        self.net = nn.Sequential(           # Sequence of steps<br>            nn.Linear(n_embd, 4 * n_embd),<br>            nn.ReLU(),<br>            nn.Linear(4 * n_embd, n_embd),<br>            nn.Dropout(dropout))<br><br>    def forward(self, x):<br>        return self.net(x)</pre><p>In the final step, we apply dropout again as a measure against overfitting.</p><h4>2.7 Attention Block</h4><p>Now it’s time to compose the attention block from the previously defined components, as shown in Fig. 2.7.1.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*ix0jvUY6Es-AGKZXQ0fuTQ.png" /><figcaption>Fig. 2.7.1: Attention block | image by author</figcaption></figure><p>Again, we define a new class called Block(). In its __init__() method, we calculate the free dimension of the <em>Q</em>, <em>K</em> and <em>V</em> matrices — the head_size — as the quotient of embedding depth n_embd and the number of attention heads n_head. Then, we instantiate the MultiHeadAttention() class in self.sa and the FeedForward() class in self.ffwd. Additionally, we define two normalization layers in the variables self.ln1 and self.ln2.</p><pre># Only one pass-through. Loop is specified in the Transformer class<br><br>class Block(nn.Module):<br>    &quot;&quot;&quot; Attention block &quot;&quot;&quot;<br><br>    def __init__(self, n_embd, n_head):<br>        super().__init__()<br>        head_size = n_embd // n_head # Free dimension of key, query and value matrices<br>        self.sa = MultiHeadAttention(n_head, head_size)<br>        self.ffwd = FeedForward(n_embd)<br>        self.ln1 = nn.LayerNorm(n_embd)<br>        self.ln2 = nn.LayerNorm(n_embd)<br><br>    def forward(self, x):<br>        x = x + self.sa(self.ln1(x)) # Residual/skip connection<br>        x = x + self.ffwd(self.ln2(x)) # Residual/skip connection<br>        return x</pre><p>In the forward() method of the class, we receive the position-enriched word embeddings of the input to the LLM as x. We normalize x using self.ln1() and pass the result through the multi-head attention self.sa(). Then, we add a skip connection, meaning we add the original values of x — without normalization and self-attention — to the results. This stabilizes training and minimizes the problem of vanishing gradients.<br>Next, we pass the updated x through the second normalization layer self.ln2 and then through the feed forward class self.ffwd. We add another skip connection — this time the variable x contains the results before <em>layer norm 2</em>, not the original input tensors to forward() — and output the result.</p><h4>2.8 Transformer Class</h4><p>Before coding the <strong>Transformer()</strong> class, let’s recall how data flows through the architecture during training and in production.</p><p>Fig. 2.8.1 shows one cycle during <strong>training</strong>. We feed the input data into the model and assign it the task of predicting the next token following to the input data. The true next token serves as our label data. We run the input data through the model up to the final <em>Feed forward</em> layer, which gives us the logits representing the token probabilities.<br>Next, we use logits and true labels to compute the cross entropy loss. The higher the loss, the greater the need to update the model parameters. Using the loss function, we backpropagate through the model and update the weights.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*41grLaTk4-7sdIofqQ0zAQ.png" /><figcaption>Fig. 2.8.1: Training loop with backpropagation | image by author</figcaption></figure><p>During <strong>production,</strong> we do not have any labels. Instead, we are interested in the model’s output as the response to the user interaction. We feed the model with the input data and pass it through the model – this time with two additional steps compared to training. After the <em>Feed forward</em> layer, the logits are softmaxed and used in the <em>Multinomial</em> distribution function to sample the next token (chapter 1.8 and 1.9).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*RtWqTggNSPZAvoSdcSzKqQ.png" /><figcaption>Fig. 2.8.2: Generation cycle| image by author</figcaption></figure><p>According to the different approaches during training and production, the Transformer() class has three methods: __init__(), forward() and generate().</p><p>Again, we start with two additional hyperparameters:</p><ul><li>n_head defines the number of parallel attention heads in the <em>Multi-head attention</em>. As mentioned earlier, this variable is set to 3.</li><li>n_layer specifies the number of sequential attention blocks (e.g. Fig. 2.8.2). In our model, this is set to 4.</li></ul><p>Within the __init__() method of the Transformer() class, we define the word embedding and the positional encoding (chapter 1.3 and 1.4). For both, we use of the nn.Embedding method, which creates a lookup table of the specified size and assigns the input values uniquely to the rows of this table.</p><ul><li>For the the word embedding, the lookup table has the size of vocab_size <em>x</em> n_embd, meaning that each token in the vocabulary corresponds to a specific row with the word embedding values.</li><li>For the positional encoding, the lookup table has the size block_size <em>x</em> n_embd, where each row of the table represent the position of the token in the input context. Since we consider block_size many tokens in the context, we need to differentiate as many positions. The number of columns again equals the embedding depth.</li></ul><p>In both cases — word embedding and positional encoding — nn.Embedding initializes the tables with random values, which remain fixed throughout the life of the LLM.</p><p>The next step in __init__() is to define a sequence of n_layer <em>Attention blocks</em> in the variable self.blocks. Since n_layer is set to 4, the data passes through 4 <em>Attention blocks</em> sequentially before continuing to the final <em>Layer Norm</em> (Fig. 2.8.1 and 2.8.2).<br>Next, we define the final <em>Layer Norm</em> and the <em>Feed forward</em> layer as a nn.linear layer in the variables self.final_ln and self.final_ff. The <em>Feed forward</em> layer translates the context-adjusted word embeddings into the probabilities of each token in the vocabulary and is of equivalent size (chapter 1.7 and Fig. 1.7.1).</p><pre># More hyperparameters<br>n_head = 3 # Number of attention heads in multi-head attention<br>n_layer = 4 # Number of attention blocks<br><br># Main class embracing all modules<br>class Transformer(nn.Module):<br><br>    # When we instantiate from the Transformer class<br>    def __init__(self):<br>        super().__init__()<br>        self.token_embedding_table = nn.Embedding(vocab_size, n_embd) # Word/token embedding<br>        self.position_embedding_table = nn.Embedding(block_size, n_embd) # Positional embedding<br>        self.blocks = nn.Sequential(*[Block(n_embd, n_head) for _ in range(n_layer)]) # Stack of attention blocks<br>        self.final_ln = nn.LayerNorm(n_embd) # Final layer norm<br>        self.final_ff = nn.Linear(n_embd, vocab_size) # Final linear layer<br><br>    # When we pass data through an instance of the Transformer class<br>    def forward(self, input, targets=None): # input and targets are both (B,T)-dimensional tensors of integers<br>        <br>        B, T = input.shape # Dimensions of input data: batch x tokens<br>        tok_emb = self.token_embedding_table(input) # (B,T,C)<br>        pos_emb = self.position_embedding_table(torch.arange(T, device=device)) # (T,C)<br>        x = tok_emb + pos_emb # (B,T,C), Python automatically adds dimension B to pos_emb<br>        x = self.blocks(x) # (B,T,C)<br>        x = self.final_ln(x) # (B,T,C)<br>        logits = self.final_ff(x) # (B,T,vocab_size)<br><br>        # Only if targets are defined --&gt; loss calculation<br>        if targets is None:<br>            loss = None<br>        else:<br>            B, T, C = logits.shape # Get dimensions of output<br>            logits = logits.view(B*T, C) # Transform to two dimensions for cross_entropy function<br>            targets = targets.view(B*T)  # Transform to one dimension for cross_entropy function<br>            loss = F.cross_entropy(logits, targets) # Calculate the losses<br><br>        return logits, loss<br><br>    # When we generate new text (production)<br>    def generate(self, idx, max_new_tokens):<br>        # idx is (B, T) tensor of indices<br>        for _ in range(max_new_tokens): # Concatenate max_new_tokens outputs<br>            # Crop idx to the last block_size tokens<br>            idx_cond = idx[:, -block_size:]<br>            # Get the predictions<br>            logits, loss = self.forward(idx_cond)<br>            # Focus only on the last token<br>            logits = logits[:, -1, :] # becomes (B, C)<br>            # Apply softmax to get probabilities<br>            probs = F.softmax(logits, dim=-1) # (B, C)<br>            # Sample from the distribution<br>            idx_next = torch.multinomial(probs, num_samples=1) # (B, 1)<br>            # Append sampled index to the running sequence<br>            idx = torch.cat((idx, idx_next), dim=1) # (B, T+1)<br>        return idx</pre><p>The forward() method is used for the training of the LLM and is also called in the generate() method. It receives the input data and, optionally, label data in the variable targets. Both are tensors of size (B, T), meaning one row for each of the batch_size many samples and one column for each of the block_size many tokens.<br>We pass the input data through the self.token_embedding_table(), which adds the corresponding word embedding values from the lookup table to each token in all of the batches in a separate dimension. This extends the size of the tensor from (B, T) to (B, T, C), where C represents the embedding depth.<br>Similarly, self.position_embedding_table() adds the embedding values to each of the T positions 0, 1, …, T-1 in the input context, resulting in a tensor of size (T, C). Next, we add the token embedding tok_emb and the positional embedding pos_emb to form the variable x. Python automatically extends the pos_emb tensor by the dimension B and repeats the content of the original tensor B-times. Thus, x has the size (B, T, C).</p><p>According to Fig. 2.8.1, the next step is the <em>Attention block</em>. We pass x through self.blocks(), then through the final <em>Layer Norm</em> self.final_ln(), and finally through the final <em>Feed forward</em> layer self.final_ff(), which gives us the tensor with the logits of size (B, T, vocab_size). As discussed earlier, the linear layer translates the word embeddings into probabilities values for the tokens in the vocabulary.</p><p>If targets is specified — during training — the next step is to transform the logits from a 3-dimensional tensor to 2 dimensions. With logits.view(B*T, C), the B batches of data are stacked on top of of each other (caution: C does now represent the vocab size). We apply the same transformation on targets, reducing it from 2D to 1D. Both transformation are required to match the input format of F.cross_entropy(), which calculates the loss value. As always, the goal of the training process is to minimize the loss. This is equivalent to getting the best prediction quality for the next token to follow the input context stored in targets.</p><p>During production, we use the generate() method. It receives a tensor of context indices idx and the number of tokens to generate max_new_tokens. The tensor idx has the size (B, T). This means we can generate B batches of outputs in parallel, but B can also be 1 if idx has only one row. The indices in idx represent the token numbers of the input context according to the dictionaries <em>itoc</em> and <em>ctoi</em> we specified in the tokenization (chapter 2.2). The parameter max_new_tokens is an “artificial” one — you will not find it in real life LLMs. We need it because we have no stop tokens in our training data, therefore we need a “hard” way to stop the process of text generation.<br>In generate() we have a loop with max_new_tokens repetitions. Inside, we condition idx in case it has more tokens than our specified block_size. In this case, we take the last block_size many tokens. Next, we call self.forward() with the input idx_cond. The method returns the logits and the loss (we do not use the loss any further). The logits tensor has the size (B, T, C) (C represents the vocab size) and contains the predicted probability values for each token in the input context — even for those where we know the next token because we can read it in the user input. That is why we limit logits to the last token with logits[:, -1, :]. This step reduces the tensor’s dimension to (B, C).<br>According to Fig. 2.8.2 the next steps are <em>Softmax</em> and <em>Mutinomial</em>. Correspondingly, we softmax the logits over the C dimension. Now, we have real probabilities, adding up to 1 over the vocabulary, and save them in probs. Next, we input probs into torch.multinomial() which samples one index according to the given probabilities. This new index is the essence of all we did because it represents the next token! We concatenate it to the given context in idx and repeat the process max_new_tokens times.</p><h4>2.9 Instantiate the Transformer</h4><p>We have defined all classes of the Fairy_Tale_GPT. So, we can instantiate it and give it a first try before we start the model training.<br>Instantiating is very simple. We call the Transformer() class and push it to device. For our information, we enumerate the model.parameters() with a generator comprehension and add up all elements. This gives us the number of parameters in our LLM.</p><pre># Instantiate an object from the Transformer class<br>model = Transformer().to(device) # &#39;model&#39; lives on the device, in my case the GPU<br><br># Print the number of parameters of the model<br>print(format(sum(p.numel() for p in model.parameters()),&quot;,&quot;), &#39;parameters&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/528/1*noMAKgaT2ezbcnTx7MkP9g.png" /></figure><p>We see that our Fairy_Tale_GPT has a little more than 1.8 million parameters.</p><p>Now, I am curious to see a first output of the LLM — knowing that the model hasn’t seen any training yet. The generate method requires a tensor of starting indices as the context. Therefore, we define a 2-d tensor containing only one 0 (random choice) and store it in context. Next, we call the generate method of the instance with model.generate(), provide it with the context and specify the max_new_tokens parameter as 200. With [0] we unpack the first batch (technically required, although we have only one batch) and transform the given PyTorch tensor into a Python list with .tolist(). Finally, we decode() the list of integer token numbers into characters according to our dictionary itoc (chapter 2.2), save the result in untrained_results and print it.</p><pre># Generate from the model<br>context = torch.zeros((1, 1), dtype=torch.long, device=device)<br>untrained_results = decode(model.generate(context, max_new_tokens=200)[0].tolist(),itoc)<br>print(untrained_results)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/778/1*W_aJzPSvy6lHAnNFMSbt5w.png" /></figure><p>Well, the output does not exactly look like English words from fairy tales, does it? It is more a sequence of purely random characters from our vocabulary. Let’s check if training can improve the result.</p><h4>2.10 Model Training</h4><p>Before we actually code the training loop for the LLM, we define some more hyperparameters and a helper function for calculating the losses during training.</p><ul><li>eval_iters defines how many training iterations are averaged in the calculation of the losses. We set it to 10.</li><li>learning_rate is the step width for updating the model parameters and is set to 1e-3. Since we use a learning rate scheduler, learning rate is only the starting value. The actual learning rate is reduced according to a cosine function during training to improve the model stability and prevent overfitting.</li><li>max_iters defines the number of training loops. It is 10,000 iterations.</li><li>eval_interval defines after how many iterations the training code prints the actual loss values, outputs the corresponding plot, and a text sample. We receive those updates every 200 iterations.</li><li>context_tensor is the matrix of indices we already used in chapter 2.9. It is required for the sample text generations during the training.</li></ul><pre># Cosine annealing learning rate scheduler<br>from torch.optim.lr_scheduler import CosineAnnealingLR<br><br># More hyperparameters<br>eval_iters = 10 # How many iterations do we average in the loss calculation<br>learning_rate = 1e-3 # Starting step size for learning<br>max_iters = 10000 # Number of training loops<br>eval_interval = 200 # How often evaluate the model performance<br>context_tensor = torch.zeros((1, 1), dtype=torch.long, device=device)<br><br><br># Function calculates losses and averages results over eval_interval values<br>@torch.no_grad() # Do not calculate any gradients for this function<br>def estimate_loss():<br>    out = {} # Empty dictionary for the results<br>    model.eval() # Set model to evaluation mode<br>    # Calculate loss for training and validaton data<br>    for split in [&#39;train&#39;, &#39;val&#39;]:<br>        losses = torch.zeros(eval_iters) # Set to 0 for start<br>        for k in range(eval_iters): # 10 loops<br>            X, Y = get_batch(split) # Get training data<br>            logits, loss = model(X, Y) # Call model and get logits and losses<br>            losses[k] = loss.item()<br>        out[split] = losses.mean() # Save average under key &#39;train&#39; or &#39;val&#39;<br>    model.train() # Set model in training mode<br>    return out</pre><p>The estimate_loss() function computes the validation and the training loss during the model training. The decorator @torch.no_grad() turns off the gradient tracking in PyTorch for the decorated function.</p><p>Inside estimate_loss() are two nested loops. The outer loop iterates over the two values ‘train’ and ‘val’ which specify training and validation. The inner loop iterates over the range(eval_iters) (which means 10 iterations) and calls the get_batch() function either for ‘train’ or for ‘val’. The returned data is fed into the model() which returns the logits and the loss. Here, we are interested in the loss and save it to the tensor losses. Outside the inner loop we compute the average over the ten values in losses and save it either in out[‘train’] or out[‘val’]. Finally, out is the return value of the function.</p><p>For the model training we define an optimizer and a learning rate scheduler. The optimizer updates the model parameters according to the gradients from backpropagation and the learning rate. We choose the AdamW optimizer which is very common. The AdamW optimizer allows to specify weight_decay which adds the sum of all weights (times the specified factor 0.03) to the loss function. This pushes the model to prefer smaller weights — again a measure against overfitting.<br>As described in the hyperparameter definition, the learning rate scheduler reduces the learning rate from learning_rate (1e-3) to eta_min (1e-5) over the course of max_iters (10,000) training iterations.</p><pre># Create a PyTorch optimizer<br>optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate, weight_decay=0.03)<br><br># Cosine annealing learning rate scheduler<br>scheduler = CosineAnnealingLR(optimizer, T_max=max_iters, eta_min=1e-5)  # Minimal LR is 1e-5</pre><p>Inside the training loop we use three lists to collect the losses. All three lists are not essential to the training itself but used to inform about the training progress every eval_interval iterations. In loss_lst_train we collect the training loss, in loss_lst_val the validation loss and in loss_lst_x we store the actual iteration numbers for the x-axis of the plot.</p><p>The training loop runs from 0 to max_iters(-1) and stores the actual loop counter in iter. We call the get_batch() function for training data and store the returned input data in xb and the corresponding labels in yb. Both are fed into the model which returns the logits and the loss. Next, we set all gradients to <em>None</em> with optimizer.zero_grad(set_to_none=True) and compute the new gradients with loss.backward(). We update the model parameters with optimizer.step() and reduce the learning rate incrementally according to a cosine function with scheduler.step().</p><pre># Empty lists of losses<br>loss_lst_train = []<br>loss_lst_val = []<br>loss_lst_x = []<br><br># Train the model over max_iter loops<br>for iter in range(max_iters):<br><br>    # Get a batch of training data<br>    xb, yb = get_batch(&#39;train&#39;)<br>    <br>    # Run the transformer model<br>    logits, loss = model(xb, yb)<br><br>    # Zero the gradients<br>    optimizer.zero_grad(set_to_none=True)<br><br>    # Calculate the gradients<br>    loss.backward()<br><br>    # Optimize parameters through backpropagation<br>    optimizer.step()<br><br>    # Update the learning rate with the scheduler<br>    scheduler.step()<br>    <br>    # Evaluate and print losses<br>    if iter % eval_interval == 0 or iter == max_iters - 1:<br>        <br>        # Caluculate losses<br>        losses = estimate_loss() # Call evaluation function<br>        loss_lst_train.append(losses[&#39;train&#39;].item()) # Append to training list<br>        loss_lst_val.append(losses[&#39;val&#39;].item()) # Append to validaton list<br>        loss_lst_x = list(range(len(loss_lst_train))) # Prepare the x values for plotting<br><br>        # Plot<br>        clear_output(wait=True) # Clear output in jupyter<br>        print(f&quot;Step {iter}: train loss {losses[&#39;train&#39;]:.4f}, val loss {losses[&#39;val&#39;]:.4f}&quot;)<br>        plt.figure(figsize=(5,3))<br>        plt.plot(loss_lst_x,loss_lst_train,label=&#39;train&#39;)<br>        plt.plot(loss_lst_x,loss_lst_val,label=&#39;val&#39;)<br>        plt.xlabel(&#39;steps (x&#39; + str(eval_interval) + &#39;)&#39;)<br>        plt.ylabel(&#39;losses&#39;)<br>        plt.title(&#39;Training and validation loss&#39;)<br>        plt.legend()<br>        plt.show()<br><br>        # Do a test generation to observe the quality<br>        print(&#39;Test word generation:&#39;)<br>        print(decode(model.generate(context_tensor, max_new_tokens=200)[0].tolist(),itoc))</pre><p>All operations inside the if-condition are optional and used to inform about the progress during training. In the first step we call the estimate_loss() function and collect the losses in the specified lists. In the middle part of the code, we clear the output in the Jupyter notebook with clear_output() and plot the training and validation loss as a matplotlib line chart over the iteration numbers. In the lower part we generate a test text in the same way as we did in chapter 2.9.</p><p>The output of the code at the end of the 10,000 training iterations looks something like this:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/784/1*VIeClrz6H76bmT0zGm4LVg.png" /></figure><p>After model training, we should save the parameters. Otherwise they will be lost after deleting the model from the PC’s memory and we would have to restart the training.</p><pre>torch.save(model.state_dict(), &#39;Name_of_your_choice.pth&#39;)</pre><p>The model.state_dict() saves the model’s parameters but not the model architecture. We need to load this first before we can load the parameters.</p><h4>2.11 Generate new Tokens</h4><p>Before we start the token generation, let us load the model parameters. Here, we assume that model itself is already loaded.</p><pre># Load the saved model state<br>state_dict = torch.load(&#39;Name_of_your_choice.pth&#39;)<br><br># Load the parameters into the transformer model<br>model.load_state_dict(state_dict)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/821/1*Rc9V3pJAwu2BicFydqixGA.png" /></figure><p>Now, let’s compare the LLM’s output before and after the training.</p><pre># Print output before training<br>print(&#39;Model output before training:&#39;)<br>print(untrained_results)<br><br># Print output after training<br>context = torch.zeros((1, 1), dtype=torch.long, device=device)<br>trained_results = decode(model.generate(context, max_new_tokens=500)[0].tolist(),itoc)<br>print(&#39;\nModel output after training:&#39;)<br>print(trained_results)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/787/1*GpR4AHlyybO8TsE2xfD_jg.png" /></figure><p>For the output of the untrained LLM we use the variable untrained_results, which we filled before the training. The text trained_results is generated ‘right now’ through the model.generate() method. Please check chapter 2.9 for an explanation of the line of code.</p><p>While in untrained_results the sequence of characters is purely random, we can clearly recognize an English word structure in trained_results, although some words are fantasy and the word order is often incorrect. Please remember that we taught the model only character sequences, not word sequences. Clearly, we would not accept a similar answer from a real world LLM. Nevertheless, I hope you accept the result as a general proof that the transformer architecture like implemented in our toy project works in principle.</p><p>We can also give a more meaningful context than just a 2d-tensor with one 0 in it. To do so, we first encode the context sentence, transform it into a PyTorch tensor and input the context tensor to the model.generate() method.</p><pre># Encode a starting sentence<br>text = [[&quot;The king was very sad and tired&quot;]]<br>context = encode(text, ctoi)<br><br># Transform context to PyTorch tensor<br>context_tensor = torch.tensor([context], dtype=torch.long, device=device)<br><br># Generate new text in response to context sentence<br>print(decode(model.generate(context_tensor[:,0,:], max_new_tokens=500)[0].tolist(),itoc))</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/786/1*kn_D8eoyqlACPSJhD0ET9A.png" /></figure><h3>Conclusion</h3><p>The transformer architecture, originally presented in <a href="https://arxiv.org/pdf/1706.03762">“Attention Is All You Need”</a>, sparked a rapid evolution of Large Language Models and other generative AI tools with fascinating performances. The evolution is far from over, and we see new and improved solutions emerging almost every month.</p><p>In part 1 of this article, we studied the logic and math behind the transformer model in depth. The core of the transformer is the attention mechanism, which captures the context in which words or tokens are used. Context is essential to correctly understand the meaning of words and sentences. The attention mechanism saves the recognized context through an adjustment of the word embedding vectors. These modified embedding vectors are further processed in fully connected neural networks to find the next token as the output of the LLM.</p><p>In part 2, we coded all steps of the transformer architecture in a toy project, the Fairy_Tale_GPT. The intention was to deepen the conceptual understanding and to demonstrate the theory in action. The example basically works, but at the same time, it illustrates how demanding human language is for computer models. In real applications, LLM model sizes exceed the trillion-parameters limit and are trained on terabytes of data.</p><p>I hope you gained a solid understanding of how Large Language Models work and enjoyed the journey. Let’s be curious about what comes next in this fascinating field of research!</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=77e44e19f466" width="1" height="1" alt=""><hr><p><a href="https://pub.towardsai.net/from-concept-to-code-unveiling-the-chatgpt-algorithm-77e44e19f466">From Concept to Code: Unveiling the ChatGPT Algorithm</a> was originally published in <a href="https://pub.towardsai.net/">Towards AI</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Building a Vision Inspection CNN for an Industrial Application]]></title>
            <link>https://medium.com/data-science/building-a-vision-inspection-cnn-for-an-industrial-application-138936d7a34a?source=rss-2628a598423a------2</link>
            <guid isPermaLink="false">https://medium.com/p/138936d7a34a</guid>
            <category><![CDATA[cnn]]></category>
            <category><![CDATA[grad-cam]]></category>
            <category><![CDATA[vision-inspection]]></category>
            <category><![CDATA[hands-on-tutorials]]></category>
            <category><![CDATA[convolutional-neural-net]]></category>
            <dc:creator><![CDATA[Ingo Nowitzky]]></dc:creator>
            <pubDate>Thu, 21 Nov 2024 19:33:33 GMT</pubDate>
            <atom:updated>2024-11-21T19:33:33.417Z</atom:updated>
            <content:encoded><![CDATA[<h4>A step-by-step approach using PyTorch</h4><p><strong>In this article, we develop and code a Convolutional Neural Network (CNN) for a vision inspection classification task in the automotive electronics industry. Along the way, we study the concept and math of convolutional layers in depth, and we examine what CNNs actually see and which parts of the image lead them to their decisions.</strong></p><h3>Table of Content</h3><p>Part 1: Conceptual background<br>Part 2: Defining and coding the CNN<br>Part 3: Using the trained model in production<br>Part 4: What did the CNN consider in its “decision”?</p><h3>Part 1: Conceptual background</h3><h4>1.1 The task: Classify an industrial component as good or scrap</h4><p>In one station of an automatic assembly line, coils with two protruding metal pins have to be positioned precisely in a housing. The metal pins are inserted into small plug sockets. In some cases, the pins are slightly bent and therefore cannot be joined by a machine. It is the task of the visual inspection to identify these coils, so that they can be sorted out automatically.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*d2sHQh31o-dMaX4GNyzzvw.png" /><figcaption>Fig. 1: Coils, housing and sockets | image by author</figcaption></figure><p>For the inspection, each coil is picked up individually and held in front of a screen. In this position, a camera takes a grayscale image. This is then examined by the CNN and classified as good or scrap.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*8WFSIPEGWWcXfNs_ofFD1Q.png" /><figcaption>Fig. 2: Basic setup of visual inspection and resulting image | image by author</figcaption></figure><p>Now, we want to define a convolutional neural network that is able to process the images and learn from pre-classified labels.</p><h4>1.2 What is a Convolutional Neural Network (CNN)?</h4><p>Convolutional Neural Networks are a combination of <strong>convolutional filters</strong> followed by a <strong>fully connected Neural Network</strong> (NN). CNNs are often used for <strong>image processing,</strong> like face recognition or visual inspection tasks, like in our case. <strong>Convolutional filters</strong> are matrix operations that slide over the images and recalculate each pixel of the image. We will study convolutional filters later in the article. The <strong>weights</strong> of the filters are <strong>not preset</strong> (as, e.g. the sharpen function in Photoshop) but instead are learned from the data during training.</p><h4>1.3 Architecture of a Convolutional Neural Network</h4><p>Let’s check an example of the architecture of a CNN. For our convenience, we choose the model <strong>we will implement</strong> later.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*Ge8zPVMM_79XlVFrdDrbjA.png" /><figcaption>Fig. 3: Architecture of our vision inspection CNN | image by author</figcaption></figure><p>We want to feed the CNN with our inspection images of size 400 px in height and 700 px in width. Since the images are grayscale, the corresponding PyTorch tensor is of size 1x400x700. If we used a colored image, we would have 3 incoming channels: one for red, one for green and one for blue (RGB). In this case the tensor would be <strong>3</strong>x400x700.</p><p>The first <strong>convolutional filter</strong> has 6 <strong>kernels</strong> of size 5x5 that slide over the image and produce 6 independent new images, called <strong>feature maps,</strong> of slightly reduced size (6x396x696). The <strong>ReLU activation</strong> is not explicitly shown in Fig. 3. It does not change the dimensions of the tensors but sets all negative values to zero. ReLU is followed by the <strong>MaxPooling</strong> layer with a kernel size of 2x2. It halves the width and height of each image.</p><p>All three layers — convolution, ReLU, and MaxPooling — are implemented a second time. This finally brings us 16 feature maps with images of height 97 pixels and width 172 pixels. Next, all the matrix values are flattened and fed into the equally sized first layer of a fully connected neural network. Its second layer is already reduced to 120 neurons. The third and output layer has only 2 neurons: one represents the label “OK”, and the other the label “not OK” or “scrap”.</p><p><strong>If you are not yet clear about the changes in the dimensions, please be patient.</strong> We study how the different kinds of layers — convolution, ReLU, and MaxPooling — work in detail and impact the tensor dimensions in the next chapters.</p><h4>1.4 Convolutional filter layers</h4><p>Convolutional filters have the task of finding <strong>typical structures/patterns</strong> in an image. Frequently used kernel sizes are 3x3 or 5x5. The 9, respectively 25, weights of the kernel are not specified upfront but learned during training (here we assume that we have only one input channel; otherwise, the number of weights multiply by input channels). The kernels slide over the matrix representation of the image (each input channel has its own kernel) with a defined <strong>stride</strong> in the horizontal and vertical directions. The corresponding values of the kernel and the matrix are multiplied and summed up. The summation results of each sliding position form the new image, which we call the <strong>feature map</strong>. We can specify multiple kernels in a convolutional layer. In this case, we receive multiple feature maps as the result. The kernel slides over the matrix from left to right and top to bottom. Therefore, Fig. 4 shows the kernel in its <strong>fifth sliding position</strong> (not counting the “…”). We see three input channels for the colors red, green, and blue (RGB). Each channel has one kernel only. In real applications, we often define multiple kernels per input channel.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tx0JWmrXI6kxT5_llBnXoA.png" /><figcaption>Fig. 4: Convolutional layer with 3 input channels and 1 kernel per channel | image by author</figcaption></figure><p>Kernel 1 does its work for the red input channel. In the shown position, we compute the respective new value in the feature map as <em>(-0.7)*0 + (-0.9)*(-0.2) + (-0.6)*0.5 + (-0.6)*0.6 + 0.6*(-0.3) + 0.7*(-1) + 0*0.7 + (-0.1)*(-0.1) + (-0.2)*(-0.1) = (-1.33).</em> The respective calculation for the green channel (kernel 2) adds up to <em>-0.14,</em> and for the blue channel (kernel 3) to <em>0.69</em>. To receive the final value in the feature map for this specific sliding position, we sum up all three channel values and add a bias (bias and all kernel weights are defined during training of the CNN): <em>(-1.33) + (-0.14) + 0.69 + 0.2 = -0.58</em>. The value is placed in the position of the feature map highlighted in yellow.</p><p><strong>Finally, if we compare the size of the input matrices to the size of the feature map, we see that through the kernel operations, we lost two rows in height and two columns in width.</strong></p><h4>1.5 ReLU activation layers</h4><p>After the convolution, the feature maps are passed through the activation layer. Activation is required to give the network <strong>nonlinear capabilities</strong>. The two most frequently used activation methods are <strong>Sigmoid</strong> and <strong>ReLU</strong> (Rectified Linear Unit). ReLU activation sets all negative values to zero while leaving positive values unchanged.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*tjaJCcac0nZMjPAhVAc3tg.png" /><figcaption>Fig. 5: ReLU activation of the feature map | image by author</figcaption></figure><p>In Fig. 5, we see that the values of the feature map pass the ReLU activation element-wise.</p><p><strong>ReLU activation has no impact on the dimensions of the feature map.</strong></p><h4>1.6 MaxPooling layers</h4><p>Pooling layers have mainly the task of reducing the size of the feature maps while keeping the important information for the classification. In general, we can pool by calculating the <strong>average</strong> of an area in the kernel or returning the <strong>maximum</strong>. MaxPooling is more beneficial in most applications because it <strong>reduces the noise</strong> in the data. Typical kernel sizes for pooling are 2x2 or 3x3.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*va4KwkmSC9gm8f9IKG08WQ.png" /><figcaption>Fig. 6: Max Pooling and Average Pooling with a 2x2 kernel | image by author</figcaption></figure><p>In Fig. 6, we see an example of MaxPooling and AvgPooling with a kernel size of 2x2. The feature map is divided into areas of the kernel size, and within those areas, we take either the maximum (→ MaxPooling) or the average (→ AvgPooling).</p><p><strong>Through pooling with a kernel size of 2x2, we halve the height and width of the feature map.</strong></p><h4>1.7 Tensor dimensions in the Convolutional Neural Network</h4><p>Now that we have studied the convolutional filters, the ReLU activation, and the pooling, we can revise Fig. 3 and the dimensions of the tensors. We start with an image of size 400x700. Since it is grayscale, it has only 1 channel, and the corresponding tensor is of size 1x400x700. We apply 6 convolutional filters of size 5x5 with a stride of 1x1 to the image. Each filter returns its own feature map, so we receive 6 of them. Due to the larger kernel compared to Fig. 4 (5x5 instead of 3x3), this time we lose 4 columns and 4 rows in the convolution. This means the returning tensor has the size 6x396x696.</p><p>In the next step, we apply MaxPooling with a 2x2 kernel to the feature maps (each map has its own pooling kernel). As we have learned, this reduces the maps’ dimensions by a factor of 2. Accordingly, the tensor is now of size 6x198x348.</p><p>Now we apply 16 convolutional filters of size 5x5. Each of them has a kernel depth of 6, which means that each filter provides a separate layer for the 6 channels of the input tensor. Each kernel layer slides over one of the 6 input channels, as studied in Fig. 4, and the 6 returning feature maps are added up to one. So far, we considered only one convolutional filter, but we have 16 of them. That is why we receive 16 new feature maps, each 4 columns and 4 rows smaller than the input. The tensor size is now 16x194x344.</p><p>Once more, we apply MaxPooling with a kernel size of 2x2. Since this halves the feature maps, we now have a tensor size of 16x97x172.<br>Finally, the tensor is flattened, which means we line up all of the <em>16*97*172 = 266,944</em> values and feed them into a fully connected neural network of corresponding size.</p><h3>Part 2: Defining and coding the CNN</h3><p>Conceptually, we have everything we need. Now, let’s go into the industrial use case as described in chapter 1.1.</p><h4>2.1 Load the required libraries</h4><p>We are going to use a couple of PyTorch libraries for data loading, sampling, and the model itself. Additionally, we load matplotlib.pyplot for visualization and PIL for transforming the images.</p><pre>import torch<br>import torch.nn as nn<br>from torch.utils.data import DataLoader, Dataset<br>from torch.utils.data.sampler import WeightedRandomSampler<br>from torch.utils.data import random_split<br>from torchvision import datasets, transforms<br>import matplotlib.pyplot as plt<br>import numpy as np<br>from PIL import Image<br>import os<br>import warnings<br>warnings.filterwarnings(&quot;ignore&quot;)</pre><h4>2.2 Configure your device and specify hyperparameters</h4><p>In device, we store ‘cuda’ or ‘cpu’, depending on whether or not your computer has a GPU available. minibatch_size defines how many images will be processed in one matrix operation during the training of the model. learning_rate specifies the magnitude of parameter adjustment during backpropagation, and epochs defines how often we process the whole set of training data in the training phase.</p><pre># Device configuration<br>device = torch.device(&#39;cuda&#39; if torch.cuda.is_available() else &#39;cpu&#39;)<br>print(f&quot;Using {device} device&quot;)<br><br># Specify hyperparameters<br>minibatch_size = 10<br>learning_rate = 0.01<br>epochs = 60</pre><h4>2.3 Custom loader function</h4><p>For loading the images, we define a custom_loader. It opens the images in binary mode, crops the inner 700x400 pixels of the image, loads them into memory, and returns the loaded images. As the path to the images, we define the relative path data/Coil_Vision/01_train_val_test. Please make sure that the data is stored in your working directory. You can download the files from my Dropbox as <a href="https://www.dropbox.com/scl/fi/z8kkui6ync57rudlx10dx/CNN_data.zip?rlkey=27787pjnnbaa3mss5nu4nbi7u&amp;st=4q3p56fh&amp;dl=0">CNN_data.zip</a>.</p><pre># Define loader function<br>def custom_loader(path):<br>    with open(path, &#39;rb&#39;) as f:<br>        img = Image.open(f)<br>        img = img.crop((50, 60, 750, 460))  #Size: 700x400 px<br>        img.load()<br>        return img<br><br># Path of images (local to accelerate loading)<br>path = &quot;data/Coil_Vision/01_train_val_test&quot;</pre><h4>2.4 Define the datasets</h4><p>We define the dataset as tuples consisting of the image data and the label, either <em>0</em> for scrap parts and <em>1</em> for good parts. The method datasets.ImageFolder() reads the labels out of the folder structure. We use a transform function to first load the image data to a PyTorch tensor (values between 0 and 1) and, second, normalize the data with the approximate mean of 0.5 and standard deviation of 0.5. After the transformation, the image data is roughly standard normal distributed (mean = 0, standard deviation = 1). We split the dataset randomly into 50% training data, 30% validation data, and 20% testing data.</p><pre># Transform function for loading<br>transform = transforms.Compose([transforms.ToTensor(),<br>                                transforms.Normalize((0.5), (0.5))])<br><br># Create dataset out of folder structure<br>dataset = datasets.ImageFolder(path, transform=transform, loader=custom_loader)<br>train_set, val_set, test_set = random_split(dataset, [round(0.5*len(dataset)), <br>                                                      round(0.3*len(dataset)), <br>                                                      round(0.2*len(dataset))])</pre><h4>2.5 Balance the datasets</h4><p>Our data is unbalanced. We have far more good samples than scrap samples. To reduce a bias towards the majority class during training, we use a WeightedRandomSampler to give higher probability to the minority class during sampling. In lbls, we store the labels of the training dataset. With np.bincount(), we count the number of <em>0</em> labels (bc[0]) and <em>1</em> labels (bc[1]). Next, we calculate probability weights for the two classes (p_nOK and p_OK) and arrange them according to the sequence in the dataset in the list lst_train. Finally, we instantiate train_sampler from WeightedRandomSampler.</p><pre># Define a sampler to balance the classes<br># training dataset<br>lbls = [dataset[idx][1] for idx in train_set.indices]<br>bc = np.bincount(lbls)<br>p_nOK = bc.sum()/bc[0]<br>p_OK = bc.sum()/bc[1]<br>lst_train = [p_nOK if lbl==0 else p_OK for lbl in lbls]<br>train_sampler = WeightedRandomSampler(weights=lst_train, num_samples=len(lbls))</pre><h4>2.6 Define the data loaders</h4><p>Lastly, we define three data loaders for the training, the validation, and the testing data. Data loaders feed the neural network with batches of datasets, each consisting of the image data and the label.<br>For the train_loader and the val_loader, we set the batch size to 10 and shuffle the data. The test_loader operates with shuffled data and a batch size of 1.</p><pre># Define loader with batchsize<br>train_loader = DataLoader(dataset=train_set, batch_size=minibatch_size, sampler=train_sampler)<br>val_loader = DataLoader(dataset=val_set, batch_size=minibatch_size, shuffle=True)<br>test_loader = DataLoader(dataset=test_set, shuffle=True)</pre><h4>2.7 Check the data: Plot 5 OK and 5 nOK parts</h4><p>To inspect the image data, we plot five good samples (“OK”) and five scrap samples (“nOK”). To do this, we define a matplotlib figure with 2 rows and 5 columns and share the x- and the y-axis. In the core of the code snippet, we nest two for-loops. The outer loop receives batches of data from the train_loader. Each batch consists of ten images and the corresponding labels. The inner loop enumerates the batches’ labels. In its body, we check if the label equals <em>0</em> — then we plot the image under “nOK” in the second row — or if the label equals <em>1</em> — then we plot the image under “OK” in the first row. Once count_OK and count_nOK both are greater or equal 5, we break the loop, set the title, and show the figure.</p><pre># Figure and axes object<br>fig, axs = plt.subplots(nrows=2, ncols=5, figsize=(20,7), sharey=True, sharex=True)<br><br>count_OK = 0<br>count_nOK = 0<br><br># Loop over loader batches<br>for (batch_data, batch_lbls) in train_loader:<br>    <br>    # Loop over batch_lbls<br>    for i, lbl in enumerate(batch_lbls):<br>        <br>        # If label is 0 (nOK) plot image in row 1<br>        if (lbl.item() == 0) and (count_nOK &lt; 5):<br>            axs[1, count_nOK].imshow(batch_data[i][0], cmap=&#39;gray&#39;)<br>            axs[1, count_nOK].set_title(f&quot;nOK Part#: {str(count_nOK)}&quot;, fontsize=14)<br>            count_nOK += 1<br>            <br>        # If label is 1 (OK) plot image in row 0<br>        elif (lbl.item() == 1) and (count_OK &lt; 5):<br>            axs[0, count_OK].imshow(batch_data[i][0], cmap=&#39;gray&#39;)<br>            axs[0, count_OK].set_title(f&quot;OK Part#: {str(count_OK)}&quot;, fontsize=14)<br>            count_OK += 1<br>    <br>    # If both counters are &gt;=5 stop looping<br>    if (count_OK &gt;=5) and (count_nOK &gt;=5):<br>        break<br>        <br># Config the plot canvas<br>fig.suptitle(&quot;Sample plot of OK and nonOK Parts&quot;, fontsize=24)<br>plt.setp(axs, xticks=[], yticks=[]) <br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OWvH0dMqHcmXQJRjRePkEA.png" /><figcaption>Fig. 7: Example for OK (upper row) and nonOK parts (lower row) | image by author</figcaption></figure><p>In Fig. 7, we see that most of the nOK samples are clearly bent, but a few are not really distinguishable by eye (e.g., lower right sample).</p><h4>2.8 Define the CNN model</h4><p>The model corresponds to the architecture depicted in Fig. 3. We feed the grayscale image (only one channel) into the first convolutional layer and define 6 kernels of size 5 (equals 5x5). The convolution is followed by a ReLU activation and a MaxPooling with a kernel size of 2 (2x2) and a stride of 2 (2x2). All three operations are repeated with the dimensions shown in Fig. 3. In the final block of the __init__() method, the 16 feature maps are flattened and fed into a linear layer of equivalent input size and 120 output nodes. It is ReLU activated and reduced to only 2 output nodes in a second linear layer.<br>In the forward() method, we simply call the model layers and feed in the x tensor.</p><pre>class CNN(nn.Module):<br><br>    def __init__(self):<br>        super().__init__()<br><br>        # Define model layers<br>        self.model_layers = nn.Sequential(<br><br>            nn.Conv2d(in_channels=1, out_channels=6, kernel_size=5),<br>            nn.ReLU(),<br>            nn.MaxPool2d(kernel_size=2, stride=2),<br><br>            nn.Conv2d(in_channels=6, out_channels=16, kernel_size=5),<br>            nn.ReLU(),<br>            nn.MaxPool2d(kernel_size=2, stride=2),<br><br>            nn.Flatten(),<br>            nn.Linear(16*97*172, 120),<br>            nn.ReLU(),<br>            nn.Linear(120, 2)<br>        )<br>        <br>    def forward(self, x):<br>        out = self.model_layers(x)<br>        return out</pre><h4>2.9 Instantiate the model and define the loss function and the optimizer</h4><p>We instantiate model from the CNN class and push it either on the CPU or on the GPU. Since we have a classification task, we choose the CrossEntropyLoss function. For managing the training process, we call the Stochastic Gradient Descent (SGD) optimizer.</p><pre># Define model on cpu or gpu<br>model = CNN().to(device)<br><br># Loss and optimizer<br>loss = nn.CrossEntropyLoss()<br><br>optimizer = torch.optim.SGD(model.parameters(), lr=learning_rate)</pre><h4>2.10 Check the model’s size</h4><p>To get an idea of our model’s size in terms of parameters, we iterate over model.parameters() and sum up, first, all model parameters (num_param) and, second, those parameters that will be adjusted during backpropagation (num_param_trainable). Finally, we print the result.</p><pre># Count number of parameters / thereof trainable<br>num_param = sum([p.numel() for p in model.parameters()])<br>num_param_trainable = sum([p.numel() for p in model.parameters() if p.requires_grad == True])<br><br>print(f&quot;Our model has {num_param:,} parameters. Thereof trainable are {num_param_trainable:,}!&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/772/1*K4a1H05zkml55uh78eAsRQ.png" /></figure><p>The print out tells us that the model has more than 32 million parameters, thereof all trainable.</p><h4>2.11 Define a function for validation and testing</h4><p>Before we start the model training, let’s prepare a function to support the validation and testing. The function val_test() expects a dataloader and the CNN model as parameters. It turns off the gradient calculation with torch.no_grad() and iterates over the dataloader. With one batch of images and labels at hand, it inputs the images into the model and determines the model’s predicted classes with output.argmax(1) over the returned logits. This method returns the indices of the largest values; in our case, this represents the class indices.<br>We count and sum up the correct predictions and save the image data, the predicted class, and the labels of the wrong predictions. Finally, we calculate the accuracy and return it together with the misclassified images as the function’s output.</p><pre>def val_test(dataloader, model):<br>    # Get dataset size<br>    dataset_size = len(dataloader.dataset)<br>    <br>    # Turn off gradient calculation for validation<br>    with torch.no_grad():<br>        # Loop over dataset<br>        correct = 0<br>        wrong_preds = []<br>        for (images, labels) in dataloader:<br>            images, labels = images.to(device), labels.to(device)<br>            <br>            # Get raw values from model<br>            output = model(images)<br>            <br>            # Derive prediction<br>            y_pred = output.argmax(1)<br>            <br>            # Count correct classifications over all batches<br>            correct += (y_pred == labels).type(torch.float32).sum().item()<br>            <br>            # Save wrong predictions (image, pred_lbl, true_lbl)<br>            for i, _ in enumerate(labels):<br>                if y_pred[i] != labels[i]:<br>                    wrong_preds.append((images[i], y_pred[i], labels[i]))<br><br>        # Calculate accuracy<br>        acc = correct / dataset_size<br>        <br>    return acc, wrong_preds</pre><h4>2.12 Model training</h4><p>The model training consists of two nested for-loops. The outer loop iterates over a defined number of epochs, and the inner loop enumerates the train_loader. The enumeration returns a batch of image data and the corresponding labels. The image data (images) is passed to the model, and we receive the model’s response logits in outputs. outputs and the true labels are passed to the loss function. Based on loss l, we perform backpropagation and update the parameter with optimizer.step. outputs is a tensor of dimension <em>batchsize x output nodes</em>, in our case <em>10 x 2</em>. We receive the model’s prediction through the indices of the max values over the rows, either <em>0</em> or <em>1</em>.</p><p>Finally, we count the number of correct predictions (n_correct), the true OK parts (n_true_OK), and the number of samples (n_samples). Each second epoch, we calculate the training accuracy, the true OK share, and call the validation function (val_test()). All three values are printed for information purpose during the training run. With the last line of code, we save the model with all its parameters in “model.pth”.</p><pre>acc_train = {}<br>acc_val = {}<br># Iterate over epochs<br>for epoch in range(epochs):<br><br>    n_correct=0; n_samples=0; n_true_OK=0<br>    for idx, (images, labels) in enumerate(train_loader):<br>        model.train()<br>        # Push data to gpu if available<br>        images, labels = images.to(device), labels.to(device)<br>        <br>        # Forward pass<br>        outputs = model(images)<br>        l = loss(outputs, labels)<br>              <br>        # Backward and optimize<br>        optimizer.zero_grad()<br>        l.backward()<br>        optimizer.step()<br><br>        # Get prediced labels (.max returns (value,index))<br>        _, y_pred = torch.max(outputs.data, 1)<br><br>        # Count correct classifications<br>        n_correct += (y_pred == labels).sum().item()<br>        n_true_OK += (labels == 1).sum().item()<br>        n_samples += labels.size(0)<br>        <br>    # At end of epoch: Eval accuracy and print information<br>    if (epoch+1) % 2 == 0:<br>        model.eval()<br>        # Calculate accuracy<br>        acc_train[epoch+1] = n_correct / n_samples<br>        true_OK = n_true_OK / n_samples<br>        acc_val[epoch+1] = val_test(val_loader, model)[0]<br>        <br>        # Print info<br>        print (f&quot;Epoch [{epoch+1}/{epochs}], Loss: {l.item():.4f}&quot;)<br>        print(f&quot;      Training accuracy: {acc_train[epoch+1]*100:.2f}%&quot;)<br>        print(f&quot;      True OK: {true_OK*100:.3f}%&quot;)<br>        print(f&quot;      Validation accuracy: {acc_val[epoch+1]*100:.2f}%&quot;)<br>        <br># Save model and state_dict<br>torch.save(model, &quot;model.pth&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*38ecsrpUnDCJ7tHfEO6Gfw.png" /></figure><p>Training takes a couple of minutes on the GPU of my laptop. It is highly recommended to load the images from the local drive. Otherwise, training time might increase by orders of magnitude!</p><p>The printouts from training inform that the loss has reduced significantly, and the validation accuracy — the accuracy on data the model has not used for updating its parameters — has reached 98.4%.</p><p>An even better impression on the training progress is obtained if we plot the training and validation accuracy over the epochs. We can easily do this because we saved the values each second epoch.<br>We create a matplotlib figure and axes with plt.subplots() and plot the values over the keys of the accuracy dictionaries.</p><pre># Instantiate figure and axe object<br>fig, ax = plt.subplots(figsize=(10,6))<br>plt.plot(list(acc_train.keys()), list(acc_train.values()), label=&quot;training accuracy&quot;)<br>plt.plot(list(acc_val.keys()), list(acc_val.values()), label=&quot;validation accuracy&quot;)<br>plt.title(&quot;Accuracies&quot;, fontsize=24)<br>plt.ylabel(&quot;%&quot;, fontsize=14)<br>plt.xlabel(&quot;Epochs&quot;, fontsize=14)<br>plt.setp(ax.get_xticklabels(), fontsize=14)<br>plt.legend(loc=&#39;best&#39;, fontsize=14)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1000/1*2SIvePZqTGCjjsl8X0SWXA.png" /><figcaption>Fig. 8: Training and validation accuracy during model training | image by author</figcaption></figure><h4>2.13 Loading the trained model</h4><p>If you want to use the model for production and not only for study purpose, it is highly recommended to save and load the model with all its parameters. Saving was already part of the training code. Loading the model from your drive is equally simple.</p><pre># Read model from file<br>model = torch.load(&quot;model.pth&quot;)<br>model.eval()</pre><h4>2.14 Double-check the model accuracy with test data</h4><p>Remember, we reserved another 20% of our data for testing. This data is totally new to the model and has never been loaded before. We can use this brand-new data to double-check the validation accuracy. Since the validation data has been loaded but never been used to update the model parameters, we expect a similar accuracy to the test value. To conduct the test, we call the val_test() function on the test_loader.</p><pre>print(f&quot;test accuracy: {val_test(test_loader,model)[0]*100:0.1f}%&quot;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/621/1*T9SGHLu5BUaqLbfPIpcS4g.png" /></figure><p>In the specific example, we reach a test accuracy of 99.2%, but this is highly dependent on chance (remember: random distribution of images to training, validation, and testing data).</p><h4>2.15 Visualizes the misclassified images</h4><p>The visualization of the misclassified images is pretty straightforward. First, we call the val_test() function. It returns a tuple with the accuracy value at index position <em>0</em> (tup[0]) and another tuple at index position <em>1</em> (tup[1]) with the image data (tup[1][0]), the predicted labels (tup[1][1]), and the true labels (tup[1][2]) of the misclassified images. In case tup[1] is not empty, we enumerate it and plot the misclassified images with appropriate headings.</p><pre>%matplotlib inline<br><br># Call test function<br>tup = val_test(test_loader, model)<br><br># Check if wrong predictions occur<br>if len(tup[1])&gt;=1:<br>    <br>    # Loop over wrongly predicted images<br>    for i, t in enumerate(tup[1]):<br>        plt.figure(figsize=(7,5))<br>        img, y_pred, y_true = t<br>        img = img.to(&quot;cpu&quot;).reshape(400, 700)<br>        plt.imshow(img, cmap=&quot;gray&quot;)<br>        plt.title(f&quot;Image {i+1} - Predicted: {y_pred}, True: {y_true}&quot;, fontsize=24)<br>        plt.axis(&quot;off&quot;)<br>        plt.show()<br>        plt.close()<br>else:<br>    print(&quot;No wrong predictions!&quot;)</pre><p>In our example, we have only one misclassified image, which represents 0.8% of the test dataset (we have 125 test images). The image was classified as OK but has the label nOK. Frankly, I would have misclassified it too :).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*thCvBPKriILDG8RiqckMEg.png" /><figcaption>Fig. 9: Misclassified image | image by author</figcaption></figure><h3>Part 3: Using the trained model in production</h3><h4>3.1 Loading the model, required libraries, and parameters</h4><p>In the production phase, we assume that the CNN model is trained and the parameters are ready to be loaded. Our aim is to load new images into the model and let it classify whether the respective electronic component is good for assembly or not (see chapter <em>1.1 The task: Classify an industrial component as good or scrap</em>).</p><p>We start by loading the required libraries, setting the device as ‘cuda’ or ‘cpu’, defining the class CNN (exactly as in chapter 2.8), and loading the model from file with torch.load(). We need to define the class CNN before loading the parameters; otherwise, the parameters cannot be assigned correctly.</p><pre># Load the required libraries<br>import torch<br>import torch.nn as nn<br>from torch.utils.data import DataLoader, Dataset<br>from torchvision import datasets, transforms<br>import matplotlib.pyplot as plt<br>from PIL import Image<br>import os<br><br># Device configuration<br>device = torch.device(&#39;cuda&#39; if torch.cuda.is_available() else &#39;cpu&#39;)<br><br># Define the CNN model exactly as in chapter 2.8<br>class CNN(nn.Module):<br><br>    def __init__(self):<br>        super(CNN, self).__init__()<br><br>        # Define model layers<br>        self.model_layers = nn.Sequential(<br><br>            nn.Conv2d(in_channels=1, out_channels=6, kernel_size=5),<br>            nn.ReLU(),<br>            nn.MaxPool2d(kernel_size=2, stride=2),<br><br>            nn.Conv2d(in_channels=6, out_channels=16, kernel_size=5),<br>            nn.ReLU(),<br>            nn.MaxPool2d(kernel_size=2, stride=2),<br><br>            nn.Flatten(),<br>            nn.Linear(16*97*172, 120),<br>            nn.ReLU(),<br>            nn.Linear(120, 2),<br>            #nn.LogSoftmax(dim=1)<br>        )<br>        <br>    def forward(self, x):<br>        out = self.model_layers(x)<br>        return out<br><br># Load the model&#39;s parameters<br>model = torch.load(&quot;model.pth&quot;)<br>model.eval()</pre><p>With running this code snippet, we have the CNN model loaded and parameterized in our computer’s memory.</p><h4>3.2 Load images into dataset</h4><p>As for the training phase, we need to prepare the images for processing in the CNN model. We load them from a specified folder, crop the inner 700x400 pixels, and transform the image data to a PyTorch tensor.</p><pre># Define custom dataset<br>class Predict_Set(Dataset):<br>    def __init__(self, img_folder, transform):<br>        self.img_folder = img_folder<br>        self.transform = transform<br>        self.img_lst = os.listdir(self.img_folder)<br><br>    def __len__(self):<br>        return len(self.img_lst)<br><br>    def __getitem__(self, idx):<br>        img_path = os.path.join(self.img_folder, self.img_lst[idx])<br>        img = Image.open(img_path)<br>        img = img.crop((50, 60, 750, 460))  #Size: 700x400<br>        img.load()<br>        img_tensor = self.transform(img)<br>        return img_tensor, self.img_lst[idx]</pre><p>We perform all the steps in a custom dataset class called Predict_Set(). In __init__(), we specify the image folder, accept a transform function, and load the images from the image folder into the list self.img_lst. The method __len__() returns the number of images in the image folder. __getitem__() composes the path to an image from the folder path and the image name, crops the inner part of the image (as we did for the training dataset), and applies the transform function to the image. Finally, it returns the image tensor and the image name.</p><h4>3.3 Path, transform function, and data loader</h4><p>The final step in data preparation is to define a data loader that allows to iterate over the images for classification. Along the way, we specify the path to the image folder and define the transform function as a pipeline that first loads the image data to a PyTorch tensor, and, second, normalizes the data to a range of approximately -1 to +1. We instantiate our custom dataset Predict_Set() to a variable predict_set and define the data loader predict_loader. Since we do not specify a batch size, predict_loader returns one image at a time.</p><pre># Path to images (preferably local to accelerate loading)<br>path = &quot;data/Coil_Vision/02_predict&quot;<br><br># Transform function for loading<br>transform = transforms.Compose([transforms.ToTensor(),<br>                                transforms.Normalize((0.5), (0.5))])<br><br># Create dataset as instance of custom dataset<br>predict_set = Predict_Set(path, transform=transform)<br><br># Define loader<br>predict_loader = DataLoader(dataset=predict_set)</pre><h4>3.4 Custom function for classification</h4><p>So far, the preparation of the image data for classification is complete. However, what we are still missing is a custom function that transfers the images to the CNN model, translates the model’s response into a classification, and returns the classification results. This is exactly what we do with predict().</p><pre>def predict(dataloader, model):<br><br>    # Turn off gradient calculation<br>    with torch.no_grad():<br>        <br>        img_lst = []; y_pred_lst = []; name_lst = []<br>        # Loop over data loader<br>        for image, name in dataloader:<br>            img_lst.append(image)<br>            image = image.to(device)<br>            <br>            # Get raw values from model<br>            output = model(image)<br>            <br>            # Derive prediction<br>            y_pred = output.argmax(1)<br>            y_pred_lst.append(y_pred.item())<br>            name_lst.append(name[0])<br>            <br>    return img_lst, y_pred_lst, name_lst</pre><p>predict() expects a data loader and the CNN model as its parameters. In its core, it iterates over the data loader, transfers the image data to the model, and interprets the models response with output.argmax(1) as the classification result — either <em>0</em> for scrap parts (nOK) or <em>1</em> for good parts (OK). The image data, the classification result, and the image name are appended to lists and the lists are returned as the function’s result.</p><h4>3.5 Predict labels and plot images</h4><p>Finally, we want to utilize our custom functions and loaders to classify new images. In the folder “data/Coil_Vision/02_predict” we have reserved four images of electronic components that wait to be inspected. Remember, we want the CNN model to tell us whether we can use the components for automatic assembly or if we need to sort them out because the pins are likely to cause problems while trying to push them in the plug sockets.<br>We call the custom function predict(), which returns a list of images, a list of classification results, and a list of image names. We enumerate the lists and plot the images with the names and the classification as headings.</p><pre># Predict labels for images<br>imgs, lbls, names  = predict(predict_loader, model)<br><br># Iterate over classified images<br>for idx, image in enumerate(imgs):<br>    plt.figure(figsize=(8,6))<br>    plt.imshow(image.squeeze(), cmap=&quot;gray&quot;)<br>    plt.title(f&quot;\nFile: {names[idx]}, Predicted label: {lbls[idx]}&quot;, fontsize=18)<br>    plt.axis(&quot;off&quot;)<br>    plt.show()<br>    plt.close()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*uc4VZ3_CcC6rNl3EKeEFkg.png" /><figcaption>Fig. 10: Classification results in production phase | image by author</figcaption></figure><p>We see that the two images on the left side have been classified a good (label <em>1</em>) and the two on the right as scrap (label <em>0</em>). Due to our training data, the model is quite sensitive, and even small bends in the pins lead to them being classified as scrap.</p><h3>Part 4: What did the CNN consider in its “decision”?</h3><p>We have gone deep into the details of the CNN and our industrial use case so far. This seems like a good opportunity to take one step further and try to understand what the CNN model “sees” while processing the image data. To do this, we first study the convolutional layers and then examine which parts of the image are specifically important for the classification.</p><h4>4.1 Study the convolutional filters’ dimensions</h4><p>To gain a better understanding of how convolutional filters work and what they do to the images, let’s examine the layers in our industrial example in more detail.</p><p>To access the layers, we enumerate model.children(), which is a generator for the model’s structure. If the layer is a convolutional layer, we append it to the list all_layers and save the weights’ dimensions in conv_weights. If we have a ReLU or a MaxPooling layer, we have no weights. In this case, we append the layer and <em>“*”</em> to the respective lists. Next, we enumerate all_layers, print the layer type, and the weights’ dimensions.</p><pre># Empty lists to store the layers and the weights<br>all_layers = []; conv_weights = []<br><br># Iterate over the model&#39;s structure<br># (First level nn.Sequential)<br>for _, layer in enumerate(list(model.children())[0]):<br>    if type(layer) == nn.Conv2d:<br>        all_layers.append(layer)<br>        conv_weights.append(layer.weight)<br>    elif type(layer) in [nn.ReLU, nn.MaxPool2d]:<br>        all_layers.append(layer)<br>        conv_weights.append(&quot;*&quot;)<br><br># Print layers and dimensions of weights<br>for idx, layer in enumerate(all_layers):<br>    print(f&quot;{idx+1}. Layer: {layer}&quot;)<br>    if type(layer) == nn.Conv2d:<br>        print(f&quot;          weights: {conv_weights[idx].shape}&quot;)<br>    else:<br>        print(f&quot;          weights: {conv_weights[idx]}&quot;)<br>    print()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/752/1*8ThvrZYSqewL3CVlZqA6sA.png" /><figcaption>Fig. 11: Layers and weights’ dimensions</figcaption></figure><p>Please compare the code snippet’s output with Fig. 3. The first convolutional layer has one input — the original image with only one channel — and returns six feature maps. We apply six kernels, each of depth one and size 5x5. Correspondingly, the weights are of dimension torch.Size([6, 1, 5, 5]). In contrast, layer 4 receives six feature maps as input and returns 16 maps as output. We apply 16 convolutional kernels, each of depth 6 and size 5x5. The weights’ dimension is therefore torch.Size([16, 6, 5, 5]).</p><h4>4.2 Visualize the convolutional filters’ weights</h4><p>Now, we know the convolutional filters’ dimensions. Next, we want to see their weights, which they have gained during the training process. Since we have so many different filters (six in the first convolutional layer and 16 in the second), we select, in both cases, the first input channel (index <em>0</em>).</p><pre>import itertools<br><br># Iterate through all layers<br>for idx_out, layer in enumerate(all_layers):<br>    <br>    # If layer is a convolutional filter<br>    if type(layer) == nn.Conv2d:<br>        <br>        # Print layer name<br>        print(f&quot;\n{idx_out+1}. Layer: {layer} \n&quot;)<br>        <br>        # Prepare plot and weights<br>        plt.figure(figsize=(25,6))<br>        weights = conv_weights[idx_out][:,0,:,:] # only first input channel<br>        weights = weights.detach().to(&#39;cpu&#39;)<br>        <br>        # Enumerate over filter weights (only first input channel)<br>        for idx_in, f in enumerate(weights):<br>            plt.subplot(2,8, idx_in+1)<br>            plt.imshow(f, cmap=&quot;gray&quot;)<br>            plt.title(f&quot;Filter {idx_in+1}&quot;)<br>            <br>            # Print texts<br>            for i, j in itertools.product(range(f.shape[0]), range(f.shape[1])):<br>                if f[i,j] &gt; f.mean():<br>                    color = &#39;black&#39;<br>                else:<br>                    color = &#39;white&#39;<br>                plt.text(j, i, format(f[i, j], &#39;.2f&#39;), horizontalalignment=&#39;center&#39;, verticalalignment=&#39;center&#39;, color=color)<br>            <br>            plt.axis(&quot;off&quot;)<br>        plt.show()<br>        plt.close()       </pre><p>We iterate over all_layers. If the layer is a convolutional layer (nn.Conv2d), then we print the layer’s index and the layer’s core data. Next, we prepare a plot and extract the weights matrix for the first input layer as an example. We enumerate all output layers and plot them with plt.imshow(). Finally, we print the weights’ values on the image so that we get an intuitive visualization of the convolutional filters.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*GxV1ZXVfBuyxxkIyY97LUA.png" /><figcaption>Fig. 12: Visualization of the 6+16 convolutional filters (input layer index 0) | image by author</figcaption></figure><p>Fig. 12 shows the six convolutional filter kernels of layer 1 and the 16 kernels of layer 4 (for input channel <em>0</em>). The model schematic in the upper right indicates the filters with a red outline. We see that the majority of values are close to 0, and some are in the range of positive or negative 0.20–0.25. The numbers represent the values used for the convolution demonstrated in Fig. 4. This gives us the feature maps, which we inspect next.</p><h4>4.3 Examine the feature maps</h4><p>According to Fig. 4, we receive the first feature maps through the convolution of the input image. Therefore, we load a random image from the test_loader and push it to the CPU (in case you operate the CNN on the GPU).</p><pre># Test loader has a batch size of 1<br>img = next(iter(test_loader))[0].to(device)<br>print(f&quot;\nImage has shape: {img.shape}\n&quot;)<br><br># Plot image<br>img_copy = img.to(&#39;cpu&#39;)<br>plt.imshow(img_copy.reshape(400,700), cmap=&quot;gray&quot;)<br>plt.axis(&quot;off&quot;)<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/527/1*aYCO2KQis-lzOHCsrSbapA.png" /><figcaption>Fig. 13: Random image as output of above code | image by author</figcaption></figure><p>Now we pass the image data img through the first convolution layer (all_layers[0]) and save the output in results. Next, we iterate over all_layers and feed the next layer with the output from the previous layer operation. Those operations are convolutions, ReLU activations or MaxPoolings. The output of each operation we append to results.</p><pre># Pass the image through the first layer<br>results = [all_layers[0](img)]<br><br># Pass the results of the previous layer to the next layer<br>for idx in range(1, len(all_layers)):  # Start at 1, first layer already passed!<br>    results.append(all_layers[idx](results[-1]))  # Pass the last result to the layer</pre><p>Finally, we plot the original image, the feature maps after passing the first layer (convolution), the second layer (ReLU), the third layer (MaxPooling), the forth layer (2nd convolution), the fifth layer (2nd ReLU), and the sixth layer (2nd MaxPooling).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*yo9KsNnDELLjOjoEZ8IORQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*rqQlKRcGSV3KkyvlM3Z-wA.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*iKECMSZxPH_5lWfN42oJ4w.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2vWmXh5SK1qrby7ns5HIbQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2yDZ5BVLFYKBeRMGd1YkRw.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x_2bGxg-ChrhCBJz9oflHQ.png" /></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*oVrS1FBCfFVhUPHSdNp7Zg.png" /><figcaption>Fig. 14: Original image and feature maps after passing the convolution, ReLU and MaxPooling layers | image by author</figcaption></figure><p>We see that the convolutional kernels (compare Fig. 12) recalculate each pixel of the image. This appears as changed grayscale values in the feature maps. Some of the feature maps are sharpened compared to the original image or have a stronger black-and-white contrast, while others seem to be faded.</p><p>The ReLU operations turn dark gray into black since negative values are set to zero.</p><p>MaxPooling keeps the images almost unchanged while halving the image size in both dimensions.</p><h4>4.4 Visualize the image areas that impact the classification the most</h4><p>Before we finish, let’s analyze which areas of the image are particularly decisive for the classification into scrap (index <em>0</em>) or good parts (index <em>1</em>). For this purpose, we use Gradient-weighted Class Activation Mapping (gradCAM). This technique computes the gradients of the trained model with respect to the predicted class (the gradients show how much the inputs — the image pixels — influence the prediction). The averages of the gradients of each feature map (= output channel of convolution layer) build the weights with which the feature maps are multiplied when calculating a heat map for visualization.<br>But let’s look at one step after the other.</p><pre>def gradCAM(x):<br>    <br>    # Run model and predict<br>    logits = model(x)<br>    pred = logits.max(-1)[-1] # Returns index of max value (0 or 1)<br>    <br>    # Fetch activations at final conv layer<br>    last_conv = model.model_layers[:5]<br>    activations = last_conv(x)<br><br>    # Compute gradients with respect to model&#39;s prediction<br>    model.zero_grad()<br>    logits[0,pred].backward(retain_graph=True)<br>    <br>    # Compute average gradient per output channel of last conv layer<br>    pooled_grads = model.model_layers[3].weight.grad.mean((1,2,3))<br>    <br>    # Multiply each output channel with its corresponding average gradient<br>    for i in range(activations.shape[1]):<br>        activations[:,i,:,:] *= pooled_grads[i]<br>    <br>    # Compute heatmap as average over all weighted output channels<br>    heatmap = torch.mean(activations, dim=1)[0].cpu().detach()<br>    <br>    return heatmap</pre><p>We define a function gradCAM that expects the input data x, an image or a feature map, and returns a heatmap.</p><p>In the first block, we input x in the CNN model and receive logits, a tensor of shape [1, 2] with two values only. The values represent the predicted probabilities of the classes <em>0</em> and <em>1</em>. We select the index of the larger value as the model’s prediction pred.</p><p>In the second block, we extract the first five layers of the model — from first convolution to second ReLU — and save them to last_conv. We run x through the selected layers and store the output in activations. As the name suggests, those are the activations (=feature maps) of the second convolutional layer (after ReLU activation).</p><p>In the third block, we do the backward propagation for the logit value of the predicted class logits[0,pred]. In other words, we compute all the gradients of the CNN with respect to the prediction. The gradients show how much a change in the input data — the original image pixels — impact the models output — the prediction. The result is saved in the PyTorch computational graph until we delete it with model.zero_grad().</p><p>In the fourth block, we compute the averages of the gradients over the input channel, as well as the height and width of the image or the feature maps. As a result, we receive 16 average gradients for the 16 feature maps that are returned from the second convolutional layer. We save them in pooled_grads.</p><p>In the fifth block, we iterate over the 16 feature maps returned from the second convolutional layer and weight them with the average gradients pooled_grads. This operation gives more impact to those feature maps (and their pixels) that have high importance for the prediction and vice versa. From now on, activations holds not the feature maps, but the weighted feature maps.</p><p>Finally, in the last block, we compute the heatmap as the average feature map of all activations. This is what the function gradCAM returns.</p><p>Before we can plot the image and the heatmap, we need to transform both for the overlay. Remember, the feature maps are smaller than the original picture (see chapters 1.3 and 1.7), and so is the heatmap. This is why we need the function upsampleHeatmap(). The function scales the pixel values to the range of 0 to 255 and transforms them to 8-bit integer format (required from the cv2 library). It resizes the heatmap to 400x700 px and applies a color map to both the image and heatmap. Finally, we overlay 70% heatmap and 30% image and return the composition for plotting.</p><pre>import cv2<br><br>def upsampleHeatmap(map, img):<br>    m,M = map.min(), map.max()<br>    i,I = img.min(), img.max()<br>    map = 255 * ((map-m) / (M-m))<br>    img = 255 * ((img-i) / (I-i))<br>    map = np.uint8(map)<br>    img = np.uint8(img)<br>    map = cv2.resize(map, (700,400))<br>    map = cv2.applyColorMap(255-map, cv2.COLORMAP_JET)<br>    map = np.uint8(map)<br>    img = cv2.applyColorMap(255-img, cv2.COLORMAP_JET)<br>    img = np.uint8(img)<br>    map = np.uint8(map*0.7 + img*0.3)<br>    return map</pre><p>We want to plot the original image and the heatmap overlay next to each other in one row. To do this, we iterate over the data loader predict_loader, run the gradCAM() function on the images and the upsampleHeatmap() function on the heatmap and the image. Finally, we plot the original image and the heatmap in a row with matplotlib.pyplot.</p><pre># Iterate over dataloader<br>for idx, (image, name) in enumerate(predict_loader):<br>    <br>    # Compute heatmap<br>    image = image.to(device)<br>    heatmap = gradCAM(image)<br>    image = image.cpu().squeeze(0).permute(1,2,0)<br>    heatmap = upsampleHeatmap(heatmap, image)<br>    <br>    # Plot images and heatmaps<br>    fig = plt.figure(figsize=(14,5))<br>    fig.suptitle(f&quot;\nFile: {names[idx]}, Predicted label: {lbls[idx]}\n&quot;, fontsize=24)<br>    plt.subplot(1, 2, 1)<br>    plt.imshow(image, cmap=&quot;gray&quot;)<br>    plt.title(f&quot;Image&quot;, fontsize=14)<br>    plt.axis(&quot;off&quot;)<br>    plt.subplot(1, 2, 2)<br>    plt.imshow(heatmap)<br>    plt.title(f&quot;Heatmap&quot;, fontsize=14)<br>    plt.tight_layout()<br>    plt.axis(&quot;off&quot;)<br>    plt.show()<br>    plt.close()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fEO-HvNouTYmF4MUUYXSVQ.png" /><figcaption>Fig. 15: Image and heatmap (inner two rows of the output) | image by author</figcaption></figure><p>The blue areas of the heatmap have low impact on the model’s decision, while the yellow and red areas are very important. We see that in our use case, mainly the contour of the electronic component (in particular the metal pins) is decisive for the classification into scrap or good parts. Of course, this is highly reasonable, given that the use case primarily deals with bent pins.</p><h3>Conclusion</h3><p>Convolutional Neural Networks (CNNs) are nowadays a common and widely used tool for visual inspection tasks in the industrial environment. In our use case, with relatively few lines of code, we managed to define a model that classifies electronic components as good parts or scrap with high precision. The big advantage, compared to classic approaches of vision inspection, is that no process engineer needs to specify visual marks in the images for the classification. Instead, the CNN learns from labeled examples and is able to replicate this knowledge to other images. In our specific use case, 626 labeled images were sufficient for training and validation. In more complex cases, the demand for training data might be significantly higher.</p><p>Algorithms like gradCAM (Gradient-weighted Class Activation Mapping) significantly help in understanding which areas in the image are particularly relevant for the model’s decision. In this way, they support a broad use of CNNs in the industrial context by building trust in the model’s functionality.</p><p>In this article, we have explored many details of the inner workings of Convolutional Neural Networks. I hope you enjoyed the journey and have gained a deep understanding of how CNNs work.</p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=138936d7a34a" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/building-a-vision-inspection-cnn-for-an-industrial-application-138936d7a34a">Building a Vision Inspection CNN for an Industrial Application</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Neural Networks are Universal Function Modelers]]></title>
            <link>https://aiadvances.org/neural-networks-are-universal-function-modelers-40eccfa80e51?source=rss-2628a598423a------2</link>
            <guid isPermaLink="false">https://medium.com/p/40eccfa80e51</guid>
            <category><![CDATA[explanation]]></category>
            <category><![CDATA[insights]]></category>
            <category><![CDATA[approximation]]></category>
            <category><![CDATA[neural-networks]]></category>
            <category><![CDATA[regression]]></category>
            <dc:creator><![CDATA[Ingo Nowitzky]]></dc:creator>
            <pubDate>Thu, 24 Oct 2024 10:45:16 GMT</pubDate>
            <atom:updated>2024-11-24T22:40:05.003Z</atom:updated>
            <content:encoded><![CDATA[<h4>Facilitate your understanding of complex neural network architectures</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*sw-LmjwUAme7l7mW7TSNyA.png" /><figcaption>Fig. 1: Fully Connected Neural Network replicating the 3D spiral | image by author</figcaption></figure><p>Complex neural network architectures like e.g. the transformer behind large language models (LLMs) are difficult to comprehend. It helped me a lot in understanding its function to break the whole architecture into blocks. Surprisingly, many of these blocks consist of fully connected neural networks that act as the computational engines within the complex structure. Similar to classic regression models, fully connected neural networks have the fascinating ability to learn any function or pattern from given training data and reproduce this pattern in the production phase.</p><p>In this article, I elaborate on this helpful perspective on fully connected neural networks and introduce them as highly sophisticated regression models —this is what I call <strong>Universal Function Modelers</strong>.</p><h3>Limitations of classic regression models</h3><p>Classic regression techniques, such as linear, polynomial, exponential, or logarithmic regression, aim to fit functions to data points in a way that captures the trends and patterns in the data as accurately as possible. However, these regression methods have <strong>several limitations</strong>:</p><ol><li>The type of function is predetermined. If it does not fit the data well, the resulting regression will also be poor.</li><li>The number of input data (independent variables) is usually limited.</li><li>The number of output data (dependent variables) is limited to one.</li></ol><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N3UUIKXHjthsk_XLQbqZlg.png" /><figcaption>Fig. 2: Classic regression techniques: linear, polynomial, exponential, logarithmic | image by author</figcaption></figure><h3>Neural Networks as regression models</h3><p>Neural networks have none of the above-mentioned limitations. They can approximate any pattern in the data and are not limited in the number of input or output variables — both can range <strong>from 1 to tens of thousands</strong>. As a result, neural networks are used in a wide variety of tasks:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*VfcghT9BrbmQgdTLKy2SeA.png" /><figcaption>Fig. 3: Examples for neural network tasks with different inputs and outputs | image by author</figcaption></figure><p>Let’s take a closer look at this important characteristic of neural networks using an example.<br>I generated scattered data points with a “crazy function” — a three-dimensional spiral with three intertwined strands.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/392/1*abhWDVVkNahkpBUUyFiF_w.png" /><figcaption>Eq. 1: Scattered 3D spiral function to produce sample data</figcaption></figure><p><em>x_i</em>, <em>y_i</em> and <em>z_i</em> are the coordinates of the scattered data points on the 3D spiral. Since we have three intertwined spirals, we have 3 stands x (<em>x</em>, <em>y</em>, <em>z</em>) = 9 coordinates, and index <em>i</em> ranges from 1 to 3.<em> R</em> is the radius of the spiral and is constant. <em>theta</em> is the first independent variable of the formula set and represents the angle, rotating from 0 to 4*pi. <em>Z</em> is the second independent variable, ranging from 0 to 20, and <em>ran</em> is a normal distributed random number which adds the scatter to the spiral.</p><p>Let us be very clear on the two remarks:</p><ol><li>We need Eq. 1 only for the data generation and we will <strong>not</strong> use it anywhere in our neural network.</li><li>The function and its data are only an <strong>arbitrary example</strong>, representing <strong>complex patterns hidden</strong> in the data. We could have used any other patterns.</li></ol><p>The task of the neural network is to learn the relationships between input and output data and, as a trained model, <strong>reproduce output values</strong> from new input data consistent with the patterns. In other words, the neural network is expected to learn the function from the data <strong>without knowing or using the mathematical formulas </strong>(Eq. 1).</p><h3>Loading the example data</h3><p>Since the data is only an example, it has been prepared in advance. Here, we load the <em>theta</em> and <em>z</em> input data as well as the <em>x1</em> to <em>z3</em> output data (9 coordinates). After loading, the output data is visualized.</p><pre>%matplotlib widget<br>import numpy as np<br>import matplotlib.pyplot as plt<br><br># Load data from Github repository<br>base_url = &#39;https://raw.githubusercontent.com/IngoNowitzky/UFM_Medium/main/&#39;<br>data_url = &#39;spiral_data.npy&#39;<br>labels_url = &#39;spiral_labels.npy&#39;<br>data = np.load(base_url + data_url)<br>labels = np.load(base_url + labels_url)<br><br># Determine parameters from the loaded data<br>n_strands = 3  # Number of strands<br>n_points = len(data)  # Number of points per spiral strand<br>colors = [&#39;b&#39;, &#39;r&#39;, &#39;g&#39;]  # Colors for the strands<br><br># Plot the loaded data<br>fig = plt.figure()<br>ax = fig.add_subplot(111, projection=&#39;3d&#39;)<br># Plot each strand of the spiral with unique colors<br>for i in range(n_strands):<br>    x = labels[:, i * 3]<br>    y = labels[:, i * 3 + 1]<br>    z = labels[:, i * 3 + 2]<br>    ax.scatter(x, y, z, color=colors[i], label=f&#39;Strand {i + 1}&#39;, alpha=0.6, marker=&#39;o&#39;)<br># Set labels for the axes<br>ax.set_xlabel(&#39;X-axis&#39;)<br>ax.set_ylabel(&#39;Y-axis&#39;)<br>ax.set_zlabel(&#39;Z-axis&#39;)<br>ax.set_title(&#39;Loaded 3D Spiral Data&#39;)<br>ax.legend()<br># Show the plot<br>plt.show()</pre><p>In the code snippet, we define base_url as the path to my GitHub repository and data_url and labels_url as the file names. We load the 9 coordinates <em>x_1</em> to <em>z_3</em> into the NumPy array labels and the two input variables <em>theta</em> and <em>z</em> into the array data. Next, we define n_stands as the number of spiral strands (needed to correctly distribute the data in the plot), n_points as the number of data points, and colors as a Python list containing the colors of the three spiral strands. Finally, we define a 3D Matplotlib figure and plot the labels data. The output of the above code is shown in Fig. 4:</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*IEaGDHwXttPz1g6Z4AwmGQ.png" /><figcaption>Fig. 4: Training data for the neural network regression | image by author</figcaption></figure><p>I tried to find a good perspective for the plot, but if you want to study the spirals in more detail, you can open an interactive view <a href="https://ingonowitzky.github.io/">here</a>.</p><h3>Defining the Neural Network</h3><p>Essentially, the task of the neural network is to learn the functional relationship between the two input values (<em>theta</em>, <em>z</em>) and the nine output values (<em>x_1</em>, <em>y_1</em>, …, <em>z_3</em>). This relationship is central to the training process and enables us to generate new outputs from new input values. As we will see, a very simple neural network is sufficient for this task.</p><h4>Data preparation</h4><p>First, we prepare the data for model training. We start by converting the NumPy arrays data and labels into the PyTorch tensors data_tensor and labels_tensor, and combine them in a dataset. Next, we instantiate the dataloader, which enables us to feed our model with randomly shuffled data batches of size 16 during training.</p><pre>import torch<br>from torch.utils.data import DataLoader, TensorDataset<br><br># Set seed for reproducibility<br>torch.manual_seed(42)<br><br># Convert to PyTorch tensors<br>data_tensor = torch.tensor(data, dtype=torch.float32)<br>labels_tensor = torch.tensor(labels, dtype=torch.float32)<br><br># Create DataLoader for training<br>dataset = TensorDataset(data_tensor, labels_tensor)<br>dataloader = DataLoader(dataset, batch_size=16, shuffle=True)</pre><h4>Model definition</h4><p>Second, we define the neural network. As mentioned above, a relatively simple fully connected neural network is sufficient for our task. <br>We need 2 input nodes for the input variables <em>theta</em> and <em>z</em>, two hidden layers with 64 nodes each (my choice), and an output layer with 9 nodes corresponding to the nine output variables — the 3D coordinates <em>x_1</em>, …, <em>z_3</em>. After each of the two hidden layers, a ReLU activation function is applied.</p><pre>import torch.nn as nn<br><br># Define the neural network model<br>class NeuralNetwork(nn.Module):<br>    def __init__(self):<br>        super(NeuralNetwork, self).__init__()<br>        self.fc1 = nn.Linear(2, 64)     # First layer with 2 input features (theta, z)<br>        self.fc2 = nn.Linear(64, 64)    # Hidden layer with 64 nodes<br>        self.fc3 = nn.Linear(64, 3*3)   # Output layer with 3 output sets (x, y, z)<br><br>    def forward(self, x):<br>        x = torch.relu(self.fc1(x))<br>        x = torch.relu(self.fc2(x))<br>        x = self.fc3(x)<br>        return x</pre><p>We define the network structure in the class NeuralNetwork() and inherit from nn.Module. As is common, the layers are defined in the __init__() method. If you are not yet familiar with neural network definitions in PyTorch, please don’t be confused: We do not define the layers in terms of nodes but rather the interconnections between the nodes. That is why we define three layers instead of four, as shown in Fig. 1.</p><ul><li>self.fc1 defines the connections between the input layer (2 nodes) and the first hidden layer (64 nodes).</li><li>self.fc2 defines the connections between the first hidden layer (64 nodes) and the second hidden layer (64 nodes).</li><li>Finally, self.fc3 defines the connections between the second hidden layer (64 nodes) and the output layer (9 nodes).</li></ul><p>Please refer to the left side of Fig. 1 for reference.</p><p>The forward() method defines the data flow in the network. It expects an input tensor x, feeds it into self.fc1(), and applies ReLU. The result is then passed to self.fc2(), with ReLU activation applied again. Finally, the output is passed through self.fc3().</p><h4>Model training</h4><p>The most interesting part of the code is the training process. We instantiate our model class NeuralNetwork() as model and select Mean Squared Error (nn.MSELoss()) as the loss function. The Adam optimizer (optim.Adam()) handles the training process.<br>We train the model for 500 epochs (n_epochs) and print the current loss every 10 epochs. In the inner for-loop, we iterate over the dataloader and process batches of 16 datasets at a time.</p><pre>import torch.optim as optim<br><br># Instantiate the model, define the loss function and the optimizer<br>model = NeuralNetwork()<br>criterion = nn.MSELoss()<br>optimizer = optim.Adam(model.parameters(), lr=0.001)<br><br># Train the model<br>n_epochs = 500<br>for epoch in range(n_epochs):<br>    model.train()<br>    running_loss = 0.0<br>    for batch in dataloader:<br>        inputs, targets = batch<br>        optimizer.zero_grad()<br>        outputs = model(inputs)<br>        loss = criterion(outputs, targets)<br>        loss.backward()<br>        optimizer.step()<br>        running_loss += loss.item()<br>    <br>    if (epoch + 1) % 10 == 0:<br>        print(f&#39;Epoch [{epoch + 1}/{n_epochs}], Loss: {running_loss / len(dataloader):.4f}&#39;)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*DJDrl-9jzg9rADAoeeE2WA.png" /></figure><p>The training loss decreases from approximately 0.29 to 0.01, indicating that the model successfully approximates the function values.</p><h3>Approximating the 3D spiral</h3><p>The model is now trained, and the decrease in loss values looks promising. Next, we want to check how well the neural network has learned the 3D spiral “formula set” from the data. To do this, we generate new <em>theta</em> and <em>z</em> values, feed them into the trained model, and compute the nine output values (<em>x_1</em>, …, <em>z_3</em>). We plot the values as solid lines and overlay them with the original points. Let’s go through the code in a little more detail:</p><p>theta_new and z_new are 1,000 new values used as input for the neural network model. We combine them to the 2D array data_new with 1,000 rows and 2 columns. The NumPy array is then converted into a PyTorch tensor named data_new_tensor. We feed this tensor in the model(), and receive the model’s response in labels_pred_tensor. Finally, the tensor is transformed back into the NumPy array labels_pred for further processing.<br>Next, we define a 3D Matplotlib figure and iterate over the true data points labels and the reconstructed points labels_pred. We plot the true labels as points and the reconstruction as solid lines. The three strands are distinguished using the colors blue, red and green.</p><pre># Generate new theta and z values for line plot<br>theta_new = np.linspace(0, 4 * np.pi, 1000)  # Generate more points for a smoother line<br>z_new = np.linspace(0, 20, 1000)  # z-axis values corresponding to theta_new<br><br># Prepare the new input data<br>data_new = np.column_stack((theta_new, z_new))<br>data_new_tensor = torch.tensor(data_new, dtype=torch.float32)<br><br># Predict using the trained model<br>model.eval()<br>labels_pred_tensor = model(data_new_tensor)<br>labels_pred = labels_pred_tensor.detach().numpy()<br><br># Plot the original and predicted points for comparison<br>fig = plt.figure()<br>ax = fig.add_subplot(111, projection=&#39;3d&#39;)<br><br># Plot the true points for each strand<br>n_strands = 3<br>colors = [&#39;b&#39;, &#39;r&#39;, &#39;g&#39;]<br>for i in range(n_strands):<br>    x_true = labels[:, i * 3]<br>    y_true = labels[:, i * 3 + 1]<br>    z_true = labels[:, i * 3 + 2]<br>    ax.scatter(x_true, y_true, z_true, color=colors[i], label=f&#39;True Points Strand {i + 1}&#39;, alpha=0.6, marker=&#39;o&#39;, s = 8)<br><br># Plot the predicted line for each strand<br>for i in range(n_strands):<br>    x_pred = labels_pred[:, i * 3]<br>    y_pred_ = labels_pred[:, i * 3 + 1]<br>    z_pred = labels_pred[:, i * 3 + 2]<br>    ax.plot(x_pred, y_pred_, z_pred, color=colors[i], label=f&#39;Predicted Line Strand {i + 1}&#39;, alpha=0.8, linewidth = 3)<br><br># Set labels for the axes<br>ax.set_xlabel(&#39;X-axis&#39;)<br>ax.set_ylabel(&#39;Y-axis&#39;)<br>ax.set_zlabel(&#39;Z-axis&#39;)<br>ax.set_title(&#39;Neural Network Regression for 3D Spirals (PyTorch)&#39;)<br>ax.legend(loc=&#39;best&#39;, fontsize= 7)<br><br># Show the plot<br>plt.show()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/698/1*hgD7PGo-9Z34-yngXetLxA.png" /><figcaption>Fig. 5: Approximation (solid lines) from neural network and original values (points) | image by author</figcaption></figure><p>Again, I tried to find a good perspective on the 3D data. If you want to study the approximation in more detail, you can explore an interactive 3D plot <a href="https://ingonowitzky.github.io/approximation_3d_spiral.html">here</a>.</p><p>The plot demonstrates that our simple neural network has successfully learned the three spiral functions from the training data and can now transform new input data into points on the spirals. This is exactly what we aimed to achieve. It is evident that the neural network models our “crazy function” well!<br><strong>This confirms that neural networks are indeed Universal Function Modelers</strong>.</p><h3>Conclusion</h3><p>Viewing fully connected neural networks as <strong>Universal Function Modelers</strong> or <strong>Computational Engines</strong> has greatly helped me understanding complex neural network architectures like the transformer used in Large Language Models (LLMs).</p><p>This analogy may also assist readers in demystifying other complex network architectures that underpin modern AI. Interpreting the fully connected neural network blocks in a complex architecture as a flexible function modeler allows to unpuzzle more sophisticated architectures build on these basic principles step-by-step.</p><p><strong>I hope this perspective is as helpful for you as it is for me in understanding advanced neural network architectures.</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=40eccfa80e51" width="1" height="1" alt=""><hr><p><a href="https://aiadvances.org/neural-networks-are-universal-function-modelers-40eccfa80e51">Neural Networks are Universal Function Modelers</a> was originally published in <a href="https://aiadvances.org/">AI Advances</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
        <item>
            <title><![CDATA[Linear Discriminant Analysis (LDA)]]></title>
            <link>https://medium.com/data-science/linear-discriminant-analysis-lda-598d8e90f8b9?source=rss-2628a598423a------2</link>
            <guid isPermaLink="false">https://medium.com/p/598d8e90f8b9</guid>
            <category><![CDATA[machine-learning]]></category>
            <category><![CDATA[failure-detection]]></category>
            <category><![CDATA[data-preparation]]></category>
            <category><![CDATA[class-separation]]></category>
            <category><![CDATA[dimensionality-reduction]]></category>
            <dc:creator><![CDATA[Ingo Nowitzky]]></dc:creator>
            <pubDate>Sat, 12 Oct 2024 17:48:27 GMT</pubDate>
            <atom:updated>2024-10-14T15:47:38.626Z</atom:updated>
            <content:encoded><![CDATA[<h4>Discover how LDA helps identify critical data features</h4><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*OQ4qE0xXKDOaRBtBAtNACg.png" /><figcaption>Classification of LDA within AI and ML Methods | image by author</figcaption></figure><p><strong>This article aims to explore Linear Discriminant Analysis (LDA), focusing on its core ideas, its mathematical implementation in code, and a practical example from manufacturing.<br>I hope you’re on board. Let’s get started!</strong></p><p>Who works with <strong>industrial data</strong> in practice will be familiar with this situation: The datasets usually have many features, and it is often unclear which of the features are important and which are less. “Important” is a relative term in this context. Often, the goal is to differentiate the datasets from each other, i.e., to classify them. A very typical task is to distinguish good parts from bad parts and to identify the causes (aka features) that lead to the failure of the parts.</p><p>A commonly used method is the well-known Principal Component Analysis (PCA). While PCA belongs to the <strong>un</strong>supervised methods, the less widespread LDA is a supervised method and thus learns from <strong>labeled data</strong>. Therefore, it is particularly suited for explaining failure patterns from large datasets.</p><h3>1. Goal and Principle of LDA</h3><p>The goal of LDA is to linearly combine the features of the data so that the labels of the datasets are best separated from each other, and the number of new features is reduced to a predefined count. In AI jargon, this is typically referred to as a<strong> projection to a lower-dimensional space</strong>.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*fnaHijTNEzDgVLrnOfsKZQ.png" /><figcaption>Principle of LDA | image modified from <a href="https://github.com/rasbt/python-machine-learning-book-3rd-edition/blob/master/ch05/images/05_06.png">Raschka/Mirjalili, 2019</a></figcaption></figure><h3>Excursus: What is dimensionality and what is dimensionality reduction?</h3><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*2w3EBVnyBuemH0l_mPZfMQ.png" /><figcaption>Dimensions and graphical representation | image by author</figcaption></figure><p>Dimensionality refers to the number of features in a dataset.<br>With just one measurement (or feature), such as the tool temperature from an injection molding machine, we can represent it on a number line. Two features, like temperature and tool pressure, are still manageable: we can easily plot the data on an x-y chart. With three features — temperature, tool pressure, and injection pressure — things get more interesting, but we can still plot the data in a 3D x-y-z chart. However, when we add more features, such as viscosity, electrical conductivity, and others, the complexity increases.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*QQFcV9IjsbKzJimlC8UtIg.png" /><figcaption>Dimensionality reduction | image by author</figcaption></figure><p>In practice, datasets often contain hundreds or even thousands of features. This presents a challenge because many machine learning algorithms perform poorly as datasets grow too large. Additionally, the amount of data required increases exponentially with the number of dimensions to achieve statistical significance. This phenomenon is known as the “curse of dimensionality.” These factors make it essential to determine which features are relevant and to eliminate the less meaningful ones early in the data science process.</p><h3>2. How does LDA work?</h3><p>The process of Linear Discriminant Analysis (LDA) can be broken down into five key steps.</p><p><strong>Step 1:</strong> Compute the <em>d</em>-dimensional mean vectors for each of the <em>k</em> classes separately from the dataset.</p><p>Remember that LDA is a supervised machine learning technique, meaning we can utilize the known labels. In the first step, we calculate the mean vectors <em>mean_c</em> for all samples belonging to a specific class <em>c</em>. To do this, we filter the feature matrix by class label and compute the mean for each of the <em>d</em> features. As a result, we obtain <em>k</em> mean vectors (one for each of the <em>k</em> classes), each with a length of <em>d</em> (corresponding to the <em>d</em> features).</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*N-VbzV-PotaEJ7nwK6oSIQ.png" /><figcaption>Label vector Y and feature matrix X | image by author</figcaption></figure><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2FiuZC2ZVaB4XyggAqvpFu2h&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2FiuZC2ZVaB4XyggAqvpFu2h&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="393" height="165" frameborder="0" scrolling="no"><a href="https://medium.com/media/8293011456cc354f1ed465ba80445186/href">https://medium.com/media/8293011456cc354f1ed465ba80445186/href</a></iframe><p><strong>Step 2:</strong> Compute the scatter matrices (between-class scatter matrix and within-class scatter matrix).</p><p>The within-class scatter matrix measures the variation among samples within the same class. To find a subspace with optimal separability, we aim to minimize the values in this matrix. In contrast, the between-class scatter matrix measures the variation between different classes. For optimal separability, we aim to maximize the values in this matrix.<br>Intuitively, within-class scatter looks at how compact each class is, whereas between-class scatter examines how far apart different classes are.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*0fCEtLfTJPYrjcJ5ihaFOQ.png" /><figcaption>Within-class and between-class scatter matrices | image by author</figcaption></figure><p>Let’s start with the <strong>within-class scatter</strong> matrix <em>S_W</em>. It is calculated as the sum of the scatter matrices <em>S_c</em> for each individual class:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2FqFN5nw43n88oPdXdbSsUJd&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2FqFN5nw43n88oPdXdbSsUJd&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="343" height="258" frameborder="0" scrolling="no"><a href="https://medium.com/media/a9e0d2aaa8eb85097867c58ffd26bdb0/href">https://medium.com/media/a9e0d2aaa8eb85097867c58ffd26bdb0/href</a></iframe><p>The <strong>between-class scatter</strong> matrix <em>S_B</em> is derived from the differences between the class means <em>mean_c</em> and the overall mean of the entire dataset:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2Fwo62DkqV8Q3R7pzfSVMJGa&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2Fwo62DkqV8Q3R7pzfSVMJGa&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="859" height="112" frameborder="0" scrolling="no"><a href="https://medium.com/media/5d2f4ea097c6d276930ea9ee9501195e/href">https://medium.com/media/5d2f4ea097c6d276930ea9ee9501195e/href</a></iframe><p>where <em>mean</em> refers to the mean vector calculated over all samples, regardless of their class labels.</p><p><strong>Step 3:</strong> Calculate the eigenvectors and eigenvalues for the ratio of <em>S_W</em>​ and <em>S_B</em>​.</p><p>As mentioned, for optimal class separability, we aim to maximize <em>S_B​</em> and minimize <em>S_W</em>​. We can achieve both by maximizing the ratio <em>S_B/S_W</em>​. In linear algebra terms, this ratio corresponds to the scatter matrix <em>S_W⁻¹ S_B</em>​, which is maximized in the subspace spanned by the eigenvectors with the highest eigenvalues. The eigenvectors define the directions of this subspace, while the eigenvalues represent the magnitude of the distortion. We will select the <em>m</em> eigenvectors associated with the highest eigenvalues.</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2FmkgXLrbYGNN2TfBBccqjUL&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2FmkgXLrbYGNN2TfBBccqjUL&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="315" height="137" frameborder="0" scrolling="no"><a href="https://medium.com/media/cabb6285ee55db461fd3e11c25786bd6/href">https://medium.com/media/cabb6285ee55db461fd3e11c25786bd6/href</a></iframe><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*gxwb2V5kGl2tg-mXITN29A.png" /><figcaption>Subspace spanned by eigenvectors | image by author</figcaption></figure><p><strong>Step 4:</strong> Sort the eigenvectors in descending order of their corresponding eigenvalues, and select the <em>m</em> eigenvectors with the largest eigenvalues to form a <em>d × m-</em>dimensional transformation matrix <em>W</em>.</p><p>Remember, our goal is not only to project the data into a subspace that enhances class separability but also to reduce dimensionality. The eigenvectors will define the axes of our new feature subspace. To decide which eigenvectors to discard for the lower-dimensional subspace, we need to examine their corresponding eigenvalues. In simple terms, the eigenvectors with the smallest eigenvalues contribute the least to class separation, and these are the ones we want to drop. The typical approach is to rank the eigenvalues in descending order and select the top <em>m</em> eigenvectors. <em>m</em> is a freely chosen parameter. The larger <em>m</em>, the less information is lost during the transformation.</p><p>After sorting the eigenpairs by decreasing eigenvalues and selecting the top <em>m</em> pairs, the next step is to construct the <em>d × m</em>-dimensional transformation matrix <em>W</em>. This is done by stacking the <em>m</em> selected eigenvectors horizontally, resulting in the matrix <em>W</em>:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2FqMwLT2TiDK3PHP5sYAWSfF&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2FqMwLT2TiDK3PHP5sYAWSfF&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="200" height="230" frameborder="0" scrolling="no"><a href="https://medium.com/media/6e4d124d4d3f9eacf26ab204f79edede/href">https://medium.com/media/6e4d124d4d3f9eacf26ab204f79edede/href</a></iframe><p>The first column of <em>W</em> represents the eigenvector corresponding to the highest eigenvalue, the second column represents the eigenvector corresponding to the second highest eigenvalue, and so on.</p><p><strong>Step 5:</strong> Use <em>W</em> to project the samples onto the new subspace.</p><p>In the final step, we use the <em>d × m-</em>dimensional transformation matrix <em>W</em>, which we composed from the top <em>m</em> selected eigenvectors, to project our samples onto the new subspace:</p><iframe src="https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fmath.embed.fun%2Fembed%2FxyKYukAaaKWpHY1tA581Qz&amp;display_name=Embed+Fun&amp;url=https%3A%2F%2Fmath.embed.fun%2FxyKYukAaaKWpHY1tA581Qz&amp;key=a19fcc184b9711e1b4764040d3dc5c07&amp;type=text%2Fhtml&amp;schema=embed" width="200" height="136" frameborder="0" scrolling="no"><a href="https://medium.com/media/dca65c1d08d60cc8dad7bdbf0c6d0565/href">https://medium.com/media/dca65c1d08d60cc8dad7bdbf0c6d0565/href</a></iframe><p>where <em>X</em> is the initial <em>n × d</em>-dimensional feature matrix representing our samples, and <em>Z</em> is the newly transformed <em>n × m</em>-dimensional feature matrix in the new subspace. This means that the selected eigenvectors serve as the “recipes” for transforming the original features into the new features (the <strong>Linear Discriminants</strong>): The eigenvector with the highest eigenvalue provides the transformation recipe for <strong>LD1</strong>, the eigenvector with the second highest eigenvalue corresponds to <strong>LD2</strong>, and so on.</p><figure><img alt="" src="https://cdn-images-1.medium.com/max/1024/1*x_DupdGQKwHFvpsAaA950A.png" /><figcaption>Projection of X onto the linear discriminants LD</figcaption></figure><h3>3. Implementing Linear Discriminant Analysis (LDA) from Scratch</h3><p>To demonstrate the theory and mathematics in action, we will program our own LDA from scratch using only numpy.</p><pre>import numpy as np<br><br><br>class LDA_fs:<br>    &quot;&quot;&quot;<br>    Performs a Linear Discriminant Analysis (LDA)<br><br>    Methods<br>    =======<br>    fit_transform():<br>        Fits the model to the data X and Y, derives the transformation matrix W<br>    and projects the feature matrix X onto the m LDA axes<br>    &quot;&quot;&quot;<br><br>    def __init__(self, m):<br>        &quot;&quot;&quot;<br>        Parameters<br>        ==========<br>        m : int<br>            Number of LDA axes onto which the data will be projected<br><br>        Returns<br>        =======<br>        None<br>        &quot;&quot;&quot;<br>        self.m = m<br><br>    def fit_transform(self, X, Y):<br>        &quot;&quot;&quot;<br>        Parameters<br>        ==========<br>        X : array(n_samples, n_features)<br>            Feature matrix of the dataset<br>        Y = array(n_samples)<br>            Label vector of the dataset<br><br>        Returns<br>        =======<br>        X_transform : New feature matrix projected onto the m LDA axes<br>        <br>        &quot;&quot;&quot;<br><br>        # Get number of features (columns)<br>        self.n_features = X.shape[1]<br>        # Get unique class labels<br>        class_labels = np.unique(Y)<br>        # Get the overall mean vector (independent of the class labels)<br>        mean_overall = np.mean(X, axis=0)  # Mean of each feature<br>        # Initialize both scatter matrices with zeros<br>        SW = np.zeros((self.n_features, self.n_features))  # Within scatter matrix<br>        SB = np.zeros((self.n_features, self.n_features))  # Between scatter matrix<br><br>        # Iterate over all classes and select the corresponding data<br>        for c in class_labels:<br>            # Filter X for class c<br>            X_c = X[Y == c]<br>            # Calculate the mean vector for class c<br>            mean_c = np.mean(X_c, axis=0)<br>            # Calculate within-class scatter for class c<br>            SW += (X_c - mean_c).T.dot((X_c - mean_c))<br>            # Number of samples in class c<br>            n_c = X_c.shape[0]<br>            # Difference between the overall mean and the mean of class c --&gt; between-class scatter<br>            mean_diff = (mean_c - mean_overall).reshape(self.n_features, 1)<br>            SB += n_c * (mean_diff).dot(mean_diff.T)<br><br>        # Determine SW^-1 * SB<br>        A = np.linalg.inv(SW).dot(SB)<br>        # Get the eigenvalues and eigenvectors of (SW^-1 * SB)<br>        eigenvalues, eigenvectors = np.linalg.eig(A)<br>        # Keep only the real parts of eigenvalues and eigenvectors<br>        eigenvalues = np.real(eigenvalues)<br>        eigenvectors = np.real(eigenvectors.T)<br><br>        # Sort the eigenvalues descending (high to low)<br>        idxs = np.argsort(np.abs(eigenvalues))[::-1]<br>        self.eigenvalues = np.abs(eigenvalues[idxs])<br>        self.eigenvectors = eigenvectors[idxs]<br>        # Store the first m eigenvectors as transformation matrix W<br>        self.W = self.eigenvectors[0:self.m]<br><br>        # Transform the feature matrix X onto LD axes<br>        return np.dot(X, self.W.T)</pre><h3>4. Applying LDA to an Industrial Dataset</h3><p>To see LDA in action, we will apply it to a typical task in the production environment. We have data from a simple manufacturing line with only 7 stations. Each of these stations sends a data point (yes, I know, only one data point is highly unrealistic). Unfortunately, our line produces a significant number of defective parts, and we want to find out which stations are responsible for this.</p><p>First, we load the data and take an initial look.</p><pre>import pandas as pd<br><br># URL to Github repository<br>url = &quot;https://raw.githubusercontent.com/IngoNowitzky/LDA_Medium/main/production_line_data.csv&quot;<br><br># Read csv to DataFrame<br>data = pd.read_csv(url)<br><br># Print first 5 lines<br>data.head()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/662/1*IHl_-rYCyoBpgbov9ZXiAQ.png" /></figure><p>Next, we study the distribution of the data using the .describe() method from Pandas.</p><pre># Show average, min and max of numerical values<br>data.describe()</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/810/1*bp5HoeXSwbekKjWXdralgw.png" /></figure><p>We see that we have 20,000 data points, and the measurements range from -5 to +150. Hence, we note for later that we need to normalize the dataset: the different magnitudes of the numerical values would otherwise negatively affect the LDA.<br>How many good parts and how many bad parts do we have?</p><pre># Count the number of good and bad parts<br>label_counts = data[&#39;Label&#39;].value_counts()<br><br># Display the results<br>print(&quot;Number of Good and Bad Parts:&quot;)<br>print(label_counts)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/791/1*CNw-Ue_g8Q_FsJPOZ4iIrw.png" /></figure><p>We have 19,031 good parts and 969 defective parts. The fact that the dataset is so imbalanced is an issue for further analysis. Therefore, we select all defective parts and an equal number of randomly chosen good parts for the further processing.</p><pre># Select all bad parts<br>bad_parts = data[data[&#39;Label&#39;] == &#39;Bad&#39;]<br><br># Randomly select an equal number of good parts<br>good_parts = data[data[&#39;Label&#39;] == &#39;Good&#39;].sample(n=len(bad_parts), random_state=42)<br><br># Combine both subsets to create a balanced dataset<br>balanced_data = pd.concat([bad_parts, good_parts])<br><br># Shuffle the combined dataset<br>balanced_data = balanced_data.sample(frac=1, random_state=42).reset_index(drop=True)<br><br># Display the number of good and bad parts in the balanced dataset<br>print(&quot;Number of Good and Bad Parts in the balanced dataset:&quot;)<br>print(balanced_data[&#39;Label&#39;].value_counts())</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/727/1*L_SVv2hkWxY_SM4F6J2F1w.png" /></figure><p>Now, let’s apply our LDA from scratch to the balanced dataset. We use the StandardScaler from sklearn to normalize the measurements for each feature to have a mean of 0 and a standard deviation of 1. We choose only one linear discriminant axis (<em>m=1</em>) onto which we project the data. This helps us clearly see which features are most relevant in distinguishing good from bad parts, and we visualize the projected data in a histogram.</p><pre>import matplotlib.pyplot as plt<br>from sklearn.preprocessing import StandardScaler<br><br># Separate features and labels<br>X = balanced_data.drop(columns=[&#39;Label&#39;])<br>y = balanced_data[&#39;Label&#39;]<br><br># Normalize the features<br>scaler = StandardScaler()<br>X_scaled = scaler.fit_transform(X)<br><br># Perform LDA<br>lda = LDA_fs(m=1)  # Instanciate LDA object with 1 axis<br>X_lda = lda.fit_transform(X_scaled, y) # Fit the model and project the data<br><br># Plot the LDA projection<br>plt.figure(figsize=(10, 6))<br>plt.hist(X_lda[y == &#39;Good&#39;], bins=20, alpha=0.7, label=&#39;Good&#39;, color=&#39;green&#39;)<br>plt.hist(X_lda[y == &#39;Bad&#39;], bins=20, alpha=0.7, label=&#39;Bad&#39;, color=&#39;red&#39;)<br>plt.title(&quot;LDA Projection of Good and Bad Parts&quot;)<br>plt.xlabel(&quot;LDA Component&quot;)<br>plt.ylabel(&quot;Frequency&quot;)<br>plt.legend()<br>plt.show()<br><br># Examine feature contributions to the LDA component<br>feature_importance = pd.DataFrame({&#39;Feature&#39;: X.columns, &#39;LDA Coefficient&#39;: lda.W[0]})<br>feature_importance = feature_importance.sort_values(by=&#39;LDA Coefficient&#39;, ascending=False)<br><br># Display feature importance<br>print(&quot;Feature Contributions to LDA Component:&quot;)<br>print(feature_importance)</pre><figure><img alt="" src="https://cdn-images-1.medium.com/max/1001/1*mXrC3tBxEQMZxmY4uFqOlg.png" /><figcaption>Feature matrix projected to one LD (m=1)</figcaption></figure><figure><img alt="" src="https://cdn-images-1.medium.com/max/639/1*tXg8b0JFGIb5FrXx6MRV7A.png" /><figcaption>Feature importance = How much do the stations contribute to class separation?</figcaption></figure><p>The histogram shows that we can separate the good parts from the defective parts very well, with only a small overlap. This is already a positive result and indicates that our LDA was successful.</p><p>The “LDA Coefficients” from the table “Feature Contributions to LDA Components” represent the eigenvector from the first (and only, since <em>m=1</em>) column of our transformation matrix <em>W</em>. They indicate the direction and magnitude with which the normalized measurements from the stations are projected onto the linear discriminant axis. The values in the table are sorted in descending order. We need to read the table from both the top and the bottom simultaneously because the absolute value of the coefficient indicates the significance of each station in separating the classes and, consequently, its contribution to the production of defective parts. The sign indicates whether a lower or higher measurement increases the likelihood of defective parts. Let’s take a closer look at our example:</p><p>The largest absolute value is from Station 4, with a coefficient of -0.672. This means that Station 4 has the strongest influence on part failure. Due to the negative sign, higher positive measurements are projected towards a negative linear discriminant (LD). The histogram shows that a negative LD is associated with good (green) parts. Conversely, <strong>low and negative measurements at this station increase the likelihood of part failure</strong>.<br>The second highest absolute value is from Station 2, with a coefficient of 0.557. Therefore, this station is the second most significant contributor to part failures. The positive sign indicates that high positive measurements are projected towards the positive LD. From the histogram, we know that a high positive LD value is associated with a high likelihood of failure. In other words, <strong>high measurements at Station 2 lead to part failures</strong>.<br>The third highest coefficient comes from Station 7, with a value of -0.486. This makes Station 7 the third largest contributor to part failures. The negative sign again indicates that high positive values at this station lead to a negative LD (which corresponds to good parts). Conversely, <strong>low and negative values at this station lead to part failures</strong>.<br>All other LDA coefficients are an order of magnitude smaller than the three mentioned, <strong>the associated stations therefore have no influence on part failure</strong>.</p><p>Are the results of our LDA analysis correct? As you may have already guessed, the production dataset is synthetically generated. I labeled all parts as defective where the measurement at Station 2 was greater than 0.5, the value at Station 4 was less than -2.5, and the value at Station 7 was less than 3. <strong>It turns out that the LDA hit the mark perfectly!</strong></p><pre># Determine if a sample is a good or bad part based on the conditions<br>data[&#39;Label&#39;] = np.where(<br>    (data[&#39;Station_2&#39;] &gt; 0.5) &amp; (data[&#39;Station_4&#39;] &lt; -2.5) &amp; (data[&#39;Station_7&#39;] &lt; 3),<br>    &#39;Bad&#39;,<br>    &#39;Good&#39;<br>)</pre><h3>5. Conclusion</h3><p>Linear Discriminant Analysis (LDA) not only reduces the complexity of datasets but also highlights the key features that drive class separation, making it highly effective for identifying failure causes in production systems. It is a straightforward yet powerful method with practical applications and is readily available in libraries like scikit-learn.</p><p>To achieve optimal results, it is crucial to balance the dataset (ensure a similar number of samples in each class) and normalize it (mean of 0 and standard deviation of 1).<br><strong>The next time you work with a large dataset containing class labels and numerous features, why not give LDA a try?</strong></p><img src="https://medium.com/_/stat?event=post.clientViewed&referrerSource=full_rss&postId=598d8e90f8b9" width="1" height="1" alt=""><hr><p><a href="https://medium.com/data-science/linear-discriminant-analysis-lda-598d8e90f8b9">Linear Discriminant Analysis (LDA)</a> was originally published in <a href="https://medium.com/data-science">TDS Archive</a> on Medium, where people are continuing the conversation by highlighting and responding to this story.</p>]]></content:encoded>
        </item>
    </channel>
</rss>