Generative AI (GenAI) techniques utilizing deep learning, neural networks, and other sophisticated machine learning have enabled tremendous advances in applications like computer vision, natural language processing, creative content generation, and more. However, the powerful capabilities of GenAI also raise critical ethical concerns regarding bias, transparency, accountability, and unintended societal consequences that must be proactively addressed as adoption continues growing rapidly.

One report breaks the types of ethical and moral concerns from AI into eight categories:

1. Lack of explainability
2. Incorporation of human bias
3. Inherent English and European bias of training data
4. Lack and impact of regulation, now and in the future
5. Fear of apocalyptic outcomes
6. Lack of accountability by developers
7. Impact to the environment
8. Effect on labor, both in the data mines and the coming battlefield of machine versus human productivity

This post provides an in-depth look at a handful of these key ethical issues posed by GenAI applications, highlighting recent problems, and analyzing potential solutions that allow the technology to be harnessed responsibly.

Bias in GenAI Models

One of the most pressing issues with GenAI systems is the perpetuation of unfair bias based on flaws in the training data or algorithms. Without careful consideration, models can learn and amplify existing societal biases around race, gender, age, ethnicity and more from biased data. This leads to discriminatory and prejudicial outcomes that can severely impact individuals and groups.

The below graphic from CSA Research illustrates how training data can lead to different types of bias in models. What makes this even more difficult to pin down is that most model developers never release even the metadata of the training data — if anything is open sourced, it is usually the fine-tuning data, which is highly curated and not representative of the full training set.

How does this bias in the training data manifest in AI? Facial analysis models have demonstrated higher error rates for women and darker-skinned individuals. Hiring algorithms have disadvantaged qualified candidates from minority backgrounds. Predictive policing systems exhibit racial biases, leading to over-policing in marginalized communities.

If we look at generative AI in particular, Bloomberg conducted a study analyzing the outputs of Stable Diffusion text-to-image models and concluded, “The world according to Stable Diffusion is run by white male CEOs. Women are rarely doctors, lawyers or judges. Men with dark skin commit crimes, while women with dark skin flip burgers.

Stable Diffusion generates images using artificial intelligence, in response to written prompts. Like many AI models, what it creates may seem plausible on its face but is actually a distortion of reality. An analysis of more than 5,000 images created with Stable Diffusion found that it takes racial and gender disparities to extremes — worse than those found in the real world.” (Link)

Mitigating bias requires a multi-pronged approach. Some considerations include:

• Preparing training data: Curating data carefully, enhancing diversity, and correcting imbalances help reduce baked-in bias.
• Fairness metrics: Rigorously defining, evaluating, and optimizing for fairness measures can quantify and mitigate algorithmic biases.
• Techniques like adversarial debiasing explicitly optimizes models to reduce learned biases.
• Inclusive design from the beginning can ensure that other use cases are considered. If data is primarily sourced from
• Testing models in different environments. One framework that the World Economic Forum offers is STEEPV. “STEEPV analysis is a method of performing strategic analysis of external environments. It is an acronym for social (i.e. societal attitudes, culture and demographics), technological, economic (ie interest, growth and inflation rate), environmental, political and values. Performing a STEEPV analysis will help you detect fairness and non-discrimination risks in practice.” (Link)
• Increasing diversity of data and teams building GenAI systems also helps offset biases. Many leading companies also have specific responsible AI teams or an AI ethics office that are engaged in the end-to-end process.

There is no straightforward answer to solving these challenges, but a mix of these methods is starting to result in a set of best practices that work toward mitigating bias.

Lack of Explainability and Interpretability

With complex GenAI models like deep neural networks, it is often not possible to fully explain the rationale behind their decisions and predictions. However, understanding model behaviors, especially for sensitive applications in areas like finance, healthcare and criminal justice, is essential to ensure transparency, trace accountability, and avoid harm.

Several approaches to enhance interpretability are being explored:

• Local explainability focuses on explaining the reasons for specific individual predictions. This fosters user confidence in the system.
• Rule-based explanations approximate the model logic through more interpretable rules and logical statements.
• Highlighting influential input features helps users understand key data driving model outputs.
• Model visualization techniques also provide valuable insights into model mechanics.

Legal and Regulatory Challenges

GenAI systems must adhere to existing regulations around data privacy, consumer protections and anti-discrimination laws. However, the rapid evolution of AI capabilities presents policy challenges. Key steps needed include:

• Developing techniques like federated learning and differential privacy to preserve user privacy.
• Establishing rigorous internal governance procedures for responsible data collection and use practices.
• Advocating for regulations specific to GenAI systems, focused on transparency and accountability.
• Adhering to ethical principles and frameworks on fairness and bias mitigation proposed by organizations like the EU and OECD.

Handling Hallucinations and Unintended Harmful Effects

A known issue with large language models is the propensity to generate false information or content that promotes harmful stereotypes due to lack of real world knowledge and common sense. For instance, ChatGPT has produced toxic text, harmful misinformation and biased narratives when prompted.

Potential solutions to mitigate the risk of societal harm include:

• Adversarial testing to catch flaws and make models more robust.
• Incorporating human feedback loops to correct bad behavior and guide the model.
• Monitoring deployments and having human oversight over high-risk applications.

The Path Forward for Ethical GenAI

Addressing the multifaceted ethical challenges posed by advancing GenAI requires sustained engagement between stakeholders from research, policy, industry and civil society. Key focal areas for enabling responsible GenAI include:

• Promoting participation of diverse voices in GenAI development and governance.
• Supporting ongoing research to improve techniques and best practices for bias detection and mitigation, explainability, transparency, and error handling.
• Advocating for regulatory policy development to ensure accountability and prevent harm from unchecked GenAI deployment.
• Incentivizing ethical engineering practices focused on safety, transparency and fairness throughout the GenAI model lifecycle.

With proactive efforts to put ethics at the core of the GenAI application development process, this transformative technology can uplift society in countless ways while minimizing risks from misuse and unintended consequences. All stakeholders have a shared responsibility to steer the GenAI revolution towards just and beneficial outcomes for all.

Conclusion

GenAI innovation carries tremendous potential for positive impact but also poses novel challenges we must thoughtfully address. This will require sustained collaboration and open dialogue between all parties to establish norms, incentives, guidelines and technical standards that enable GenAI technology to be deployed safely, ethically and for the benefit of humanity as a whole. If we make the necessary investments and tradeoffs, the GenAI applications of the future will represent the best of human values like compassion, creativity and justice.

Earlier this month, I had the privilege of attending the International Conference on Computer Vision (ICCV) in Paris. After years of virtual research and other technology conferences, I was excited to revisit the experience of meeting those involved in the state-of-the-art in an open research-oriented forum like ICCV. It did not disappoint.

In this article, I share a retrospective covering:

1. Some of the state-of-the-art research presented at the conference.
2. What excited me the most given other momentum in the Bay Area right now, and how this research can be incorporated into startups.
3. How I personally experienced the conference and what impressed me about the inclusiveness at ICCV 2023.
4. What we can take away from all of it to apply elsewhere.

1. A glimpse into the state-of-the-art

It can be daunting to try to keep up with any given research area, especially the AI or machine learning space right now, when it seems like there are so many papers published to Arxiv daily. This makes it extremely difficult to either understand the macro direction of research trends or what is pertinent to the area of application an individual is working on.

Looking at the growth of submissions to ICCV alone, interest and research in the computer vision space has significantly increased since 2017.

Thankfully, the conference organizers and committees review a massive number of submissions and distill top papers by area into the orals presented at the event, with other relevant papers available via posters. Everyone has an opportunity to ask questions or discuss the papers with the authors.

This year, over 8,260 papers were submitted, of which 7,000~ reviewers narrowed down the pool to 2,161 accepted papers, and only 152 of them made it to orals.

Taking a look at the accepted papers by the top ten topic areas, it is interesting to observe the following (from slide 21 here) as we think about research trends in the computer vision:

1. 3D from multiview and sensors — Includes simultaneous localization and mapping (SLAM) problems used in autonomous vehicles, robotics, 3D scene generation (neural radiance views), visual querying of video, augmented reality, and other use cases.
2. Image and video synthesis — Generating relevant images and video.
3. Transfer, low-shot, continual, long-tail learning — Creating performant deep models from a large number of images that follow a long-tailed class distribution. For example, these models can be easily biased towards dominant classes (e.g. cats and dogs) and perform poorly on tail classes (e.g. elephants).
4. Low-level and physics based vision —Detailed features that are used to describe an image. Low-level vision includes considering things like corners, angles, and colors while physics based vision examines lighting, weathering, reflections, shape, etc.
5. Vision and language — Targeting truly multimodal models where both vision augments language and language augments vision.
6. Segmentation, grouping, and shape analysis — Grouping and segmentation processes with issues of shape perception and
   representation that includes topics such as abstraction, interpolation between early and middle vision, ocluded figures, and contours.
7. 3D from a single image and shape-from-x —Reconstructing a 3D image from a single image or a shape using point clouds, geometry estimation, and other methods.
8. Self-, semi-, meta-, unsupervised learning — Developing better models using self-directed (no labeled data), semi (some labeled data), or meta (changing the model based on learning) unsupervised learning.
9. Recognition: Detection — Conducting object recognition and distinguishing subjects from the background.
10. Adversarial attack and defense — Crafting perturbations to inputs that are difficult to perceive by humans but that cause models to product incorrect predictions (attack) and the corresponding methods to defend against this (defense).

We will not delve into all of these papers, but there were a few trends and topic areas below that I wanted to highlight in more detail.

Video and robotics as the future

Video is really hard tech, but an area ripe for research and innovation.

• Used in two ways: exteroception which teaches us about the external world. We build mental models of behavior (physical, social..) and use them to interpret, predict, and control
• Proprioception tells me about my current state in the world. Helps produce an episodic memory situated in space and time, and guides action in a context-specific way (ego centric views as area of research)

Scaling token-based LLM-like models is not the answer, as this is not feasible in its current state. Capturing the essence of the 4D world and complexity translates to a much higher token count than text, as shown in the workshop slide example below. To address these, new methods should be developed.

Robotics is also a good focus area for further research because it combines everything that computer vision does with the difficulties of the physical world. For example, robotics must handle multiple spatiotemporal scales to even take an action where the finest scale can be understood as physical action, but the larger scales are best understood in terms of goals and intentionality. So, a single action can be thought of as a combination of movements and goals within the complex constraints of an environment.

Data poisoning, safety, and adversarial attacks

Foundation and deep learning models are vulnerable to multiple forms of attack, including data poisoning and adversarial attacks. While I do not see as much discourse on this subject as generative AI in practice, it is critical that we pay attention to research in this space to ensure applications built with (generative) AI are secure.

In fact, it is commonly known that CNNs are highly vulnerable to adversarial attacks and so there has been ongoing research into making CNNs more robust. One paper proposes an adjustment to Mixture of Experts that adds adversarial training to it where the “robustness of routers
(i.e., gating functions to select data-specific experts) and
robustness of experts (i.e., the router-guided pathways defined by the subnetworks of the backbone CNN)” are alternately optimized for.

Using this framework, what the authors call “AdvMoE”, or Adversarial Mixture of Experts, they see robustness increased up to 4% and training cost reduced by more than 50% while maintaining accuracy.

Vision-Language Pre-trained models (VLP) are also highly subject to adversarial attacks. As we see text-to-image models becoming more popular, it is important to recognize that now multimodal encoders can be exploited to transfer vulnerabilities across modalities.

The paper “Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models” looks at the current state of vulnerabilities (below) and proposes a new method of attack that further exploits the multimodal nature of modern foundation models that also goes beyond white box (full access to model) settings.

Compared to existing adversarial attacks, the proposed method, Set-level Guidance Attack (SGA), exploits the inherent need for transferability in these models by taking advantage of complex cross-modal interactions:

The paper cites impressive results that against fused (ALBEF) and aligned (TCL) multimodal modules can be up to 30% more effective on both white box and black box settings, and some improvement by architecture (CLIP VIT versus CLIP CNN) with CNNs performing worse.

On the other hand, some of these adversarial techniques are now also helping artists and others protect their art from being wrongfully used for training.

There is also research that looks at evaluating how we assess data purification methods and how to apply more memory efficient techniques to improve adversarial purification. One paper in this space that stood out to me was “Robust Evaluation of Diffusion-Based Adversarial Purification”, where the authors instead of relying on the adjoint method, which depends on the performance of an underlying numerical solver, the authors propose using a surrogate back propagation process to approximate gradients from iterative procedures, revealing that some recent works might be less robust than claimed.

Attacks and defenses like these highlight the need to ensure models are robust to adversarial attacks as the adoption of text-to-image or other multimodal models increases and for further conversation on responsible applications of AI.

NERF, poses, and the art of 3D

When looking at augmented reality applications or images generated by vision models, some results can look like something straight out of a horror movie with how distorted the figures are. Humans appear in virtual reality with half of their bodies going through the bench, and generated human poses can have limbs appearing out of nowhere.

A lot of posters and papers presented made a lot of progress in this space, such as realistic reconstructions of poses like here incorporating actual constraints or those that incorporate real-world physics through kinematic observation, as below.

The progress in generating scenes (e.g., NERF), camera angles, models, and more has exciting implications for the future of video, robotics, augmented and virtual reality, and other applied areas of computer vision.

2. Translating research into fuel for startups

Robotics as an allegory for teaching machines

Networks have traditionally only been used in robotics to encode states for policy learning, but now roboticists are thinking about how to use the embedding space effectively and more creatively. That includes using LLMs to interact with the world, how to interpret the world through hierarchal representations, and how to efficiently represent image observations. Examples of compelling research include:

• Learning a controller in the latent space can make robot learning more data efficient e.g., learning functional distances based on goal and observation using contrastive learning: https://arxiv.org/pdf/2303.08135.pdf
• Learning to generalize grasping policies across geometry, pose, category, etc. Dynamic vision-based policy learning without knowing object state: https://arxiv.org/abs/2303.00938
• Incorporating game theory for autonomous vehicle decision making: https://arxiv.org/pdf/2107.01110.pdf

Improving robotics also teaches us about teaching machines more generally — these are all things we also need if we want to apply foundation models more creatively either through reinforcement learning or other ways of automated decision-making based on observations.

Personalization on another level

Advancements in computer vision will also improve personalization on multiple levels. One creative way is through egocentric views from a given person’s perspective or scene. Example papers include:

• Egocentric video — EgoLoc: Revisiting 3D Object Localization from Egocentric Videos with Visual Queries Jinjie Mai, Abdullah Hamdi, Silvio Giancola, Chen Zhao, Bernard Ghanem
• EMR-MSF: Self-Supervised Recurrent Monocular Scene Flow Exploiting Ego-Motion Rigidity Zijie Jiang, Masatoshi Okutomi
• ClothPose: A Real-world Benchmark for Visual Analysis of Garment Pose via An Indirect Recording Solution Wenqiang Xu, Wenxin Du, Han Xue, Yutong Li, Ruolin Ye, Yan-Feng Wang, Cewu Lu

Metadata and data management — retail analytics or what kinds of images that contain x are viewed

With enhancements in computer vision, it also seems like there will be improved metadata and collection of metadata about how users are interacting with images or models. For example, the Segment Anything paper results will allow retailers and others to automatically extract and catalogue data about images that consumers are interacting with.

Other papers to look at include:

• Towards Open-Vocabulary Video Instance Segmentation Haochen
  Wang, Cilin Yan, Shuai Wang, Xiaolong Jiang, Xu Tang, Yao Hu, Weidi Xie, Efstratios Gavves
• Progressive Spatio-Temporal Prototype Matching for Text-Video
  Retrieval Pandeng Li, Chen-Wei Xie, Liming Zhao, Hongtao Xie, Jiannan
  Ge, Yun Zheng, Deli Zhao, Yongdong Zhang

This is pretty exciting to consider how analytics can be improved to enhance the user experience.

Including fairness, privacy, and explainability in vision applications

A common approach to applying machine learning models now is to download an existing model (over 10,000 now on HuggingFace alone) and then fine-tune it. However, it is commonly recognized that there are social biases in all pre-trained models, and it is not apparent what those biases are. Nonprofits and research institutions are doing manual expert testing of some large language models (LLMs) to identify, for example, political biases in some of the popular LLMs. This is ultimately because we have no visibility into what data was used for training the original model (not fine-tuning data).

I found several interesting papers at ICCV related to these topics around mitigating bias, enhancing privacy, and practical methods we can apply as we implement generative AI systems to ensure fairer outcomes.

The authors of Overwriting Pretrained Bias with Finetuning Data approached the question of do the biases in pre-trained models transfer when you fine-tune them for downstream tasks, and, if so, how can people mitigate these? They specifically tackled the two most common forms of bias as defined in computer vision for the paper as seen in the image below from their presentation.

This is also a useful framework for us more generally to consider, as many struggle to identify, quantify, and then create testing methods to address bias in generative AI and other types of machine learning.

What stands out in the paper is how they apply a scientific approach to setting up the experiment, testing, and then proposing a solution that can help mitigate the underlying biases of the original pre-trained model through careful fine-tuning. This type of testing is one we should rigorously do to test for bias before putting any out-of-the-box model into production.

For the experiment, the study starts by analyzing bias as spurious correlations, where a sensitive attribute, like gender, might be incorrectly correlated with other attributes in a model. Specifically, they look at those in the CelebA dataset labeled “male,” and that have eyebags as a spurious correlation. They also note that the image of “male” is also not gender inclusive, as this all relies on human or semi-automated (based on human input) labeling. However, by reducing the strength of the correlation with only 100 images in the eyebag example, the bias was reduced by half.

The key takeaway for this paper is that while pre-trained models may introduce bias into finetuned models, careful curation and manipulation of the finetuning dataset can correct for these biases, thus reconciling performance with fairness in model outcomes.

Another paper that impressed me at ICCV on this topic was ITI-GEN: Inclusive Text-to-Image Generation. The authors here also target the biases in pre-trained models but from a different lens. Their goal is to align models to the human definition of inclusive, which they define as where generative images are uniformly distributed across attributes of interest. Below is an example the authors provide of the type of bias they then attempt to tackle:

Many current solutions rely strictly on text-based debiasing. However, as previous research shows, there are limits to this kind of naive solution. Since text-based debiasing can only convey so much information, it struggles with things like negation (not wearing glasses) or skin tone.

The novel approach from this paper is that, whereas many multimodal approaches have text augmenting images, they use images to augment text since detailed attributes (e.g., glasses) can be conveyed via information-rich images.

These visual differences can be translated through the method below into natural language, the output of which is actually inclusive tokens that can be used for alignment across models.

Using these embeddings could help build more inclusive models in a much more scalable and generalizable way.

That being said, text-based debiasing and testing will still be a key component of any AI Safety strategy. Red teaming is still necessary to identify bias in foundation models. However, one of the hardest things to do is to actually uncover bugs. Models likely hallucinate more than we think they do, it is simply the case that we do not recognize every hallucination.

The authors of Adaptive Testing of Computer Vision Models propose a way to improve text-based testing and debiasing by using multimodal models to prompt us, the humans, with more creative and out-of-distribution (OOD) tests.

To address this problem, the authors propose adaptive testing where the model suggests prompts using related embeddings that move toward useful (“hill climb”) tests that move closer to out-of-distribution questions as shown in the image below.

These can also be generalized to other images and models. Fine-tuning on the identified examples can also fix those bugs in the future, helping developers determine where to deploy and what is safe to implement through efficient testing.

The last paper I will mention in this section is PGFed: Personalize Each Client’s Global Objective for Federated Learning. Federated learning has not been a prominent topic lately, but the authors of this paper propose a more efficient and privacy-preserving method for using personalized models for federated learning with explicit knowledge transfer versus the conventional global model that is then distributed. From the engineering side, I found this to make the use of federated learning more feasible for those interested in deploying in a more privacy-oriented way.

This section of papers was one of the most interesting to me and offered insights into how, on the applied side, we might translate research into better responsible AI practices.

3. Diversity and inclusion at the conference

Last but not least, I was personally very impressed with how the ICCV organizers supported diversity at the conference. It was significantly better than what I see at industry events run by the private sector.

Their support included everything from having an actual dedicated committee at all to travel scholarships, on-site healthcare, and even volunteer events at local high schools to teach young women about computer vision.

Diversity efforts resulted in a better balance in gender not only because of the committee efforts for providing scholarships and support, but also through the highlighting of the actual work of underrepresented groups (not just by gender). This includes the Women in Computer Vision workshop, LatinX in Computer Vision workshop, and even an Arabic language speakers meetup.

It would be great to see big tech and others adopt similar efforts or policies to support better representation at tech conferences.

Conclusion

This retrospective turned out to be a lot longer than I expected, but I had so many thoughts after the conference that I wanted to capture at least some of them for future reference and to share more broadly.

I hope this provided an overview of some of the key research highlights (I intentionally spent more time on less known areas, otherwise the entire retrospective would have only been about NERFs or gaussian splats), and some of the things I’m excited about going forward.

Reference

• ICCV 2023 Outreach Event
• Accueil

Papers

Gao, I., Ilharco, G., Lundberg, S., & Ribeiro, M. T. (2023). Adaptive testing of computer vision models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 4003–4014).

Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., … & Girshick, R. (2023). Segment anything. arXiv preprint arXiv:2304.02643.

Lu, D., Wang, Z., Wang, T., Guan, W., Gao, H., & Zheng, F. (2023). Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 102–111). Link.

Luo, J., Mendieta, M., Chen, C., & Wu, S. (2023). PGFed: Personalize Each Client’s Global Objective for Federated Learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 3946–3956).

Shan, S., Ding, W., Passananti, J., Zheng, H., & Zhao, B. Y. (2023). Prompt-Specific Poisoning Attacks on Text-to-Image Generative Models. arXiv preprint arXiv:2310.13828.

Wang, A., & Russakovsky, O. (2023). Overcoming Bias in Pretrained Models by Manipulating the Finetuning Dataset. arXiv preprint arXiv:2303.06167.

Zhang, Y., Cai, R., Chen, T., Zhang, G., Zhang, H., Chen, P. Y., … & Liu, S. (2023). Robust Mixture-of-Expert Training for Convolutional Neural Networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 90–101). Link.

Zhang, C., Chen, X., Chai, S., Wu, C. H., Lagun, D., Beeler, T., & De la Torre, F. (2023). ITI-GEN: Inclusive Text-to-Image Generation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (pp. 3969–3980).

Imagine you have a very complex language machine. This machine has many internal pieces (called weights) that were optimized for general use.

Now you want to tweak this machine a bit to do a specific new task, like answering questions about a given subject area in-depth, that branches off from the main general purpose trunk. But if you try to adjust every single part of the machine separately, it will take a huge amount of effort and be really slow.

The key idea of the Low-Rank Adapter (LoRA) research is that you likely only need to tweak a few key parts of the machine to get it to do that new task well. You don’t need to adjust every single part, but rather just a few in a very efficient way.

Background

In my last article on quantization, we discussed how low-rank adapters reduce barriers to hardware and increase performance through improved memory usage.

However, it is important to delve into the specifics of the original Low-Rank Adaptation (LoRA) paper in order to gain a deeper understanding of the overall concept that the authors are conveying, which extends beyond just performance and even beyond LLMs.

In general, it is very common for people to rely on fine-tuning to apply one large-scale model to many specific downstream tasks. You might want to apply a 40B parameter Falcon model to the domain of finance, and to do this you would go to the Hugging Face website, take Falcon, and fine-tune it on your curated finance dataset. The problem with this, as the authors point out, is that the new model contains just as many parameters as the original model.

Said more mathematically in the paper, each fine-tuned model has “a set of parameters ∆Φ whose dimension |∆Φ| equals |Φ0|. Thus, if the pre-trained model is large (such as GPT-3 with |Φ0| ≈ 175 Billion), storing and deploying many independent instances of fine-tuned models can be challenging, if at all feasible.”

LoRA addresses this by doing the following:

LoRA allows us to train some dense layers in a neural network indirectly by optimizing rank decomposition matrices of the dense layers’ change during adaptation instead, while keeping the pre-trained weights frozen…

Said again more mathematically:

…the task-specific parameter increment ∆Φ = ∆Φ(Θ) is further encoded by a much smaller-sized set of parameters Θ with |Θ| << |Φ0|. The task of finding ∆Φ thus becomes optimizing over Θ:

Where a pre-trained autoregressive language model PΦ(y|x) is parametrized by Φ Each downstream task is represented by a training dataset of context-target pairs: Z = {(xi , yi)}i=1,..,N , where both xi and yi are sequences of tokens.

There are four major implications to this approach and research:

1. A pre-trained model can be shared and used to build many small LoRA modules for different tasks.
2. LoRA can boost training efficiency and lower hardware requirements up to three times by using adaptive optimizers that eliminate the need to calculate gradients or maintain optimizer states for most parameters.
3. No inference latency.
4. Interoperable with other techniques. After conducting a thorough analysis of previous research in this field, the authors discovered that this approach could be integrated with other adapter or quantization methods.

Technical details and benchmarks

The key technical idea in this research is that when you fine-tune a pre-trained model on a new task, the changes to the weights (represented by ∆W) have low intrinsic dimensionality or “rank” — you don’t need to update all the weights, just a small subset.

LoRA exploits this by parametrizing the weight updates ∆W using low-rank matrices. Concretely, they decompose ∆W = BA, where B and A are low-rank matrices that are trained on the downstream task while keeping the original pre-trained weights W frozen.

We can break this down into more detail across three components.

1. Parametrization of ∆W:

• For each weight matrix W in the model, the authors introduce two smaller matrices B and A to represent the update ∆W.
• B has the same number of rows as W but much fewer columns (rank r).
• A has r rows like B, and the same number of columns as W.
• The update is computed as ∆W = BA, which is low-rank since B and A have rank r.

2. Training:

• The original weights W are frozen, and only B and A are trained.
• This means the gradient is only computed for B and A, reducing memory and compute.
• The forward pass computes h = Wx + ∆Wx = Wx + BAx.

3. Choice of rank r:

• The authors find that a very low rank like r=1 or r=2 works surprisingly well for large models like GPT-3.
• This suggests the intrinsic dimensionality needed for adaptation is very small.
• They analyze the subspaces learned by different choices of r and find the top singular vectors overlap.

To gauge the effectiveness of this method, the authors tested LoRA on the GLUE benchmark using RoBERTa, DeBERTa, GPT-2, and GPT-3 with other adapter methods. In every instance, LoRA either matched or exceeded full fine-tuning, even with significantly fewer parameters. For example, the authors were able to show a 10,000x reduction in trainable parameters for GPT-3, without any loss in performance.

The paper also analyzes the learned low-rank updates ∆W and shows they amplify certain directions in the original weight matrix W that are useful for the task but not emphasized during pre-training.

This suggests that the low-rank adaptation matrix potentially amplifies the important features for specific downstream tasks that were learned but not emphasized in the general pre-training model.

In summary, by constraining the update ∆W to be low-rank, LoRA allows efficient adaptation of large pre-trained models with minimal performance loss. The key factors are the parametrization of ∆W, keeping W fixed, and finding that a very low rank (r) works well empirically.

Conclusion

Many people are interested in using LLMs to complete particular downstream tasks that are related to a specific field or situation. Nevertheless, fully fine-tuning an LLM is not always possible for everyone. Fortunately, LoRA provides a solution that not only offers efficient computing but also a scalable method for applications to specific new downstream tasks.

In summary, the benefits of this method include:

• Easy to switch between tasks just by swapping the small LoRA modules.
• Much lower memory usage and faster training compared to full fine-tuning since we don’t need to store the optimizer state for the original parameters (frozen W).
• Faster training as there are fewer gradients to compute.
• No increase in inference latency or computation compared to the original model.

Furthermore and worth following, this research is not just applicable to just LLMs:

The principles outlined here apply to any dense layers in deep learning models, though we only focus on certain weights in Transformer language models in our experiments as the motivating use case.

LoRA Paper.

LoRA GitHub repository.

PEFT library on HuggingFace for ease of use.

LLM-driven agents can help us do any number of tasks more efficiently, whether that is code review, writing emails, etc. Some tasks that we want them to do require more domain knowledge and expertise than others, so we might want to fine-tune our own model (more to come on this). There are 40 billion parameter models now open-sourced, so why not fine-tune these with your data and knowledge without feeding it into a third-party environment?

However, there are significant hardware (and financial) challenges to fully leveraging the power of LLMs, especially if we want to fine-tune them on our own proprietary data and expertise. Not everyone has several thousand (or tens of thousands of) dollars to invest in GPUs readily available, and they can be hard to provision on cloud service providers because of high demand.

Since we have been on the topic of memory lately, I want to share another area I am excited to see developments in related to a different area of memory, which is quantization. This is a very practical technique that can be used to help everyone take advantage of fine-tuning their own models and reduce barriers caused by hardware limitations.

What is Quantization?

In the context of machine learning, quantization is a process that reduces the precision of the numerical weights used in a model. The aim of quantization is to make the model smaller and more computationally efficient, at the cost of a slight reduction in accuracy.

In a typical neural network, the weights are represented as 32-bit floating-point numbers. However, these high-precision numbers can be memory-intensive and computationally demanding, especially for large models with billions of parameters.

Quantization tackles this problem by reducing the precision of these weights. For example, the weights might be rounded off to the nearest 16-bit, 8-bit, or even 4-bit numbers, as in the case of QLoRA, which we will explore in further detail.

This process greatly reduces the memory requirements and computational complexity of the model, which can make it possible to run large models on hardware with limited memory and computational capacity, such as mobile devices or single GPUs. Research shows that “moving from floating-point representations to low-precision fixed integer values represented in four bits or less holds the potential to reduce the memory footprint and latency by a factor of 16x; and, in fact, reductions of 4x to 8x are often realized in practice in these applications.”

It’s important to note that while quantization can slightly degrade the performance of the model, research and engineering efforts such as QLoRA aim to mitigate this reduction in accuracy, so the quantized model performs nearly as well as the fully fine-tuned model that has no quantization. We should also note that quantization tends to work better with LLMs than it does with foundational computer vision models for various reasons, even though quantization was historically popularized more by the computer vision world.

Historical Developments of Quantization

Quantization in machine learning has evolved significantly over the years, driven by the need to deploy powerful deep learning models on devices with limited memory and computational power, like mobile phones or edge devices. In the early days of machine learning, models were small enough that quantization wasn’t really necessary.

However, as models started to grow, the need for more memory-efficient representations became evident. Researchers started exploring lower-precision arithmetic, starting with 16-bit half-precision floating points. Below are some of the different types of quantization that can happen at different points of the model development process.

1. Fixed-Point Quantization: Fixed-point quantization, which involves representing weights as integers rather than floating-point numbers, was one of the earliest techniques used. One of the key papers here was the first to propose this technique to quantize deep convolutional networks (DCNs) and showed that fixed-point implementations of DCNs combined with optimization of bit-allocation across layers actually improved both memory and accuracy of them when benchmarked against CIFAR-10. However, the limited dynamic range of fixed-point numbers can make this type of quantization less suitable for other networks.
2. Floating-Point Quantization: Researchers also continued using lower-precision floating-point numbers, like the 16-bit half-precision floating points mentioned earlier. One of the early publications to develop this technique was in using an 8-bit floating point representation (FP8) for training deep neural networks (DNNs), where they created a hybrid FP8 format to match different precision requirements in the model training process and a new end-to-end distributed training method. This allowed for a larger dynamic range than fixed-point numbers and applicability to more DNN models.
3. Weight Quantization: In weight quantization, weights between neurons are approximated and constrained, unlike traditional stochastic gradient descent, which evaluates the gradient on the current iterate. For example, BinaryConnect acknowledges that typical DNN computations involve a lot of multiplication and addition operations, also known as multiply-accumulate operations. In digital hardware, multipliers consume the most power and take up the most space. So, if the weights between neurons are binary during forward and backpropagation, these multiply-accumulate operations can be simplified into mere addition operations. This results in efficient computation and improved performance.
4. Activation Quantization: Researchers then started quantizing not just the weights but also the activations in the network. This allows for further memory savings and can also significantly speed up computation, especially on hardware designed for lower-precision arithmetic.
5. Post-Training vs Training-Time Quantization: Initially, most quantization was done post-training, meaning the model was first trained at full precision and then quantized. However, this can lead to significant accuracy loss. To address this, researchers started to use training-time quantization, where the model is trained with quantization from the start. One example of this is the Data Quality-aware Mixed-precision Quantization (DQMQ) framework, which adapts the quantization bit-width dynamically according to the quality of the data. This adaptation is guided by a bit-width decision policy that can be learned during training and is modeled as a hybrid reinforcement learning task. This can allow the model to better adapt to the quantization, rewarding outcomes that improve performance.
6. Advanced Techniques: More recently, researchers have been exploring more advanced quantization techniques. For example, this paper from NVIDIA on mixed-precision training proposes using both single and half precisions. They keep a single-precision copy of the weights that accumulate the gradients after each optimizer step; this copy is then rounded to half-precision for the forward and backward propagation steps. Loss scaling helps ensure smaller half-precision values do not vanish. QLoRA uses a novel double quantization technique to further reduce memory usage.

It is no surprise that with the development of foundation models, such as LLMs, this area has regained much focus, and new research on these various methods continues to be published. One of the recent publications that has shown significant promise at the implementation level is QLoRa, and by proxy LoRA, which we will discuss next in the context of LLMs specifically.

Quantizing LLMs: LoRa and QLoRa

QLoRA is an efficient fine-tuning technique for machine learning models, specifically for large language models. It is designed to allow LLMs to be fine-tuned on a single GPU with limited memory, even when the model itself has a huge number of parameters, up to 65 billion.

The key technique QLoRA uses is backpropagation of gradients through a 4-bit quantized version of the pre-trained model into structures called Low Rank Adapters (LoRA). In practice, an adapter is a small module that is inserted into each layer of a pre-existing model. The adapter is trained to adapt the inputs of the layer to the specific task at hand, while the pre-existing parameters of the model are kept fixed (or “frozen”). They are referred to as “Low Rank” because of their origins in linear algebra — a matrix that can be factored into a product of two matrices with smaller dimensions. This factorization significantly reduces the number of parameters that need to be learned and stored.

This quantization process allows the model to drastically reduce memory usage without a significant drop in task performance. The paper cites that QLoRA reduces the average memory requirements of finetuning a 65B parameter model from >780GB of GPU memory to <48GB without degrading the runtime or predictive performance compared to a 16-bit fully finetuned baseline.

The QLoRA-based model, named Guanaco, has performed very well on the Vicuna benchmark, which measures the performance of conversational AI models. Despite its more compact size and the reduced memory required for its fine-tuning, it achieves nearly the same performance as ChatGPT.

QLoRA introduces three novel techniques to save memory and achieve this performance:

1. 4-bit NormalFloat (NF4): A new data type that’s optimal for normally distributed weights. By using only 4 bits, it saves a lot of memory compared to higher precision data types.
2. Double quantization: An extra layer of quantization on top of the 4-bit quantization. This involves quantizing the quantization constants using the LoRA method, leading to even further memory savings.
3. Paged optimizers: These manage memory usage by handling memory spikes that might otherwise cause problems. They make sure that the memory usage stays within the limit of the GPU memory.

QLoRA was used to fine-tune more than 1,000 models, including different types of models and models of different scales, some of which would be unmanageable using traditional fine-tuning techniques. They found that QLoRA could produce state-of-the-art results even with smaller models and small high-quality datasets.

However, they also found that current chatbot benchmarks aren’t always reliable for accurately measuring chatbot performance. They provided a detailed breakdown, or “lemon-picked analysis”, of where Guanaco doesn’t perform as well as ChatGPT e.g., mathematics, obscure questions, and some types of assumptions as part of the preamble. The authors also recognize that there is a need for more testing at larger scales, as this is still a computationally-intensive task.

Finally, they released all of their models and the code used to implement QLoRA, including the CUDA kernels required for 4-bit training. Check out their GitHub repository here for details.

I personally tried implementing their code to quantize several fine-tuned LLMs in the 7B (works even on a free-tier single instance GPU in Google Colab or Kaggle) to the 40B parameter range (check out Vast.ai for cheaper GPU alternatives, or if you have some you can rent out!) with significantly reduced hardware requirements and minimal impact on performance for specific use cases. Techniques like QLoRA (and LoRA) should be further developed and can truly unlock the potential of LLMs for a larger pool of people.

Conclusion

With the development of increasingly larger foundation models, and LLMs in particular, the area of quantization and its role in memory-efficient processing has become critical to truly democratizing the power of LLMs to a broader audience. These techniques, developed initially to make deep learning more efficient on mobile and edge devices, are now helping to make LLMs and specifically using fine-tuned LLMs more accessible to all.

References

Courbariaux, M., Bengio, Y., & David, J. P. (2015). Binaryconnect: Training deep neural networks with binary weights during propagations. Advances in neural information processing systems, 28. Link.

Dettmers, T., Pagnoni, A., Holtzman, A., & Zettlemoyer, L. (2023). Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314. Link.

Gholami, A., Kim, S., Dong, Z., Yao, Z., Mahoney, M. W., & Keutzer, K. (2021). A survey of quantization methods for efficient neural network inference. arXiv preprint arXiv:2103.13630. Link.

Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., … & Chen, W. (2021). Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685. Link.

Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., … & Kalenichenko, D. (2018). Quantization and training of neural networks for efficient integer-arithmetic-only inference. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 2704–2713). Link.

Lin, D., Talathi, S., & Annapureddy, S. (2016, June). Fixed point quantization of deep convolutional networks. In International conference on machine learning (pp. 2849–2858). PMLR. Link.

Micikevicius, P., Narang, S., Alben, J., Diamos, G., Elsen, E., Garcia, D., … & Wu, H. (2017). Mixed precision training. arXiv preprint arXiv:1710.03740. Link.

Sun, X., Choi, J., Chen, C. Y., Wang, N., Venkataramani, S., Srinivasan, V. V., … & Gopalakrishnan, K. (2019). Hybrid 8-bit floating point (HFP8) training and inference for deep neural networks. Advances in neural information processing systems, 32. Link.

Van Den Oord, A., & Vinyals, O. (2017). Neural discrete representation learning. Advances in neural information processing systems, 30. Link.

Wang, Y., Guo, J., Guo, S., & Zhang, W. (2023). Data Quality-aware Mixed-precision Quantization via Hybrid Reinforcement Learning. arXiv preprint arXiv:2302.04453. Link.

Zhou, C., Liu, P., Xu, P., Iyer, S., Sun, J., Mao, Y., … & Levy, O. (2023). Lima: Less is more for alignment. arXiv preprint arXiv:2305.11206. Link.

Introduction

Large language models (LLMs) are advanced machine learning models with many parameters trained on a massive library of text. From a technical standpoint, language modeling (LM) is a crucial technique for generating advances in machine language intelligence. The primary objective of LM is to develop models that accurately estimate the probability of generating word sequences, thereby facilitating the estimation of probabilities associated with future tokens or those that are absent.

Recent developments in natural language processing and related disciplines have been facilitated by large language models (LLMs) such as ChatGPT. Research has demonstrated that when these models reach a certain scale, they exhibit emergent behaviors, such as the capacity to engage in “reasoning”. By incorporating models with “chain-of-thought” methods, i.e. reasoning examples or a straightforward prompt such as “Let’s think step by step,” these models can respond to questions with explicit logical steps.

One example we can look at that LLMs can handle is: “All whales are mammals, and all mammals have kidneys; therefore, all whales have kidneys.” This particular example has generated significant excitement within the research community. The reason behind this enthusiasm is that the ability to reason logically is considered a fundamental characteristic of human intelligence and is often seen as lacking in current artificial intelligence systems.

While large language models (LLMs) have demonstrated remarkable performance on specific reasoning tasks, their generalizable reasoning capabilities and the extent to which they can engage in reasoning processes remain debatable. This uncertainty has triggered ongoing debates and different perspectives among the research community. For instance, some say that LLMs are “adequate zero-shot reasoners”, whereas other authors have concluded that “LLMs are still a long way from achieving adequate results on everyday planning/reasoning tasks that pose no challenge for humans.”

In this article, we begin to explore the role of memory in advancing LLMs, some emerging architectures, and opportunities for their application. Subsequent articles will cover more of the implementation details and specifics of different approaches and design patterns.

LLMs with Long Term Memory

LLMs are founded on an architecture for deep learning called the Transformer Model. The Transformer Model employs self-attention mechanisms to determine the relationships between words or elements in a text sequence. This and related foundational developments enable the model to comprehend and generate text with greater context and coherence.

To improve the capabilities of LLMs, researchers have considered incorporating mechanisms for long-term memory into these models. Long-term memory allows a model to store information for an extended period and utilize it when developing responses. There are currently two primary ways in which long-term memory can be incorporated.

One way that LLMs can incorporate long-term memory is by utilizing additional memory components, such as third-party databases including vector databases or (knowledge) graph databases. These components store factual data or embeddings that the model can access and use to generate text. The LLM can provide more precise and accurate responses by accessing this external memory.

Alternatively, memory can be incorporated into the model itself. For instance, the model may include an internal memory module (e.g., an index, a modified attention mechanism, etc.) that stores pertinent data during text processing. This internal memory can be modified and accessed to affect the model’s behavior. LLMs can demonstrate more consistent behavior and create appropriate responses for the given context by employing long-term memory in the actual model architecture.

Architectural Design Considerations:

Several architectural factors must be accounted for when developing LLMs with long-term memory. A few crucial considerations include:

• Storage: The model’s memory should be large enough to store and retrieve pertinent information. The memory component’s size and architecture must be designed to accommodate the intended quantity of long-term data.
• Storage update: The model should include mechanisms for revising long-term memory in response to new input or data. This makes sure the memory is up-to-date and pertinent to the current context.
• Memory retrieval: The model requires efficient access and retrieval mechanisms for information stored in long-term memory. This retrieval process must be quick and precise so that the model can utilize the stored information during text generation without causing excess latency. This is especially true of methods utilizing external memory components, such as a third-party database.
• Incorporation with attention mechanisms: The model’s long-term memory should be incorporated with self-attention mechanisms. This enables the model to pay attention to pertinent portions of memory during text generation and make informed decisions based on the stored data.
• Training and Selection: Training LLMs with long-term memory demands a thorough understanding of how the memory is updated and utilized during training. To ensure efficient learning and utilization of stored data, robust selection techniques must account for the memory component.

To that end, there are some new emerging architectures in the research community that try to account for these design considerations, while also improving cost and long-memory efficiency. One example, as seen in the following diagram, is adding a decoupled caching layer that stays fresh through training to supplement pre-trained models.

Example architecture with augmented memory mechanism from Wang et al. (2023)

Another example shown below is the “Hyena Hierarchy,” which looks at breaking the quadratic barrier (causes scaling cost and compute time) of the attention mechanism by replacing the attention mechanism with subquadratic operators in a new dense, attention-free architecture.

Example architecture using Hyena Hierarchy from Poli et al. (2023)

These new paradigms and others coming out of the research and developer communities will be explored in further detail in future articles.

Advancements and Opportunities in GenAI Applications

The development of LLM-based GenAI applications has created new opportunities in numerous fields. As models become more adept at comprehending and remembering extended context, they can be utilized in increasingly complex and dynamic situations.

Advancements in LLMs

Improving Context Understanding and Coherence in Text Generation

The capability of GenAI applications to comprehend and generate text with enhanced context comprehension and coherence is one of their significant benefits. Modern LLMs, such as GPT-3.5 and above, have been trained on enormous quantities of data and may produce more appropriate and contextually appropriate responses. This enhancement enables more meaningful and precise interactions across multiple applications. This is furthered by the improvement of long-term memory mechanisms in areas such as those listed below.

• Improved Accuracy and Specificity of Generated Responses:

The ability to fine-tune LLMs to acquire domain-specific expertise is one of the main benefits of GenAI applications. This goes beyond using only “preamble” or prompt engineering of an existing general LLM. By introducing the model to an enormous amount of data from a specific domain and tuning its parameters accordingly, we can train it to generate responses demonstrating a profound understanding of the topic. This specialization permits the model to provide more precise and contextually pertinent data.

On top of this, GenAI applications excel at information retrieval by employing these refined LLMs. By integrating mechanisms for long-term memory, these models may retrieve information with even better precision and relevance to the collected data, allowing the model to provide more accurate results to user queries.

• Consistent Behavior and Reduction of Biases in LLMs:

Ensuring the consistency of, reducing hallucinations in, and eliminating biases within LLMs present enormous challenges. By utilizing better long-term memory mechanisms in LLMs, we can potentially improve bias in the following areas:

• Bias Reduction: AI models acquire knowledge from data. The model will exhibit the same biases if the training data contains biases. Therefore, it is essential to curate a balanced and diverse training dataset. This requires precise data selection and evaluation to reduce implicit and explicit bias. Better long-term memory can improve the quality of outputs, reducing hallucinations and potentially biased outputs.
• Fairness Evaluation: Implementing and monitoring impartiality metrics can assist in quantifying bias. These metrics may serve as guidelines for the LLM’s behavior. Part of this also requires looking at the input (prompts) and outputs (responses) of the LLM at the user interaction level. This is where having an external long-term memory mechanism can be helpful as well. Conducting regular audits and evaluations throughout the entire process is essential to identify and correct any biases in the AI system.
• Algorithmic Transparency: Most LLMs are still very opaque in their decision-making. While there are explainable AI techniques that can help identify inherent biases by revealing how the model makes its decisions, there are also opportunities with emerging memory mechanisms within model architectures that could make models more transparent. For example, techniques that can freeze or update only certain parts of a model or its running long-term memory could keep models up-to-date but also make sure they adhere to regulations such as the “right to be forgotten” without requiring retraining.

Opportunities:

Generative AI is a subfield of artificial intelligence that concentrates on developing models and systems to generate new and unique content. In contrast to traditional AI systems, intended to identify and categorize existing data, generative AI seeks to generate new data that mimics the trends and features of the training data to which it has been exposed. Having more advanced memory in LLMs powering generative AI applications can significantly impact the following areas:

• Content Creation: AI has significantly advanced in generating written content, including poetry, news articles, and technical papers. For instance, businesses can use AI to generate marketing copy or provide content suggestions, drastically reducing the time spent creating and modifying content. In addition, Generative AI models use methods such as generative adversarial networks (GANs), variational autoencoders (VAEs), and RNN’s to generate new content such as pictures, videos, text, and music. These models discover the fundamental patterns and frameworks of the training data and then use this information to produce new samples with comparable features. Having longer-term memory can help evolve content based on a longer context and reference back key elements from previous statements or works.
• Personalized Experiences: Whether it be through the generation of personalized purchasing experiences or the creation of unique learning paths in education, Generative AI has the potential to personalize numerous aspects of our everyday lives. Improved memory would provide better personalization capabilities that could account for a longer period of time of preferences.
• Drug Discovery: Generative AI is already used to forecast the potential properties of molecules and generate new potential drug candidates in drug discovery. This already has proven to significantly accelerate the drug discovery and development process. Better long-term memory could increase the potential of these applications even more by incorporating more context about each molecule or drug candidate.

Conclusion

Despite the exciting developments and opportunities, substantial challenges lie ahead. It is computationally challenging to train these advanced LLMs, and their complexity makes them more difficult to comprehend and control. Researchers are still discovering how to manage these trade-offs effectively. A key part of this discussion is the role of long-term memory in LLMs and GenAI more broadly.

Concerns regarding the ethical implications of such advanced AI models continue to grow. There are still unresolved privacy concerns, particularly when considering models that can remember information for extended periods. Also of critical concern is whether these models will remember and perpetuate detrimental or biased behavior.

Researchers, developers, regulators, and end-users must collaborate to address these challenges. In order to responsibly uncover every potential of LLM-based GenAI applications, it is necessary to develop novel methods for model comprehension, ethical standards for AI usage, and more effective privacy protection methods.

In conclusion, the rapid development of GenAI applications based on developments in memory and context represents a significant milestone in the evolution of AI. As we continue to stretch the limits of what AI can recall, reason over, and recognize, the distinction between human and artificial intelligence tasks becomes increasingly fuzzy. The developments in LLMs offer intriguing opportunities in various industries but also present new challenges. As we continue to evolve artificial intelligence, the road ahead is stimulating and full of potential.

References

Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473. Link.

Kojima, T., Gu, S. S., Reid, M., Matsuo, Y., & Iwasawa, Y. (2022). Large language models are zero-shot reasoners. Advances in neural information processing systems, 35, 22199–22213. Link.

Marcus, G. 2020. The next decade in ai: four steps towards robust artificial intelligence. ArXiv preprint, abs/2002.06177. Link.

Paananen, V., Oppenlaender, J., & Visuri, A. (2023). Using Text-to-Image Generation for Architectural Design Ideation. Link.

Terrence J. Sejnowski; Large Language Models and the Reverse Turing Test. Neural Comput 2023; 35 (3): 309–342. doi: https://doi.org/10.1162/neco_a_01563

Wang, W., Dong, L., Cheng, H., Liu, X., Yan, X., Gao, J., & Wei, F. (2023). Augmenting Language Models with Long-Term Memory. Link.

Wei, J., Wang, X., Schuurmans, D., Bosma, M., Xia, F., Chi, E., … & Zhou, D. (2022). Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35, 24824–24837. Link.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., … & Polosukhin, I. (2017). Attention is all you need. Advances in neural information processing systems, 30. Link.

Yosef, M. A., Bauer, S., Hoffart, J., Spaniol, M., & Weikum, G. (2012, December). Hyena: Hierarchical type classification for entity names. In Proceedings of COLING 2012: Posters (pp. 1361–1370). Link.

Credit to Joss Taylor Olson (jtojoss@yahoo) for brilliant creative writing expertise.

I don’t have a lot of memories of my mother, but there is one evening I remember distinctly.

“You know why our little village is called a ‘Moon Village’?” she asked. The moon was full and bright in the night sky. From where we sat, I could see more of the foothills below where our encampment, Guryong Village, was located.

I couldn’t fathom why our miserable home would have such a pretty name. Looking down at the desolate lean-tos and with my stomach empty from not having been able to eat all day, I responded with a complaint as children often do: “No. I don’t care. I’m hungry. It’s cold.”

She scoffed a bit. I’m sure she was hungry and cold, too. “It’s because we have the best view of the moon!” She stood and lifted me up onto her shoulders. “See, raise your hands. We’re so high up on Mt. Guryong that you can touch the moon!”

“You’re our little Miracle on the Han River,” she’d tell me.

I raised my face up to the moonlight and lifted my hands to the night sky, feeling on top of the world.

“Why do we have to live here?” I asked my grandmother once, after my mother had passed from breast cancer. I was still perpetually hungry. I watched a few roaches scuttle around the ground in front of me. Sometimes they would run over my legs while I sat to talk with her. Sometimes I gave them names based on the bullies I despised.

She motioned for me to follow her. We started walking up the hill where my mother and I used to sit. “It’s because we’re vagrants.”

“What’s a ‘vagrant’?”

When we reached the top of the hill, she asked me to sit next to her. The sun was brutal against our skin when it wasn’t freezing. We had such extreme weather conditions in the Moon Village and no way to deal with them.

“People the government deemed bad and stopping the modernization of our country,” she explained further, “See, we wouldn’t have amazing buildings like that if we didn’t sacrifice and work hard.” She gestured to the office buildings and apartments in the distance.

“We used to live over there,” she pointed in another direction, “where the Olympic Stadium is. We used to live in an apartment there. One day, they announced that we needed to make sacrifices for our country, and forced us to get on a bus, so they could use that land for the stadium. They left us here with nothing. We did what we could to build a village.”

I didn’t understand until much later the larger truth behind such statements. That this was part of a large “social purification” campaign — taught to us today as the “modernization” campaign — to…” make Korea an ‘economically modern’ country with the intent of bidding for the Olympics. The stronger the economy, the more attractive the bid.” This era of development was in fact described in history books as, “The Miracle on the Han River”. The coup de grâce to show the world the fruits of this modernization campaign, that we had fully transformed from a ‘least developed country’ to a ‘developed country’, was the 1988 Seoul Summer Olympics.

It was in this modern Korea, no longer ravaged by the war of the 1950s and four previous decades of excruciating Japanese colonization, that there was no place for ‘vagrants’.

She switched to Japanese, which was sometimes easier for her than Korean because of her experience living through the Japanese occupation. “We couldn’t afford to show any weakness to the Westerners coming to visit us. So the government, police, and Chaebols [conglomerate companies like Samsung] — they moved everyone that didn’t fit. That included us, homeless people, children, pro-democracy protestors, students, anyone that spoke up for human rights, labor rights, or otherwise deemed as slowing down economic progress. Anyone the police didn’t like, too. You’re lucky to live here.”

I was appalled. “Lucky?”

“Some people, including your uncle, never survived police encounters. Some families were separated or sent to forced labor camps to support modernization. I bet you don’t even remember him, your uncle, do you? To think I survived Japanese occupation only to outlive my own children in the name of national greed.”

She paused. Sighed. “What place do girls have in economic development? Nobody will care about you. We will all die here.”

I stayed up that night, oscillating between believing and refusing to believe that I had no place in the universe. It’s hard to reject something you’ve been told your whole life is true.

“Why do you have that stupid thing?” my father shouted angrily, throwing a Soju bottle that shattered near me. Roaches scattered. “Get out of here.” He held his head in his hands. I think he was crying.

My grandmother was in pain from cancer, which we did not have the means to treat. Just like mom. I was still hungry. I could never escape the feeling of being hungry.

I held the object in question, a cartoony tiger pin of Hodori, close to me in a moment of stillness before fleeing over broken glass. Ignoring the cuts, I ran on bloody feet to the top of the hill and paused. Hodori was the mascot for the 1988 Seoul Summer Olympics. Mom had given me this pin because I was her Miracle on the Han River.

The irony was that all tigers in Korea had been killed during the Japanese occupation. Of this symbol of national pride, only the symbol remained.

I was Mom’s Miracle on the Han River. But the Miracle on the Han River — who was it for?

I stood pensively under the moonlight. Determination filled me as I tightened my grip on the pin. I began to make plans, to run through scenarios, to learn how to escape this system. How to make it better. I would rather die fighting than give up, as I was sure the tigers must have, and do everything in my power to build a world where nobody else had to either.

I had no idea at the time that, in just a few short years, I would attend university in the United States at the age of 13. I had no idea that the entire village would fall under the shadows of the newly built Gangnam Samsung Luxury Apartment Towers. They boasted 70 floors, fully automated apartments, and helipads on the roofs.

I didn’t care. I was hungry. I was cold. We all were.

References to learn more for people interested:

Seoul ‘moon village’ becomes a tourist trap. https://www.koreaherald.com/view.php?ud=20220926000213.

Miracle on the Han River. https://en.wikipedia.org/wiki/Miracle_on_the_Han_River.

Park Chung Hee. https://en.wikipedia.org/wiki/Park_Chung_Hee.

Samsung Tower Place. https://en.wikipedia.org/wiki/Samsung_Tower_Palace.

Seoul’s last remaining slums. https://www.bloomberg.com/news/videos/2020-12-17/seoul-s-last-remaining-slum-video.

South Korea covered up mass abuse, killing of ‘vagrants’. https://apnews.com/article/south-korea-crime-2020-tokyo-olympics-busan-olympic-games-43ad81ac09564c90bc6e6b3cd005d73d.

Brother’s Home: South Korea’s 1980s ‘concentration camp’. https://www.bbc.com/news/world-asia-52797527.

신자유주의 시대의 무허가 정착지 경관에 대한 비판적 해석 : 강남구 개포동 구룡마을을 사례로. https://academic.naver.com/article.naver?doc_id=79758734.

서울 중계동 백사마을의 건축적 특성 연구. https://academic.naver.com/article.naver?doc_id=176945228.

Are large cats compatible with modern society on the Korean Peninsula? Ecological Restoration. 34 (3): 173–183. doi:10.3368/er.34.3.173. S2CID 88992035.

Hodori. https://en.wikipedia.org/wiki/Hodori.

Introduction

Ever wondered how you could represent molecules in Python?

In this post, we’ll take a journey through the molecular world and explore five oxygen-containing molecules: water, carbon dioxide, ozone, hydrogen peroxide, and glucose.

We’ll learn how to represent them using Python data structures and also create a molecular graph using the networkx package.

Meet the Molecules

Here are the five Oxygen-containing molecules we will look at, their structures, and their overall role in the ecosystem:

1. Water (H2O): Water is a simple molecule composed of two hydrogen atoms and one oxygen atom. It is essential for life on Earth and is involved in many biological and chemical processes.
2. Carbon dioxide (CO2): Carbon dioxide consists of one carbon atom and two oxygen atoms. It is produced as a byproduct of cellular respiration in living organisms and is also a greenhouse gas that plays a significant role in Earth’s climate.
3. Ozone (O3): Ozone is a molecule made up of three oxygen atoms. It is found in Earth’s atmosphere, primarily in the stratosphere, where it forms the ozone layer that protects life on Earth from harmful ultraviolet radiation.
4. Hydrogen peroxide (H2O2): Hydrogen peroxide is a chemical compound with two hydrogen atoms and two oxygen atoms. It is a powerful oxidizer and is used as a disinfectant, bleaching agent, and antiseptic.
5. Glucose (C6H12O6): Glucose is a simple sugar and an essential energy source for living organisms. It is a carbohydrate that consists of six carbon atoms, twelve hydrogen atoms, and six oxygen atoms. During cellular respiration, glucose is broken down to produce energy in the form of ATP and releases carbon dioxide and water as byproducts.

Now, let’s compare both a traditional Python and graph approach to representing these molecules.

Representing Molecules in Python

In this section, we will explore how to represent the five molecules and their bonds using Python data structures like dictionaries and lists. Our goal is to create a simple and intuitive representation of these molecules, making it easy to understand their composition and structure.

Using Dictionaries and Lists

Python dictionaries and lists provide a convenient way to represent molecules and their bonds. We can create a dictionary where the keys represent the molecule names, and the values are lists of dictionaries describing the bonds between atoms within the molecule.

Here’s a code snippet illustrating how to represent the five molecules using dictionaries and lists:

The dictionary called molecules contains the names of the molecules as keys and lists of dictionaries describing their bonds as values. The dictionaries representing bonds contain keys for the bonded atoms and the number of bonds.

Displaying Molecules and Their Bonds

Now that we have represented the molecules and their bonds in a Python dictionary, we can display this information in a user-friendly format. The following code snippet prints the molecules and their bonds:

The output of this code will be:

This representation allows us to easily visualize the molecules and their bonds, but it lacks the graphical representation that the networkx package provides, which we will explore in the next section.

Visualizing Molecules as a Graph with NetworkX

In this section, we’ll learn how to represent the five molecules as a graph using the networkx package. This powerful Python library allows us to create, manipulate, and study the structure, dynamics, and functions of complex networks. We'll start by installing the package and then create a molecular graph to visualize the relationships between atoms in our molecules.

Installing the NetworkX Package

To install the networkx package, simply run the following command:

pip install networkx

Creating a Molecular Graph

With the networkx package installed, we can now create a graph to represent our molecules. A graph consists of nodes (atoms, in our case) and edges (bonds between atoms). We'll also include edge attributes to store the molecule name for each bond.

Here’s a code snippet that demonstrates how to create a molecular graph using networkx:

This script creates a networkx graph with nodes for the different atoms (H, O, C, and N) and adds edges representing the bonds between these atoms in each molecule. The graph is then visualized using matplotlib, with the molecule name labeled on each edge representing a bond in the respective molecule.

Please note that this representation is highly simplified and doesn’t take into account the detailed molecular structure or the specific locations of atoms and bonds within the molecules. We also intentionally duplicate molecule representations (not necessary in the graph) for the purpose of visualizing it here.

For an example of how to represent the graph with molecule logic consolidated in a single place but not optimized for visualization, see below:

Interpreting the Molecular Graph

The molecular graph allows us to visualize the relationships between the atoms in our molecules. In the graph, each node represents an atom, and each edge represents a bond between two atoms. The edge labels indicate which molecule the bond belongs to.

By visualizing the molecules as a graph, we gain a better understanding of the relationships between atoms and the overall structure of each molecule. However, it’s important to remember that this representation is a simplification and may not accurately depict the 3D structure of the molecules.

Conclusion

In this Medium post, we explored different approaches to represent and visualize oxygen-containing molecules using Python.

We started by discussing five molecules that use oxygen: water, carbon dioxide, ozone, hydrogen peroxide, and glucose. Then, we demonstrated how to represent these molecules using both traditional methods in Python and the networkx library. Although this approach provided a simple visualization of the atomic relationships, it may not be the most accurate way to depict molecular structures.

When representing and visualizing molecules in Python, it’s essential to choose the appropriate tools based on the specific requirements of your project. For simple graph-based representations, networkx can provide a quick and straightforward solution. However, for more accurate and detailed molecular visualizations, specialized libraries like RDKit are recommended.

By understanding the various tools and techniques available for molecular visualization in Python, you’ll be better equipped to tackle projects involving chemistry, biology, and other fields where molecular structures play a critical role.

Happy coding!

References

NetworkX:

• Documentation: https://networkx.org/documentation/stable/index.html
• Tutorial: https://networkx.org/documentation/stable/tutorial.html

RDKit:

• Homepage: https://www.rdkit.org/
• Documentation: https://www.rdkit.org/docs/index.html
• Getting Started: https://www.rdkit.org/docs/GettingStartedInPython.html

Cheminformatics and Computational Chemistry:

• Cheminformatics in Python: https://chem.libretexts.org/@go/page/1796
• Computational Chemistry: https://en.wikipedia.org/wiki/Computational_chemistry
• Cheminformatics Tools: https://en.wikipedia.org/wiki/List_of_cheminformatics_software

Background

I grew up playing the game of Go and currently work in the field of applied machine learning. As advances in reinforcement learning eventually led to AlphaGo and the Netflix documentary on its development, I began to think of the game differently.

Is it really “solved” for?

Many Go engines utilize Monte Carlo Tree Search (MCTS) combined with other CNNs or deep learning algorithms. MCTS is a search algorithm that consists of four main steps: Selection, Expansion, Simulation, and Backpropagation. The algorithm iteratively performs these steps to build a search tree and determines the best move based on the tree’s structure.

Here’s a high-level example of how it works:

1. Selection: Starting from the root node (the current game state), traverse the tree by selecting the child node with the highest Upper Confidence Bound applied to Trees (UCT) value, which balances exploration and exploitation. The traversal continues until a leaf node is reached.
2. Expansion: If the leaf node represents a non-terminal game state, expand the node by generating all possible moves and adding them as child nodes.
3. Simulation: Run a random playout (simulation) from the newly-expanded node’s game state until a terminal game state (win, loss, or draw) is reached. In Go, this involves randomly placing stones on the board while following the game’s rules until the game ends.
4. Backpropagation: Update the nodes in the path from the leaf node to the root node based on the simulation’s result. Each node keeps track of the number of simulations that have passed through it (visits) and the total score obtained from those simulations.

The algorithm repeats these steps until a predefined time limit or a maximum number of iterations is reached. The move corresponding to the child node of the root with the most visits is chosen as the best move.

But does the way a Go engine interprets the game really convey the depth behind the game? Is there still value in playing such a complicated game?

Let’s explore a bit further by understanding a bit of the history behind the game and then a specific game example.

What is Go?

Go is an ancient board game that originated in China over 2,500 years ago, and it is known as “Weiqi” in Chinese, “Baduk” in Korean, and “Igo” in Japanese. It is a strategic two-player game played on a grid of 19x19 lines, although smaller boards (such as 9x9 or 13x13) are also used for faster games or educational purposes.

The objective of the game is to surround and capture more territory than your opponent. The game is played with two sets of stones, one black and one white. Players take turns placing their stones on the intersections of the grid, with black going first. Once placed, a stone cannot be moved unless it is captured by the opponent.

Capturing occurs when a group of stones is completely surrounded by the opponent’s stones, causing them to be removed from the board. Additionally, the concept of “liberties” is important in Go, which refers to the empty points directly adjacent to a stone or group of stones. When a group of stones has no more liberties, it is captured.

Go is a game of immense strategic depth, with a large number of potential moves and complex interactions between groups of stones. As a result, this is what makes it a popular subject for artificial intelligence research.

A famous game & move example

One of the most famous series of moves in Go history is the “ear-reddening move” (耳赤の一手), which occurred during a game between two legendary Japanese Go players, Honinbo Shusaku (本因坊秀策) and Gennan Inseki (厳然因碩), in 1846. Shusaku was just a teenager at the time, while Gennan was a top player in his 50s and considered one of the strongest of his era.

The ear-reddening move is a single move (move 127) played by Shusaku during this game. It was a surprising and deep move that demonstrated Shusaku’s exceptional reading ability and strategic insight. Legend has it that Gennan’s ears turned red with embarrassment upon realizing the implications of Shusaku’s move, hence the name “ear-reddening move.”

This move eventually led to Shusaku’s victory and further solidified his reputation as one of the greatest Go players of all time. The game itself is also famous and is known as the “ear-reddening game” (耳赤の一局). It is still studied by Go players today for its high level of play and the insights it offers into the game’s strategic depth.

A more detailed analysis

1. Opening (Fuseki): Shusaku, playing as Black, opens with a diagonal move in the lower right corner (4–4 point, position D16). White’s response is the 3–4 point in the upper left corner (C4). Shusaku’s opening strategy is known as the “Shusaku Fuseki,” which focuses on rapid development and strong positions.

2. Moves 31–60: The players work on expanding their frameworks while keeping an eye on their opponents’ weaknesses. Shusaku’s moves during this phase, such as move 35 at R9, R10, aim to reinforce his territorial claims and secure groups. White, on the other hand, looks for ways to invade or reduce Black’s potential territory.

3. Move 70: White invades Black’s moyo (a large framework with the potential to become territory) at position O10. This move is critical because it challenges Black’s territorial claims in the center and the left side of the board.

4. Moves to 90: In response to White’s invasion, Black tries to surround and attack White’s group, aiming to strengthen his own position while putting pressure on White. A key move to note is Black’s move at position O5 earlier, which challenges White’s attempts to secure his invaded group.

5. Moves 91–126: White manages to create a living group on the left side of the board, reducing Black’s potential territory. However, Black’s moves, such as move 119, continue to pressure White’s groups and create opportunities for future attacks.

6. Move 127 (Ear Reddening Move): Shusaku plays at K11, a subtle move that threatens to connect his stones on the left side while also reducing White’s influence in the center. This move shifts the balance of power in the game and marks the beginning of the end for White. Experts have praised this move as a brilliant example of Shusaku’s deep understanding of the game, as it creates multiple threats and opportunities without directly engaging the opponent.

1. Moves 128-End: As the game progresses, Shusaku manages to exploit the weaknesses exposed by move 127, leading to a decisive advantage. White is unable to recover from the position created by the Ear Reddening Move, and Black eventually secures the victory.

An analysis like this highlights the strategic planning and fighting spirit displayed by both players, particularly Shusaku’s ability to identify and exploit opportunities.

A modern Go engine might be able to provide a similar recommendation on this series of moves, but what other lessons can we learn from this game?

Lessons beyond the game

The Ear Reddening Game, like many other classic Go games, offers valuable lessons that can be applied to life more generally.

Here are seven things we can learn from the game:

1. Patience and foresight: Shusaku’s Ear Reddening Move (move 127) demonstrates the importance of patience and the ability to see long-term consequences of one’s actions. In life, being patient and considering the long-term effects of our decisions can lead to better outcomes.
2. Balance: In Go, players must balance attack and defense, territorial gains, and influence. Similarly, in life, it’s essential to maintain a balance between different aspects, such as work and personal life, short-term and long-term goals, and risk and reward.
3. Adaptability: Throughout the game, both players had to adapt their strategies in response to their opponent’s moves. In life, being adaptable and flexible in the face of changing circumstances can help us navigate challenges more effectively.
4. The importance of small moves: In the Ear Reddening Game, seemingly small moves had a significant impact on the final outcome. In life, small actions can sometimes have far-reaching consequences, so it’s essential to pay attention to details and not underestimate the value of small efforts.
5. Learning from others: The game offers an opportunity to learn from the strategies and tactics of two great Go players. In life, we can grow by learning from the experiences, knowledge, and wisdom of others.
6. Resilience and perseverance: The players in this game faced setbacks and challenges but continued to play until the end. In life, resilience and perseverance are crucial qualities that enable us to overcome obstacles and achieve our goals.
7. Appreciating the beauty in complexity: The Ear Reddening Game is a beautiful example of the depth and complexity of Go. In life, we can learn to appreciate the beauty and intricacies of complex systems, relationships, and situations.

Even if Go engines can get to an advanced level of gameplay, studying and learning from classic Go games like the Ear Reddening Game can offer valuable insights and teach us important life lessons that extend beyond the game board.

Reference

GoKifu replay. Link.

Sensei’s Library. Link.

Background

In our last article, Algorithmic Approaches to Music Theory: Conventional and Graph-Based Methods, we used conventional Python and graph-based approaches to explore the foundations of music theory, starting with the Western major and minor scales.

Today, let’s take a look at the Indian Shruti system, which is based on 22 Shruti and not 12 notes from an equal temperament system. We will take the same approach and better understand the music theory behind the Shruti system through the implementation of basic scales using both a conventional Python method and a graph-based approach. Then, we can analyze the learnings from an implementation and from a topic perspective.

Shruti major scale: Shruti Shuddha Saptak

The Indian Shruti major scale, also known as the “Shruti Shuddha Saptak,” is a scale in Indian classical music derived from the 22-Shruti system. The term “shruti” refers to the smallest musical interval in Indian classical music. The Indian Shruti major scale is analogous to the Western major scale.

Here is the algorithm to define the Indian Shruti major scale:

1. Start with the fundamental frequency (tonic), called “Sa” or “Shadja” in Indian music. This is the first note of the scale.
2. Calculate the following frequency ratios for the subsequent notes:

• Sa (Shadja): 1/1
• Re (Rishabh): 9/8
• Ga (Gandhar): 5/4
• Ma (Madhyam): 4/3
• Pa (Pancham): 3/2
• Dha (Dhaivat): 27/16
• Ni (Nishad): 15/8

3. Multiply the fundamental frequency by each of these ratios to get the frequencies of the corresponding notes in the Indian Shruti major scale.

4. The final note, “Sa” (Shadja) in the upper octave, has a ratio of 2/1, which means it’s exactly double the frequency of the starting note.

For example, if the starting note (Sa) has a frequency of 240 Hz:

1. Sa (Shadja): 240 * (1/1) = 240 Hz
2. Re (Rishabh): 240 * (9/8) = 270 Hz
3. Ga (Gandhar): 240 * (5/4) = 300 Hz
4. Ma (Madhyam): 240 * (4/3) = 320 Hz
5. Pa (Pancham): 240 * (3/2) = 360 Hz
6. Dha (Dhaivat): 240 * (27/16) = 405 Hz
7. Ni (Nishad): 240 * (15/8) = 450 Hz
8. Sa (Shadja) upper octave: 240 * (2/1) = 480 Hz

Keep in mind that these are only approximate values for the Indian Shruti major scale. In actual performance, musicians may make small adjustments to these frequencies based on the specific raga being performed or the melodic context.

Python implementation

Below is the conventional way to code the algorithm for the Shruti Suddha Saptak using Python.

When you run this code, it will output the frequencies of the Indian Shruti major scale based on the provided tonic frequency:

Graph (networkx) implementation

Now, here’s the Python code to represent the Indian Shruti major scale as a NetworkX graph:

This code creates a directed graph with nodes representing the notes of the Indian Shruti major scale and edges representing the frequency ratios between the notes.

We can also visualize the graph with the code below:

The draw_graph function uses NetworkX and matplotlib to visualize the graph with a circular layout. The graph shows the notes with their names and the edges with the frequency ratios.

Now, what if we try to add the minor scale equivalent to both implementations?

Shruti Minor Scale: Asavari Thaat

In the Indian Shruti system, there isn’t a direct equivalent of the Western minor scale. However, you can create a scale with a similar mood or tonality by selecting different swaras (notes) from the 22-shruti system. One possible example is the Asavari Thaat, which is a scale used in Hindustani music and has a somewhat similar tonality to the natural minor scale.

Here’s the algorithm for the Asavari Thaat scale:

1. Start with the fundamental frequency (tonic), called “Sa” or “Shadja” in Indian music. This is the first note of the scale.
2. Calculate the following frequency ratios for the subsequent notes:

• Sa (Shadja): 1/1
• Re (Rishabh): 9/8
• Ga (Gandhar): 32/27
• Ma (Madhyam): 4/3
• Pa (Pancham): 3/2
• Dha (Dhaivat): 8/5
• Ni (Nishad): 16/9

3. Multiply the fundamental frequency by each of these ratios to get the frequencies of the corresponding notes in the Asavari Thaat scale.

4. The final note, “Sa” (Shadja) in the upper octave, has a ratio of 2/1, which means it’s exactly double the frequency of the starting note.

For example, if the starting note (Sa) has a frequency of 240 Hz:

1. Sa (Shadja): 240 * (1/1) = 240 Hz
2. Re (Rishabh): 240 * (9/8) = 270 Hz
3. Ga (Gandhar): 240 * (32/27) = 284.44 Hz
4. Ma (Madhyam): 240 * (4/3) = 320 Hz
5. Pa (Pancham): 240 * (3/2) = 360 Hz
6. Dha (Dhaivat): 240 * (8/5) = 384 Hz
7. Ni (Nishad): 240 * (16/9) = 426.67 Hz
8. Sa (Shadja) upper octave: 240 * (2/1) = 480 Hz

Keep in mind that Indian classical music is based on ragas, which have more intricate melodic structures and phrasings than the scales found in Western music. This is just an example of how to create a scale similar to the Western minor scale within the 22-shruti system.

Python implementation

Here is the conventional Python implementation of the Asavari Thaat scale.

When you run this code, it will output the frequencies of the Asavari Thaat scale based on the provided tonic frequency:

We can change the tonic_frequency variable to generate the scale based on a different starting frequency.

Graph (networkx) implementation

Here’s how we can add the Asavari Thaat scale to the existing major scale NetworkX graph:

Let’s clean up the graph to make it more readable and at scale:

This outputs a graph like below, although it can also be represented in a single directed graph.

Analysis

Learnings on the two implementation approaches

When comparing the implementation of the Indian Shruti major scale and Asavari Thaat scale using Python code versus a NetworkX graph representation, we can observe several differences:

1. Representation:

• In the Python implementation, scales are represented as lists of frequency values. This approach focuses on the numeric values of the frequencies.
• In the NetworkX graph representation, the scales are represented as graphs with nodes and edges. This approach emphasizes the relationships between the notes, making it easier to visualize and understand the structure of the scales.

2. Visualization:

• The Python implementation does not inherently provide a visual representation of the scales. If you want to visualize the relationships between the notes, you would need to use additional libraries or tools.
• With the NetworkX graph approach, you can easily create a visual representation of the scales, clearly showing the relationships between the notes and their frequencies. This can be beneficial for understanding the overall structure of the scales and their connections.

3. Scalability:

• In the Python implementation, adding more scales or modifying the existing scales requires changes to the code itself, potentially making it more complex as new scales are added.
• In the NetworkX graph representation, adding more scales or modifying the existing scales can be done by simply adding nodes and edges to the graph. This allows for easier scaling and modification of the representation as new scales are added or existing ones are updated.

4. Flexibility:

• The Python implementation is more focused on the specific calculations of frequencies and may not easily support additional information or properties related to the scales.
• The NetworkX graph representation can store additional information and properties related to the notes, such as names or other attributes, making it a more flexible representation that can be extended to include more information as needed.

While the Python implementation is more focused on the numeric calculations of frequencies, the NetworkX graph approach provides a more visually appealing and flexible representation that emphasizes the relationships between the notes. Depending on the requirements of your specific use case, you may choose one approach over the other based on the advantages and disadvantages of each method.

Learnings on the Shruti method

From the two implementations (Python code and NetworkX graph), we can also learn various aspects of the two scales (Indian Shruti major scale and Asavari Thaat scale) and the Shruti system:

1. Frequency Ratios:

• Both implementations allow us to calculate the frequency ratios between the consecutive notes of the scales. These ratios are essential in understanding the intervals between notes, which define the unique characteristics of each scale.

2. Scale Structure:

• The NetworkX graph implementation helps us visualize the structure of the scales and their connections. This visualization makes it easier to understand the relationships between notes in each scale and compare the two scales.

3. Scale Differences:

• Comparing the Python implementations or the NetworkX graph representation, we can observe differences in the frequency ratios between the two scales. For example, the Indian Shruti major scale has a major third (Ga), while the Asavari Thaat scale has a minor third (Ga). This difference in intervals leads to the distinct sound of each scale.

4. Shruti System:

• The Python implementations illustrate the algorithmic approach to generating the scales in the Shruti system, highlighting the importance of frequency ratios in defining the scales. By comparing the two scales, we can see how the Shruti system can generate different scales by modifying the intervals between notes.

5. Flexibility and Extensibility:

• The NetworkX graph representation shows the potential for easily adding more scales or modifying the existing scales, allowing us to analyze a broader range of scales within the Shruti system. This flexibility can help us better understand the relationships between various scales in the system and their properties.

Conclusion

In conclusion, the Python code and NetworkX graph implementations provide different perspectives on the Indian Shruti major scale, the Asavari Thaat scale, and the Shruti system. By examining these implementations, we can gain insights into the structure, relationships, and unique characteristics of the scales and better understand the principles underlying the Shruti system.

Background

As an experienced engineer with a past life in professional music, I have always been fascinated by the convergence of music and technology. Both domains share commonalities, such as mathematical underpinnings and logical structures.

In this article, my objective is to explore and compare conventional and graph-based techniques for developing algorithms that encapsulate music theory concepts. By investigating these methodologies, it is interesting to understand how the basis for modern music theory works.

Approach

Step 1: Establishing the algorithm for a major scale

A major scale in music can be defined by its pattern of whole and half steps. In Western music, the major scale consists of seven notes, following the pattern: whole, whole, half, whole, whole, whole, half.

Here’s an algorithmic representation of generating a major scale for a given root note (assuming a 12-tone equal temperament tuning system):

1. Define the pattern of whole and half steps for a major scale: [2, 2, 1, 2, 2, 2, 1].
2. Define the list of note names in chromatic order: [“C”, “C#”, “D”, “D#”, “E”, “F”, “F#”, “G”, “G#”, “A”, “A#”, “B”].
3. Determine the index of the root note in the chromatic list (e.g., “C” would be 0, “C#” would be 1, “D” would be 2, and so on).
4. Initialize an empty list to store the major scale notes.
5. Iterate through the pattern of whole and half steps, adding the current step value to the index of the root note, and modulo 12 to wrap around the chromatic list if necessary.
6. For each resulting index, append the corresponding note from the chromatic list to the major scale notes list.
7. Return the list of major scale notes.

Step 2: Comparing the conventional and graph approaches to representing the major scale algorithm

The conventional approach for this would be to implement the following:

Testing this out by printing the C and D major scales provides the following outputs:

Now, we can build a graph version of the major scale algorithm with the following implementation done in networkx (Python):

When we output the C and D major scales, we receive the same result as with the conventional method.

The elements of the graph can also be visualized in various ways that can be intuitive to a broader audience of users. It is also interesting to note in the visual below the recursive nature of the root notes (tonic).

Here is one mechanism for implementing the visualization so:

While there is more overhead in building the graph initially, the logic can be incrementally built upon in an intuitive way. We can observe this by adding the logic for a minor scale.

Let’s compare the two approaches again.

Step 3: Adding the minor scale

The natural minor scale has the following pattern: whole, half, whole, whole, half, whole, whole.

In the conventional approach, we create a separate function for the minor scale. This is in no way connected to the original snippet for the major scale.

We can test this with the below code:

The composable nature of the graph allows users to see a major scale in relation to a minor scale while also having a single place for the logic. We see this when we layer in the minor scale algorithm on top of the existing foundation of the major scale code.

Let’s try outputting C and D major and minor scales. We can query a single graph for both.

Similar to the major scale graph, this combined graph with both major and minor scales can also be visualized. This graph can be customized and made much more interactive if written in D3, but for this exercise, we will leave it in matplotlib.

Conclusion

In this article, we explored music theory algorithms through two approaches — a conventional python implementation and a graph-based approach. From this exercise, we can see how to construct the algorithms for major and minor scales and start to visualize how they relate to each other.

Check out the repo here for a notebook with all of the code in one place.

What other music theory algorithms should we include next?