Lukas Hauzenberger·Oct 7, 2024Efficient LLM Pretraining: Packed Sequences and Masked AttentionTraining large language models (LLMs) is a computationally demanding task. It requires vast amounts of data, powerful hardware, and clever…
Lukas Hauzenberger·May 26, 2024Introducing pytora — a minimal PyTorch implementation of LoRATLDR - here is the code: https://github.com/sirluk/pytora
Lukas Hauzenberger·Jan 16, 2024Multilabel Classification using Mistral-7B on a single GPU with quantization and LoRALLMs have impressed with there abilities to solve a wide variety of tasks, not only for natural language but also in a multimodal setting…A response icon11A response icon11
Lukas Hauzenberger·Dec 18, 2023What I learned doing Data Science Interviews in the 2020sFew disciplines have experienced hype periods comparable to that of Data Science (DS) and Machine Learning (ML) since approximately 2016…
Lukas Hauzenberger·Apr 7, 2023Training and serving PyTorch models in the Google Cloud with Vertex AI pipelinesWith each new feature Vertex AI is making it easier to put Machine Learning (ML) prototypes into production. Nowadays you do not really…A response icon3A response icon3
Lukas Hauzenberger·Aug 22, 2022Adverserial TrainingLarge language models following the transformer architecture have been incredibly successful since their introduction by Vaswani et. al. in…
Lukas Hauzenberger·Feb 23, 2022An practical introduction to Diff-Pruning for BERTIn this article I would like to give a short introduction for an elegant technique to make finetuning of large language models more storage…A response icon4A response icon4