Skip to content

Andrew #1738

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 6 commits into from
Sep 26, 2024
Merged

Andrew #1738

Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
8f133a6
remove latest blog post
LinuxAzzamean Sep 25, 2024
694c3be
New Fixed Blog Post
LinuxAzzamean Sep 25, 2024
68f6641
fix title
LinuxAzzamean Sep 25, 2024
df28bcd
Merge branch 'site' of https://github.com/pytorch/pytorch.github.io i…
LinuxAzzamean Sep 25, 2024
cccfa75
fix v1
LinuxAzzamean Sep 26, 2024
d7782a9
fix v2
LinuxAzzamean Sep 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
13 changes: 3 additions & 10 deletions _posts/2024-09-25-pytorch-native-architecture-optimization.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,9 @@
---
layout: blog_detail
title: "CUDA-Free Inference for LLMs"
title: "PyTorch Native Architecture Optimization: torchao"
author: Team PyTorch
---

# PyTorch Native Architecture Optimization: torchao

By Team PyTorch

We’re happy to officially launch torchao, a PyTorch native library that makes models faster and smaller by leveraging low bit dtypes, quantization and sparsity. [torchao](https://github.com/pytorch/ao) is an accessible toolkit of techniques written (mostly) in easy to read PyTorch code spanning both inference and training. This blog will help you pick which techniques matter for your workloads.

Expand Down Expand Up @@ -61,15 +58,11 @@ from torchao.quantization import (
float8\_dynamic\_activation\_float8\_weight,
)

![](/assets/images/Figure_1.png){:style="width:100%"}

<<<<<<< HEAD:_posts/2024-09-25-pytorch-native-architecture-optimization.md
We also have extensive benchmarks on diffusion models in collaboration with the HuggingFace diffusers team in [diffusers-torchao](https://github.com/sayakpaul/diffusers-torchao) where we demonstrated 53.88% speedup on Flux.1-Dev and 27.33% speedup on CogVideoX-5b
=======
We also have extensive benchmarks on diffusion models in collaboration with the HuggingFace diffusers team in [diffusers-torchao](https://github.com/sayakpaul/diffusers-torchao) where we demonstrated 53.88% speedup on Flux.1-Dev and 27.33% speedup on CogVideoX-5b

![](/assets/images/Figure_1.png){:style="width:100%"}

We also have extensive benchmarks on diffusion models in collaboration with the HuggingFace diffusers team in [diffusers-torchao](https://github.com/sayakpaul/diffusers-torchao) where we demonstrated 53.88% speedup on Flux.1-Dev and 27.33% speedup on CogVideoX-5b
>>>>>>> 97898699f7101b847da377106274783ced03bb3d:_posts/2024-09-25-pytorch-native-architecture-optimizaion.md

Our APIs are composable so we’ve for example composed sparsity and quantization to bring 5% [speedup for ViT-H inference](https://github.com/pytorch/ao/tree/main/torchao/sparsity)

Expand Down