index.rst

TorchServe

⚠️ Notice: Limited Maintenance

This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches. Users should be aware that vulnerabilities may not be addressed.

TorchServe is a performant, flexible and easy to use tool for serving PyTorch models in production.

What's going on in TorchServe?

• High performance Llama 2 deployments with AWS Inferentia2 using TorchServe
• Naver Case Study: Transition From High-Cost GPUs to Intel CPUs and oneAPI powered Software with performance
• Run multiple generative AI models on GPU using Amazon SageMaker multi-model endpoints with TorchServe and save up to 75% in inference costs
• Deploying your Generative AI model in only four steps with Vertex AI and PyTorch
• PyTorch Model Serving on Google Cloud TPUv5
• Monitoring using Datadog
• Torchserve Performance Tuning, Animated Drawings Case-Study
• Walmart Search: Serving Models at a Scale on TorchServe
• Scaling inference on CPU with TorchServe
• TorchServe C++ backend
• TorchServe with Intel® Extension for PyTorch*
• Grokking Intel CPU PyTorch performance from first principles: a TorchServe case study
• Grokking Intel CPU PyTorch performance from first principles( Part 2): a TorchServe case study
• Case Study: Amazon Ads Uses PyTorch and AWS Inferentia to Scale Models for Ads Processing
• Optimize your inference jobs using dynamic batch inference with TorchServe on Amazon SageMaker
• Using AI to bring children's drawings to life
• Model Serving in PyTorch
• Evolution of Cresta's machine learning architecture: Migration to AWS and PyTorch
• Explain Like I’m 5: TorchServe
• How to Serve PyTorch Models with TorchServe
• How to deploy PyTorch models on Vertex AI
• Quantitative Comparison of Serving Platforms

.. customcardstart::

.. customcarditem::
   :header: TorchServe Quick Start
   :card_description: Learn how to install TorchServe and serve models.
   :image: https://user-images.githubusercontent.com/880376/83180095-c44cc600-a0d7-11ea-97c1-23abb4cdbe4d.jpg
   :link: getting_started.html
   :tags: Quick Start

.. customcarditem::
   :header: Running TorchServe
   :card_description: Indepth explanation of how to run TorchServe
   :image: https://raw.githubusercontent.com/pytorch/serve/master/docs/images/dogs-after.jpg
   :link: server.html
   :tags: Running TorchServe

.. customcarditem::
   :header: Why TorchServe
   :card_description: Various TorchServe use cases
   :image: https://download.pytorch.org/torchaudio/tutorial-assets/thumbnails/streamreader_basic_tutorial.png
   :link: use_cases.html
   :tags: Examples

.. customcarditem::
   :header: TorchServe GenAI Use Cases
   :card_description: Showcasing GenAI deployment scenarios and use cases
   :image: https://raw.githubusercontent.com/pytorch/serve/master/examples/LLM/llama/images/llama.png
   :link: genai_use_cases.html
   :tags: Use Cases

.. customcarditem::
   :header: Performance
   :card_description: Guides and best practices on how to improve perfromance when working with TorchServe
   :image: https://raw.githubusercontent.com/pytorch/serve/master/benchmarks/predict_latency.png
   :link: performance_guide.html
   :tags: Performance,Troubleshooting

.. customcarditem::
   :header: Metrics
   :card_description: Collecting and viewing Torcherve metrics
   :image: https://user-images.githubusercontent.com/5276346/234725829-7f60e0d8-c76d-4019-ac8f-7d60069c4e58.png
   :link: metrics.html
   :tags: Metrics,Performance,Troubleshooting

.. customcarditem::
   :header: Large Model Inference
   :card_description: Serving Large Models with TorchServe
   :image: https://raw.githubusercontent.com/pytorch/serve/master/docs/images/ts-lmi-internal.png
   :link: large_model_inference.html
   :tags: Large-Models,Performance

.. customcarditem::
   :header: Troubleshooting
   :card_description: Various updates on Torcherve and use cases.
   :image: https://raw.githubusercontent.com/pytorch/serve/master/benchmarks/snake_viz.png
   :link: Troubleshooting.html
   :tags: Troubleshooting,Performance

.. customcarditem::
   :header: TorchServe Security Policy
   :card_description: Security Policy
   :image: https://user-images.githubusercontent.com/880376/83180095-c44cc600-a0d7-11ea-97c1-23abb4cdbe4d.jpg
   :link: security.html
   :tags: Security

.. customcarditem::
   :header: FAQs
   :card_description: Various frequently asked questions.
   :image: https://raw.githubusercontent.com/pytorch/serve/master/docs/images/NMTDualTranslate.png
   :link: FAQs.html
   :tags: FAQS

.. customcardend::