Text-Generation-Inference is a solution build for deploying and serving Large Language Models (LLMs). TGI enables high-performance text generation using Tensor Parallelism and dynamic batching for the most popular open-source LLMs, including StarCoder, BLOOM, GPT-NeoX, Llama, and T5. Text Generation Inference is already used by customers such as IBM, Grammarly, and the Open-Assistant initiative implements optimization for all supported model architectures, including:

• Tensor Parallelism and custom cuda kernels
• Optimized transformers code for inference using flash-attention and Paged Attention on the most popular architectures
• Quantization with bitsandbytes or gptq
• Continuous batching of incoming requests for increased total throughput
• Accelerated weight loading (start-up time) with safetensors
• Logits warpers (temperature scaling, topk, repetition penalty ...)
• Watermarking with A Watermark for Large Language Models
• Stop sequences, Log probabilities
• Token streaming using Server-Sent Events (SSE)

Currently optimized architectures

• BLOOM
• FLAN-T5
• Galactica
• GPT-Neox
• Llama
• OPT
• SantaCoder
• Starcoder
• Falcon 7B
• Falcon 40B

Check out the source code 👉

• the server backend: https://github.com/huggingface/text-generation-inference
• the Chat UI: https://huggingface.co/spaces/text-generation-inference/chat-ui

Check out examples

• Introducing the Hugging Face LLM Inference Container for Amazon SageMaker
• Deploy LLMs with Hugging Face Inference Endpoints