Efficient deployment and inference of GPU-accelerated LLMs​

NVIDIA TensorRT-LLM is an open-source software that delivers state-of-the-art performance for LLM serving using NVIDIA GPUs. It consists of the TensorRT deep learning compiler and includes optimized kernels, pre- and post-processing steps, and multi-GPU/multi-node communication primitives.​ During this session, I will present TensorRT-LLM features and capabilities and walk the audience through the steps needed to build and run a model in TensorRT-LLM on both single GPU and multi-GPUs. I will also show how to use TRT-LLM backend and Triton Inference Server for deployment.​