Prices shown in US dollars.
Find my training Basket 0
NVIDIA

Model Parallelism: Building and Deploying Large Neural Networks (NV-MP-DEPLOY-NETW)

Very large deep neural networks (DNNs), whether applied to natural language processing (e.g., GPT-3), computer vision (e.g., huge Vision Transformers), or speech AI (e.g., Wave2Vec 2) have certain properties that set them apart from their smaller counterparts. As DNNs become larger and are trained on progressively larger datasets, they can adapt to new tasks with just a handful of training examples, accelerating the route toward general artificial intelligence. Training models that contain tens to hundreds of billions of parameters on vast datasets isn’t trivial and requires a unique combination of AI, high-performance computing (HPC), and systems knowledge. Virtual Learning This interactive training can be taken from any location, your office or home and is delivered by a trainer. This training does not have any delegates in the class with the instructor, since all delegates are virtually connected. Virtual delegates do not travel to this course, Global Knowledge will send you all the information needed before the start of the course and you can test the logins.

GK #GK847007Company EventVirtual Learning
Duration1 dayFormatsCompany Event · Virtual LearningTopicsApplications Development And Programming

What you’ll learn

Course Objectives

  • Train neural networks across multiple servers
  • Use techniques such as activation checkpointing, gradient accumulation, and various forms of model parallelism to overcome the challenges associated with large-model memory footprint
  • Capture and understand training performance characteristics to optimize model architecture
  • Deploy very large multi-GPU models to production using NVIDIA Triton™ Inference Server

Course Content

Module 1:  Introduction

 

Module 2: Introduction to Training of Large Models      

  • Learn about the motivation behind and key challenges of training large models.
  • Get an overview of the basic techniques and tools needed for large-scale training.
  • Get an introduction to distributed training and the Slurm job scheduler.
  • Train a GPT model using data parallelism.
  • Profile the training process and understand execution performance.

 

Module 3: Model Parallelism: Advanced Topics

  • Increase the model size using a range of memory-saving techniques.
  • Get an introduction to tensor and pipeline parallelism.
  • Go beyond natural language processing and get an introduction to DeepSpeed.
  • Auto-tune model performance.
  • Learn about mixture-of-experts models.

 

Module 4: Inference of Large Models

  • Understand the challenges of deployment associated with large models.
  • Explore techniques for model reduction.
  • Learn how to use TensorRT-LLM.
  • Learn how to use Triton Inference Server.
  • Understand the process of deploying GPT checkpoint to production.
  • See an example of prompt engineering.

 

Module 5: Final Review

  • Review key learnings and answer questions.
  • Complete the assessment and earn a certificate.
  • Complete the workshop survey.

Upcoming training

No upcoming dates are listed yet. You can still enquire with flexible dates, and we’ll check options for your team.