Episode 26: Developing and Training LLMs From Scratch

Vanishing Gradients

Hugo speaks with Sebastian Raschka, a machine learning & AI researcher, programmer, and author. As Staff Research Engineer at Lightning AI, he focuses on the intersection of AI research, software development, and large language models (LLMs).

How do you build LLMs? How can you use them, both in prototype and production settings? What are the building blocks you need to know about?

​In this episode, we’ll tell you everything you need to know about LLMs, but were too afraid to ask: from covering the entire LLM lifecycle, what type of skills you need to work with them, what type of resources and hardware, prompt engineering vs fine-tuning vs RAG, how to build an LLM from scratch, and much more.

The idea here is not that you’ll need to use an LLM you’ve built from scratch, but that we’ll learn a lot about LLMs and how to use them in the process.

Near the end we also did some live coding to fine-tune GPT-2 in order to create a spam classifier!

LINKS

• The livestream on YouTube
• Sebastian's website
• Machine Learning Q and AI: 30 Essential Questions and Answers on Machine Learning and AI by Sebastian
• Build a Large Language Model (From Scratch) by Sebastian
• PyTorch Lightning
• Lightning Fabric
• LitGPT
• Sebastian's notebook for finetuning GPT-2 for spam classification!
• The end of fine-tuning: Jeremy Howard on the Latent Space Podcast
• Our next livestream: How to Build Terrible AI Systems with Jason Liu
• Vanishing Gradients on Twitter
• Hugo on Twitter