Stage 4: Deep Learning and Neural Networks
Open-source LLM textbook from Zhejiang University covering architecture, prompt engineering, fine-tuning, model editing, and RAG
Foundations of Large Language Models (ZJU-LLMs)
Overview
Foundations of LLMs is a comprehensive educational textbook and learning resource created by the Database and Big Data Analytics Laboratory (DAILY Lab) at Zhejiang University. This open-source project systematically explains foundational knowledge and introduces cutting-edge technologies related to Large Language Models (LLMs) for readers interested in this rapidly evolving field.
Repository: https://github.com/ZJU-LLMs/Foundations-of-LLMs
Stars: 11.4k+ on GitHub
Format: PDF textbook with chapter-by-chapter materials and paper lists
Project Philosophy
The author team is committed to:
- Listening to suggestions from the open-source community and experts
- Providing monthly updates to keep content current
- Creating an accessible, rigorous, and in-depth textbook on large models
- Tracking the latest technological developments through curated paper lists for each chapter
Content Structure
The first edition consists of six main chapters, each using a different animal as a thematic background to illustrate specific technologies:
Chapter 1: Traditional Language Models
- Foundation concepts in language modeling
- Historical context and evolution of language models
Chapter 2: Large Language Model Architecture
- Evolution of LLM architectures
- Key architectural innovations and design principles
- Comparative analysis of different model structures
Chapter 3: Prompt Engineering
- Techniques for effective prompt design
- Prompt optimization strategies
- Applications and best practices
Chapter 4: Parameter-Efficient Fine-Tuning (PEFT)
- Methods for efficient model adaptation
- Low-resource fine-tuning techniques
- LoRA, Prefix-tuning, and other PEFT approaches
Chapter 5: Model Editing
- Techniques for modifying model knowledge
- Knowledge updating and correction methods
- Preserving model integrity during edits
Chapter 6: Retrieval-Augmented Generation (RAG)
- Integration of retrieval systems with generative models
- Enhancing LLM outputs with external knowledge
- RAG architectures and implementation strategies
Available Resources
1. Complete Textbook
- Full PDF version available in Chinese
- English version also available
- Comprehensive coverage of all six chapters
2. Chapter-by-Chapter Content
- Individual PDF files for each chapter
- Allows focused study on specific topics
- Easy navigation and reference
3. Paper Lists
- Curated collections of relevant research papers for each chapter
- Continuously updated with latest research
- Tracks cutting-edge developments in each area
Future Directions
The author team plans to expand the textbook with additional chapters covering:
- LLM Inference Acceleration: Techniques for faster model inference
- LLM Agents: Autonomous agents powered by large language models
- Additional emerging topics in the LLM landscape
Target Audience
This learning resource is designed for:
- Students and researchers in AI and NLP
- Practitioners working with LLMs
- Anyone interested in understanding the foundations of large language models
- Developers building LLM-based applications
Unique Features
- Visual Learning: Each chapter features a unique animal theme to make concepts more memorable and engaging
- Community-Driven: Open to issues and feedback from the community
- Regular Updates: Monthly updates ensure content remains current
- Comprehensive Paper Lists: Each chapter includes curated research papers
- Free and Open: Completely open-source and freely accessible
- Bilingual Support: Available in both Chinese and English
Academic Rigor
The content is based on the author team's exploration and understanding of relevant research directions. The team actively welcomes corrections and suggestions through GitHub issues, ensuring continuous improvement and accuracy.
Contact Information
For questions or suggestions related to the textbook:
- Email: xuwenyi@zju.edu.cn
- GitHub Issues: Submit feedback and suggestions directly on the repository
Why This Resource Matters
Large Language Models have become one of the most revolutionary technological advancements in artificial intelligence. This textbook provides:
- Systematic Foundation: Build understanding from traditional models to modern LLMs
- Practical Techniques: Learn actionable methods for working with LLMs
- Research Connections: Stay connected to the latest research through paper lists
- Progressive Learning: Move from basic concepts to advanced techniques
Community Recognition
With over 11,400 stars on GitHub, this project has gained significant recognition in the AI/ML community as a valuable educational resource for understanding Large Language Models.