13C: Fullstack LLMs

Materials:

Date: tbd

Pre-work:

1. LLM Intro
2. LLM Ops
3. XAI Tutorial by Hima Lakkaraju, Julius Adebayo, Sameer Singh
4. UQ Tutorial by Balaji Lakshminarayanan

ML Engineering

• LLaMA Stack - a full stack LLaMA-centered APIs for inference, safety, agentic system, among others.
• MLFlow LLMs - tool calling, agents, evaluation, RAGs, serving and more
• Ray LLMs
• MLFlow Tracing observability for LLMs
• Others popular stacks LlamaIndex, LangChain
• DeepEval

XAI

• XAI @ Harvard, Spring’23, Explainable AI by Prof.Hima Lakkaraju
• Publications by Hima Lakkaraju
• LLMs for XAI
  • Can Large Language Models Simplify Explainable AI
• XAI for LLMs
  • Studying Large Language Model Generalization with Influence Functions
  • Do Influence Functions Work on Large Language Models?

UQ

• Quantifying Uncertainty in Natural Language Explanations of Large Language Models
• Conformal Prediction with Large Language Models for Multi-Choice Question Answering code

Security

• NeMO Gaurdrails
• LlaMA Gaurd 7B Model, paper