Working Notes: a commonplace notebook for recording & exploring ideas.
Home. Site Map. Subscribe. More at expLog.

Transformers

date: 2023-09-23
last-updted: 2023-10-14

Have the potential to change programming; spending the time to understand how they work, and work backwards to understand what would be valuable with them.

Exploring

the basics behind transformers, with a running commentary on important papers
programming equivalent to transformers, RASPy etc.
performance, and mechanical sympathy for transformers
fine tuning and similar shenanigans to make them easier to work with

Ideally I can use this to build simple models for personal projects (shell completion, etc.) and also to write about mechanical sympathy for models.

Terms

Term	One line definition	Detailed Notes
Attention
Cublas
Decoder
Encoder
Flash Attention [2]
FP8
MKL
NCCL
Pipeline Parallelism
Transformer

Resources

Sasha rush puzzles (https://github.com/srush/LLM-Training-Puzzles)
[Annotated transformer](Collected resources, mostly sourced from Threads:)
nanoGPT, Andrej Karpathy
Implementation of llamav2 from scratch
- https://news.ycombinator.com/item?id=37059479
- https://blog.briankitano.com/llama-from-scratch/

— Kunal