Gilesthomas Domain Overview🔒 SSL Verified

Table of Contents

• ↓ Building intuition about LLM parameter counts
• ↓ poppy the training box, part 1: the beginnings
• ↓ Writing an LLM from scratch, part 34b -- from bigrams to GPT-2, one component at a time (in JAX)
• ↓ Writing an LLM from scratch, part 34a -- building a JAX training loop for an LLM training run
• ↓ Thoughts on Role Confusion
• ↓ Flax debugging: making a hash of things
• ↓ Top Alternatives to gilesthomas.com

6-Month Traffic Trend Estimate

Official Contact & Profiles

Visual representation extracted from gilesthomas.com

Content Snippets from gilesthomas.com

"When I was building my GPT-2 implementation in JAX, I started with just token embeddings for the input, and a separate output head (as I was not using weight tying). It wasn't an LLM -- no Transformer blocks, no attention, no feed-forward networks."

"I was somewhat surprised when I noticed that even that stripped-down model had 77 million parameters with the "small" settings I was using to train -- specifically, an embedding dimension of 768. However, I realised I shouldn't be -- with a vocab size of 50,257, each of those components is essentially a 768×50,257 matrix, and that is indeed over 38 million numbers."

"But the finished LLM at the end of the project was only 163 million parameters -- that meant that the input and output components alone were almost half of it. That felt like a surprisingly large percentage."

"I had a similar shock when I was first looking into the feed-forward network, and realised that it had roughly twice as many parameters as the attention layers."

Key Topics & Focus Areas for gilesthomas.com:

Building intuition about LLM parameter counts

poppy the training box, part 1: the beginnings

Writing an LLM from scratch, part 34b -- from bigrams to GPT-2, one component at a time (in JAX)

Writing an LLM from scratch, part 34a -- building a JAX training loop for an LLM training run

Thoughts on Role Confusion

Flax debugging: making a hash of things