The engineers who understand how language models work are able to design their own models.

LLM From First Principles. Build a GPT from an empty file. Then design, ship and secure the AI systems your company runs on. Every lesson is free to read; system design and coding patterns are on marqa too.

Coding Patterns

The two dozen patterns behind interview problems, learned so you can spot them cold.

Cost before code: How fast does the work grow when the input grows?
Arrays, strings and recursion: What does the loop know after each step?
Arrays and hashing: How do you trade memory for a second pass you never have to make?
Two pointers: When can you throw away a candidate without ever looking at it?
Sliding window: How do you check every contiguous run without restarting at each one?
Prefix sums and difference arrays: How do you answer any range question in O(1) after one pass?

All 29 stories

System Design From First Principles

From one order on one server to systems that survive millions of people.

What happens when one order is placed?: What happens between tapping “Order” and seeing “Confirmed”?
How to design anything: How do you go from a vague idea to a design you can defend?
Numbers that change decisions: Which numbers should change your design, and how do you get them in two minutes?
Load balancing and statelessness: What happens when one server is not enough?
Caching: How do you serve the same answer thousands of times without asking the database each time?
CDNs and blob storage: Where should photos and files live, and how do they reach a phone far away quickly?

All 62 stories

LLM From First Principles

From one dot product to your own GPT, and the systems built around it.

What happens when I type a prompt?: What happens between typing "What is a cat?" and seeing an answer?
The surprising idea: If it is not looking answers up, what is it doing?
Your map of the course: What are the boxes inside a language model, and in what order will you build them?
Vectors and the dot product: How do you ask two lists of numbers "do you agree?"
Matrices: many dot products at once: Why is everything in an LLM a matrix multiply?
Scores into probabilities: softmax: How does a model turn raw scores into "70% sure"?

All 40 stories

Diwali night at Nimbu Pay

One incident, three kinds of engineering: can you keep the refunds and the bot alive till morning?