Large Language Models

Large language model (LLM)
Reinforcement learning from human feedback (RLHF)
I am going to use some of the stunning graphics from this Deep Learning video course by educator Grant Sanderson. Highly recommended.
- Large Language Models explained briefly.
- Transformers, the tech behind LLMs.

Pre-training for Generating text. Learning probabilities of next word as weights.
From Large Language Models explained briefly video in the above Deep Learning course.

Training set can be random human texts. Vast numbers of them.
Can pass in sentence minus one word. Or page of text minus one word.
Compare machine prediction with correct answer - true last word - and adjust weights.
Texts provide contradictory examples of next word. No single answer. So learn probabilities.
One issue is whether LLM when generating text should always pick the no.1 probability only. Why might that be a bad idea?

Transformers

Transformer model (started the AI explosion in 2017)
Generative pre-trained transformer (GPT)

"Predicting" an entire story from seed text. Predict one word. Feed back in. Predict another word. Feed back in.
I like the way he says it feels like this should not really work, but it can.
From Transformers, the tech behind LLMs video in the above Deep Learning course.

Break input into pieces - tokens. In language, that might be a word / piece of a word.
Each token gets a (large) vector of numbers.
This vector encodes the token's "meaning".
Token vector length ("hidden dimension") can be very large. e.g. In largest version of GPT-3, vector length was 12,288 numbers.
To help the architecture, these vectors are the same length.
Words with similar meanings have similar vectors, like similar locations in n-dimensional space.

From the previous.
Attention mechanism is used for nearby tokens to update the numbers (i.e. the meaning) of the vector of a token.
Here the vector for "model" ends up entirely different in the two situations.

From the previous.
An embedding matrix is a trainable lookup table that converts tokens into high-dimensional numeric vectors.

See long history of Word embedding in NLP.
As the model learns, it embeds words as points in n-dimensional space (maybe thousands of dimensions). The way this happens is that different dimensions take on different meanings.

From the previous.
To find the female equivalent of "uncle", we can look at the vector difference between "man" and "woman", and add it to the vector for "uncle" and then see what word is nearby.
Learning encoded information on sex into one of the dimensions.
But normally dimensions have less clear meaning.

From the previous.
In GPT-3 there are 50,257 tokens (not exactly words) and vector length is 12,288 numbers.
That is a matrix of 617 million weights.

Vector for a word starts with its entry in the embedding matrix. Ends up being modified - for not for all instances of this word. It is for instance of word in this context.
Context size is how wide a context you can give to the tokens. How many faraway tokens can influence it.
GPT 3 context size = 2048. i.e. Data flowing through the network is 2048 columns (not 50,257 columns).
The end matrix has one end column for the next token/word. The end column is combined with the unembedding matrix that maps it to a probability distribution over the tokens/words.
Unembedding matrix has a row for each token. (For GTP 3, 50,257). Columns is the same as the embedding vector size. (For GTP 3, 12,288). So again, 617 million weights.