← AI HoodmanNo. 005 · Embeddings

Word2Vec

How did the model learn coffee and tea are similar?

01of 08
Use the arrows, swipe or the arrow keys to turn the slides. Click a slide to read it page by page.
  1. Slide 1 of 8: Nobody labeled coffee and tea as similar. Word embeddings learned from raw text Cities Drinks london paris juice tokyo coffee tea latte Vehicles car truck bus So how did the model learn they're similar?
  2. Slide 2 of 8: The key idea: The text labels itself. I drink hot coffee I drink hot tea Same neighbors coffee every every morning morning Different word tea pulled close together Words with similar neighbors get similar embeddings.
  3. Slide 3 of 8: Training data: Slide a window over the text. Center word Context: up to 2 words on each side Window 1 I drink hot coffee every morning (drink, I) (drink, hot) (drink, coffee) Window 2 I drink hot coffee every morning (hot, I) (hot, drink) (hot, coffee) (hot, every) Window 3 I drink hot coffee every morning (coffee, drink) (coffee, hot) (coffee, every) (coffee, morning) Every word takes a turn as the center. One sentence, many training pairs.
  4. Slide 4 of 8: Two flavors: Skip-gram and CBOW. drink Skip-gram hot Center word predicts each neighbor separately. No averaging. every 4 neighbors → 4 training pairs. morning Better for rare words. coffee CBOW drink hot coffee every Most common Neighbors are averaged into one vector, which predicts the center word. average 4 neighbors → 1 training example. morning Faster to train. Rare words matter? Skip-gram. Need speed? CBOW.
  5. Slide 5 of 8: Skip-gram model: Two tables, one row per word. Center table Context table Used when a word is the center Used when a word is a neighbor tea [0.7, 0.2, 0.5] morning [0.5, 0.0, 0.6] coffee [0.8, 0.1, 0.5] hot [0.9, 0.3, 0.4] car [-0.4, 0.9, 0.0] giraffe [-0.6, 0.7, -0.2] Both tables start as random numbers. Training fills them in. Pair (coffee, hot): dot product of the two rows coffee 0.8 0.1 0.5 hot 0.9 0.3 0.4 Multiply 0.72 0.03 0.20 Add up 0.72 + 0.03 + 0.20 = 0.95 Each training pair from the window is one dot product.
  6. Slide 6 of 8: From score to probability: Squash the score with a sigmoid. sigmoid(score) = sigmoid(0.95) = 1 Any score becomes a number from 0 to 1 1 + e−score 1 1+e −0.95 = 1 1.39 = 0.72 1 Training step: 0.72 (coffee, hot) is real, so the target is 1. The model said 0.72. 0 0.95 score Nudge both rows so the next score is closer to 1. After training, the center table is your word embeddings.
  7. Slide 7 of 8: But the original version is slow: The fix: Negative sampling Original: softmax 100,000+ dot products per pair Compare coffee with every word in the vocabulary. Negative sampling 6 dot products per pair Compare coffee with its real neighbor and 5 random words. Pair Where it came from Target (coffee, hot) the text 1 (coffee, giraffe) random word 0 (coffee, tax) random word 0 (coffee, piano) random word 0 (coffee, cloud) random word 0 (coffee, sock) random word 0 Push real pairs toward 1 and random pairs toward 0.
  8. Slide 8 of 8: Takeaway: Predict a word's neighbors. Similar neighbors, similar embeddings.

Keep reading

Word2Vec · 1 / 8
  1. Slide 1 of 8: Nobody labeled coffee and tea as similar. Word embeddings learned from raw text Cities Drinks london paris juice tokyo coffee tea latte Vehicles car truck bus So how did the model learn they're similar?
  2. Slide 2 of 8: The key idea: The text labels itself. I drink hot coffee I drink hot tea Same neighbors coffee every every morning morning Different word tea pulled close together Words with similar neighbors get similar embeddings.
  3. Slide 3 of 8: Training data: Slide a window over the text. Center word Context: up to 2 words on each side Window 1 I drink hot coffee every morning (drink, I) (drink, hot) (drink, coffee) Window 2 I drink hot coffee every morning (hot, I) (hot, drink) (hot, coffee) (hot, every) Window 3 I drink hot coffee every morning (coffee, drink) (coffee, hot) (coffee, every) (coffee, morning) Every word takes a turn as the center. One sentence, many training pairs.
  4. Slide 4 of 8: Two flavors: Skip-gram and CBOW. drink Skip-gram hot Center word predicts each neighbor separately. No averaging. every 4 neighbors → 4 training pairs. morning Better for rare words. coffee CBOW drink hot coffee every Most common Neighbors are averaged into one vector, which predicts the center word. average 4 neighbors → 1 training example. morning Faster to train. Rare words matter? Skip-gram. Need speed? CBOW.
  5. Slide 5 of 8: Skip-gram model: Two tables, one row per word. Center table Context table Used when a word is the center Used when a word is a neighbor tea [0.7, 0.2, 0.5] morning [0.5, 0.0, 0.6] coffee [0.8, 0.1, 0.5] hot [0.9, 0.3, 0.4] car [-0.4, 0.9, 0.0] giraffe [-0.6, 0.7, -0.2] Both tables start as random numbers. Training fills them in. Pair (coffee, hot): dot product of the two rows coffee 0.8 0.1 0.5 hot 0.9 0.3 0.4 Multiply 0.72 0.03 0.20 Add up 0.72 + 0.03 + 0.20 = 0.95 Each training pair from the window is one dot product.
  6. Slide 6 of 8: From score to probability: Squash the score with a sigmoid. sigmoid(score) = sigmoid(0.95) = 1 Any score becomes a number from 0 to 1 1 + e−score 1 1+e −0.95 = 1 1.39 = 0.72 1 Training step: 0.72 (coffee, hot) is real, so the target is 1. The model said 0.72. 0 0.95 score Nudge both rows so the next score is closer to 1. After training, the center table is your word embeddings.
  7. Slide 7 of 8: But the original version is slow: The fix: Negative sampling Original: softmax 100,000+ dot products per pair Compare coffee with every word in the vocabulary. Negative sampling 6 dot products per pair Compare coffee with its real neighbor and 5 random words. Pair Where it came from Target (coffee, hot) the text 1 (coffee, giraffe) random word 0 (coffee, tax) random word 0 (coffee, piano) random word 0 (coffee, cloud) random word 0 (coffee, sock) random word 0 Push real pairs toward 1 and random pairs toward 0.
  8. Slide 8 of 8: Takeaway: Predict a word's neighbors. Similar neighbors, similar embeddings.