Mastering Word Embeddings and Vector Representations
In the previous lesson on Natural Language Processing Basics, we explored how machines process text. However, computers do not understand words, sentences, or paragraphs the way humans do. They only understand numbers. Word embeddings are the bridge that transforms human language into a mathematical format that Large Language Models (LLMs) can process. This lesson explores how we turn words into dense vectors and why this is the foundation of modern AI.
What are Word Embeddings?
Word embeddings are a type of word representation that allows words with similar meanings to have a similar representation in a multi-dimensional space. Instead of treating a word as a simple string of characters, we represent it as a list of numbers (a vector). These numbers capture the semantic meaning, relationships, and context of the word.
Example of a 3-dimensional vector:
"Apple" -> [0.85, -0.23, 0.45]
"Banana" -> [0.79, -0.15, 0.40]
"Car" -> [-0.10, 0.95, -0.30]
In the example above, "Apple" and "Banana" have vectors that are numerically closer to each other than they are to "Car". This proximity tells the model that fruits are more related to each other than they are to vehicles.
The Evolution: From One-Hot Encoding to Dense Vectors
The Problem with One-Hot Encoding
Before modern embeddings, researchers used One-Hot Encoding. If a vocabulary had 10,000 words, each word was represented by a vector of size 10,000 with a single "1" and 9,999 "0"s. This approach had two major flaws:
- Sparsity: Huge vectors filled mostly with zeros are computationally expensive and inefficient.
- No Semantic Meaning: In one-hot encoding, the vector for "Hotel" is just as different from "Motel" as it is from "Bicycle". There is no mathematical relationship between related words.
The Solution: Dense Embeddings
Dense embeddings solve this by using a fixed, smaller number of dimensions (typically 50 to 1024). Every position in the vector is a continuous value (a float), allowing the model to capture subtle nuances in meaning.
How Vectors Capture Meaning: The Diagram
Imagine a 2D plane where the X-axis represents "Gender" and the Y-axis represents "Royalty".
Royalty ^
| [King] [Queen]
|
|
| [Man] [Woman]
+--------------------------->
Gender
In this vector space, the distance and direction from "Man" to "King" is almost identical to the distance and direction from "Woman" to "Queen". This allows for vector arithmetic: Vector("King") - Vector("Man") + Vector("Woman") = Vector("Queen").
Popular Word Embedding Models
- Word2Vec: Developed by Google, it uses two architectures: Continuous Bag of Words (CBOW) and Skip-gram. It learns embeddings by predicting a word based on its neighbors or vice versa.
- GloVe (Global Vectors): Developed by Stanford, it focuses on global word-word co-occurrence statistics from a large corpus.
- FastText: Developed by Facebook, it treats words as a collection of character n-grams, allowing it to handle out-of-vocabulary words and spelling errors effectively.
Practical Java Example: Simulating Vector Similarity
While most LLM training happens in Python, Java developers often use libraries like DeepLearning4j to implement these concepts. Below is a conceptual representation of how we might calculate the similarity between two vectors in Java.
public class VectorSimilarity {
public static double cosineSimilarity(double[] vectorA, double[] vectorB) {
double dotProduct = 0.0;
double normA = 0.0;
double normB = 0.0;
for (int i = 0; i < vectorA.length; i++) {
dotProduct += vectorA[i] * vectorB[i];
normA += Math.pow(vectorA[i], 2);
normB += Math.pow(vectorB[i], 2);
}
return dotProduct / (Math.sqrt(normA) * Math.sqrt(normB));
}
public static void main(String[] args) {
double[] apple = {0.85, -0.23, 0.45};
double[] banana = {0.79, -0.15, 0.40};
double similarity = cosineSimilarity(apple, banana);
System.out.println("Similarity: " + similarity);
}
}
Real-World Use Cases
- Recommendation Systems: If a user likes a product, the system finds other products with similar vector representations.
- Search Engines: Moving beyond keyword matching to "Semantic Search," where the engine understands the intent behind the query.
- Sentiment Analysis: Understanding if a review is positive or negative based on the emotional "direction" of the word vectors.
- Machine Translation: Mapping vectors from one language space to another.
Common Mistakes to Avoid
- Ignoring Context: Traditional embeddings like Word2Vec assign one vector per word. This means "Bank" (river bank) and "Bank" (financial institution) have the same vector. Note: Modern LLMs solve this with "Contextual Embeddings."
- Overfitting on Small Datasets: Training embeddings on a tiny dataset will not capture meaningful relationships. It is usually better to use pre-trained embeddings.
- Assuming High Dimensions are Always Better: Increasing dimensions (e.g., from 300 to 3000) can lead to diminishing returns and increased computational costs.
Interview Notes for Developers
- What is the difference between CBOW and Skip-gram? CBOW predicts the target word from context, while Skip-gram predicts the context from a target word. Skip-gram generally performs better with large datasets and rare words.
- How do you measure distance between vectors? Cosine Similarity is the most common metric because it measures the angle between vectors rather than their magnitude, making it robust to document length.
- What are static vs. dynamic embeddings? Static embeddings (Word2Vec) never change once learned. Dynamic embeddings (BERT/GPT) change based on the surrounding words in a sentence.
Summary
Word embeddings transformed NLP by moving from discrete, meaningless numbers to continuous, semantic vector spaces. By representing words as dense vectors, we allow machines to perform mathematical operations on language, enabling the complex reasoning seen in modern Large Language Models. Understanding vectors is the first step toward mastering Topic 6: Neural Network Architectures for NLP.