Deep Learning Specialization - Coursera
1.0.0
1.0.0
  • Introduction
  • Neural Networks and Deep Learning
    • Introduction to Deep Learning
    • Logistic Regression as a Neural Network (Neural Network Basics)
    • Shallow Neural Network
    • Deep Neural Network
  • Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization
    • Practical Aspects of Deep Learning
    • Optimization Algorithms
    • Hyperparameter Tuning, Batch Normalization and Programming Frameworks
  • Structuring Machine Learning Projects
    • Introduction to ML Strategy
    • Setting Up Your Goal
    • Comparing to Human-Level Performance
    • Error Analysis
    • Mismatched Training and Dev/Test Set
    • Learning from Multiple Tasks
    • End-to-End Deep Learning
  • Convolutional Neural Networks
    • Foundations of Convolutional Neural Networks
    • Deep Convolutional Models: Case Studies
      • Classic Networks
      • ResNets
      • Inception
    • Advice for Using CNNs
    • Object Detection
      • Object Localization
      • Landmark Detection
      • Sliding Window Detection
      • The YOLO Algorithm
      • Intersection over Union
      • Non-Max Suppression
      • Anchor Boxes
      • Region Proposals
    • Face Recognition
      • One-Shot Learning
      • Siamese Network
      • Face Recognition as Binary Classification
    • Neural Style Transfer
  • Sequence Models
    • Recurrent Neural Networks
      • RNN Structure
      • Types of RNNs
      • Language Modeling
      • Vanishing Gradient Problem in RNNs
      • Gated Recurrent Units (GRUs)
      • Long Short-Term Memory Network (LSTM)
      • Bidirectional RNNs
    • Natural Language Processing & Word Embeddings
      • Introduction to Word Embeddings
      • Learning Word Embeddings: Word2Vec and GloVe
      • Applications using Word Embeddings
      • De-Biasing Word Embeddings
    • Sequence Models & Attention Mechanisms
      • Sequence to Sequence Architectures
        • Basic Models
        • Beam Search
        • Bleu Score
        • Attention Model
      • Speech Recognition
Powered by GitBook
On this page
  • Word2Vec
  • GloVe

Was this helpful?

  1. Sequence Models
  2. Natural Language Processing & Word Embeddings

Learning Word Embeddings: Word2Vec and GloVe

The basic concept behind learning word embeddings is to use the word vectors of one or more words in a sentence to obtain the word vector for another word in the sentence.

For example,

I want a glass of orange ______.

The word vector of the last word can be learned by passing a vector containing the vectors of one or more previous words, concatenated together, through a neural network and obtaining the output from a softmax layer. It can then be compared to other words in the database and may be similar to the vector for the word juice.

There are some efficient approaches to learning word embeddings, including Word2Vec and GloVe.

Word2Vec

Word2Vec (word to vector) aims to learn a word vector for a given word, such that words that appear in a similar context have similar vectors. It is based on the skip-gram model. Another version of Word2Vec is based on the CBOW (Continuous bag of Words) model.

In the skip-gram model, a single word is used to predict its neighboring words i.e. its context.

In the CBOW model, a certain word is predicted using its neighboring words.

The skip-gram model is slow to train depending on the size of the vocabulary. Therefore, a technique called negative sampling is used.

GloVe

GloVe stands for "Global Vectors for word representation". While Word2Vec is a "predictive" approach, GloVe is a "count-based" approach.

GloVe learns by constructing a co-occurrence matrix that basically counts how many times a word appears in a given context. This matrix is then factorized to obtain word vectors and context vectors.

PreviousIntroduction to Word EmbeddingsNextApplications using Word Embeddings

Last updated 5 years ago

Was this helpful?