Deep Learning Specialization - Coursera
main
main
  • Introduction
  • Neural Networks and Deep Learning
    • Introduction to Deep Learning
    • Logistic Regression as a Neural Network (Neural Network Basics)
    • Shallow Neural Network
    • Deep Neural Network
  • Improving Deep Neural Networks: Hyperparameter tuning, Regularization and Optimization
    • Practical Aspects of Deep Learning
    • Optimization Algorithms
    • Hyperparameter Tuning, Batch Normalization and Programming Frameworks
  • Structuring Machine Learning Projects
    • Introduction to ML Strategy
    • Setting Up Your Goal
    • Comparing to Human-Level Performance
    • Error Analysis
    • Mismatched Training and Dev/Test Set
    • Learning from Multiple Tasks
    • End-to-End Deep Learning
  • Convolutional Neural Networks
    • Foundations of Convolutional Neural Networks
    • Deep Convolutional Models: Case Studies
      • Classic Networks
      • ResNets
      • Inception
    • Advice for Using CNNs
    • Object Detection
      • Object Localization
      • Landmark Detection
      • Sliding Window Detection
      • The YOLO Algorithm
      • Intersection over Union
      • Non-Max Suppression
      • Anchor Boxes
      • Region Proposals
    • Face Recognition
      • One-Shot Learning
      • Siamese Network
      • Face Recognition as Binary Classification
    • Neural Style Transfer
  • Sequence Models
    • Recurrent Neural Networks
      • RNN Structure
      • Types of RNNs
      • Language Modeling
      • Vanishing Gradient Problem in RNNs
      • Gated Recurrent Units (GRUs)
      • Long Short-Term Memory Network (LSTM)
      • Bidirectional RNNs
    • Natural Language Processing & Word Embeddings
      • Introduction to Word Embeddings
      • Learning Word Embeddings: Word2Vec and GloVe
      • Applications using Word Embeddings
      • De-Biasing Word Embeddings
    • Sequence Models & Attention Mechanisms
      • Sequence to Sequence Architectures
        • Basic Models
        • Beam Search
        • Bleu Score
        • Attention Model
      • Speech Recognition
Powered by GitBook
On this page

Was this helpful?

  1. Structuring Machine Learning Projects

Setting Up Your Goal

Every ML project has a specific goal to be achieved, and we use evaluation metrics to determine if the goal is satisfactorily achieved.

One thing to keep in mind is to use a single number evaluation metric. This helps us easily and quickly compare the outcomes of each iteration of a certain model or compare the performance of multiple models. For example, instead of using both precision and recall, we could use the f1-score.

Sometimes, we might not be able to use a single metric. In such cases, we could try using optimizing and satisficing metrics. Say we have two metrics: accuracy and runtime. We could choose a model that has the highest accuracy with runtime<=threshold. So, accuracy would be the optimizing metric and runtime would be the satisficing metric. In general, if we have n metrics, we could have 1 optimizing metric and n-1 satisficing metrics.

Another thing to keep in mind is that the dev and test sets must come from the same distribution. Also, they must contain data that is similar to the kind of data that the model is to be run on. This would ensure that the model would perform well once deployed.

If we notice that the metric isn't able to correctly identify the better model, we must promptly change the metric. Also, if we see that the dev/test set is not indicative of the data that the model will be used on, we must change the dev/test set as well to ensure that the model would be able to generalize well.

PreviousIntroduction to ML StrategyNextComparing to Human-Level Performance

Last updated 4 years ago

Was this helpful?