Mismatched Training and Dev/Test Set

Say we have 100,000 training images from a web search and 20,000 images from phone cameras. We want our model to be deployed on mobile phones. In such a case, a model trained entirely on images from the web may not generalize to images captured by mobile phones.

Since we have established earlier that the dev and test sets must come from the same distribution, we could use 10,000 mobile camera images along with the other 100,000 training images for training the model and we could use the remaining 10,000 mobile phone images in the dev and test sets (5000 each).

Another solution worth trying out would be to try and obtain/synthesize more training data similar to the data that the model would be deployed on, i.e. more similar to the dev/test set data.