Clothing recommendation story – Machine Learning Stories

A chain of shoe stores asked you to create a clothing recommendation system based on their own database. They have the pictures of shoes of people buying the shoes (standing at the checkout), and the model of the shoes they bought.

A person standing at the checkout:

Shoes he is wearing:

Shoes he bought:

False False True False False

The company got 400 examples of real data from one shop with the system installed. Installation of the cameras is expensive, and the project is risky, so the company decided to use artificial data they already have.

You’ve been given 400 real examples and 2000 images of THE COMPANY WORKERS shoes and their liking. The company thinks it is possible to recommend the shoe from the current offer, based on such artificial dataset.

What metrics seem to be most sensible for the task?

	F1-score
	Accuracy Accuracy is ok
	AUCROC (Area Under the ROC curve)
	Log-loss
	User defined metric

Correct

Incorrect

Estimating human level performance will let you set up your goals.

	True
	False

Correct

Incorrect

On your order to estimate human level performance the experiment has been made. Salesmen were told to guess what will the customer buy just after he emerged in the shop. The accuracy was 65%!

	You can guess the best possible accuracy (aka 100% - Bayes error) is a little higher than the human level performance.
	You can’t say where the Bayes error isl, even knowing the human level performance
	Bayes error is between 0 and 35%

Correct

Incorrect

The company says you’ll be getting 1000 more artificial examples each week. You are thinking about the validation. As a rule of thumb, quality of the system is estimated better using:

	Hold-out set
	Cross-validation

Correct

Incorrect

In the presented problem, it’s better to use

	Hold-out set
	Cross-validation

Correct

Incorrect

What split should you do?

	In neural networks you make splits more like train/dev/test 98%/1%/1%
	Use random 10% as test, random 10% as dev, and the rest as a train
	Use all the real data as test, 20% of artificial as dev, and the rest as a train
	Use half of the real as test, second half of the real as dev, and artificial data as train set

Correct

Incorrect

After getting more and more data, you collected 400 real examples, and 10000 artificial examples.

	The test and dev set should be extended equally by the artificial data to gain a better proportion between the train and test/dev (like 80/10/10)
	Train data should be extended and dev/test should remain the same

Correct

Incorrect

The chain of stores is happy with your work because there’s a lot of talk about Neural Networks in media. They decided to gather some more real data for you, and now you have 10 000 of artificial examples, 400 real examples you had, and 10 000 real examples from the current offer.

	You may think about the better representation of the distribution in the test and dev sets
	It is impossible to train the system anymore because new data is from a different period and you can’t generalize between periods well
	You should never change the dev/test sets.

Correct

Incorrect

It turns out ShoeMakers has hired one of your competitors to build a system as well. Your system has higher accuracy! However, when ShoeMakers tries out your and your competitor’s systems, they conclude they actually like your competitor’s system better, because even though you have higher overall accuracy, you have more false positives (happening to recommend definitely wrong shoe). What should you do?

	Rethink the appropriate metric and tune the model to the new one
	Tune the model to minimize false positives
	Choose the model that had lowest false positives number
	From now start to minimize both logloss and false positives

Correct

Incorrect

You’ve handily beaten your competitor, and your system is now deployed, and the performance of your system slowly degrades because your data is being tested on a new type of data.

	The very new project should be launched, to deal with a new, different data
	New type new shoes can be added to your training set
	New type new shoes can be added to your dev/test sets
	New type new shoes should be added to your training set
	New type new shoes should be added to your dev/test sets

Correct

Incorrect

ShoeMakers are very happy with your work. They think you could use your system to recommend the hats for the befriended company.

	True
	False

Correct

Incorrect

Leave a Comment Cancel reply