Turning colourers into cooks

Trained on 12,000 children’s drawings, Otsuka’s Sketch Cook brings families together over mealtime by using machine learning to turn their art into recipes they can cook with their parents.

The experiment generated over 500,000 recipes, and $3 million USD of media coverage.

Here’s how it worked:

Client

Otsuka Pharmaceutical

Agency

Dentsu Inc

Machine Learning

Qosmos

App production

Coconoe

Launched

March 2018

Official Sketch Cook Website

First, kids drew what they want to have for dinner.

The Sketch Cook app then recognises the drawing...

... and transforms it into a 'real' picture.

Depending on the nutritional value of their drawing, healthy side dishes are recommended...

... and a recipe is generated.

Then all that's left is to make it!

The app was trained on a dataset of 12,000 drawings of the 30 most popular children's dishes in Japan.

Three machine learning algorithms were used. The first, 'Pix2Pix' transformed the drawing into a realistic image.

The second, 'Super-resolution' increased the detail of the generated image.

And a third image classifier matched it against 3 million recipes in the Rakuten Recipe database to find the closest match.