Can artificial intelligence make a tastier chocolate chip cookie recipe than a human being?

At the risk of upsetting millions of grandmothers everywhere, we set out to find an answer.

We recruited Dan Souza, chief content officer for America’s Test Kitchen, for our experiment. He matched the Test Kitchen’s Perfect Chocolate Chip Cookie recipe against recipes from two AI programs, ChatGPT and DishGen.

Why chocolate chip cookies?

“Cookies, when you make small changes to ingredients, we find that you get some pretty massive differences,” Souza told Morning Edition’s A Martinez. So, it’s kind of a fun one to use as a litmus test for how successful a recipe development could be.”

How did the experiment work?

Souza asked each AI program to come up with a chocolate chip cookie recipe. The results were nearly identical. Souza said there’s a good reason for that.

“What it gave me was a pretty traditional chocolate chip cookie recipe. If you look on the back of Toll House [chocolate chip] morsels, which is where most chocolate chip cookie recipes kind of originate, it was a pretty good mimic to that. You have your classic ingredients, you have your flour, you’ve got white sugar and a little bit of brown sugar, a couple of eggs,” Souza said. “What you find with these engines is they’re pulling from all over the place and so you get sort of an average output, and it looked like a really average cookie to me.”

One of these cookies is different from the other. The one on the left was created from a recipe generated by ChatGPT. The cookie on the right came from a recipe app called DishGen.

‎/America’s Test Kitchen

Advertisement

hide caption

toggle caption

‎/America’s Test Kitchen

Advertisement

Souza said both of the programs were fairly straightforward. He typed in prompts of what he was looking for and the results popped up quickly.

DishGen had a “modify” button if, for example, you wanted to change the recipe to make the cookies chewier. But there were some frustrations, as well.

“The craziest thing is I would do the search — I did it multiple times on different days — and I actually got completely different recipes. So, the same prompt but I had a different recipe, which I found like totally infuriating. If I had something that I liked and I wanted to make it again, I couldn’t.”

The taste test

In our view, the cookies from ChatGPT and DishGen were pretty good but a little boring. A variation on the ChatGPT recipe that was intended to make the cookies more chewy actually made them too chewy.

The Test Kitchen’s Perfect Chocolate Chip Cookie recipe lived up to its name. The cookies were crunchy on the edges and chewy in the middle. They had a nutty flavor with a hint of toffee.

Souza said the Test Kitchen recipe has many innovations that the AI engines could not pick up on.

“It’s seemingly simple, but there’s a lot going on there,” Souza said. “One of the things we’ve done is we browned the butter, which does two things. It adds tons of rich, nutty flavor. But liquefying the butter also means that you get a denser cookie, which eats chewier. So, it’s really, really valuable.”

Souza said the AI recipes have a lot of potential, but that the technology is not quite there yet.

“It’s missing the people part of it. So, if you’re tasting a chocolate chip cookie recipe that your mom has made forever, there’s a big emotional pull there. And that actually influences how you taste something. You know, we eat with our eyes, but we also eat with all of our emotions and kind of everything that we bring into the picture. You’re never going to get that from an AI.”

Reena Advani edited the radio version of this story. Obed Manuel edited the digital.