I am pleased (I think) to announce that I have just submitted to ChatGPT an exam, consisting entirely of questions taken from recent final exams in my sophomore-level intermediate economics class, and it has earned a score of zero. Not only did it earn a score of zero, but several of its answers would have merited negative scores if I were allowed to give them. The answers are in every case egregiously wrong, showing absolutely zero understanding of the basics.
I am frankly a little surprised; I had expected it to get at least a few things right.
Edited to add: If you are looking for the details, look here.
Any chance we could see the test and its responses, or do you need to keep them off the Internet so you can reuse them? It would be interesting to compare this to the very different results Bryan Caplan got.
I would recommend trying GPT4. ChatGPT wasn’t able to solve a simple game theory problem (a two-stage extensive form game) I created for it, but GPT4 solved it correctly.
Have you seen this? Bryan Caplan did the same thing with GPT-4 and got different results. https://betonit.substack.com/p/gpt-4-takes-a-new-midterm-and-gets
I wonder if you used GPT-4 or the free version. The former is much smarter.
But then ChatGPT *did* pick the wrong econ textbook! https://gregmankiw.blogspot.com/2023/04/is-chatgpt-wise-or-just-flattering-me.html
As other have said, this is extremely surprising, especially if gpt4 was used. Assuming you were using the free version, I’m happy to volunteer to run the experiment with gpt4 (and to agree to keep the questions secret if that’s your preference).
> Have you seen this? Bryan Caplan did the same thing with GPT-4 and got different results. https://betonit.substack.com/p/gpt-4-takes-a-new-midterm-and-gets
Bryan Caplan’s test was purely a test of guessing the teacher’s password – i.e., regurgitating his political positions back to him. Given that ChatGPT has surely consumed more than enough Bryan Caplan blog posts to synthesize Caplan’s positions (especially given that the questions mention Caplan), it’s not surprising that it was able to flatter Caplan into awarding a grade of A. I see that I’ve read enough Caplan blog posts for my lifetime, so I actually unsubscribed from his Substack.
> I’ve read enough Caplan blog posts for my lifetime, so I actually unsubscribed from his Substack.
Too bad! I find I still have some use for Caplan’s thoughts in my life, even if just to intelligently challenge my views in the cases I disagree with him.