I had an experience this week which forcefully reminded me that ChatGPT and Google’s Gemini were great but not perfect. And to be clear, I have jumped into the AI pool with both feet and am enthusiastic about the long-term prospects. However, I believe that we need to tap the brakes on the irrational exuberance and belief that AI can do everything, everywhere all at once.
The specific project that broke ChatGPT’s back was obscure on the one hand but should not really have been that tough. My daughter is finishing her doctoral dissertation and was trying to generate a map that compared the borders of the Byzantine Empire in the years 379 AD versus 457 AD.
Here is the prompt that I used against deep research:
Create a detailed map that overlays the borders of the Byzantine empire in 379AD at the start of the reign of Theodosius the Great versus the borders in 457AD at the end of the reign of Marcian. I need both borders shown clearly on a single map.
Use a historical map style and highlight major cities.
The Deep Research option is powerful but often time-consuming. As it runs, I enjoy watching the play-by-play in the details window. ChatGPT did an excellent job of generating a text analysis of the changing borders, major cities, and historical events.
The wheels fell off the bus when I asked ChatGPT to turn its text analysis into an easy-to-read map.
Without digging too deeply into the minutiae of the fifth century world, the point is that it made up names, misspelled names and placed cities at random. Notice that Rome appears twice on the Italian peninsula. What is particularly frustrating about this effort is that the names and locations were correct in the text.
I tried patiently asking for spelling corrections and proper placements of well-known cities without success. Finally, I told ChatGPT that its results were garbage and threw up my hands. To its credit, ChatGPT took the criticism in stride. It replied “Thank you for your candor. You are right to expect better “. Unfortunately, things did not get better.
After a few minutes of cursing out that platform I decided to give Google Gemini a shot at the identical query. Shockingly its results were even worse. If you look at the image below, you will see “Rome” in the middle of the Iberian Peninsula. Antioch appears three or four times across Europe, but many of the other names are right out of fantasy novels.
I was complaining about this mapping chaos to a friend. He shared a similar story. He entered a photo from a small offsite meeting into ChatGPT. He asked it to add the words “Mahalo from Hawaii 2025” above a photo of a group of colleagues. Instead of just adding the text, the engine totally changed the image. It made people skinnier; it changed men into women and an Asian into a Caucasian. Another friend told me that an AI generated biography of him talked about his twin children which he does not have. It even provided a link to a non-existent source. Yikes.
Ronald Reagan used to say: Trust but verify.
My point is not to suggest that we run away from AI and cancel all our subscriptions. Rather, it is to remind everyone (me included) that we cannot hand the keys to the AI engines and walk away. They are tools that can assist us but, in the end, we need to look at the output, see if it looks and smells right, and decide whether to accept it or not. It is clear that the performance of AI engines is uneven; excellent at some projects and terrible at others–such as mapping.
We will probably see the rise of the machines someday–but today is not the day.
It was just last month that OpenAI boss Sam Altman claimed that Meta had been trying to poach his to...
Read More →
If you’ve had enough of AI-generated images filling up your search results, then the DuckDuckGo se...
Read More →
The Mac apps community is a wonderful place to find utilities that can supercharge your computing ex...
Read More →
As AI tools improve, we keep getting encouraged to offload more and more complex tasks to them. LLMs...
Read More →
Google is steadily rolling out contextual improvements to Gemini that make it easier for users to de...
Read More →
Amazon is taking some inspiration from the TV shows available on its Prime Video streaming platform,...
Read More →
MidjourneyWhen it comes to AI image generators, you’ve got your choice from dozens these days. Two...
Read More →
OpenAIFollowing its U.S. debut in January, OpenAI’s Operator AI agent will soon be expanding to ...
Read More →
AI has been a part of the Google Shopping experience for a while now. In October last year, Google s...
Read More →
Comments on "That moment when I told ChatGPT it needed a history lesson, and it agreed with me" :