gauging the emerging risks of consumer-facing model drift

Stanford and UC Berkely Researchers recently discovered that ChatGPT was getting worse at many tasks, including math and writing working code, but better at visual reasoning and more hesitant about answering tricky questions.  

While poring over their research paper, my wife approached me with a real problem. The Scottish Power app had refused to accept that her new meter reading of 0014340 was clearly more than the last reading of 0014289 from a ‘smart meter’. Had Scottish Power fired up ChatGPT for their new app update or simply botched their data validator? 

In the old English…