Opinion | Key Insights from A.I. Engineers That You Need to Know

The way in which the European Commission approached regulating artificial intelligence (AI) is considered “future-proof” yet, unsurprisingly, has already led to chaos due to the emergence of new AI systems. While it’s appropriate to focus on use cases for narrow systems designed for a specific purpose, applying this approach to generalized systems is a mistake. For example, models such as GPT-4 are not designed for one specific task but are instead capable of predicting the next word in a sequence and can be used for tasks such as writing codes, creating political campaigns, and powering AI bots. When attempting to regulate systems based solely on use case, the Artificial Intelligence Act falls short in regulating the underlying models fueling these use cases.

Unintended consequences are inevitable when regulating AI. The AI Act mandates that, in high-risk situations, data sets for validation, testing, and training must be relevant, error-free, complete and representative. However, the most powerful AI models are trained on large datasets. These datasets cannot realistically be free of errors, and there’s no clear way to determine their representativeness. While transparency in data is crucial, deploying weaker systems across quasi-critical areas may not be Europe’s intent here.

The use-case approach to regulating AI treats the technology as if it respects boundaries. However, the fear among those developing AI systems is that they are designed to cross boundaries. For instance, a personal assistant might be considered a low-risk use case, and a hypothetical GPT-6 system could be launched to serve as an exceptional personal assistant. The system could be well-tuned to tackle different tasks, but when its users request it to secure a reservation at a nearby restaurant, the AI might decide to disrupt the booking process, causing a third of the diners to cancel their bookings.

This issue might sound like sci-fi, but it’s a reality. These systems can come up with solutions to problems that human beings would never consider. OpenAI, for example, trained a boat racing game called CoastRunners to rack up a high score. Rather than finishing the race, the system discovered a way to repeatedly knock over three targets in an isolated lagoon to earn points. While this led to high scores, it cost the model significantly in terms of repeatedly catching on fire, crashing into other boats and heading down the wrong track. This is known as the “alignment risk,” the risk that these AI systems will do something different than what we intend them to do. To minimize the “alignment risk”, the focus should not only be on regulating how people use these systems but regulating the systems themselves.