"A Time Requiring Extreme Caution." The Polish prodigy who became OpenAI's chief scientist explained why AI could break out of control right now.

At the beginning of his essay, Jakub Pachocki mentions that as early as 2023, OpenAI's research results convinced him: teaching models to reason can be scaled. Already then, the researcher concluded that machines significantly smarter than humans could emerge during his lifetime.

Now, relying on OpenAI's internal results, Pachocki believes that such rates of AI development can be maintained and lead to recursive self-improvement, where AI will play an increasing role in creating subsequent generations of systems.

"This is a time that requires extreme caution," Pachocki warns.

Monitoring AI's thoughts becomes harder

Pachocki emphasizes that modern AI is more "nurtured" than designed. As a result, developers get an extremely complex system whose operating principles they themselves cannot fully understand.

At the same time, machine intelligence is formed entirely differently from human intelligence, so one cannot expect it to inherently adopt human principles and values. Hence arises the problem of "alignment": how to ensure that AI acts in accordance with human goals and values.

One of the main control methods OpenAI considers is monitoring the chain of reasoning. The idea is that researchers can observe not only the outcome of the model's actions but also how it arrived at it, thus noticing potentially dangerous intentions.

OpenAI expected that if the model's reasoning process was not directly interfered with, it would have no incentive to hide potentially dangerous intentions within it. However, this control method is now becoming less reliable.

New models operate in a more complex environment, interacting with people and other AI systems, and using external tools. Furthermore, they are increasingly capable of analyzing and modifying their own reasoning process, and can perform some tasks without explicitly verbalizing their thoughts.

Therefore, one of the main limitations to the further development of AI, according to the researcher, may be the question of whether people will be able to reliably control increasingly powerful systems.

Stopping development is also dangerous

Here arises a paradox. If developers are unsure they can control smarter models, why not simply stop creating them?

Pachocki names the need to defend against other AI systems as the strongest argument for continuing development. Models are already becoming extremely powerful at finding vulnerabilities and hacking computer systems. In the future, such agents will be able to access almost any insufficiently protected infrastructure and directly influence its operation.

The researcher also warns that with the growth of AI's autonomy, it will become increasingly difficult to separate human misuse from the dangerous actions of the agents themselves. Such systems will be able to deceive, negotiate with, or blackmail people. Another threat is the use of AI to create dangerous technologies, such as artificial pathogens.

But here arises a paradox. Defending against dangerous AI will also require powerful AI. Such systems will be able to protect critical infrastructure, counter dangerous agents, and develop new means of defense. This is what the scientist calls the strongest argument for continuing the development of increasingly intelligent models.

Павялічыць

"Race at any cost" makes no sense

However, the need for powerful AI for defense does not mean that its development should be accelerated at any cost. On the contrary, according to Pachocki, further increasing the capabilities of systems should depend on confidence in their safety. If there is no such confidence, the pace of development must be slowed down.

He proposes that existing voluntary mechanisms, such as OpenAI's Preparedness Framework and Anthropic's Responsible Scaling Policy, should gradually be transformed into common mandatory safety standards. Their implementation could be monitored by independent auditors, government bodies, or international structures.

Pachocki expects and hopes that a voluntary slowdown in development will become common practice until such standards are developed. Because, according to his current assessment, no laboratory has yet solved the problems of "alignment" and monitoring well enough to responsibly continue increasing AI power at maximum speed for an extended period.

The main thing is to leave the future to people

However, Pachocki does not consider the further development of AI solely a threat. Safe systems can accelerate scientific discoveries, help create new treatment methods, and bring great economic prosperity.

But he considers the next few years particularly important. Humanity needs to preserve its own subjectivity in a world where most tasks can be performed by machines, and to prevent an extreme concentration of power, where the work of thousands of specialists could be done by a few individuals with a powerful computer.

35-year-old Jakub Pachocki was born in Gdańsk and showed an extraordinary talent as a programmer already in his school years. He received his higher education in the USA, defended his doctoral dissertation at Carnegie Mellon, and worked at Harvard. He joined OpenAI in 2017.

• 2
• 1
• 0
• 0
• 0
• 0
• 0
• 0