OpenAI, the non-profit research laboratory that develops artificial general intelligence (AGI), is forming a new team to bring "superintelligent" AI under control. The team, called Superalignment, will be led by Ilya Sutskever, OpenAI's chief scientist and one of the company's co-founders.

Sutskever and Jan Leike, a lead on the alignment team at OpenAI, wrote in a blog post that they believe AI with intelligence exceeding that of humans could arrive within the decade. This AI, assuming it does, indeed, arrive eventually, won't necessarily be benevolent, necessitating research into ways to control and restrict it.

"But humans won't be able to reliably supervise AI systems much smarter than us," Sutskever and Leike wrote. "We need to start thinking about how to design AI systems that are aligned with our values, even if they become much smarter than us."

The Superalignment team will focus on four areas of research:

• Value alignment: How to ensure that AI systems share our values, such as a desire to avoid harm and promote well-being.
• Alignment incentives: How to design AI systems that are incentivized to behave in a way that is aligned with our values.
• Safety: How to prevent AI systems from causing harm, even if they are not designed to do so.
• Explainability: How to ensure that we can understand how AI systems make decisions, so that we can hold them accountable if they do something wrong.

The Superalignment team will have access to 20% of the compute that OpenAI has secured to date. The team will also be joined by scientists and engineers from OpenAI's previous alignment division as well as researchers from other orgs across the company.

The formation of the Superalignment team is a significant step forward in the effort to ensure that AI is developed safely and responsibly. However, it is important to note that there is no guarantee that we will be able to control superintelligent AI. The future of AI is uncertain, and it is important to be prepared for the possibility that we may not be able to control it.

Despite the risks, Sutskever and Leike are optimistic about the future of AI. They believe that AI has the potential to solve some of the world's most pressing problems, such as climate change and poverty. However, they also believe that we need to be careful about how we develop AI, so that it does not pose a threat to humanity.

"We believe that the potential benefits of AI are enormous," Sutskever and Leike wrote. "But we also believe that the risks are real. We need to start thinking about these risks now, so that we can ensure that AI is used for good."

The formation of the Superalignment team is a step in the right direction. It is important that we continue to research ways to control superintelligent AI, so that we can reap the benefits of AI without putting ourselves at risk.