Microsoft CEO Satya Nadella said Sunday that the pursuit of superintelligence — AI technology that could surpass humans across virtually every cognitive task — should focus on "helping humanity" and remain under "human control."

In a post on X, Nadella called for broader AI adoption through what he described as a "frontier ecosystem" where both closed- and open-source AI models can "thrive" across countries, communities and businesses.

"Any pursuit of superintelligence has to be grounded in the core principle that if the AI we build is not helping humanity and under human control, it's not worth pursuing," Nadella wrote. "We also need to accelerate and spread the benefits of AI, such that they are diffused broadly across countries, communities, and companies. This requires a frontier ecosystem in which both closed and open-source models can thrive."

He added that organizations should be able to build AI systems using their own data rather than becoming dependent on a single model provider.

"For firms, it’s imperative that they retain full control over their unique and tacit knowledge," Nadella wrote. "Every organization should be able to build its own continuous learning loop/hill climbing machine, without becoming dependent on any one model provider, and have the ability to embed its own knowledge into models and weights they control."

Nadella called for a deliberate approach to AI development, saying Microsoft supports concepts such as "embedded evaluators."

"So, in this context, we welcome the research, focus, and deliberate pacing needed to get alignment right as the design goal," he said. "We also welcome ideas like ‘embedded evaluators’ and the broader efforts to develop the mechanisms to make this more than just talk."

Nadella also said advanced AI development should not be controlled by a handful of companies.

"The key is that this cannot be controlled by a handful of entities, but must have broad representation across the ecosystem, countries, and fields, including academia," he said. 

"This is the approach we are taking: broad access and choice at every layer of the AI stack; enterprise control of learning loops and models; and the ‘Code of Conduct’ that underlies our own first party MAI models that we’ll publish tomorrow for public consultation."

His comments come as debate intensifies over the rapid pace of AI development and the risks posed by increasingly capable systems.

"I won't lie to you – there are real dangers," Anthropic CEO Dario Amodei said in an interview with CBS News. "And I think for too long the industry lied to people about the fact that this technology had risks."

"Last week, former Anthropic researcher Jacob Coxon warned on social media that Anthropic and OpenAI are "gambling with our lives" by pursuing self-improving superintelligence, adding that AI has a greater than 10% chance of "kill[ing] all humans" within "the next decade."