“Large language models use an internal attention mechanism to focus on the most relevant words in a sentence, helping them understand and generate better responses,” Zhou told the Post on Friday.

Existing attention mechanisms experience diminishing returns on efficiency due to the way they determine what information is most relevant, which necessitates trade-offs between accuracy and computational costs.

These costs compound exponentially as inputs become longer, making it expensive to train sophisticated AI agents, whose practical usefulness lies in their ability to autonomously execute tasks over extended periods.

Instead of having the models compare each new piece of data with all previous ones, the paper proposed using a “gate” to help the models decide what information to discard, thereby improving their training stability and ability to handle long inputs, according to Zhou.

Crucially, the paper validated the innovation with “extensive evidence” through more than 30 experiments across models of different sizes and architectures, meaning that the technique would likely be widely adopted, according to the 14 leading AI experts that made up the conference award committee.

“This paper represents a substantial amount of work that is possible only with access to industrial scale computing resources,” the judges wrote.

“The authors’ sharing of the results of their work … is highly commendable, especially in an environment where there has been a move away from open sharing of scientific results around LLMs.”

“We will continue open-sourcing Qwen models to foster research, drive industry adoption, and make cutting-edge AI accessible to all,” he said.

Alibaba Cloud’s achievement marked the second year in a row that a Chinese team had secured NeurIPS’s top award, following last year’s win by researchers from TikTok-owner ByteDance and Peking University.

Three of the four best papers at this year’s NeurIPS had researchers from China listed as their lead authors, demonstrating the country’s leading role in the global AI research landscape.

“The most impressive thing about Chinese AI researchers is their engineering ability, which is really solid and amazing” said Yao Shunyu, a senior staff research scientist at Google DeepMind. “The industry there is definitely going upwards.”

This year’s best papers will be presented during the main conference this coming week, which will be held in two locations for the first time – San Diego and Mexico City – due to difficulties in securing US visas for international researchers.