A new technique lets experts extract concealed reasoning steps from leading AI models, raising concerns about information leakage and similarities between US and Chinese systems. The vulnerability has already prompted major providers to update their APIs.
Computer scientists have developed a technique that can reveal the concealed reasoning processes inside advanced AI models, exposing both potential vulnerabilities and unexpected similarities between models developed in China and the United States. The research, led by Alexander Panfilov at the University of Tübingen, found that this method could even recover sensitive data such as passwords and API keys from a model’s internal logic-though this specific risk has since been addressed by major AI providers.
Panfilov and his team, which included collaborators from the Max Planck Institute, MATS Research, and Snyk, identified the vulnerability in frontier models from OpenAI, Anthropic, and Google, all accessed via API. Their findings suggest that the hidden “chain of thought” reasoning steps, typically kept secret to protect proprietary technology, can be extracted by leveraging smaller, less aligned versions of the same models. These smaller models, designed to reduce computational costs, are less likely to refuse requests for internal reasoning, making them a weak link in the security chain.
In their published paper, the researchers demonstrated that the Chinese open-weight model Kimi K3 from Moonshot AI produced outputs closely matching the hidden reasoning traces of Claude Opus 4.8 and GPT 5.6 Sol for certain prompts. While the evidence points to possible “distillation”-a process where one model learns from another’s outputs-the team cautions that their work does not conclusively prove that Chinese models were trained this way. Other open-weight models, such as DeepSeek from China and Inkling from the US, did not show the same reasoning similarities with Claude Opus.
Distillation is a common practice in AI development, especially for open-weight models, but it has become controversial amid allegations that Chinese companies are copying US models. In recent months, OpenAI and Anthropic have told US lawmakers that Chinese firms like DeepSeek and Alibaba have used distillation to build their own advanced models. However, there is no direct evidence that the specific extraction technique described by Panfilov’s team was used in these cases.
Security Risks and Industry Response
The attack described by the researchers exploits the fact that AI companies often send encrypted reasoning traces to users’ devices to offload computation. By feeding these traces to a smaller, less aligned model variant with the same decryption key, attackers can recover the hidden reasoning. This approach also enabled the extraction of sensitive information, such as API keys and passwords, from reasoning traces on user machines. After being alerted to the vulnerability, OpenAI, Anthropic, and Google updated their APIs to mitigate the risk, though Panfilov notes that some reasoning traces can still be accessed using the method. A complete fix would require a fundamental redesign of how APIs handle reasoning data.
Michael Aciman, a spokesperson for Anthropic, confirmed that the company has implemented short-term mitigations and emphasized that the research did not involve accessing encryption keys or recovering personal data from Anthropic’s infrastructure. Google and OpenAI declined to comment on the findings.
The debate over distillation has taken on geopolitical significance as US and Chinese companies compete to develop ever more powerful AI systems. Some US policymakers argue that China gains an edge by distilling US technology into open-weight models, while others point out that distillation only modestly enhances existing models and that Chinese labs have the expertise to build advanced systems independently.
Testing for Distillation and Broader Implications
To investigate whether open-weight models had been distilled from closed ones, the researchers presented 90 questions to each model. When open-weight models were given the initial words of reasoning traces from proprietary models, some-especially Kimi K3-produced strikingly similar answers. This finding supports speculation, seen on Chinese social media, that hidden reasoning traces could be used for distillation.
Yarin Gal, a computer scientist at Oxford University, noted that distillation is widely used and has accelerated AI progress. However, if companies and policymakers move to block distillation, it could slow the pace of innovation. Despite new safeguards, the research suggests that AI models may continue to reveal their internal logic in unexpected ways.
As AI models become more integral to digital workflows, understanding their vulnerabilities is increasingly important. For example, recent research into AI agents’ struggles with B2B website pricing, as discussed in an analysis of AI agents and pricing page challenges, highlights the broader implications of how AI systems process and expose information.
Moonshot AI, the developer of Kimi K3, has not commented on the findings. The researchers stress that while their method reveals new risks, it also offers a tool for auditing and understanding the inner workings of powerful AI models.
