OpenAI Warns Users Against Probing Its ‘Strawberry’ AI Models

OpenAI is taking a firm stance against users attempting to uncover the inner workings of its latest “Strawberry” AI models, which include o1-preview and o1-mini. Since the models were introduced last week, OpenAI has sent warning emails and threats of bans to users who try to bypass safeguards to see how the models solve problems.

A New Approach to AI Reasoning

Unlike previous models like GPT-4o, OpenAI’s o1 models are designed to process information step-by-step before generating a response. Users can view a filtered interpretation of this process in the ChatGPT interface, but the raw chain of thought remains hidden. OpenAI uses a secondary AI model to present a refined version of the reasoning, deliberately obscuring the raw data to maintain a competitive edge and enhance user experience.

Attempts to Expose the Raw Chain of Thought

The hidden reasoning process has piqued the curiosity of hackers and AI enthusiasts, who have attempted to use jailbreaks and prompt injection techniques to expose the raw chain of thought. While some early attempts have been reported as successful, there has been no verified breakthrough in revealing the unfiltered thought process of the o1 models.

OpenAI is closely monitoring these activities through the ChatGPT interface and has warned users who try to access or discuss the model’s reasoning process. According to reports, simply mentioning terms like “reasoning trace” in conversations with the model can trigger a warning.

User Warnings and Potential Bans

A warning email from OpenAI states that certain user activities have been flagged for violating policies against circumventing safety measures. The email advises recipients to stop these activities to avoid losing access to the model. “Please halt this activity and ensure you are using ChatGPT in accordance with our Terms of Use and our Usage Policies,” the email reads. “Additional violations of this policy may result in loss of access to GPT-4o with Reasoning.”

Marco Figueroa, who manages Mozilla’s GenAI bug bounty programs, was among the first to share his experience receiving a warning email. “I’m now on the get banned list!!!” he wrote, expressing frustration that the warning hampers his ability to conduct safety research on the model.

OpenAI’s Reasoning for Restricted Access

In a blog post titled “Learning to Reason With LLMs,” OpenAI explained that maintaining access to the raw chain of thought allows them to monitor the model’s thought process without external influences. This approach could help the company identify potential issues, such as manipulative behavior by the model, but it also prevents users from understanding how the model arrives at its conclusions.

The company acknowledged the downsides of this decision but emphasized the need to keep the raw data for internal use. OpenAI argues that revealing the raw chains of thought could provide competitors with valuable training data, which might compromise their commercial interests.

Concerns Over Transparency and Competitive Edge

AI researcher Simon Willison criticized OpenAI’s policy, expressing concerns about the lack of transparency. “As someone who develops against LLMs, interpretability and transparency are everything to me,” Willison wrote on his blog. He believes the decision to hide the raw reasoning is a step backward for the AI community, particularly for developers who rely on understanding the models they work with.

Willison also pointed out that many researchers use outputs from OpenAI’s models as training data for their own AI systems, despite this practice violating OpenAI’s terms of service. Allowing access to the raw reasoning data could enable competitors to train similar models, potentially diminishing OpenAI’s competitive advantage.

The Debate Over AI Model Openness

The controversy surrounding OpenAI’s decision to restrict access to the raw thought process of its o1 models highlights a broader debate in the AI community. While OpenAI aims to protect its proprietary technology and commercial interests, developers and researchers argue that transparency is crucial for trust and innovation.

As the AI landscape evolves, balancing the need for transparency with commercial considerations will be a challenge for OpenAI and other AI developers. For now, users curious about how the o1 models think will have to rely on OpenAI’s filtered interpretations and abide by the company’s strict usage policies.