California Revises AI Safety Bill SB 1047 Ahead of Final Vote

California’s AI safety bill, SB 1047, designed to prevent potential disasters from large-scale AI systems, has been significantly revised following input from key industry players, including AI firm Anthropic. The bill, which faced strong opposition from various Silicon Valley stakeholders, now incorporates several amendments intended to address these concerns.

On Thursday, SB 1047 successfully passed through California’s Appropriations Committee, moving it closer to becoming law, though not without notable changes. Senator Scott Wiener, the bill's sponsor, confirmed these adjustments in a statement to TechCrunch.

Key Changes to SB 1047

“We accepted a number of very reasonable amendments proposed, and I believe we’ve addressed the core concerns expressed by Anthropic and many others in the industry,” Senator Wiener stated. “These amendments build on significant changes to SB 1047 I made previously to accommodate the unique needs of the open source community, which is an important source of innovation.”

The bill still aims to hold developers accountable for AI models that could cause catastrophic events, such as widespread harm or cybersecurity breaches costing over $500 million. However, the latest revisions reduce the authority of California’s government to enforce these measures proactively.

Reduced Authority for California’s Attorney General

One of the most significant changes is the removal of a provision that allowed California’s attorney general to sue AI companies for negligent safety practices before a disaster occurs. This change, suggested by Anthropic, means that legal action can only be taken after an incident has occurred. The attorney general retains the ability to seek injunctive relief, compelling companies to halt operations deemed dangerous.

Revised Oversight and Certification Requirements

Additionally, the bill no longer mandates the creation of a new agency, the Frontier Model Division (FMD). Instead, the responsibilities initially assigned to the FMD will be handled by an expanded Board of Frontier Models within the existing Government Operations Agency. The board, now enlarged from five to nine members, will continue to set compute thresholds, issue safety guidance, and regulate auditors.

Senator Wiener also softened the requirements for AI labs regarding safety certifications. Labs are no longer required to submit safety test results under penalty of perjury; instead, they will provide public statements outlining their safety practices without facing criminal liability.

Protections for Open-Source Developers

Further amendments introduce more lenient language regarding developers' responsibility to ensure AI models' safety. The bill now requires “reasonable care” rather than “reasonable assurance” that AI models do not pose significant risks of causing a catastrophe.

A specific protection for open-source developers has also been added. Developers who spend less than $10 million fine-tuning a covered model are not considered liable under SB 1047; instead, liability remains with the original developer.

Ongoing Controversy and Next Steps

Despite strong opposition from U.S. congressmen, AI researchers, and major tech companies, SB 1047 has progressed through California’s legislative process with relative ease. The latest amendments are likely to appease some critics, potentially offering Governor Gavin Newsom a less contentious bill to sign.

Although Governor Newsom has not publicly commented on SB 1047, his prior statements reflect a commitment to fostering AI innovation in California. Meanwhile, Anthropic is reviewing the latest changes before taking an official position. Not all of the firm’s suggested amendments were incorporated into the bill.

“The goal of SB 1047 is—and has always been—to advance AI safety, while still allowing for innovation across the ecosystem,” stated Nathan Calvin, senior policy counsel for the Center for AI Safety Action Fund. “The new amendments will support that goal.”

However, the bill's core premise—holding developers accountable for the dangers posed by their AI models—remains controversial. Critics argue that the recent amendments do little to address the fundamental issues with the bill.

“The edits are window dressing,” tweeted Andreessen Horowitz general partner Martin Casado. “They don’t address the real issues or criticisms of the bill.”

Final Vote Approaches

Shortly after the bill passed on Thursday, eight U.S. Congress members from California urged Governor Newsom to veto SB 1047, stating it would harm the state’s startup ecosystem, scientific development, and even efforts to protect against potential AI-related harms.

SB 1047 now heads to California’s Assembly floor for a final vote. If it passes, it will return to the Senate for approval of the amendments before heading to Governor Newsom’s desk for a final decision.