加州州长纽森签署了SB 53法案，对大型AI公司提出了新的监管要求。该法案旨在通过强制性披露和报告机制，来评估和缓解AI可能带来的“灾难性风险”。具体而言，大型AI公司需要发布“前沿AI框架”，说明其如何评估和应对潜在的灾难性风险，例如可能导致大规模伤亡或巨额经济损失的事件。同时，他们还需发布“模型卡”，总结模型的风险评估结果，并向加州紧急事务办公室报告内部部署的风险评估和关键安全事件。此外，该法案还包含对内部举报人的保护条款，以及禁止公司就灾难性风险做出虚假或误导性陈述。此举标志着加州在AI安全监管方面迈出了重要一步，但未来仍有进一步完善的空间。

⚖️ **强制性风险披露与评估框架：** SB 53法案要求年收入超过5亿美元且训练模型计算量超过10^26 FLOP的大型AI公司，发布一份“前沿AI框架”。该框架需详细说明公司如何识别、评估和缓解可能导致大规模伤亡（超过50人死亡或重伤）或单次事件造成超过10亿美元损失的“灾难性风险”，涵盖CBRN武器协助、自主犯罪和失控等场景。

📄 **模型卡与透明度报告：** 除了前沿AI框架，这些公司还需发布“模型卡”，作为一种透明度报告。模型卡应总结对模型潜在灾难性风险的评估结果，说明第三方评估的参与程度，以及为实施前沿AI框架所采取的其他措施。这有助于公众和监管机构了解AI模型的风险状况。

🚨 **关键安全事件报告机制：** 法案规定，大型AI公司必须向加州紧急事务办公室（Cal OES）报告其模型出现的“关键安全事件”。这包括灾难性风险的实际发生、模型权重被未经授权地转移或修改、模型失控导致死伤，以及模型使用欺骗性手段规避控制并增加灾难性风险的情况。报告需在事件发生后15天内提交。

🛡️ **强化举报人保护：** SB 53法案特别加强了对内部举报人的保护。如果员工有合理理由相信公司的AI活动对公众健康或安全构成“具体且实质性的危险”，或公司违反了SB 53法案，他们可以向州总检察长、联邦机构或其他相关方披露信息，而不会受到报复。

💰 **违规处罚与未来展望：** 对于不遵守SB 53法案要求的公司，将面临由州总检察长执行的民事处罚，最高可达每项违规100万美元。该法案的实施是加州在AI治理领域的重要尝试，但未来可能受到国会相关立法的潜在影响。

Published on September 29, 2025 11:29 PM GMT

California Governor Gavin Newsom signed SB 53 on September 29. I think it’s a pretty great step, though I certainly hope future legislation builds on it.

I wrote up my understanding of what the text actually does; I welcome any corrections (and certainly further analysis for what this all means for AI safety)!

Very short summary

The law requires major AI companies to:

Publish a frontier AI framework describing how the developer “approaches” assessing and mitigating catastrophic risks, defined as risks of death/serious injury to >50 people or >$1B in damage from a single incident related to CBRN uplift, autonomous crime, or loss of control, and keep the published version of the framework up to date.Publish model cards summarizing the assessments of catastrophic risks from the model, the role of third parties in those assessments, and how the frontier AI framework was implemented.Report to California's Office of Emergency Services 1) assessments of catastrophic risks from internal deployment and 2) critical safety incidents, defined as a materialization of catastrophic risks, unauthorized transfer/modification of the weights, loss-of-control resulting in death/bodily injury, and deceptive behavior that increases catastrophic risks.Allow whistleblowers to disclose information about the frontier developer's activities to the Attorney General, a federal authority, a manager, and certain other employees if they have reasonable cause to believe that those activities pose a "specific and substantial danger to the public health or safety resulting from a catastrophic risk," or that the frontier developer has violated SB 53.Not make “any materially false or misleading statement” about catastrophic risk from its frontier models, its management of catastrophic risk, or its compliance with its frontier AI framework.

Note that violations are punishable by fines up to $1M per violation, as enforced by the California Attorney General, and that the bill would not apply if Congress preempts state AI legislation.

Longer summary

What the bill requires of large frontier developers

“Large frontier developers” are defined as developers of models trained with >10^26 FLOP who also had >$500M in revenue the previous calendar year. They must do the following.

Publish a "frontier AI framework" (no longer "safety and security protocol") that "describes how the large frontier developer approaches" the following:
"incorporating national standards, international standards, and industry-consensus best practices into its frontier AI framework,""defining and assessing thresholds used by the large frontier developer to identify and assess whether a frontier model has capabilities that could pose a catastrophic risk,"
These are defined as a "foreseeable and material risk" that a frontier model will "materially contribute to the death of, or serious injury to, more than 50 people" or more than $1B in damage from a single incident involving a frontier model providing "expert-level assistance" in creating a CBRN weapon; cyberattacks, murder, assault, extortion, or theft with "no meaningful human oversight"; or "evading the control of its frontier developer or user." Explicitly excluded, as of the new amendments, are risks from information that's publicly accessible "in a substantially similar form" without a frontier model and lawful activity of the federal government.This is a somewhat narrower scope than before, which included assistance with cyberattacks and "limited" rather than "no meaningful" human oversight
"applying mitigations to address the potential for catastrophic risks based on the results of [those] assessments,"assessing those mitigations as part of internal or external deployment decisions,third-party assessments,"cybersecurity practices to secure unreleased model weights from unauthorized modification or transfer by internal or external parties,""identifying and responding to critical safety incidents,"
These are defined as including materialization of a catastrophic risk; "unauthorized access to, modification of, or exfiltration of, the model weights of a frontier model" or "loss of control of a frontier model" that (as of the new amendments) results in "death or bodily injury"; and a frontier model using "deceptive techniques against the frontier developer to subvert the controls or monitoring of its frontier developer outside of the context of an evaluation designed to elicit this behavior and in a manner that demonstrates materially increased catastrophic risk."
internal governance practices to implement the above, and"assessing and managing catastrophic risk resulting from the internal use of its frontier models, including risks resulting from a frontier model circumventing oversight mechanisms."
Publish any changes to that framework and justification for such changes within 30 daysPublish model cards, defined as "transparency reports," that include summaries of the following (the "summaries of" is new): assessments of catastrophic risks from the model, the results of those assessments, the extent to which third-party evaluators were involved, and other steps taken to fulfill the frontier AI framework.Transmit to California's Office of Emergency Services "a summary of any assessment of catastrophic risk" resulting from "internal use of its frontier models" "every three months or pursuant to another reasonable schedule specified by the large frontier developer."
Yes, this suddenly makes Cal OES an important governing body for frontier AI.
Report critical safety incidents related to frontier models to Cal OES within 15 days. These will also not be subject to FOIA-type laws. Cal OES will write an annual, anonymized/aggregated report about these incidents to the governor and legislature.
OES can designate a federal incident reporting standard as being sufficient to meet these requirements, and complying with that means you don't have to report to OES.
(TLDR, whistleblower protections) Not make, adopt, enforce, or enter into a rule, regulation, policy, or contract that prevents "covered employees" from disclosing, or retaliates against a covered employee for disclosing, information to the Attorney General, a federal authority, a person with authority over the covered employee, or another covered employee who has authority to investigate, discover, or correct the reported issue, if the covered employee has reasonable cause to believe that the information discloses that the frontier developer’s activities pose a "specific and substantial danger to the public health or safety resulting from a catastrophic risk or that the frontier developer has violated" SB 53, and to create a mechanism for employees to anonymously report such dangers or violations.
The Attorney General will write an annual, anonymized/aggregated report about these disclosures to the governor and legislature.
Not make "any materially false or misleading statement about catastrophic risk from its frontier models or its management of catastrophic risk" or its implementation of the frontier AI framework. (Exception, as of the new amendments, if it was "made in good faith and reasonable under the circumstances.")

Other notes about the bill

All the published stuff can be redacted; they have to justify the redactions on grounds of trade secrets, cybersecurity, public safety, or the national security of the United States or to comply with any federal or state law.Some, but not all, of the above requirements also apply to "frontier developers" who aren't "large frontier developers," i.e. they've trained 10^26 FLOP models but have <$500M/yr in revenue.There are civil penalties for noncompliance, including violating their own published framework, to be enforced by the state AG; these are fines of up to $1M per violation."This act shall not apply to the extent that it strictly conflicts with the terms of a contract between a federal government entity and a frontier developer."The bill could be blocked by going into effect if Congress preempts state AI regulation, which it continues to consider doing.

Discuss