文章探讨了创办AI安全组织时，非营利与营利模式之间的选择。作者建议优先考虑非营利模式，尽管它可能存在取悦资助者的激励扭曲，但资助者通常仍致力于改善世界。相比之下，营利模式会因客户更关注自身利益而非社会效益而导致更严重的激励扭曲。营利模式虽然能获得更多资金，但可能导致方向错误。作者强调，无论选择何种模式，都必须清晰界定方向并高效执行，同时指出AI安全研究可能成本高昂，在某些情况下，营利模式可能是唯一选择，但需谨慎设计以避免偏离初衷。

🎯 **模式选择的核心权衡：激励扭曲与资金规模**。文章指出，在创办AI安全组织时，非营利模式和营利模式各有优劣。非营利模式可能面临取悦资助者的激励扭曲，但资助者目标通常与改善世界一致，故扭曲有限。而营利模式则因客户需求的多样化（更关注自身利益而非社会效益）而导致更严重的激励扭曲，可能使组织偏离AI安全这一核心目标。

🚀 **方向与速度：AI安全工作的关键要素**。作者强调，AI安全工作的成效取决于“方向”和“速度”两个关键因素。即使拥有大量资金（速度），但如果方向错误，即所做之事无助于AI安全，其影响力也将大打折扣。尤其在AI安全领域，很容易误以为自己在提供帮助，实则可能适得其反，因此明确正确的方向至关重要。

💰 **资金获取与研究成本**。AI安全研究的成本差异很大，理论性研究可能相对便宜，但实际执行往往非常昂贵，以至于只有少数资助者能够负担。如果尝试申请资助但未获批准，营利模式可能成为唯一的选择。在这种情况下，组织需要花费大量精力确保营利模式不扭曲其AI安全的核心目标，避免成为“摩洛克的奴隶”。

Published on August 15, 2025 1:52 PM GMT

This is one of the most common questions I get when people use my AI safety career advice service. 

My short answer is: yes, you can start a for-profit AI safety org, but you should try non-profit first. 

It mostly comes down between a trade-off between distorted incentives and amounts of funding. 

Non-profits slightly distort your incentives, trying to do things that please funders rather than directly doing what most improves the world. 

But usually the funders also want to improve the world, so it’s mostly just “distorting” it to their judgement of what’s impactful, rather than moving away from impact entirely. 

For-profits massively distort your incentives. 

Your customers usually don’t care about their purchases making the world better. They care about making their lives immediately better. 

Yes, for-profits you have a higher potential upside of raising far more funding. 

However, broadly speaking, there are two important inputs into how much impact you have: 

DirectionSpeed 

Going fast in the wrong direction doesn’t matter. This would be having a lot of money to do things, but the things don’t actually help with AI safety. 

Especially given how relatively easy it is in this field to think you’re helping but actually making things worse, this should be a very strong consideration.

Of course, conversely, knowing the right direction but not moving anywhere is pointless. And doing AI safety research can be cheap (if it’s mostly theoretical work and you can find good people who are willing to work for little), but often it’s very expensive. So expensive that only a handful of funders can realistically fund your work. So if you’ve tried applying to them and they’ve taken a disliking to you for whatever reason, often going for-profit is the only real option left. 

In that case, it can be worth trying for-profit. Just spend a lot of time and energy figuring out ways to make sure that it doesn’t distort your end goal and end up making you a slave to Moloch. 

This is a cross-post from my Substack

Discuss