“What AI companies can do today to help with the most important century” by Holden Karnofsky

I’ve been writing about tangible things we can do today to help the most important century go well. Previously, I wrote about helpful messages to spread and how to help via full-time work.

This piece is about what major AI companies can do (and not do) to be helpful. By “major AI companies,” I mean the sorts of AI companies that are advancing the state of the art, and/or could play a major role in how very powerful AI systems end up getting used.¹

This piece could be useful to people who work at those companies, or people who are just curious.

Generally, these are not pie-in-the-sky suggestions - I can name² more than one AI company that has at least made a serious effort at each of the things I discuss below (beyond what it would do if everyone at the company were singularly focused on making a profit).³

I’ll cover:

• Prioritizing alignment research, strong security, and safety standards (all of which I’ve written about previously).
• Avoiding hype and acceleration, which I think could leave us with less time to prepare for key risks.
• Preparing for difficult decisions ahead: setting up governance, employee expectations, investor expectations, etc. so that the company is capable of doing non-profit-maximizing things to help avoid catastrophe in the future.
• Balancing these cautionary measures with conventional/financial success.
• I’ll also list a few things that some AI companies present as important, but which I’m less excited about: censorship of AI models, open-sourcing AI models, raising awareness of AI with governments and the public. I don’t think all these things are necessarily bad, but I think some are, and I’m skeptical that any are crucial for the risks I’ve focused on.

Source:

https://www.cold-takes.com/what-ai-companies-can-do-today-to-help-with-the-most-important-century/

---