Observations on AI in the enterprise
For at least the past year, my job responsibilities have revolved around AI in some way, shape or form. Both at Deloitte and at my current job at MoFo, I feel like I’ve gotten to see a range of ways AI adoption is happening within large enterprises at the ground level, so I wanted to put down some thoughts on how I see things at this snapshot in time.
AI is changing constantly, so I don’t expect a lot of these details to hold true for very long. But I think the major themes I’ve noticed have been pretty consistent for a while now, and I expect them to stay applicable for at least the short-term future.
I: The range of perceived AI capabilities is super wide, and growing.
This is an expansion of a lot of what Karpathy tweeted recently about the “funnel” of LLM capability and understanding. Aaron Levie has also been tweeting a lot about this as well.
Outside of tech/AI Twitter, a lot of regular people think of AI as a ChatGPT-4 era model that they tried a few times and came away unimpressed. Even in the workplace, a lot of people’s exposure to AI at their jobs is just whatever form of Microsoft CoPilot that their company has half-assedly enabled.
This is a shame because CoPilot sucks! It sucks even when you set the model to Claude Opus since the harness and the integration is so bad. Of course, trying to explain a “harness” to an AI-newbie is already too much jargon, so it can be hard to explain to convince people of the power of AI when all they’ve ever used is CoPilot and it feels shoved down their throats (I really hate that my work laptop has a CoPilot button instead of a left-ctrl key). But CoPilot is just not a good representative of what AI is actually capable of when you put it in the right environment.
An analogy I like to think of for using AI effectively is to think of it like driving a Formula 1 car. A crude breakdown of what makes an F1 car successful is: 1/3 the engine, 1/3 the chassis (the rest of the car), and 1/3 the driver. AI is kind of similar: the engine is like the model, it powers everything. But it’s only effective if it has the right chassis (harness and apps) engineered around it. And of course, the driver (end user) is the one who decides whether they go into the wall or onto the podium.
So chances are, even if you do have direct access to Claude and/or ChatGPT at work, unless you work at a tech company, it’s pretty likely that it’s not being maximized. You might have the engine, but you might not have the right infrastructure around the models or the right savviness to know how to use it. Meanwhile, if you’ve ever substantively used a product like Codex, Legora, or Claude Cowork, the value of using frontier models in a capable harness is pretty obvious.
This gap isn’t necessarily caused by AI-skepticism or cheapness within organizations. IT, InfoSec, procurement, and other related parties within most large organizations are not that well equipped to figure this stuff out (more on this below). There are a lot of valid reasons why these friction points exist.
But this leads most white collar professionals feeling disappointed, frustrated, and confused about what value AI is supposed to provide them. This is certainly a substantial driver of AI skepticism, and it may take a while to disabuse people of these early impressions. As frontier development keeps accelerating faster than the rate of enterprise adoption, I would expect this gap to continue widening.
II: AI models are getting better faster than organizations can adapt their approaches and understanding around them.
A challenge I think enterprises are facing is that trying to consolidate an approach around where AI capabilities are today will mostly not be relevant in 6 months.
Enterprises move slow. I’ve seen this firsthand within Deloitte, the IRS (my primary client at Deloitte), and at MoFo. I’ve been a part of efforts to put together AI trainings that take months to actually deliver after their initial conception. I’ve seen it take several weeks to get new connectors enabled in Claude. I’ve been on teams that have spent weeks trying to try and upgrade a tool to a new open-source AI model; by the time we switched, the next version had already been released.
Maybe at some point model progression will plateau because of exogenous regulation or internal pacing or a lack of compute or a natural slowdown of scaling laws. But trying to anticipate that is like trying to catch a falling knife. For organizations, this applies to efforts like trainings, sales pitches, governance, infosec, wrappers, internal tools, workflows, etc. As Levie wrote, “if you codified your processes 3-6 months ago you’re likely already behind again.”
The lessons and best practices of using AI are constantly shifting. One of the last things I did before I left Deloitte in May was deliver an AI 101 training to other employees, which included stuff like how to effectively prompt AI. Less than 6 months later, that kind of stuff just seems totally irrelevant.
Another example: a coworker recently told me about his friend that teaches an AI course to law school students, where one of the lessons is about showing the difference in AI outputs from a bad prompt and a good prompt. When he recently went to show the example for this semester, the “bad” prompt actually returned a better output than the “good” prompt!
In many ways this is just a further extension of the “bitter lesson.” The frontier models (and harnesses) are just constantly getting better to the point where all this extraneous scaffolding might become entirely obsolete by the time the next version of Sonnet comes out. So if you’re a large organization, do you need an internal AI chatbot, like Deloitte’s Sidekick? Do you need orchestration systems, like LangChain? Do you even need bespoke AI tools, like Harvey or Devin? (Just recently, Claude announced a new set of Artifacts for Legal that seem to be lifted right out of the Legora product suite.)
These are hard things for organizations to anticipate and adapt to! When some companies are still getting up to speed on using Sharepoint or Slack, it’s hard to imagine enterprises writ large adapting quickly enough to the breakneck pace of AI.
III: OpenAI and Anthropic are still not fully mature enterprise technology businesses yet.
What makes things even more complicated is that even Claude and ChatGPT are hard to stay on top of for organizations, and OpenAI and Anthropic don’t always make it easy.
As one of the admins on our firm’s ChatGPT and Claude accounts, I find that managing these products is way more annoying to deal with than you might think. Both have their own growing pains as enterprise companies, but OpenAI is squarely behind Anthropic in terms of enterprise support and delivery.
I think this is well supported through the financials, where Anthropic has been beating them for a while. And I think this is supported narratively by OpenAI’s various diversions, “side quests” and appetite for consumer growth with non-enterprise features like Sora (rip), dots, image generation, and their free ad-supported tier.
Meanwhile, Anthropic has been more focused on acquiring enterprise business, and you can certainly tell when you are responsible for using and administering the platform for thousands of people on a daily basis.
To be more specific, Anthropic is more thoughtful on back-end things like user analytics, APIs for automating control of user spend limits, setting up an admin page that is actually intuitive and easy to use. They provide more documentation and resources on how to best set up Claude for your enterprise. Features like projects and skills are intuitive to use, share, and -- maybe most importantly -- understand (although sometimes it can be hard to keep track of what’s a connector, a plug-in, an add-in, etc.) And they have been better at branding and naming features since the beginning. Now the other AI labs are giving names to their models (Luna, Astra, Argon, etc.), copying what Anthropic has been doing with Haiku/Sonnet/Opus for a few years now.
I cannot say all the same things when it comes to OpenAI. The ChatGPT admin page is a nightmare to use and is constantly changing; the admin APIs are not as robust and have only recently started to catch up to Claude for things like analytics and usage controls. Abrupt transitions like the recent sunsetting of custom GPTs are hard to manage within an enterprise.
Customer support and user experience is something where more niche “wrapper” companies like Harvey or Legora may have some advantage over the labs. These companies can be more receptive to the needs and pain points of specific industries and customers and be more hands-on with their users to help shape the product. At least at MoFo, our reps and customer support from Legora is more robust than either OpenAI or Anthropic.
But we’ll see how durable this advantage is. At the end of the day, capabilities probably matter more than customer service. And this kind of stuff just might not matter that much to the labs: as the Anthropic S1 shows, most of their revenue comes from big whales. Most of the F500 is basically a drop in the bucket for them at this point. Trying to chase the ideal product-market fit might not really matter if you’re about to make a country of geniuses in a data center.
Conclusion
A lot of people have compared AI diffusion to that of electricity. It took several years after the advent of electricity for it to really transform workflows and processes. We had to completely configure factories and production lines; entire new machines and jobs had to be invented to take advantage of this new capability. AI is inevitably going to be similar. We are going to encounter a lot of growing pains as we figure out how to fit this into our jobs, workflows, and organizations; this is not going to just happen automatically. And we are still super early.
I’m partially writing this piece as a response to AI evangelists, labs, and accelerationists: AI take-off is going to hit a lot of real-world friction points in the non-tech world, and a lot of these speed bumps come from legitimate constraints within organizations. If you want AI adoption to go faster, you should be aware of these challenges and empathetic to people’s efforts to deal with them.
I’m also partially writing this piece as a response to AI skeptics: the capabilities of AI that you might have at work today are a far cry from the upper limits of what can actually be done with it. To bring back the car metaphor -- researchers at OpenAI using ChatGPT are like Max Verstappen in a 2023 Red Bull car; your coworker using CoPilot might be more like a teenager in driver-ed in his mom’s minivan. But it won’t be like this forever, and it won’t behoove you to dismiss AI’s capabilities entirely just based on how things are right now.
With all the doom and catastrophe talk lately, it can seem like the future of AI takeover / utopia is already written for us. But at least within the enterprise, I don’t think that’s necessarily the case. The future of AI in corporate America is going to be influenced by billions of individual choices made by millions of people in thousands of different organizations. In that sense, we all have some agency and responsibility in shaping the role of AI in our jobs. As the pope himself has said, the future of AI is not pre-determined; its future depends on the choices we make.