The Revenge of the Middle: Managers, Memory, and the Real AI Bottleneck

Val: There's literally not enough rare earth minerals in the world, and not enough manufacturing fabs, to produce the memory required today. This memory wall is us hitting a wall as an industry — not being able to satisfy the overwhelming demand for tokens, and needing a much more clever approach.

Serena: The biggest constraint in AI right now isn't compute, it's memory — the ability to move data fast enough to keep GPUs fed, to handle the massive context windows that make AI useful, and to serve inference at scale that enterprises actually need. We're in the middle of a memory shortage that's reshaping the entire AI infrastructure landscape. The organizations that solve for it can deploy bigger models, serve more users, and iterate faster. The ones that don't will hit a wall, no matter how good their models are.

My guest today is Val Bercovici. He's spent his entire career defining how the industry stores and moves data — from writing the standards others build on, to leading infrastructure at scale. Now he's at WEKA, and there's no one better to break down what this memory shortage actually means. Val, welcome to Deep Geeks.

Val: Great to be on again, Serena.

Serena: I just came back from a conference where every company was showing off a dashboard of token usage — by team, by function, even as a leaderboard. Everyone was so proud. And I kept thinking: when we all get the bill next quarter, we're not going to be nearly as happy. Then I got back and heard that Uber's COO announced they weren't getting the ROI they'd hoped for from AI — it's getting so expensive that token usage isn't translating into features for customers. Val, are you surprised by any of this?

Val: Not at all. If anything, I'm surprised it took this long. My mind goes back to the early cloud days, when a handful of developers and a few executives close to them were super excited about what they could do — the agility of building and deploying in the cloud was exhilarating. Then, a few months later, the bills came in, the sticker shock hit, and this whole category of FinOps — financially engineering applications so they don't break the bank — became a real discipline not long after the rise of cloud.

We're seeing the same thing play out, probably faster now. I expected the sticker shock, but I was surprised by the token-maxing trend we saw this spring, where everyone was almost inflating token numbers just to game leaderboards. Everybody wants to be at the top of a leaderboard, but we knew that couldn't last.

Serena: A lot of companies were surprised — both because they weren't giving the right incentives to users, and because the cost of AI is surprising in a very unpleasant way. I'd love to dig into the economics of AI — tokenomics. Is that a word, Val?

Val: Absolutely. Some people in the crypto world have their own definition, but like many things, the AI industry has co-opted it to mean its own thing.

Serena: I think people are stuck in an old narrative — cost goes down, demand goes up, Econ 101, and eventually cost normalizes. I don't think that's actually happening. What's driving the cost, and what's underneath it?

Val: It's easy to look at only one side of the coin. You could correctly claim the unit cost of inference is plunging — the actual per-token cost. DeepSeek recently made headlines, not just with the V4 model announcement — V4 Pro and Flash — but with shockingly low pricing, particularly for cache-read tokens. So you could argue the price of inference is plunging.

But the other side of that same coin is Jevons Paradox: every time a valuable commodity drops in price, consumption dwarfs that reduction. The math is roughly — for every 100x reduction in price, we see something like a 10,000x increase in consumption. So the net effect is that the final cost of inference ends up almost 100x more than people projected, which is why we're seeing annualized bills of $5,000, $10,000, even $15,000 a day for people who token-maxed. At $5 million annualized cost per user, the math just doesn't work.

Serena: Wait — $15,000, not per month, per day, just on AI?

Val: The biggest token-maxing boasters were at that figure, because, hey, it was a great figure to brag about.

Serena: That's going to be shocking to a lot of CFOs who think about budget on a longer-term basis. Presentations have been saying unit cost is falling, but no one prepared for a forecast of 100x, even 1,000x usage. That bill is going to surprise people in a year no one is prepared for. The other uncomfortable part of this is that a lot of companies are making workforce decisions based on the assumption that AI is cheaper than humans — and we're not really seeing that, are we?

Val: No. In fact, one of the most popular new commands across agents — Claude Code, OpenClaw, Hermes — is the /goal command. You specify an ambitious project, and the agent goes off: it swarms, spawns sub-agents, and runs not for seconds or minutes like a chat, but for hours or even days — iterating, testing, improving, and ultimately delivering a highly functional app with built-in security, enterprise authentication, auditing, everything you'd expect. But the token cost for that project can run into the tens of thousands of dollars. What people are realizing is that token consumption isn't as efficient as humans in many cases, and it costs more than humans when applied incorrectly.

Serena: That's a good reminder — we weren't prepared for that. Our brains like to simplify things because ambiguity is hard, so there's a narrative that there's "one AI" with one price. But you mentioned there are actually different tiers of AI cost — most people using AI day to day might be aware of it, but leadership often isn't. Can you walk us through the tiers?

Val: That phenomenon actually has a name now — "AI psychosis." The further you are from hands-on use of these tools, the simpler your mental model tends to be. One trap leaders fall into, especially if they're not hands-on, is applying "chat mentality" to what agents actually are. They're fundamentally different in almost every way. We think of chat as a single conversation with a model — if we don't like it, we start a new chat with a better model, or a cheaper one if the price feels too high.

Agents don't work that way. They don't use just one model, and they're not a single prompt and response — they run multiple turns, dozens or hundreds of prompt-and-response cycles, often with huge amounts of context: not just one PDF, but entire libraries or a corpus of material, entire codebases, entire Wikipedia-sized knowledge bases. That's a very different experience, which means we need segmentation. As any market matures, it segments. At minimum, we're seeing three common tiers emerge, and Anthropic has done a good job branding them: Opus as the high-end, premium model with premium pricing; Sonnet as more of a mid-tier, higher-volume model at slightly lower quality; and Haiku as the bargain tier — really high-volume, bulk tokens, where you're not expecting deep reasoning, just execution of explicit tasks. Developers use a combination of all three, either for latency reasons or because they have an unlimited budget. But with a real budget in hand, we're mixing and matching models, because right tool for the right job applies in AI too.

Serena: I love that — right tool for the right job, and "right tool" means the right tier. Have you seen cases of people using the wrong tier — where the job doesn't need an agent at all, but one got created anyway?

Val: One of the genuinely positive impacts of OpenClaw coming onto the scene in December and going viral is that it made the technology far more accessible — not just to developers who love hacking on it, but to tinkerers and hobbyists who do knowledge work nine-to-five. People quickly did the math, realized they were blowing through their personal token budgets at home, and told their agents in plain English: "Reduce my costs." The agents figured out how — using more than one model, using simpler models for simpler work. Some of these agent frameworks even run scheduled tasks — what developers call a cron job — like checking the weather or a stock price. The OpenClaw hackers quickly realized, "I don't need a model to run this cron job — I can just run the cron job," instead of having an agent call a tool and add an artificial layer of latency and cost.

What I love is that these agents went out, researched best practices, and developed their own internal policies: "Here's a complex task, I'll use a premium model. Here's a lower-complexity task, I'll use a budget model. Here's simple execution, I'll use the bargain-bin model." That's a hierarchy the agents learned and are deploying autonomously.

Serena: I love that, it's so smart. There was also a recent headline about Microsoft moving away from Claude Code because of cost, pushing people back to GitHub Copilot. Do you think that's a tier decision, or just overall cost-cutting?

Val: That's an important headline — I noticed it too. I'll speculate there are at least two things going on. There's an optics problem when a company's own developers are using a competitor's product instead of their own — you need to be dogfooding your own product for lots of reasons, so that likely contributed. But I also think it's representative of what I'm seeing at other startups, software companies, and dev-heavy enterprises: token costs are really coming home to roost. We like to say people are moving from token-maxing to signal-maxing — trying to extract more ROI, more signal, from the noise of all those tokens. It's now clear that beyond Anthropic and OpenAI, the open-weight models — resurgent U.S. open source models, and especially the Chinese models — are genuinely excellent. We're seeing a natural hierarchy emerge: maybe only a few frontier models are true premium tier, but instead of paying a major lab for their mid-tier model, you can go to DeepSeek, Kimi, GLM, or Minimax and get roughly equivalent mid-tier functionality at about 10% of the price — which lets your agents think and validate more.

Serena: I remember being impressed by DeepSeek two versions ago, and it's just incredible now.

Val: They keep delivering. They went dark for about 484 days — a dangerous thing to do, because their Chinese competitors were shipping almost every week, leapfrogging each other, which was breathtaking to watch. Then DeepSeek came back over a year later with a genuinely game-changing release. What I love about DeepSeek — not to be too much of a fanboy — is that the industry respects them because they publish so much. It's not just about shipping a model, it's the white papers. They've said it's no longer that important to compete at the top of the benchmark leaderboards, because the models are converging in some sense — benchmarks force everyone to train toward the same target. So the real competition is on cost, and DeepSeek innovated there: they introduced scalable, workable compression that cut memory consumption from the KV cache by 90%, where most competitors couldn't compress that much without sacrificing quality. That's the magic of DeepSeek — lower prices because they use less memory without sacrificing quality.

Serena: I'm a fan too. Let me pause and give our audience a quick summary. Before you sign or renew an AI contract: ask which tier your workflows actually require. If you're paying premium prices for basic work, that's expensive, and you have a procurement problem on your hands.

Val: Right, and it goes back to first principles. Most procurement teams are used to negotiating with SaaS vendors on a per-seat basis. What we're realizing — and it's messy for the industry and confusing for customers right now — is that there's no such thing as per-seat production AI pricing. There might be some per-seat entry floor, but everyone is moving toward consumption-based pricing. Thinking of software as a utility is very new for procurement teams, because it's currently somewhere between hard and impossible to budget for. The prediction and measurement tools will improve over time, but we're not there yet as an industry. So you have to do real due diligence before signing these AI contracts, because in some sense you're writing a blank check.

Serena: That will keep any CFO up at night — a blank check on AI. Let's talk about token maxing. I started with those usage dashboards, and I'm also seeing organizations incentivize AI usage from the people side — "use AI more, show us the numbers." I get it — adoption is hard, especially for large organizations, and boards are pushing for proof that AI is making a difference. But what's wrong with token maxing, and what would you recommend instead?

Val: I like to think about these trends as a bell curve. At the front are people who've gone through the early adoption pains and come out the other side with true ROI — they're signal-maxing. But most of the industry, the meat of the bell curve across the Global 2000, is still in the "how do I apply AI at work, and where's the business value" phase. There's no question about individual productivity benefits — it's hard to find an internal slide deck, spreadsheet, or dashboard today that isn't AI-generated. It's group productivity that people are still struggling with.

A lot of executives, especially a couple of quarters ago on earnings calls, felt pressure to say their company wasn't behind the trend and was AI-savvy — investors wanted to hear it. So leadership has been encouraging, even incentivizing, aggressive AI use — in some cases literally giving bonuses to whoever topped the usage leaderboards. That drove a lot of internal recognition, promotions, and clout — and, as you said, unaligned behavior, because the incentives were wrong. It's not about how much gas you burn, it's about the mileage you're getting.

Serena: Yes — I love that. Incentives really matter. So what should leadership incentivize instead, in terms of signal maxing? What actually is "signal"?

Val: There's a term engineers have used for a while: "goodput" — output that's useful, as opposed to output that just keeps the system busy. Even in AI training, we used to measure how busy we could keep the GPUs, because idle GPUs are expensive waste. So people ran a lot of inefficient training just to keep utilization high — which kept GPUs busy and made you feel like you were getting your money's worth, but didn't actually ship model checkpoints any faster. Efficient training used to be measured by how quickly you could checkpoint, how short your epoch times were. Today it's really about tokens.

When you're signal-maxing, you're getting verified, useful output — an accurate spreadsheet, a relatively bug-free application, a dashboard that works and has a good user experience — without burning a billion tokens for it, which has absolutely happened. You might use a million, maybe a hundred million tokens instead of a billion. It's not just about using the most expensive model for everything — some models are better at planning, others at delegating, others at doing delegated work, others at verifying accuracy because there's a clear success criterion. Engineering these together is what produces a great signal-to-noise ratio — good signal-maxing and less token-maxing over time.

Serena: I like that — signal per token is a much better measure, and it matters for anyone setting goals for their team next year: reinforcing that AI is important without incentivizing the wrong behaviors. I really like the business metrics you mentioned — not just useful output, but impact on the eventual customer. You built a dashboard — great, but what's the user experience? Do people actually use it? Does it lead to better decisions? That's the KPI successful businesses have always measured — not hours worked, or in this case, tokens burned.

Val: Exactly. You can't manage what you can't measure.

Serena: That's right. Now let's shift to memory, and give our listeners a feel for what's really driving the cost. A lot of people have heard of GPUs but will never touch one, and the cost is confusing — in the SaaS world, per-seat pricing tends to go down. Why is it going up here? Let's set this up for a VP of AI or a CFO negotiating an AI contract — how should they think about the bill, and where does memory come in?

Val: The stock market is actually helping educate a lot of leaders here. Look at the stock prices of memory and flash storage suppliers — they've become meme stocks over the last three to six months. That's a direct reflection of the AI industry shifting from a pure focus on training to monetizing the models we've spent billions training. Inference and training are fundamentally different workload profiles — you'd almost build separate data centers for each. We're seeing new vendors emerge around this: Nvidia's high-profile acquisition of Groq — with a Q — an inference-only accelerator, not even a GPU anymore but an LPU, a language or linear processing unit. The Cerebras IPO was recently oversubscribed because they're focused on inference, not training.

People are realizing inference is dominating the infrastructure discussion, and inference is far more memory-centric than compute-centric. Rough math: 100,000 tokens — a pretty typical prompt for an agent today, about a megabyte of actual data — used to translate into 30 to 50 gigabytes of actual, expensive, high-bandwidth memory — some of the most precious real estate in the world by silicon square millimeter. That's not tenable, which is one of the innovations DeepSeek and others brought to market: shrinking that memory amplification from 30-50 gigabytes down to three to five gigabytes per unit. But as we discussed with Jevons Paradox, when you shrink the memory footprint per unit, the number of units explodes — 100x, maybe 10,000x, netting out to roughly 100x. So we're memory-bound, and there simply isn't enough memory. When you look at the supply chain, analysts will tell you there aren't enough rare earth minerals in the world, and not enough fabrication capacity, to produce the memory required today.

This memory wall is us hitting a wall as an industry — unable to satisfy the overwhelming demand for tokens, and needing a much cleverer approach. DeepSeek addressed one dimension by optimizing KV cache size. One of the things I've been working on at WEKA — the personal reason I joined — is bringing abundance to this environment: taking the cost basis of storage but making it behave like memory — much faster, much lower latency — which adds abundance to an otherwise scarce infrastructure. That combination — better per-unit efficiency from approaches like DeepSeek's, and abundant memory infrastructure from WEKA — is what the industry needs to scale the memory wall.

Serena: That might be the best use of the word "abundance" alongside "scarcity" I've heard in a long time — I may have to turn that into a hashtag. I want to give our listeners a picture they can take home. What I'm hearing is that the GPU is like the stove in a kitchen — you can have a great stove, top of the line — but memory is more like the prep station next to it. You can have the best stove cooking amazing food quickly, but if your prep station can't keep up, you still can't produce as many dishes as you'd like. How does that land with you?

Val: If you're a fan of the show The Bear, it lands perfectly. Whether it's a kitchen reality show or a movie about restaurants, you see the bottlenecks plainly: very expensive equipment sitting idle 99% of the time because the system around it isn't efficient — which is a waste, and it also produces very hangry, unhappy customers.

Serena: Yes, and we don't want hangry customers. And I think utilization is another metric companies need to think about — if your GPU cluster is running at only 30% utilization, you're still paying for the full bill, right, Val?

Val: Oh yeah, the bill always comes home.

Serena: If I were to summarize the memory constraint a lot of companies and leaders aren't thinking about: costs aren't falling the way the presentations promised. It's a bit like your cell phone bill — remember when texts used to cost money per text? Now you pay one bill, you can send images and videos, the cost per text is almost zero, but your bill didn't go down — it went up, because you're using it so much more. It's the same for AI. Whichever analogy you prefer — the phone bill or the kitchen — it's worth keeping in mind for long-term budget planning, and for not making workforce decisions too quickly based on assumed AI costs, because you could be in for an unpleasant surprise.

Val: The kitchen is a perfect analogy, and so is the phone bill — I remember those thick monthly envelopes in the early mobile era, everything itemized, and the bottom line always higher than expected. We're in that exact same era for AI procurement now: very complex, utility-oriented, high-consumption bills, where the bottom line isn't what we expect. We have to go through this maturation of AI FinOps to understand, measure, and eventually engineer these bills down — we're not there yet, and it needs focus.

The other thing I'd flag is that a lot of technology leaders carry a legacy cloud mentality into AI — never thought I'd put those two words together — where you provision a big CPU instance and optionally add memory and storage. That's not how AI is built, and not how it works. Performance requirements are so high and latency tolerances so low that AI has to package GPUs and GPU memory — high-bandwidth memory — together in the same unit; you can only provision them as a pair. Which means if you're training versus inferring, you almost want different hardware: one kind of GPU for training, with a certain memory profile, and much more memory with less GPU for inference — and you can't buy and deploy systems that way yet, but that's coming. That's where WEKA's abundance comes in — we're among the first to disaggregate compute from memory for inference, which is a better mental model match for technical leaders from the cloud era. I think we'll see this play out more in 2026, and we'll finally get back to that familiar cloud-era concept, even for AI data centers.

Serena: There are companies starting to prepare for this, and others who will be very unpleasantly surprised. Val, you've written about this era of artificially low AI prices, where inference providers are essentially subsidizing costs to capture market share and drive adoption. What happens when those subsidies end? What does AI pricing actually look like in 12 to 18 months?

Val: This is where the Uber analogy applies — buying market share and getting consumers used to a certain way of operating was the VC-subsidized, immediate post-ChatGPT era of AI. What we're seeing now isn't even primarily in pricing — it's in token rate limits. There simply aren't enough raw materials to satisfy token demand, so surge token pricing became a real thing late last year and this year, reflecting the true market cost of token consumption rather than that low advertised unit cost. We're seeing providers — famously Anthropic — impose strict rate limits even on their best, highest-paying customers, because there isn't enough token supply to meet demand. Rate limits, even before huge bills, were the first pain point people hit. And I think we'll see that reflected in higher pricing to avoid hitting rate limits — you're not going to casually reach for something like Opus fast mode, which has an astronomical multi-hundred-dollar-per-million-token price point we've never seen before. As new, higher-quality model tiers appear, they'll set new pricing ceilings, much like when Uber had to go public and investors demanded real returns — we started paying real market pricing as rideshare consumers, because we like the convenience. The same thing is coming for AI: can we actually afford it? Do we rely on one model for everything, or do we have to financially engineer a mix of models?

Serena: So if the subsidies end, does the "AI is cheaper than people" thesis not survive?

Val: Not in its simplest form — the idea of one model handling real work, in an agent swarm with multiple concurrent agents, long context, and multiple turns, is already proven unsustainable and unaffordable. Even benchmarks like ARC-AGI are starting to factor in the actual number of tokens and cost per solution, not just per-token cost. The industry is maturing and segmenting — and interestingly, it's not just human engineers designing that segmentation, agents themselves have enough autonomy to build multi-model integrations on their own, just from natural-language policy. It's fun to watch this evolution, but it's becoming a mature, intricate, and complex industry for the uninitiated.

Serena: We're seeing companies learn this the hard way — some laid off a large share of their workforce assuming agents could do it, then had to walk it back publicly. There's a capability gap for sure, but there's also a cost realization that doesn't always make the headlines — that AI costs surprised people enough to change their workforce strategy.

Val: That's so important right now. There are almost two realities. One is a zero-sum mentality: expecting to cut costs by replacing humans with AI — which either hasn't worked because quality isn't there yet, or it has worked but it's really just swapping one cost for another with no net benefit. The more interesting conversation — and not enough companies are having it yet, though the leaders here will create followers — is a positive-sum mentality: AI lets you fundamentally rethink your business, rethink the tasks your staff performs. The jobs are still there — get this business function done, deliver this unit of value to a customer, process this supply efficiently — but the tasks change. The most progressive companies are increasing revenue and productivity, not laying off, because they're realizing the tasks have changed, not the jobs.

Serena: I've been running small focus groups with AI users, and my favorite question is: what are you able to do with AI that you absolutely could not do before — not faster, but literally couldn't do at all? The answers are fascinating — people talk about reshaping a job they now like a lot better, not just doing the same things faster.

Val: My favorite example is dashboards. There's always information you want aggregated and synthesized that used to sit in a frustrating in-between zone — useful, but not worth hiring a developer for, explaining requirements, iterating for weeks or months, and then finally getting an expensive dashboard. Now that can happen in an hour. When you compound that productivity benefit, it's significant. There's a darker example too: what happened with Mythos a month ago, when Anthropic released Project Glasswing — this new class of powerful model was able to creatively chain together attack sequences that were individually defensible before but collectively break a lot of software. Thousands of new vulnerabilities were discovered in mature, hardened software in the past month alone. That's a reminder to reimagine both the positive and negative — the tool use and the weapon use — of this technology, because it's possible, and it will be exploited for benefit or for loss.

Serena: For good or for evil.

Val: Exactly.

Serena: I felt that dashboard impact myself. A while back, building a dashboard from scratch used to take my whole analytics team multiple sprints — front-end developers, designers, QA testers, the whole team. I did it in under an hour. I was skeptical — I thought the filters would break, that it couldn't export anything — but it was one prompt, and everything worked, and it was actually quite beautiful. Those moments of genuine awe are where a lot of innovation comes from — and at the same time, it's a little scary for people worried about what it means for their job. Do you have encouraging words for someone who sees the power of AI but worries about their future?

Val: I don't want to be one-sided, but it's been genuinely positive at WEKA. As Chief AI Officer I have two roles — inbound and outbound. Outbound is evangelizing the benefits of AI and shaping product strategy. Inbound, a year ago, was mostly convincing people to even try these tools. Now that developers and knowledge workers have come around, our designers aren't out of work — they're helping everyone at the company become better designers, and they're busier than ever. We actually need to hire more, because people are seeing the productivity benefits, especially around dashboards.

There's a group-level benefit too: since memory — DRAM and high-bandwidth memory — and storage like flash and NVMe are in very short supply industry-wide, the dashboards we built for our own procurement and service-management teams helped us make better decisions, and we're now in the enviable position of not being short on supply — we actually hold inventory of one of the industry's hottest commodities: storage that works like memory. So there have been real individual and company-wide benefits from realizing that tasks improve, and we're seeing compounding benefits play out over three to six months in our case.

Serena: I like thinking about AI at different levels — individual, team, organization — and always coming back to outcomes: measure those instead of tokens and usage. That's going to be key for leaders planning their next chapter. The leaders who win won't necessarily have the cheapest AI or the most tokens — they'll be the ones who know what they want out of AI and measure the outcomes, instead of being surprised by a price. If you're building your strategy around people, make sure your AI cost assumptions are correct and current — because pricing will reset once the subsidies end. It's time to revisit, not time to panic.

Serena: Val, thank you so much for today's conversation. Any last words for us?

Val: Just a fun, hidden narrative rising to the surface right now — what I call the revenge of middle management. Middle management is always the first thing to get cut, and leaders proudly say they eliminated waste by removing that layer. But look at where AI productivity actually comes from: agents. What do the best agents do? They organize in swarms — they literally form an org chart. They take an ambitious goal, understand how to split it and delegate subtasks, monitor and measure those subtasks for quality and efficiency, and synthesize it all back up into a final business outcome. That's a classic MBA middle-management skill set, and those skills are now some of the hottest things you can use to train AI agents.

I'm not saying companies will suddenly go hire armies of middle managers again — but middle managers should realize that if they're on the job market, their skill set — setting goals and policies, delegating, measuring, summarizing, essentially programming an agent — is exactly what's needed to make AI work in business. So that's one of my hot takes: there will be a revenge of the middle managers, because middle-management skills are what make agents work, which is what makes AI deliver the ROI businesses expect.

Serena: I love that — it's such an encouraging, needed message today, when people feel anxiety everywhere. Thank you, Val, so much for joining us. It's always a pleasure, and I look forward to our next conversation.

Val: Can't wait. Always enjoy these.

Serena: Thanks for listening to Deep Geeks. A huge thank you to my guest today, Val Bercovici. If today's episode made you think differently about how AI gets built or powered, share it with someone who needs to hear it. Find Deep Geeks on Spotify, YouTube, or wherever you get your podcasts. Until next time.