AI: Yet another AI Chip Bottleneck in Asia. AI-RTZ #1132

If it was just like packaging Lay’s potato chips.

Regulars here at AI-RTZ and ARD podcast know of the long litany of AI bottlenecks this AI Tech Wave through the end of this decade at least. To the AI GPUs, AI memory, AI Data Centers and Power, AI Talent and others, add chip packaging technologies, aka CoWoS, explained below.

Especially as we move into a plethora of AI devices, gadgets and local compute that will be essential to take AI from ‘Search, Ask and Do’ functions of LLM AIs to AI doing things for us in the physical world. Be they AI devices, smart glasses, robots, robotaxies, drones and so many more things not yet invented.

The NY Times outlines this well in “How a Niche Technology became a Choke Point for AI”:

“Advanced chip packaging, which boosts computing power for artificial intelligence, has made the United States more reliant on Taiwan than ever.”

“Chips don’t work without packaging. The process, also called assembly, typically wraps bare pieces of silicon in protective plastic with connectors to pass signals to other chips on a circuit board.”

“These days, chips are frequently placed on an intermediary layer called a substrate, which is typically made of plastic and glass fiber and embedded copper wiring.”

These were technologies developed in the US, specifically at places like IBM, then sub-contracted to Asia, to do the then low cost, low margin work of assembling the substrates to acts as communication bridges between the chips themselves.

And the company that ran with the ball, why Taiwan Semiconductor, aka TSMC of course, in Taiwan. Yes, the one I’ve been calling the ‘Fed’ of the AI Tech Wave.

With a fair bit of help from Nvidia along the way.

“TSMC offers advanced packaging called CoWoS, short for chip on wafer on substrate. Nvidia’s new Rubin processor, for example, uses CoWoS to bundle two large A.I. chips with eight stacks of high-speed memory that each contain 12 chips, amassing 336 billion transistors in one package. By 2029, TSMC predicts a 48-fold boost in computing transistors per package compared with 2024.”

And it’s the new bottleneck in AI systems of all types, be they for Nvidia, or the entire universe of big tech companies needing AI chips and systems for all sorts of things going forward.

“Taiwan Semiconductor Manufacturing Company, which makes cutting-edge chips for Nvidia and other A.I. leaders, also packages nearly all of them. Its key suppliers and partners are mainly in Taiwan, too, facing the same threat from China that caused U.S. policymakers to funnel billions of dollars into boosting domestic chip fabrication.”

“TSMC is already struggling to catch up with A.I.-driven orders. Its CoWoS production is about 30 percent short of demand, estimated Handel Jones, an analyst at International Business Strategies, who said TSMC accounted for about 95 percent of all advanced packaging.”

“You can’t make them without advanced packaging,” said Mark Gardner, an Intel vice president and general manager for packaging and testing. “We’d be in a very different place in the A.I. world without it.”

““All I see is demand continuing to go higher and higher,” said Kevin Zhang, a TSMC senior vice president. “It is certainly going to cause a lot of constraints.”

“And costs remain high. An advanced chip package may cost $500, said Jan Vardaman, president of the research firm TechSearch International. Simpler packages may cost closer to $40, industry executives said.”

“Some chip start-ups are deliberately avoiding designs relying on CoWoS, which can require months to design the required interposer.”

“The packaging bottleneck has become a hot topic in Silicon Valley as TSMC has struggled to keep up with demand.”

The whole piece is worth a full read, particularly on the historical ins and outs of how the US lost its opportunity to build these technologies here vs Asia. And how it’s still behind the curve on that front.

So add CoWoS bottlenecks this AI Tech Wave to why computers and other gadgets will continue to go up in price for the forseeable future. And it’s not just RAMageddon for Apple and others.

And it’s far tougher than packaging Lay’s potato chips. Stay tuned.

(NOTE: The discussions here are for information purposes only, and not meant as investment advice at any time. Thanks for joining us here)