Summary:

In this episode, we argues that to effectively limit China's ambitions in AI, export control policies need to broaden their focus beyond integrated circuits to include DRAM and HBM memory technologies with greater specificity. We emphasize that these memory technologies are critical for AI systems' performance, and without restrictions on them, efforts to contain Chinese advancements in AI could be ineffective. We explore the importance of DRAM and HBM in AI development, the current state of these technologies, and the challenges facing their future development. Finally, we highlight the major players in the DRAM and HBM market, including the export restrictions implemented by different countries, and calls for a more comprehensive approach to control the export of these critical technologies.

Questions to consider as you read/listen:

What are the current export restrictions on DRAM and HBM technologies, and how are these restrictions impacting China’s AI development?

How are the key players in DRAM technology, particularly Samsung, SK Hynix, and Micron, responding to the growing demand for HBM for AI applications?

What are the challenges facing DRAM technology in the future, and how are companies like Micron attempting to overcome these obstacles to maintain their leadership in AI-related memory?

Long format:

Memory Wars: Why DRAM and HBM Must Be the Next Front in AI Export Restrictions: How Memory Tech Could Shape China’s Superpower Ambitions

(One sentence thesis: To effectively limit China's announced desire to be THE global leader for AI technology and its application, export control policies must broaden their focus beyond integrated circuits to equally prioritize DRAM and HBM memory technologies, as these are critical components in high-performance AI systems)

By Justin James McShane.

TL;DR:

While export controls on integrated circuits (ICs) are crucial to limit China’s announced intention to develop into THE global leader for AI technology and its application, DRAM and HBM memory technologies are just as vital. These memory components drive AI performance, and without robust restrictions on them, efforts to contain Chinese ambitions in AI could fall short. Policymakers from the US, Japan, South Korea and Taiwan should prioritize DRAM and HBM export controls alongside ICs to create a more comprehensive strategy in safeguarding AI-related technology leadership.

Background

Reported today (8 November 2024) in DigiTimes Asia was an article about the HBM3E “wars”. (https://www.digitimes.com/news/a20241108PD211/micron-hbm-competition-samsung-hbm3e.html ) The article points to the previous dominance of South Korea’s Samsung Electronics and SK Hynix in this technology but notes that Micron Technology (US) is coming on strong due in part to CHIPS Act subsidy based spending. Micron's HBM3E is considered to be more power and thermally efficient than its competitors. Micron's HBM3E 12-high capacity is 50% higher than the current HBM3E 8-high, which allows larger AI models to run on a single processor. Micron is looking to expand its market share through HBM3E installations and early HBM4 work. However, SK Hynix began volume production of the world's first 12-Layer HBM3E. SK Hynix claims to be 8.8 times more efficient than Samsung and Micron in HBM production. It is a fun “battle” to watch as the pace of innovation is quite high paced.

Introduction:

As artificial intelligence (AI) rapidly transforms global industries, the technologies driving its evolution demand careful scrutiny, particularly concerning national security and economic competitiveness. Integrated circuits (ICs), often at the heart of AI systems, have garnered considerable focus in efforts to regulate China’s access to advanced computing capabilities. However, dynamic random-access memory (DRAM) and high-bandwidth memory (HBM) are equally critical to the infrastructure powering these systems. This oversight could limit the effectiveness of export controls. This article argues that curbing China's ambitions in AI requires prioritizing and equating DRAM and HBM restrictions alongside IC regulations. By exploring the current landscape of DRAM and HBM technology, this piece highlights the vital need to inventory existing restrictions on these technologies and calls for further, more comprehensive actions if the policy goals of curbing Chinese ambitions in AI are to be realized.

Why does this matter?

There has been a lot of attention placed on artificial intelligence (AI). And with that attention most of the conversation focuses on the subject of integrated circuits, otherwise known as semiconductors or simply chips. A lot of focus goes on these little important physical units, the chips and for good reason as they are the fundamental building blocks of AI. With this article, I wish to go to a deeper level of beyond the building blocks to the house itself which is DRAM (pronounced D-RAM) technology.

DRAM technology in the context of AI

DRAM (dynamic random access memory) is a type of memory that is critical for artificial intelligence (AI) applications and is in high demand. DRAM is a type of RAM (random access memory) that stores data and program code in computers. It's a volatile memory, meaning it only saves data while the device is powered on. DRAM is used in many devices, including PCs, laptops, smartphones, and tablets. AI applications require high-performance computing (HPC) systems to process large amounts of data and complex computations. DRAM is a key component of data processing, and AI servers need six times the amount of DRAM as standard servers. High Bandwidth Memory (HBM) is a type of DRAM that uses stacked chips to achieve high-speed data transfer and low power consumption. HBM is used in AI applications, graphics cards, and supercomputers. The increasing use of AI is driving demand for memory and storage. This is expected to lead to more DRAM capacity expansion in laptops and servers.

High Bandwidth Memory (HBM) explained

High Bandwidth Memory (HBM) is a computer memory technology that offers high data speeds and low power consumption. It's used in high-performance computing applications, AI, and other areas where fast data access is required. HBM uses 3D stacking to pack more memory chips into a smaller space, which reduces the distance data needs to travel between the memory and processor. HBM's high bandwidth and low latency architecture makes it a good choice for AI applications that require large amounts of memory. It also has a small form factor compared to Dynamic Random Access Memory Dual In-Line Memory Module (DRAM DIMMs) where the computer memory that contains one or more DRAM chips is on a printed circuit board (PCB) that are commonly used in desktops, laptops and servers.

The different levels of High Bandwidth Memory (HBM) chips are:

HBM: The first generation of HBM has a data rate of 1.0 GB/s and a bandwidth of 128 GB/s per device

HBM2: The second generation of HBM has a data rate of 2.0 GB/s and a bandwidth of 256 GB/s per device

HBM2E: The third generation of HBM has a data rate of 3.6 GB/s and a bandwidth of 461 GB/s per device

HBM3: The fourth generation of HBM has a data rate of 6.4 GB/s and a bandwidth of 819 GB/s per device

HBM3E: is the curent state of the art and has the fastest and highest capacity high-bandwidth memory for advanced AI innovation with 8-high, 24GB cube that delivers over 1.2 TB/s bandwidth at superior power efficiency.

HBM4: The next generation of HBM will have a larger physical footprint and double the channel count per stack compared to HBM3

The current best state of the art for DRAM AI technology

As of today, the state-of-the-art DRAM technology is considered to be the "1α" (1-alpha) manufacturing process, which offers significant improvements in bit density, power efficiency, and performance, currently being produced by companies like Micron. This represents the most advanced DRAM process technology available, pushing the boundaries of scaling and density within the current DRAM architecture.

The Key aspects of the current state-of-the-art DRAM include:

Advanced node scaling: Utilizing the latest manufacturing nodes, like 1α, to achieve smaller transistors and higher density on the chip

High Bandwidth Memory (HBM): Stacking multiple DRAM dies vertically to achieve significantly higher memory bandwidth compared to traditional planar DRAM.

3D stacking techniques: Utilizing wafer bonding technology to stack different components within the DRAM chip, enabling more complex architectures

Material innovations: Exploring new materials for capacitors to improve storage capacity and reduce leakage current.

The challenges for future DRAM technology

There are physical limitation issues. As transistors become smaller, maintaining sufficient cell capacitance and signal integrity becomes increasingly difficult. There are power consumption issues. Balancing performance with power consumption as scaling progresses is not going to be easy. There are manufacturing complexity issues. The increasing complexity of 3D stacking and advanced manufacturing techniques.

DRAM assembly details

DRAM is assembled in a number of steps that include: thinning of the wafer, attaching the wafer to an adhesive backing, dicing the wafer into individual dimes using a diamond edge saw, picking the individual dies from the after, placing the dies on the circuit board, connecting the sold gold wire to connect each chip to the circuit board and encapsulating each die into a protective plastic package.

DRAM/PCB equipment

Key equipment used in PCB assembly includes: solder paste printing machines, solder paste inspection (SPI) machines, pick-and-place machines, reflow soldering machines, wave soldering machines, glue dispensing machines, and automated optical inspection (AOI) machines, all used to precisely apply solder paste, place components on the board, and inspect for defects throughout the assembly process.

DRAM design and assembly work flow

A CAD department maps out each layer of the PCB. The assembly starts after the CAD design is submitted. The manufacturing process begins with Surface-Mount Technology (SMT). The screen printer is the first step for the loaded components. After solder is placed, an automated inspection occurs, then surface mounting of resistors, capacitors, and components like DRAM chips. These PCBs are then passed through the reflow oven, where the solder is cured by high temperature cycles. After reflow, the products undergo a final inspection. Next, products go through the labeling system, important to tag the product part number and provide security features. For modules to work, though, they have to go through Automatic Serial Presence Detect (AutoSPD), which programs them to be identifiable and accessible by computers. Some products at this point undergo functional testing then further assembly for heatspreaders. It is then tested in real world conditions and visually inspected.

The major players in DRAM technology in AI

The major companies in the global DRAM technology market for AI are in listed order of highest marketshare: Samsung Electronics (South Korea), SK Hynix (South Korea), and Micron Technology (USA). These three companies collectively hold the majority of the market share, making it highly concentrated. Collectively, these three companies hold 90% of the global marketshare.

China has several companies that produce DRAM chips such as ChangXin Memory Technologies (CXMT), Fujian Jinhua Integrated Circuit (JHICC) (part of China’s Made in China 2025 program) and Tsinghua Unigroup. However, none of these currently make HBM chips at scale but they are ramping up efforts to do so. Without HBM DRAM chips, you don’t have AI chips.

HBM DRAM technology and export restrictions

In 2022, the US Department of Commerce , Bureau of Industry and Security banned export of any DRAM memory chips of 18nm half-pitch or less. The US is reportedly considering tightening restrictions to capture all HBM2, HBM3 and HBM3E chips as well as the tools required to make them.

Japanese restrictions primarily target the equipment needed to manufacture high-performance DRAM chips with smaller node sizes and not older generation DRAMs. Japanese companies like Nikon, Tokyo Electron, and Screen Holdings are subject to these export controls, as they produce key semiconductor manufacturing equipment (SME)

I could not find specific references to confirm that Taiwan has similar restrictions or that they do not have similar restrictions. I found some suggestions that it may be controlled as a Strategic High-Tech Commodities (SHTC) and under the “catch-all” control measure.

South Korea is reportedly considering export restrictions on DRAM chips. Just as I previously wrote when I highlighted that despite pressure from the US that South Korea stands alone as the only integrated chip fabricator that does not have export restrictions at all, South Korea is concerned about its impact on its economy if these proposed restrictions were to go in place as China is its major trade partner. The US government has granted Samsung Electronics and SK Hynix an indefinite waiver on restrictions to export advanced chip-making equipment to China. This waiver is expected to help the two companies maintain their competitive advantage in China's semiconductor supply chain.

Conclusion:

In conclusion, as the competitive race to lead AI advancements accelerates, it is essential for global leaders to recognize that export controls on integrated circuits alone may be insufficient to curb Chinese ambitions. DRAM and HBM memory technologies are integral to AI functionality, making them as critical to monitor and restrict. Given the growing strategic value of these memory technologies, coordinated, robust restrictions are imperative to preserve economic stability, national security, and the upper hand in AI development. Moving forward, international policymakers from the US, Japan, Taiwan and South Korea must extend the scope of export controls to include DRAM and HBM more specifically, creating a robust framework that effectively responds to the complexities of modern technological competition if the goal is to curb Chinese ambitions of dominance in AI.

Sources

https://www.digitimes.com/news/a20241008PD212/sk-hynix-samsung-hbm-production-micron.html#:~:text=SK%20Hynix%20highlights%20its%20HBM,better%20than%20Samsung%20and%20Micron

https://www.micron.com/about/blog/applications/ai/micron-continues-memory-industry-leadership-with-hbm3e-12-high-36gb#:~:text=Micron%20HBM3E%2012%2Dhigh%20boasts%20an%20impressive%2036GB,avoiding%20CPU%20offload%20and%20GPU%2DGPU%20communication%20delays

https://www.atpinc.com/blog/what-is-ai-artificial-intelligence-ai-servers-memory

https://www.driehaus.com/perspectives/High-Bandwidth-Memory-Technology-for-AI-Applications#:~:text=Prakash%20Vijayan%2C%20CFA-,High%20Bandwidth%20Memory%20Technology%20for%20AI%20Applications,bandwidth%2C%20density%2C%20and%20efficiency

https://www.lenovo.com/us/en/glossary/what-is-dram/#:~:text=DRAM%20stands%20for%20%E2%80%9Cdynamic%20random,is%20running%20programs%20or%20applications

https://www.techtarget.com/searchstorage/definition/DRAM

https://www.asml.com/en/technology/all-about-microchips/microchip-basics#:~:text=Memory%20chips%20store%20information.,device's%20power%20is%20turned%20on

https://www.atpinc.com/blog/what-is-ai-artificial-intelligence-ai-servers-memory

https://www.driehaus.com/perspectives/High-Bandwidth-Memory-Technology-for-AI-Applications#:~:text=Prakash%20Vijayan%2C%20CFA-,High%20Bandwidth%20Memory%20Technology%20for%20AI%20Applications,bandwidth%2C%20density%2C%20and%20efficiency

https://www.forbes.com/sites/tomcoughlin/2024/06/09/ai-is-driving-memory-and-storage-demand-and-product-introductions/#:~:text=The%20growing%20implementation%20of%20AI,increased%20demand%20to%20support%20AI