Micron develops advanced memory stacking technology for GPUs

Recent advancements in memory technology have brought attention to the integration of High Bandwidth Memory (HBM) and High Bandwidth Flash (HBF) in the same stack for enhanced performance in data center GPUs. This innovative approach allows both types of memory to be utilized simultaneously, optimizing memory interfaces on GPUs by enabling operations with either HBM or HBF independently. The proposed combination specifically benefits applications like large language model inference, where both low-latency access and substantial data storage are required. The new memory stack architecture enables a significant potential to enhance the performance and efficiency of GPUs used in demanding computational tasks, such as AI and deep learning. HBM is employed primarily for caching attention matrices while HBF can store the model weights—a setup that promises to streamline processes and improve data handling. Current designs suggest that a single stack can accommodate up to 8 HBM die, each with 4 GBytes of capacity, along with 8 HBF die, each supporting 64 GBytes. In this design, one or two logic die may facilitate enhanced functionalities through shared circuits. Despite advantages, there are challenges associated with this integrated memory stack approach. If the height of the combined stack is limited to 8 high, then opting for a mixed stack would mean sacrificing capacity on both HBM and HBF. Such decisions hinge on specific use cases, as dedicated stacks of solely HBM or HBF may serve better in scenarios demanding exclusive capabilities. Understanding the contexts in which these technologies thrive will be essential to exploiting their full potential. Micron emphasizes their focus on customizing logic die for select customers, such as NVIDIA, to cater to the increasing data demands of AI. As Micron advances this technology, there is an inherent push towards optimizing memory density and performance. With designs looking to expand memory capacity and capabilities, the anticipated introduction of 16 memory die in a single stack, as well as collaborative logic die functions, will likely set new benchmarks in the industry. The synergy between HBM and HBF signifies a turning point in memory technology, paving the way for more adaptable and powerful GPU architectures tailored for future applications.