SOCAMM is a memory standard initially claimed to be tailored for NVIDIA, and Team Green has been an early adopter. For those unaware, SOCAMM is based on the LPDDR DRAM, which is traditionally used in mobile and low-power devices, but unlike solutions like HBM and LPDDR5X, the SOCAMM is upgradable. It's not soldered to the PCB, and it is claimed to be a solid option to work alongside HBM to handle memory-bound tasks. According to a Hankyung report, NVIDIA, Qualcomm, and AMD are exploring integrating SOCAMM modules into upcoming AI racks.

It is claimed that both AMD and Qualcomm are exploring an approach different from NVIDIA's, which is to arrange a 'square' module with two DRAMs in two separate rows. This is intended to enable power control (via the PMIC) directly on the module itself, allowing for a more effective regulation mechanism and ensuring that the SOCAMM runs at extreme speeds without any trouble. By moving the PMIC onto the module itself, AMD and Qualcomm will also reduce motherboard manufacturing complexity by eliminating the need for power circuitry.

On the other hand, given that SOCAMM adoption widens out, the DRAM utilization figures for the memory type will rise as well, considering that with agentic AI, having a short-term storage alongside HBM has become a necessity, and SOCAMM allows having TBs of memory per CPU, allowing an agent to have millions of tokens "active" in its memory. While throughput relative to HBM is slow, SOCAMM remains a viable medium and a power-friendly option.

For now, NVIDIA intends to offer SOCAMM 2 with Vera Rubin AI clusters, and considering that AMD and Qualcomm are exploring the solution, we could expect the memory type to be featured in their next-gen AI clusters soon as well.

By: DocMemory
Copyright © 2023 CST, Inc. All Rights Reserved