Marvell and Penguin Solutions aim to end the RAMpocalypse with photonic memory fabric
Appliances can share 16 TB of DRAM and half a terabyte of HBM across up to 16 systems
Celestial AI’s dream of high-bandwidth memory (HBM) at a distance will soon be a reality as part of a new collaboration between Marvell Technology and system integrator Penguin Solutions to build a photonic memory fabric.
Marvell acquired the underlying technology for the fabric from Celestial AI for $3.5 billion in February. It comprises a memory expansion module with two stacks of HBM3e totaling 72 GB along with support for eight channels of DDR5 stacked atop a silicon photonic interposer with 7.2 Tbps (900 GB/s) of bidirectional bandwidth.
“What we provide in this appliance is what we call a Photonic Fabric memory module… We provide these modules to Penguin, and they build the box, do all the power management, do the software, and all the management layer,” Preet Virk, senior VP of Marvell’s photonic fabric biz, tells El Reg.
REG AD
Penguin’s first appliances based on the tech will support 16 of these memory modules for up to 16 TB of DDR5 and 1 TB of HBM, with future systems capable of scaling to twice the DDR5 capacity.
REG AD
With 57.6 Tbps of bandwidth distributed via a Marvell-designed optical fiber shuffle, each appliance supports up to 16 servers either via a network interface card, or via co-packaged (CPO) or near-packaged optical (NPO) modules.
If you’re not familiar, CPO modules are typically integrated directly into the compute or switch ASIC, while NPO modules are a sort of half measure. They connect electrically to the host, whether that’s a CPU, GPU, or some other ASIC, and the rest of the fabric optically. NPO is expected to play a major role in the next generation of rack-scale systems from Nvidia and AMD, as it provides better serviceability and lower blast radius than baking photonic chiplets directly into CPUs or GPUs.
Through a software layer, which we’re told was developed by Penguin, each system can access the entire shared memory space, similar to how systems connecting to network-attached storage would have access to the appliance's full capacity.
As we understand it, a lot of this functionality is made possible today through CXL 3.1 compatible NICs and host CPUs, like AMD’s shiny new Venice CPUs. However, for NPO- and CPO-based memory, Virk tells us that CXL can be bypassed entirely.
The implications of this tech should be obvious. If memory can be shared and pooled across multiple servers, each of those servers doesn't need as much of its own DRAM, possibly saving some money.
In addition to facilitating communications, Virk says, the software also allows for several memory tiering modes by which customers can expose the resources to connected systems. For the most part, these follow typical CXL memory tiering conventions, with the exception of an HBM cache-through mode that Virk says will enable users to extract the performance of HBM with the capacity of DDR5.
This mode essentially treats the HBM as a read/write cache. Frequently accessed files are written to and read from HBM first before being pushed into standard DDR5 as they get stale.
Marvell and Penguin are positioning the new memory appliance as an alternative to storage-based key value (KV) cache offload. AI inference generates large KV caches, which are used to track model state across chat sessions, and long-running agents. Depending on the context length, these caches can be both computationally expensive to recompute and large to keep in memory.
REG AD
Because of this, inactive KV caches are often offloaded to DRAM or flash memory, since it’s faster to store and copy them than to recompute them if and when a session is restored. Marvell contends that by offloading KV caches from HBM directly to these memory appliances, customers can achieve better performance and higher memory utilization.
Beyond AI-adjacent workloads, this technology also has potential to reduce the amount of DRAM required for individual systems, a concept we explored in greater depth earlier this year, and one that Meta has already embraced in its own datacenters.
Marvell expects Penguin’s first generation of Photonic Fabric memory expansion systems to arrive sometime in the second half of next year. ®
KioskNews shows a cleaned-up reading view extracted from the publisher’s page — the original always lives on their site, not ours.