A homelab builder on Reddit has run Moonshot AI’s 2.8-trillion-parameter Kimi K3 AI model on discontinued second-gen Optane PMem 200 memory, using a 40-core 3rd-gen Xeon Scalable CPU and not a GPU for inference. The build runs hot, and the builder’s advice is that you “treat active DIMM cooling as mandatory,” with dual 12W blower fans aimed at the modules to get temperatures under control. The project, a follow-up to a separate user’s first-gen Optane build earlier this year, runs at a sluggish 0.85 tokens per second even with 1-bit quantization.
Kimi K3 is absolutely not built for home use, in our estimation, with Moonshot AI recommending data center inference engines to run it, and even the smallest community quantization needs at least 610GB of RAM. Using secondhand Optane, instead, is one of the few ways to reach that capacity at home. It doesn’t help that DRAM prices have skyrocketed, and the builder says the Optane memory costs “a fraction” of the equivalent.
Intel launched its Optane Persistent Memory 200 Series (PMem 200) in 2020 for its 3rd-gen Xeon Scalable CPUs in 128GB, 256GB, and 512GB capacities. On two-socket Xeon platforms, it runs at a higher effective MT/s rate than the first generation (PMem 100) that was used on the first project, and it can deliver an average of up to 32% more memory bandwidth. Optane can run in either Memory Mode or App Direct Mode; both builds had platforms allowing for each.
Memory Mode has the Optane act as system memory with DRAM as its cache, the method used in the first build, while App Direct exposes Optane as a separate, persistent region that the software maps directly, the mode used in the second. This can give “more control over where the model’s tensors live,” according to the first build’s creator, u/APFrisco, who planned to try it.
Performance does take a hit by going with Optane over DRAM, and with the higher 3-bit quantization, the speed went down even more to 0.75 tokens per second. One commenter points out that this is very slow, with a 50k-token task estimated to take 13 hours at about 1 token per second, but another notes that overnight batches don’t care as much about speed. Both speeds compare to the first build’s roughly 4 tokens per second, although in that case the builder had an RTX 3060 handling part of the model in addition to Memory Mode, a 2-bit quantization, and the smaller, earlier 1-trillion-parameter K2.5 model.
The latest builder also had to deal with heat output, with the Optane DIMMs getting “extremely hot” with sustained inference, creating a situation the builder says can endanger neighboring DRAM. That heat is consistent with Intel’s own specs, as the manufacturer rates the modules with forced airflow of 2.4 to 2.7 meters per second for cooling, which is what you expect to find in data centers. You might not normally have this level of airflow pointed directly at your memory sticks.
The expected operating conditions put each Optane module at 12W or 15W, regardless of capacity. If the build used four sticks, that would mean up to 60W of heat placed right next to the system memory. Throttling on these occurs at 83 degrees Celsius, with a thermal trip shutdown at 85 degrees, compared to the 85 degrees Celsius and up to 95 degrees in an extended range that typical DDR4 server memory is rated for. A lot of heat has to be removed from the system to keep things running.
Get Tom's Hardware's best news and in-depth reviews, straight to your inbox.
The performance numbers given are pre-tuning — this means without tuning thread counts or NUMA affinity — so there is room for improvement, and one commenter also asked about using Memory Mode with DRAM caching for the most-used experts. The builder plans to add GPUs and to also load parts of the model into DRAM as a faster tier alongside the Optane, with “more benchmarks to come.”
8 hours ago
47
(Image credit: u/IndividualSurvival on Reddit)
English (US) ·