Jarkko Sakkinen
@jarkko@social.kernel.org
@eduzsh I'm going to keep in phase of doing at min two inferences engines annually from scratch but still polish each to as high quality I can. What I want to understand is this:
1. Let's assume we have a SoC.
2. Let's imagine it is capable of doing inference and has special features.
3. The micro-architecture design can make any feasible sacrifices on anything related training post-training (does not have to but I don't give it any weight).
What would be best architecture provide lift up for let's say up 500B parameter models. It's also definitely an area were Nvidia dose not have any tech leadership. Blackwell hardware design is sloppy and dysoptimal if thinking from this "you had one job" angle.
Next model I'm still going to do on Ryze 5 Pro (common laptop CPU from decade ago, Zen 2 architecture) I need to make it scale to GPT-OSS-120B. That's my end goal for this CPU. I'm planning to reach it with 2bit quantization. I have full MoE implementation for 20B version. The magical "model streaming" part was weird. This was discussed either in the context Dwarf Star 4 or Colibii. I mean one always mmaps huge files instead of copying anything and page fault handler brings up the "experts". Still don't get what model streaming is but I'd guess it is just a silly term for the most common activity (never checked this from their implementation).
Vezi original ↗
1. Let's assume we have a SoC.
2. Let's imagine it is capable of doing inference and has special features.
3. The micro-architecture design can make any feasible sacrifices on anything related training post-training (does not have to but I don't give it any weight).
What would be best architecture provide lift up for let's say up 500B parameter models. It's also definitely an area were Nvidia dose not have any tech leadership. Blackwell hardware design is sloppy and dysoptimal if thinking from this "you had one job" angle.
Next model I'm still going to do on Ryze 5 Pro (common laptop CPU from decade ago, Zen 2 architecture) I need to make it scale to GPT-OSS-120B. That's my end goal for this CPU. I'm planning to reach it with 2bit quantization. I have full MoE implementation for 20B version. The magical "model streaming" part was weird. This was discussed either in the context Dwarf Star 4 or Colibii. I mean one always mmaps huge files instead of copying anything and page fault handler brings up the "experts". Still don't get what model streaming is but I'd guess it is just a silly term for the most common activity (never checked this from their implementation).
