2 posts tagged with "MoE"

A working notebook on Vision MoE — from V-MoE's hard top-$k$ routing to Soft MoE's continuous slot-based formulation.

A hand-written LSTM with a sparse Mixture-of-Experts FFN between two recurrent layers, trained on Tiny Shakespeare. Anchor: Shazeer et al., Outrageously Large Neural Networks (2017) — the original LSTM-with-MoE paper.

Full runnable notebook: createcentury/jax-flax-optax-lab — 02-lstm-moe.