Sujet : Microsoft is plagiarizing my Invention [LLMs under the hood]
De : janburse (at) *nospam* fastmail.fm (Mild Shock)
Groupes : comp.lang.prologDate : 08. Oct 2024, 15:00:56
Autres entêtes
Message-ID : <ve3dum$5j97$1@solani.org>
References : 1 2 3 4 5
User-Agent : Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:91.0) Gecko/20100101 Firefox/91.0 SeaMonkey/2.53.19
will probably never get a Turing Award or something
for what I did 23 years ago. Why is its reading
count on research gate suddently going up?
Knowledge, Planning and Language,
November 2001
I guess because of this, the same topic takled by
Microsofts recent model GRIN. Shit. I really should
find some investor and pump up a start up!
"Mixture-of-Experts (MoE) models scale more
effectively than dense models due to sparse
computation through expert routing, selectively
activating only a small subset of expert modules."
https://arxiv.org/pdf/2409.12136But somehow I am happy with my dolce vita as
it is now... Or maybe I am decepting myself?
P.S.: From the GRIN paper, here you see how
expert domains modules relate with each other:
Figure 6 (b): MoE Routing distribution similarity
across MMLU 57 tasks for the control recipe.