Two reasons: 1) it loses a lot of the scale advantages you get from ml systems right now, due to being much more expensive than inference with all the tricks people have figured out, and 2) it has the "infinite data" issue mentioned in the article, where you start to need an insane amount of storage per user or you start to run into catastrophic forgetting
Oh yeah i agree, its really cool and compelling. I could even imagine tweaking the learning rates lower as your system "ages" which almost certainly happens with humans... but its very new, and probably will take a long time to compete with transformers if ever
Two reasons: 1) it loses a lot of the scale advantages you get from ml systems right now, due to being much more expensive than inference with all the tricks people have figured out, and 2) it has the "infinite data" issue mentioned in the article, where you start to need an insane amount of storage per user or you start to run into catastrophic forgetting
Oh yeah i agree, its really cool and compelling. I could even imagine tweaking the learning rates lower as your system "ages" which almost certainly happens with humans... but its very new, and probably will take a long time to compete with transformers if ever