I think the main problem with linux is that the effort is so distributed. People don't need 30 different distros to choose from they just need 1 really polished and stable experience.
I don’t see how that helps, unless you actually mean open source, rather than open weights like most people do. Without everything that goes into the model, including training data, these things are opaque.
The raw training data is so large that very few parties could host it for free even if there weren't copyright barriers.
But I think you could have a full open source training software pipeline that's set up to work with Wikipedia, Common Crawl, Books3, Library Genesis, Anna's Archive, and whatever other useful data sets people can name. There would just be a step where you have to provide your own copy of Library Genesis (or whatever subset of it you have managed to obtain).
That may very well be the case. In fact, I'm nearly certain that you're right. But it doesn't change the fact that open weight models are altogether insufficient on a number of important dimensions regarding freedom and transparency. And so often (such as the comment I replied to, I think), even technical people seem to just ignore the difference. Open weights are just weights. No amount of open-washing changes that.
Honest question, I wonder why that is? Surely we have smart humans that did not read and learn "all the books". Can AI not be trained by re-reading material multiple times to reinforce?
Start up a seti at home style of open source LLM training! Assuming there is an ability to merge the sub models trained on each user's home PC into a larger model...
That's not something that is known how to do in a reliable fashion, right? It sounds quite like the problem where transformers are unable to be updated/taught over time.
reply