I think your opinion is bad while also misrepresenting machine learning.
> It's not transformative at all, nor quoted for study/discussion, and that's exactly what makes it a problem.
This model works in the same way as GPT-3. It's just predicting what the next most likely word will be given the previous words. It does this by creating a generalizaiton over the content that was used to train it. This generalization should be similar (or close to the same) to if it had a completely different set of training data.
In the same way that you could create 60,000 new training images comparable to the MNIST dataset, and get a model that works at an equivalent level, you could do the same with source code. It doesn't matter where the original data is from because the end result should be very similar when models reach higher levels of accuracy.
> but as it is right now github are simply outlaw and with no excuse at all.
This is an insane view on machine learning. When you read content, neural weights are adjusted in your brain to keep it as a memory. Is that copyright infringement too? Do you now owe Disney royalties because you stored their movie in your memories?
It's also an incredibly harmful view. Machine learning is dependent on data. Services like search, translation, recommendations, etc. are all dependent on training data. And, you're making the case, that since you don't like that it was trained on source code, it should all be illegal.
> This model works in the same way as GPT-3. It's just predicting what the next most likely word will be given the previous words. It does this by creating a generalizaiton over the content that was used to train it. This generalization should be similar (or close to the same) to if it had a completely different set of training data.
It's not enough of a generalization to be acceptable IMO.
Let's see GPT-3 as something analogous to a brain. Human brains can do multiple things:
- learn through experience and re-use concepts, but writing completely original code,
- copy a program's structure without copy-pasting the code, effectively re-writing it,
- copy the exact code, with a few minor details changed.
Currently, Copilot does a lot of the last point [1]. So just like humans can choose different actions and the legality depends on the action, ML models can do different things, and these aren't automatically legal by virtue of coming from an ML model.
It's not enough to argue about how machine learning works. ML can do tons of things, and GitHub's current methodology leads to something close to copy-paste. Maybe it can learn a semi-original way to write common things like searching in a list, but for more exotic uses like complex algorithms, being unable to actually understand how to code, it basically has to act as a search engine for existing implementations.
> Do you now owe Disney royalties because you stored their movie in your memories?
That argument makes no sense. Nobody would be complaining if GitHub had just trained a model on that code. The potentially illegal part is offering a service that creates "new" Disney-like movies by assembling parts of Disney's IP.
> It's not enough to argue about how machine learning works. ML can do tons of things, and GitHub's current methodology leads to something close to copy-paste. Maybe it can learn a semi-original way to write common things like searching in a list, but for more exotic uses like complex algorithms, being unable to actually understand how to code, it basically has to act as a search engine for existing implementations.
I don't disagree that it is over-trained on certain sequences of words, but, overall, I think the generalization fine. It's often pre-prompted with content it has not seen before, resulting in unique new content. There's nothing copy-pasted about this, just a statistical understanding of what usually happens next.
If the pre-prompt is something very specific, ex. "a dog runs in the park while during the middle of the day, while the sky is the color _____" it will obviously output blue. The same can be true when there are only very specific known algorithms that have been used more frequently than others. And, in the cases where it does commit something arguably comparable to copyright infringement, it would probably on the programmer and not on the model for deciding to use it.
> The potentially illegal part is offering a service that creates "new" Disney-like movies by assembling parts of Disney's IP.
In very specific cases only does it output copyrighted content. Most of the time it is just outputting a generalization of what is expected. It isn't Disney's content, but human content. Also, just creating content that has some similarity isn't copyright infringement. Satire has been well accepted as fair use.
You are assuming that laws/rights operate on basic properties like learning or remembering, but they operate on goals (e.g. humans can run, but there is no speed limit for a pedestrian. But if we learned to run at 100mph somehow, a limit would be introduced). The goal is to motivate creators to create on terms that would be fair enough and useful for a society. If you “generalize” their work at such scale, it is a very different situation, like when you are using a sidewalk for robots running at 100mph and demotivate anyone from using it.
Services like search, translation, recommendations, etc. are all dependent on training data
There is an obvious benefit in being represented in search results and recommendations.
> If you “generalize” their work at such scale, it is a very different situation, like when you are using a sidewalk for robots running at 100mph and demotivate anyone from using it.
I don't understand what you're trying to state.
If you make fun of someone on national TV, someone doesn't like it either. It doesn't mean that it should be illegal.
The point of attribution and copyright is to create a creator/inventor-friendly environment. The copyright law used realistic points in a problem space to provide it. These new AIs enlarged the problem space, but it doesn’t mean that the initial idea of copyright is not applicable to them by default, or should not be. It’s okay to vote for or against that, but the potential systemic effect of that vote should also be kept in mind.
I hope this makes more clear why I used this analogy.
> The point of attribution and copyright is to create a creator/inventor-friendly environment.
They're building a class, but, beyond that, I'm not really confident there are any parties that feel like they've been harmed by what they're claiming is a violation. Microsoft didn't just copy repositories from GPL licensed code, but from other companies too. Why didn't these companies care?
It is a good counter point though, as Waymo has released datasets that can only be used under a license. While at the same time, GitHub did not initially use discretion for people who had agreed to their terms of service and were using their hosted repositories.
While I think the legal system might lean towards enforcing consent to use the data in certain circumstances, on the technical side, if it's really a generalization that is resulted from training, the initial data is meaningless in the final model. It could be argued that it did not harm this company economically because the model trained without the data would still have an almost equivalent financial impact.
> It’s okay to vote for or against that, but the potential systemic effect of that vote should also be kept in mind.
My position above was clear, I do not think the law should be changed yet. The systematic change is from someone stating a violation happening when I do not believe that it is a violation. This lawsuit could result in models like Stable Diffusion becoming illegal, which I view as incredibly harmful to the future of artificial intelligence research.
What if we extrapolate a little into the furure. Copilot-likes and SDs become useful tools for knowledge/mastery extraction and application, much better than today. Will people want to create new knowledge, or will they be satisfied with reshuffling existing one? Will opinions change if people realize that there is no point in creating something new (it will be generalized), and existing is easily reproduced without them (classic “took our jobs”). Can it all happen? How do you see it?
> Incrediblest full of all the harmfulsts! Not only Machine learning:
I do not agree that it's a good idea to make using copyrighted data for training illegal. The people who are upset that GPL code was used for training should be happy that a large corporation believes that training on copyrighted works is fair use. It's a more democratized position.
> It is illegal as the attribution is missing. Also copyright applies etc. see the lawsuit. Its less a technical thing, more a legal one.
I think in reality, this is just a lawyer or group of lawyers who realized they had a case and a reactionary audience to possibly make them millions of dollars.
> It's not transformative at all, nor quoted for study/discussion, and that's exactly what makes it a problem.
This model works in the same way as GPT-3. It's just predicting what the next most likely word will be given the previous words. It does this by creating a generalizaiton over the content that was used to train it. This generalization should be similar (or close to the same) to if it had a completely different set of training data.
In the same way that you could create 60,000 new training images comparable to the MNIST dataset, and get a model that works at an equivalent level, you could do the same with source code. It doesn't matter where the original data is from because the end result should be very similar when models reach higher levels of accuracy.
> but as it is right now github are simply outlaw and with no excuse at all.
This is an insane view on machine learning. When you read content, neural weights are adjusted in your brain to keep it as a memory. Is that copyright infringement too? Do you now owe Disney royalties because you stored their movie in your memories?
It's also an incredibly harmful view. Machine learning is dependent on data. Services like search, translation, recommendations, etc. are all dependent on training data. And, you're making the case, that since you don't like that it was trained on source code, it should all be illegal.