On many natural language tasks there can be significant overlap, making it difficult to judge performance.
That's why I like more complex code generation tasks such the dataset we used for AlphaCode.
The "problem" is that as you move up the levels of abstraction, you need fewer people to do the same amount of work. Unless the complexity of the work scales as well. I've always felt that programmers would be the first class of knowledge workers to be put out of work by automation. This may be the beginning of the end for the programming gravy train.
> The "problem" is that as you move up the levels of abstraction, you need fewer people to do the same amount of work.
This will lower the entry barrier to developing software so more people will go into the field. Before you needed to know a programming language, now you will just have a dialogue with a language model.
> I've always felt that programmers would be the first class of knowledge workers to be put out of work by automation.
We've been automating our work for 70 years, and look how many programmers are employed now. The more we automate, the more capable our field becomes and more applications pop up.
>This will lower the entry barrier to developing software so more people will go into the field.
Indeed. The ideal future of programming is something out of star trek. I often noticed how everyone on the ship is a programmer of a sort, they whip up a simulation as the problem warrants regardless of their field. But in this future, the job of programmer basically doesn't exist. As a programmer, I should be allowed to have mixed feelings about that.
Let your imagination fly. We always want more than it's possible, our wishes fill up any volume like an expanding gas. Humans are going to be crucial to orchestrate AI and extract the most utility out of it.
> as you move up the levels of abstraction, you need fewer people to do the same amount of work
Yes, but the total amount of work (and surrounding complexity) also increases with it. Just look at the evolution of the software industry over the last few decades.
History isn't a great guide here. Historically the abstractions that increased efficiency begat further complexity. Coding in Python elides over low-level issues but the complexity of how to arrange the primitives of python remains for the programmer to engage with. AI coding has the potential to elide over all the complexity that we identify as programming. I strongly suspect this time is different.
Yes, this is how you increase prosperity (see: agricultural revolution, industrial revolution, etc). You can now create more with the same number of people.
If you are interested in this topic, I recommend "Crazy Like Us" by Ethan Watters - it describes how mental illnesses get exported by media coverage and cultural attitudes from Western countries, and how cultural expectations shape the manifestation of mental illness. It was also reviewed in depth by Scott Alexander: https://astralcodexten.substack.com/p/book-review-crazy-like...
Self diagnosis waves on tiktok seem like the next evolution of this, but even faster and more globalised.
I've been wondering if this phenomenon could be harnessed to reduce the impact of mental illness: purposely spread content that encourages self-diagnosis with the mildest possible symptoms.
Have you considered to also expose this information in an interactive web interface? Using a zoomable timeline view (https://www.tensorflow.org/tensorboard/tensorboard_profiling...), both for after the fact analysis (taking a fixed N second trace and then inspecting it) as well as interactive visualization (automatically scrolling timeline with option to pause and scrub).
Yes, we've designed the overall architecture of the system to be modular so that the telemetry can be consumed by a number of different UIs --- we'd love to see someone write web interfaces and/or native GUIs for the console data. I have basically no web development experience whatsoever, though, so I went with the terminal app, because not having to learn JavaScript first made it a lot easier to get started :)
We're also thinking about factoring out the Tokio Console command-line application's internal data model and client code into its own library (https://github.com/tokio-rs/console/issues/227) to make it easier to build other UIs on top of that.
The author keeps referring to the "PSPACE" complexity of chess and Go in the context of AlphaGo; this is incorrect - these games are only PSPACE for arbitrary large board size N, at fixed board size as actually used by humans and current AI they are just constant O(1), complexity class is not relevant for this.
The article was also written before the best evidence we currently have for this was published: scaling laws for natural language understanding (https://arxiv.org/abs/2001.08361), performance of RL algorithms with respect to data (eg AlphaGo Elo vs training time), image model accuracies, etc all show that exponentially increasing amounts of data/computation are required for linear improvements in performance.
