Python serves several huge ones: web, datascience, 3d graphics, sysadmin... all with different needs, different conventions, different “standard” tools to do this and that. The community is orders of magnitude bigger than ruby by now, and anything that requires coordination is much harder to accomplish than in a lot of other ecosystems.
Python is now on par with ecosystems like C/C++ (where different build and packaging conventions are legions) or Java (where ever 10 years they write a new build tool on top of the previous one(s) to hide the complexity, and still, like python, they struggle to make simple executables 30 years on...). It’s just a hard problem to cater for so many different needs.
That’s the “oh we can’t use methods from others, we are too special” argument, which is always BS. Coupling the misapprehension to a snooty misconception is doubly unfortunate.
> Python serves several huge ones: web, datascience, 3d graphics, sysadmin... all with different needs, different conventions, different “standard” tools to do this and that.
Them having different conventions and standard tools is the failure, not a constraint.
What are the different needs that these ecosystems have which mean they couldn't use a common package manager? I don't believe there are any.
For instance, in the scientific computing world, compiled extensions are hugely important, whereas they're relatively rare in web development.
There's probably no logical reason why a single common package manager for these domains is impossible. But they have different priorities, and people write tools that solve their own problems. Who is going to decide which package managers are unnecessary, and convince all of the users who are happy with them to switch to something else?
I think something people are forgetting when comparing Python's packaging system to other languages and runtimes is how old Python and its ecosystem is. For instance, Python's public release predates Linux's public release(Feb 1991 versus Aug 1991). The first release of setuptools was 2004 and it isn't like people weren't sharing Python code even before then (https://github.com/pypa/setuptools/commit/8423e1ed14ac1691c2...). That's over 5 years before npm was released (https://en.wikipedia.org/wiki/Npm_(software)). That means npm got to learn from Python and any other project that predated it. It also means they didn't have Python's momentum to contend with (I certainly never expected Python to become _this_ popular, and that makes it very hard to change as you literally have tens of millions of people to re-educate when you change something).
So while it's fine to say you wished Python's packaging approach was as good as another languages, please do so know that it is not fair to say that if one community got something to work for their needs it means Python should have been able to as well; it's not an apples-to-apples comparison considering the trees were quite possibly planted at different times and are different varietals.
I'm not sure how old you are, but if you don't know the name "Borland" you might not realize how novel it is to have a compiler on every major OS. While Python's age means it has been around to see a lot of changes occur, that also means it had pre-existing practices and code to also consider whenever a shift in the industry occurred. Plus in hindsight things look obvious, but during the transition it isn't obvious what will stick and what won't.
It's pretty remarkable that things have continued to function since my 17 years with Python alone have seen things like:
- Compilers becoming available on all OSs
- Linux becoming a thing
- Laptops becoming common
- Smartphones
- Web-based email clients
As I have said a few times in comments on this thread, it's fine to feel like there's room for improvement, but trying to "stick it" to Python and its packaging ecosystem in some way for not being perfect is pointless for you and demotivating for those who are actually in a position to try to make things better for you (and even if you aren't a Python user you should care to some extent about it as Instagram runs on Python and thus it is helping to deliver you those cat photos while you're stuck at home ).
It sounds a bit more understandable if you rephrase it as "a small group of largely unpaid volunteers should find a solution for which millions of users and several billion-dollar businesses will happily abandon their current long-running and well-tested solutions".
First, programmers have choices, and Python's lack of a good user experience on this front will turn off people that have better options with other languages which have found a way to deliver a better experience given the same constraints as python.
Second, there is the python foundation, which funds people to work on areas of python. This should be a priority, and can be done by paid contributors.
People in the Python ecosystem have come up with plenty of new things over the years - eggs, wheels, conda, poetry... Coming up with something new is the easy bit. Getting everyone to agree that a new thing is better and stop using/recommending other things is much harder - almost impossible, if those other things serve some use cases fairly well.
The fragmented ecosystem is one of the main complaints about Python packaging today. And that's really tricky, because the main way the open source world knows to solve problems is to create new tools, and that doesn't help when the problem is 'too many tools'.
And it (and by "it" I mean the Python community) absolutely did! The main problem that it has come up with numerous different somethings that have been competing to become standards, across a wide range of related, but separate areas (packaging, versioning, environments...), and only a few of them (like pip) managed to become partial (but not at all universal) de facto standards.
not at all. from my previous experience with ruby i was using gems with compiled bindings to libraries for: mysql, memcached, json, xml, regexp and http. and probably some more i don't recall right now.
Python does all of that and more. There are still occasional issues with the thorniest setups, but binary wheels absolutely took care of the average case. You even get a massive desktop toolkit luke QT with a simple “pip install pyqt”.
Getting there might not be the nicest experience, but most packages nowadays can and do get there.
The issue is that the environment varies a lot and there's usually a lot of C++ code that needs to be compiled to interact with the python plugins based on different environments with 500+ libraries mixing & matching...
I would not use Node as an example of good packaging. It really hasn't been that long since left-pad. Then there's the fact that you still have multiple different tools to do the same thing (npm vs. yarn) and the end result node_modules/ is a fucking mess anyways.
I think you can certainly use Node as an example of much better developer experience. Even the difference between npm and yarn is fairly minimal, and there is a solid community 'default' in Yarn.
I think part of the reason we ended up with so many trivial libraries and dependencies in the Node ecosystem was precisely because it is so easy to create and publish packages.
I love Python as a language, but working with the packaging even just to install dependencies is like pulling teeth every time you start a new project - nevermind publishing packages yourself.
No it's not... 99.9% of the time, installing dependencies is a breeze. Packaging is where the issue lies, but pulling those dependencies down is painless. You run one command and everything works. Unlike say npm/node/yarn where 50% of the time after coming back to work on a project your node_modules is broken and you have to either reinstall npm or node or node + npm.
The problem we're talking about isn't "installing a dependency", the problem is creating reproducible builds across people and between dev and production. The secondary problem is managing dependencies over the long run (updating, transitive dependencies, etc). Ruby/Bundler does this so much better than requirements.txt by a long shot.
Node's packaging story sucks in many ways, but IMO python's is worse. I spent quite some time working with python when there was no first-party dependency sandboxing (i.e. there was virtualenv but no venv) and even today there are tools that work with one but not the other.
That said, PEPs 517 and 518 sound like they're going in the right direction from the article.
ied ( http://gugel.io/ied/ ) had a great idea for reducing redundant installations in node_modules/, I wish the project got some traction. It worked as advertised when I used it 4 years ago.
Maven is “so good” that people wrote another tool on top (Gradle) so they don’t have to touch the Mavenstrosity. More or less like Maven was built to get rid of that other nightmare, Ant.
Yes, Python’s packaging is partly bad because it’s based on C extensions, which are intrinsically bad, but it’s also just plain bad compared to its peers, like Ruby and NPM.
NPM also has support for C extensions, and they mostly work reliably (the only problems I've had are forgetting to `npm rebuild` when switching between node versions, or when switching between native mac builds and linux-in-docker builds.
In my personal experience, extensions written in C don't necessarily compile at first attempt because the dependency management there is even worse. May be it fails with a missing openssl or readline library and I have to go find whatever OS specific thingy I need to do to recover.
C/C++ have no equivalent of npm/bundler/pip. Instead there are incompatible flavors of Makefiles for incompatible C compilers. It's a hard problem space to work in.
