I highly recommend Poetry as a way to manage Python projects. It's my favorite bit of software I've had the pleasure of using, period; if any Poetry devs see this, kudos.

However, with pip's native freeze and install -r abilities, I'm also curious what you mean by hunter gatherer style installs. I've not yet had a problem with installing Python modules.

What are "hunter-gatherer-style installs?"

I take that as "keep running pip install on yet another discovered dep until the script works."

And heaven help you if you accidentally get conflicting versions installed in the same venv. (By conflicting, I mean two libraries that both depend on different versions of the same underlying library. Odds of this increase dramatically the more you use.)

Neither of these problems have happened to me since pip introduced their new dependency resolver (I think sometime during 2020). All dependencies are installed, and packages in the default repo usually have consistent dependency lists.

It's another story for conda, which flat-out refused to install my list of around 10 packages (that seems to stems from the fact that most packages are in conda-forge, and requirements are often inconsistent with the default repo).

Pip now checks this, but you usually use something like Poetry to give you a concrete lockfile with the correct transient dependencies locked in.

My "heaven help you" was more that, if you are in that situation, you don't really have an exit strategy in python. Do you?

yeah people are often lazy about adding a requirements.txt

Use shiv: http://shiv.readthedocs.io

Installs are now an scp.

Now of course, you gotta remember that installing something is usually more than copying the code, and there is nothing to help you with that, in any language.

Welcome to the world of deployment, which is why things like ansible and docker exist.

Is it really a problem of Python itself rather than project developers being too lazy to do proper package management?

Yes. If you are going to get predictable, consistent practices for this you need a more or less official way of doing this or projects will just end up choosing one of several ways of doing this. Or worse: they will invent their own.

And this is demonstrated in the thread: people start listing ways you can streamline installs and try to avoid version conflicts. And there are, as you can see, several. Some don't even anticipate what kinds of problems the user will encounter with versioning conflicts, old Python versions coming with the OS etc.

What is a bit disheartening is that people actually thought they were providing helpful suggestions for solutions when what they did was only to prove my point. But they won't necessarily see it that way.

I'm not sure one can really end up with one true way of packaging per language - sooner or later someone will think "this is too complicated, I can do it better and simpler!" and you end up with another way of software distribution for that language (which is just as complicated as the previous one, once the author actually notices all the usecases).

This is why I prefer using the target OS dependent package management - there is usually just a single one such official system, which is also language agnostic and much more robust than the various per language kludge.

It is often more subtle. For instance, for languages that compile to binaries, you have already solved about 95% of the problem. The remaining is mostly things like static vs dynamic linking and how you actually get the binary from a distribution point into some directory on a hard-drive (which is, in part, a separate problem domain - except when it isn't :-)).

Then you have languages that are "half way there", like Java. I suppose there are still people who make what I refer to as "splat style projects", where you unpack some horror-zip containing thousands of files and litter your surroundings with JAR files, config files and other detritus. But you can build everything into a single JAR file that contains all the dependencies and can be run without any other requirement than a sufficiently new Java runtime. And when shown how to do this, most people tend to adopt it as their default build product.

Languages like Python don't have a well-established build product that is self-contained. This places an undue burden on those who package software for distribution. It forces them to involve themselves in concerns that should be contained within the project - not leak out onto everyone's floor and potentially cause accidents.

Packagers have to make sure that "all arms and legs are inside the vehicle" for all possible permutations of system configurations. Which is easy when you have a statically linked binary. Less so when the software in question is more like a cranky, writhing toddler.

What really made the problem visible for me was when I worked in an organization where suddenly the place was filled with "data scientists". Mostly math or statistics people. Or more precisely: when more non-engineers started writing code and proved to be not only unable, but unwilling, to learn sufficient software engineering to ensure other people than themselves could actually run the code they wrote.

Which was kind of unfortunate because researchers who were unable to reproduce their own computations was a _regular_ occurrence. They simply couldn't figure out how to get their old code to run on their new computer, for instance.

And it isn't because they are jerks. It is because they use tools to get work done. They are not as interested in the tools as the people who make those tools are. To them the rest is noise that wastes their time.

This highlighted the fact that you shouldn't have to be a software engineer in order to produce programs that are easy to distribute - it has to be part of the path of least resistance.

The fact that Python says "not my problem" isn't helpful.

(I think there are interesting lessons to learn from Go. There are a lot of things I don't like about Go, but almost every instance where the language developers put their foot down and said "this is how we do it" actually made things better for everyone. It might not be your favorite way of doing things, but someone has made a choice - which is better than "I don't know...do whatever you like")

Good point about scientists being quite terrible and software mainteinability - I've encountered it quite a few times myself and from what I've heard its not much better elsewhere.

I blame the publish-or-perish mechanics and grant approval process not actually motivating the participants in any way to write maintainable or reusable code. Like, I don't say they should make it good enough for industry to consume, but at least for the followup scientist to build in the work to run!

But thinking about it, even in the computer science course I studied there was hardly any emphasis on software maintenance, cove versioning or even how to collaborate with others - all non technical courses were basically about being and analyst and planning how to write a project, with zero interaction with others.

As for self contained deployment mechanisms - I do agree that effectively packing everything into a single static compiled binary like AFAIK Go and Rust effectively do has a lot of benefits. But comming from the distro background (Fedora) it also terrifies my quite a bit!

With dynamic linking you can patch CVEs by rebuilding the system library everyone uses (and not just encryption libraries can get CVEs!), you can recompile shared libraries with hardening flags and you actually know if something no longer builds from source next time rebuilding one of the parts fails.

Compared to that a developer provided massive static binary is quite a significant black box that can have multiple unpatched CVEs, compiled in fixed version of patched libraries or even non-publicly available code (meaning the binary can't be in-depndently rebuilt from source).

This scenario kinda describes a fully open source project - I guess for proprietary stuff a lot of these downsides does not really apply though out of necessity.

I see your point about using shared libraries to fix problems in multiple applications at once, but there is a flip side to that: what you're running isn't what you built so you may not be able to predict what your software will do when installed on different users computers.

That's true - but I'm not sure one can always 100% assure the environment is the same - or even should!

Ideally one should make the software as robust as possible so that it runs with high probability even in unforeseen environments - and if it fails, it should do that noticeably so that the developers will learn about it and can fix it.

It pretty much works like this in Fedora - lot of the various API breakages and regressions show up when the bits and pieces first land in the rolling distro called Rawhide, but thats fine as hardly anyone runs Rawhide as a daily driver. By the time the next Fedora release is branched from Rawhide and goes via Beta and RC all these issues are ironed out and the end result is pretty stable yet very up to date.

It's usually all the proprietary or heavy bundling software that have the biggest problems with dependency updates as they don't go with the flow or avoid updating the dependency for so long to end up with an insane jump to do.

I agree that there is little to be gained from requiring perfection, I think there is considerable value in trying to do everything that is within reason. And there are a lot of reasonable things one can do to be nice to users. Allowing the user to install a piece of software without having to alter their system at all is a good thing.

And this brings us back to where this discussion started: it takes real effort to write non-trivial Python code that will run just by installing the program itself. The path of least resistance is to dump all the problems in the lap of the user, which many Python programmers tend to do in practice. That's not very nice to the consumer.

I think Python could benefit from developing empathy with the consumer.

Well, we have both Pipenv and Poetry now. It's a just a matter of people using them.

You are kind of proving my point there.