Here is a simple workflow with mutable systems like Fedora that I think a lot of people are missing. AI could be brought into this workflow also for those who want that:
(1) Take a snapshot of your current system (snapper+btrfs on Linux, bectl on FreeBSD+ZFS)
(2) Make destructive changes like install a new windows manager, some drivers etc.
(3) If everything worked out well, continue
(4) If something failed badly, restore from (1) using the snapshot restore -- Your system is as good as before
This workflow replicates many of the benefits of NixOS without the complex nix scripting that can be often needed.
Of course, a declarative and textual rendition of the configuration is better than bash commands entered on the command line but sometimes you don't need that level of precision.
A snapshot of your build folder. Not even the sources. This is my other problem with mainstream Distros. Extending them is completely opaque. NixOS is source based and anything and everything can be updated by the user. Need some patch from kernel ML? 1 line of code. Need a Bugfix in your IDE that hasn't landed in a release? 1 line of code.
There is no distinction between package maintainers and end users. They have the same power.
In the meantime i dont expect Debian users to ever write a package themselves or to modify one.
Arch Linux also has a long history of people writing their own package specs (AUR) and is relatively simple too of course.
Let me put it differently. The documentation of NixOS treats package maintainers and users as kind of equal.
This has benefits and downsides. Benefit is that everyone is treated as a power user. Downside is that power users are horrible at writing docs and this philosophy is my main theory why NixOS docs are so .... Bad
Fedora (and RHEL) end user and developer docs are written for quite different audiences
Yes I just replied to your other comment with the same observation. It reminds me of an article by Paul Graham, I forget which, who expressed the difficulty of explaining to programmers who lack an abstraction just how good the abstraction is. Anything you can do with NixOS, you can do with any distribution, because it isn't magic. But somehow, more stuff becomes possible because it gives you a better way to think.
(As for why the docs are so bad, I think it's because of the lack of good canonical documentation. There's too many copies of it. Search engines ignore the canonical version because it's presented as one giant document. Parts of the system aren't documented at all and you have to work out what you've got by reading the code. The result is that you have no idea what to do if you want to improve the situation - it seems like your best option is to create new documentation. And now you have the same basic level of documentation that didn't help the first hundred times it was rewritten. And I don't really think submitting a PR to nixpkgs is exactly userfriendly, so it probably discourages people from doing the "I'm just trying to understand this, so I'll fix up the documentation as I learn something" thing.)
yes i think you've hit the nail on the head. I tend to view NixOS not as a distribution, but as a distribution framework. The system configuration is the sources for an immutable distribution as much as it as system configuration.
You're in no way bound by decisions of the nixpkgs contributors: as you say, we can add a patch. Or we can also decide we totally disapprove of the way they've configured such-and-such a service and write our own systemd service to run it.
Anyone can write a local debian package which adds a patch, and build and install it. And anyone can write a systemd service and use it instead of the distribution's systemd service. But on NixOS, these are equal to the rest of the system rather than outside it. Nixpkgs is just a library which your configuration uses to build a system.
But note that I did caveat my suggestion: "Of course, a declarative and textual rendition of the configuration is better than bash commands entered on the command line but sometimes you don't need that level of precision."
That’s a great way to get one of the benefits of nix. But you still can check that snapshot into version control, share it with all your machines, etc.
You're right ... you cant check that snapshot into version control and share with your machines etc. When you need that level of control and need to scale your configuration to other machines NixOS sounds like the right choice. If it's for your own machine and you just want to try out a new windows manager non-destructively use snapshots.
Fedora also offers immutable distros which are (I've heard) much more user-friendly than Nix. Sure you can make a hacky pseudo-immutable workflow on a mutable distro but that's literally more effort for a worse result.
> There is also community-maintained support for FreeBSD, though I have not used it personally
I have tried to use the nix package manager on FreeBSD recently. I tried doing some basic things without success. Seems quite broken and unusable, which is a pity because nix on macOS seems decent. FreeBSD is much closer to Linux so there is no technical reason why nix can't be a success on FreeBSD.
nix on FreeBSD just needs more contributors to fix bugs and make popular packages work ! I wonder if it will ever happen. FreeBSD is niche and nix is somewhat niche (still). It's a double niche problem !
+1, Guix is quite good with some tricks up it's sleeve compared to Nix.
I am not a fan of S-expressions but using scheme is more reasonable than nix+bash to me.
On the negative side, guix can be slow. It is also not a very pragmatic os. NixOS does non-free firmware and drivers without issue. You need to jump through some hoops for this with Guix. This is not an issue if you plan to run guix in a VM though.
Excellently explained writeup. Kudos on explaining the shockingly multiple kernel bugs in a (a) simple (b) interesting way.
TL;DR the main issue arises because the context switch and sampling event both need to be written to the `ringBuffer` eBPF map. sampling event lock needs to be taken in an NMI which is by definition non-maskable. This leads to lock contention and recursive locks etc as explained when context switch handler tries to do the same thing.
Why not have context switches write to ringBuffer1 and sampling events write to ringBuffer2 (i.e. use different ringBuffers). This way buggy kernels should work properly too !?
> Why not have context switches write to ringBuffer1 and sampling events write to ringBuffer2 (i.e. use different ringBuffers)
That would work, but at the cost of doubling memory usage, since you then have two fixed-size ring buffers instead of one. Also, in our particular cases, the correct ordering of events is important, which is ~automatic with a single ring buffer, but gets much trickier with two.
> This way buggy kernels should work properly too !?
We have a workaround for older/buggy kernels in place. We simply guard against same-CPU recursion by maintaining per-CPU state that indicates whether a given CPU is currently in the process of adding data to the ring buffer. If that state is set, we discard events, which prevents the recursion too.
> There is one advantage of Linux over FreeBSD, which is not widely known. Linux has a huge database of known bugs in various peripheral devices, including Ethernet NICs, and when one of those is recognized it applies workarounds for the bugs.
Almost universal hardware support and workarounds for quirks is precisely the reason why everyone uses Linux even if they might want to use FreeBSD or something else. In other words, this advantage is known and is the reason why Linux is dominant in the server space today.
It is widely known that Linux has "better" hardware support for many peripheral devices, but few know what "better" means, i.e. that a lot of devices that you can buy have various bugs and in most cases their vendors do not document the bugs in any way, because they provide Windows drivers that contain workarounds for the bugs, and they do not care about other operating systems.
The users of other operating systems must discover the bugs and how to handle them by reverse engineering, and here Linux has the advantage of a much greater user base than any alternative, so in most cases the bugs will be identified by some Linux user, and then either the user or the maintainers of the corresponding Linux subsystem will implement a workaround for the bug.
Usually workarounds/quirks need the highest level understanding of the device driver and the device. There are few people in the world with such mastery and fewer people capable of translating this to another operating system. In concept this is simple but in most cases it is difficult.
Why not create a boot environment and try out FreeBSD 15 with pkgbase ? If the experiment is successful just make the boot environment the default otherwise throw it away...
With such powerful tools I find it fascinating that FreeBSD users are not more willing to experiment !
Well, all those years ago, my testbed for installing and trying out FreeBSD was a 486 with 8MB of memory. That was a heck of a machine compared to the ones BSD grew up on, and it ran great. No GUI on that setup of course, but all the Unixy stuff... vi editor, C toolchain, NFS etc.
I don't know what's the minimum system to run no-GUI mainline Linux on these days. I'm sure BSD has gotten bloated too, but I'll bet not as much.
On openSUSE Tumbleweed, it is. Each Upgrade creates two snapshots, one before, one after, and if anything goes wrong, I can boot into a snapshot where the world was still in order.
I have a higher opinion of ZFS than I do of btrfs, but FWIW snapper+btrfs has worked well for me on openSUSE Tumbleweed for ten years now, too.
Last I heard (~8 years ago), the RAID-like functionality in btrfs was very unstable and crash-prone. The impression I got was that there was not a lot of interest in fixing this. Then bcachefs came and ... appears to have gone nowhere AFAICT.
The non-RAID part of btrfs appears to be stable. It's the default filesystem on openSUSE and SLES. But I don't think it's ever going to reach feature parity with ZFS.
And to the grandparent post's point, since the split with the kernel there've been two big new feature releases: reconcile, which puts our data and drive management head and shoulders above other filesystems - and erasure coding was just released, 1.37 came out a few days ago.
btrfs is suffering from a lot of old bad publicity and some poor design decisions around RAID.
But by now it is a great file system if you don't go near RAID5/6. btrfs has its flaws (ZFS has its own flaws!). However:
- It's used a lot, especially by facebook and Redhat (on fedora)
- Gets a lot of testing
- Sees a lot of bug fixes
- Has a lot of features
I haven't read btrfs code but given that it is a popular file system and Linux code quality tends to be good in popular subsystems I would hesitate to say its code quality is worse than ZFS in any way.
> but real life workload are not "Geometric Mean".
A good benchmark suite consists of good benchmarks chosen carefully. These benchmarks are not chosen randomly. They represent diverse ways to "stress" or exercise the system. Real life workloads are indeed closer to "Geometric Mean" of various benchmarks by definition because real life workloads are diverse. Not everything would be like sqlite3 which is single pattern of file system usage.
Geekbench, Cinebench, 3DMark etc. are all averages or geometric means of various benchmarks also.
> When I use firefox, sqlite performance matter more than any random benchmark.
You've selected a single benchmark (sqlite) and said it's so important to you that it overrides everything else when you are comparing ZFS vs btrfs.
If you feel that a single benchmark like sqlite is good enough then that is fine -- your decision. I am hesitant to do the same and prefer geometric mean.
Be specific. Why do you need LVM? What for, what do you do with it?
Secondly: are you aware that ZFS includes what LVM does on Linux, and so you don't need a separate tool for it? This makes the comparison tricky but it's important to consider.
I am impressed by the achievements of the Zig team. I use the ghostty terminal emulator regularly -- it is built in Zig and it is super stable. It is a fantastic piece of software.
This makes me feel that the underlying technology behind Zig is solid.
But I prefer Rust over Zig. The main difference is Rust chooses a "closed world" model while Zig chooses an "open world" model: in Rust, you must explicitly implement a trait while in Zig as long as the shape fits, or the `.` on a structure member exists (for whichever type you pass in), it will work (I don't use Zig so pardon hand wavy description).
This gives Zig very powerful meta programming abilities but is a pain because you don't know what kind of type "shapes" will be used in a particular piece of code. Zig is similar to C++ templates in some respects.
This has a ripple effect everywhere. Rust generated documentation is very rich and explicit about what functions a structure supports (as each trait is explicitly enrolled and implemented). In Zig the dynamic nature of the code becomes a problem with autocomplete, documentation, LSP support, ...
> But I prefer Rust over Zig. The main difference is Rust chooses a "closed world" model while Zig chooses an "open world" model: in Rust, you must explicitly implement a trait while in Zig as long as the shape fits, or the `.` on a structure member exists (for whichever type you pass in), it will work (I don't use Zig so pardon hand wavy description).
Do you happen to have a more specific example by any chance? I’d be interested in what this looks like in practice, because what you described sounds a bit like Go interfaces and from my understanding of Zig, there’s no direct equivalent to it, other than variations of fieldParentPtr.
They are referring to `anytype`, which is a comptime construct telling the compiler that the parameter can be of any type and as long as the code compiles with the given value, it's good.
It's an extremely useful thing, but unconstrained, it's essentially duck typing during compile time. People has been wanting some kind of trait/interface support to constrain it, but it's unlikely to happen.
Conceptually it's quite similar to how C++ templates work (not including C++20 concepts, which is the kind of constraining you're talking about).
I quite like it when writing C++ code. Makes it dead easy to write code like `min` that works for any type in a generic way. It is, however, arguably the main culprit behind C++s terrible compiler-errors, because you'll have standard library functions which have like a stack of fifteen generic calls, and it fails really deeply on some obscure inner thing which has some kind of type requirement, and it's really hard to trace back what you actually did wrong.
In my (quite limited) experience, Zig largely avoids this by having a MUCH simpler type system than C++, and the standard library written by a sane person. Zig seems "best of both worlds" in this regard.
(1) Take a snapshot of your current system (snapper+btrfs on Linux, bectl on FreeBSD+ZFS)
(2) Make destructive changes like install a new windows manager, some drivers etc.
(3) If everything worked out well, continue
(4) If something failed badly, restore from (1) using the snapshot restore -- Your system is as good as before
This workflow replicates many of the benefits of NixOS without the complex nix scripting that can be often needed.
Of course, a declarative and textual rendition of the configuration is better than bash commands entered on the command line but sometimes you don't need that level of precision.
reply