> Both `echo -ne 'weird\xffname\0' > list0` and `printf 'weird\xffname\0' > list0` seem to work fine for me on Linux. Is this macOS-specific?

Neither of those create a non-UTF-8 filename. (Both files are named "list0", which is valid UTF-8.) They have non-UTF-8 content, but that's not weird.

But it's not too hard to get a non-UTF-8 filename:

  touch $'\xff'

Both zsh & bash support that syntax.

(You could also use process substitution with printf, but that's more steps than necessary. So, something closer to your example would be,

  touch "$(printf '\xff')"

You can't put a \0 in the filename, as there's no way to pass that string in C.)