I started reading through the Narrowlink URL, but the frequency of typos was frustrating enough the I came right back to HackerNews to read the comments and synopsis of what the kerfuffle is about.
Thanks for the pointer to headscale - will take a look ;)
MAJOR two thumbs up on this one - the pibox is a very cool product, great for learning, easy to deploy on, and powerful enough for smaller tasks. the Kubesail folks are responsive (both in email and on their Discord server). Couldn't be happier!
Another option for open source aficionados might be https://www.pgedge.com - with their spock extension they also achieve multi-master replication. Looks very interesting!
Just to add context - I do have backups, but they're spread across a myriad of different hard drives. I had used Amazon Photos to consolidate, and benefit from the facial recognition to be able to quickly locate pictures of family members quickly.
I think that I'll give DigiKam another go, against the NAS loaded up.
Not that it helps you now, but i also keep all our family photos in the cloud (iCloud in my case), but at the same time i have a small ARM machine at home that keeps a mirror of the iCloud data.
That ARM machine also has the responsibility of making backups, local to a USB drive, as well as to another cloud. Not mirrors, but proper versioned backups (as in Restic, Borg, Arq, Duplicacy, Kopia, etc).
I also maintain a couple of USB drives with yearly updated mirrors of the entire photo library. The drives are stored at geographically different locations, and surface scanned, updated and rotated yearly.
And finally, as a "last ditch recovery", i maintain an archive of M-disc Blu-Ray discs that contain a complete copy of our family photo library. Every year i make an identical set of discs containing the past years photos, and these sets are stored alongside the USB drives.
I don't bother archiving documents as everything that is important is stored on government servers anyway, or exists in hardcopy. Also, if every step in my normal 3-2-1 backup scheme has failed and i need to recover from the archive, i probably have bigger issues than retrieving my budget for this years finances.
As a fellow small ARM machine owner, what's your strategy for getting the photos from iCloud? Is there a tool one shouldn't feel weird to give their iCloud credentials to?
I was doing just this for a while but stopped because my Mac Mini was too old for the last few OS releases. I then switched to using the Windows iCloud client but a bug from ~2 years ago that consumes tremendous CPU cycles made that less than ideal. (The best you can do is lock it to a single thread, which will then use 100% 24/7)
Now I just don't backup my iCloud, though I do remove everything older than one year every new years to my home server which follows a good 3-2-1 backup strategy.
TL;DR: If you go this route, try to get a Mac Mini that can run a supported macOS for some time to come.
I've been using Windows iCloud client to backup to a VPS. It works okay for files, but for photos it pegged disk usage even when there weren't any new photos to download and my VPS provider wasn't happy. So far my solution is Windows iCloud client for files, and then OneDrive on my iPhone for photo backup, with OneDrive again on the VPS.
I like how the backup is outside my house, but I'm about to add Yubikey to my iCloud account and I'm not sure the Windows iCloud client is going to like that.
I was actually running ESXi on my Mini for a while and successfully installed macOS in a VM on it. The performance was horrendous though, so much of macOS depends on GPU acceleration which I didn't get. I think I've read newer macOS builds don't even have a software video fallback, though that might just be the Apple Silicon builds which wouldn't apply to me.
It was definitely a fun project even if not terribly useful.
It doesn’t talk about not being able to access person tags. Not being able to programmatically access the data about who google thinks is in each of my photos has been an annoying pain point for me for years. Last I checked, the data is also not included in Google Takeout dumps.
i have been using icloudpd for years. there is a docker image that makes it simple to install on my synology. because of 2fa, i have to re-authenticate it from the terminal every couple of months, but that takes one minute.
I tried this but I think my issue is the library is too large. It can pull a few dozen or a hundred or so photos just fine, but then it will time out but not quit out, so I have to babysit the process. Maybe there's a flag I missed in the documentation to retry downloads, but it basically was a nonstarter to me in the state it was in. I think its apples fault though; I can't get a big zip file of icloud data to succesfully download with their website either, it will also time out. Likewise when I try and update the bootcamp drivers on my intel bootcamp machine, I will get 30% of the way there and then it times out. I'd blame my home connection, but from cursory glances at various forums this is apparently a widespread issue with these sort of downloads form Apple.
I have recently switched to a M1 Mac Mini, and just have each family member sign in to that using Remote Desktop. It brings the added bonus of working as a content cache for anything iCloud.
My only gripe is that it downloads the shared photo album (new in iOS 16) once for each account, and when your photo library is 1.8TB, that suddenly becomes a lot of wasted space. When it comes to backing it up the backup software deduplicates the data, but not for the initial storage.
I really wish Apple would implement some kind of method for backing up photos stored in the cloud without the need for mirroring them.
Before the M1 I was using iCloud photo downloader ( https://github.com/icloud-photos-downloader/icloud_photos_do... ) on a Raspberry Pi 4 which also worked well, but in the end I got tired of iCloud credentials expiring every ~90 days, requiring each family member to login again through a console.
Considering the M1 idles at roughly 20% more than a RPi4 (M1 at 4.5W) it was an easy sell. I just got the cheapest model and added a large USB drive. Using a Mac also gives you the possibility of using something like Backblaze Personal with unlimited backup storage, if that’s your thing :-)
I use Healthchecks.IO ( https://healthchecks.io/ ) to keep an “eye” on the backup status (and other more mundane tasks like monitoring the power state of my summerhouse)
You have to fuck around with the storage format because by default scanning sofware, even when scanning documents, will basically just take poorly compressed high resolution images. Maybe there's some service which sorts that out for you but so far I have a little webapp which stores an index of paper copies. Once I figure out how to automate scanning, compressing and OCRing documents then I think it will be worth storing digital copies. But for now it works.
I do scan some paper but there's increasingly less of it and I'm likely to need future access to so little of it that it's mostly not worth the trouble.
I still do a 3-2-1 backup of documents with 2 versioned backups, one at home and one in another cloud provider, just like with photos.
The archive however is the recovery if I’m not able to retrieve my normal cloud copy (hacked, ransomware, loss of credentials, etc), I cannot access my local mirror copy (ransomware, dead disk, etc), I cannot access my local backup (dead disk, separate from the mirror disk), and I cannot access my cloud backup either.
For all of those things to go wrong a the same time, something major has to happen. Besides, where I live, most required documents (drivers license, passport, birth certificate, tax records, etc) exists in government databases, so all I have at home will be various documents that maybe have sentimental value, but not exactly needed.
Furthermore, documents change “frequently” where photos tend to be somewhat more static, so I can archive photos, and maybe get <10% “duplicates” due to later edits, archiving documents will pretty much be a lot of duplicates each year.
That being said, I think we have like 1GB documents in total, so it would be easy to fit in the archive.
I do occasionally want to retrieve older documents. As you say, they're smaller and the effort to put individual docs into a "I might want this someday" folder seems more trouble than it's worth. Of course, having a vast sea of basically write-only docs does make finding things harder especially as I don't put as much effort into organizing things any longer.
> I think that I'll give DigiKam another go, against the NAS loaded up.
I'm also looking for something FOSS that can do basic face recognition + maybe even more, but last time I checked DigiKam's detection didn't work so well, or maybe I got spoiled by the detection in Google Photos. If you do give it a go, would be nice if you reported back on your experience :)
Not sure if it has been mentioned already but photoprism added face recognition a few releases ago and it is working well for me: https://github.com/photoprism/photoprism
The quality of this software is overall extremely good. It is a solo-developer as well so you might want to consider sponsoring them if you end up using it.
Seconding photoprism. The face recognition was pretty meh when it was first introduced, more of a gimmick than anything. But (I assume) a recent update pushed it into "wow, this is actually really good" territory.
Thirding photoprism, got it hosted on a raspberry pi and made it accessible outside of my home via Tailscale. Seems to work fairly well... though I could probably make it faster with an SSD vs an HDD over USB.
To photoprism users: do you know how well it handles identifying photos of kids over the span of years?
Google photos absolutely blows me away with its ~1% false positive rate (that is, given a photo, correctly identifying with kid(s) are in it) identifying each of my kids, nieces, and nephews. I don’t really know what its false negative rate (a false negative would be an uploaded photo of John but that doesn’t get tagged as having John in it) is, though.
I can search for each of my kids and see photos of them going back to their birth.
Photoprism is absolutely amazing. It was the quickest donation I’ve ever made to an OSS project.
I have a Photoprism instance running on my home server backed by a raidz1 ZFS pool (1st backup). Photos are periodically synced to a Backblaze bucket with versioning enabled (2nd backup). The source of most photos is an iOS device with iCloud enabled (main copy). I rely on PhotoSync to periodically sync from the iOS device to Photoprism.
What I found in my previous search is that either they're really good at managing large collections of photos, or they're really good at doing various AI/recognition, but none of what I've found have been good at both.
Hence my interest in DigiKam as their management features for large collections is second to none.
Isn't that OK? You should be able to find some reasonable workflow around using the recognizer to tag the photo in its EXIF/IPTC metadata, and then load that into the organizer.
It's not not OK :) Just not what I'm looking for, I spend enough time professionally with writing glue code, that I'm looking for a good out-of-the-box experience for something like this.
There's a difference between face detection and face recognition. Last I checked (less than a year ago) it could find faces, but was abysmal at recognizing them. I spend a few days on it, tweaking this and that, taking to the devs. It was crap.
I would be careful with hard drives. As I learned couple of days ago when I got my 10 year old hard drive from storage, I was unable to read most of the data as the drive would start to intermittently disconnect from the computer.
Most likely the drive is okay, but the board has developed an issue - maybe capacitors gone bad or something.
Now, I am yet to check if it is the one that has hardware encryption (I supposed to get rid of all of them and copy data to alternative drives) - there are drives that encrypt the data by default and if you don't set up the key, the encryption still takes place with some default key. If the board dies for some reason, then you won't ever be able to decrypt the data - even if technically was unencrypted, even by swapping the board from a working hard drive. This happened to me once and I lost 3TB of important data couple of years ago.
That being said - I think the best additional backup is to store important things on BluRay - producers estimate they should last 80-100 years and so far never had a bad disk, even those burned years ago.
Latest release failed the tests (with some memory corruption yikes) even. Doesn't inspire a lot of confidence that it won't eat your photos.
But the v7 uses DNN from OpenCV for face detection that supposedly has 91% accuracy on CPLFW which is pretty good! So how come people here say it sucks?
I'm away from my desktop computer, but it wasn't half bad when I last updated its DB. I don't take much portraits, so I don't use that feature much, but give it a look when I can and update you.
I'm glad you haven't really 'lost' the photos since you have copies elsewhere. It is surprising how many people these days rely on some cloud service to store and preserve their only copy of some important and unreproducible data.
You are right about file systems doing a lousy job of helping you organize and locate photos, especially when they are spread across many different HDD and SDD drives attached to various computers. They are equally bad at organizing other forms of content (documents, videos, logs, music, software, etc.)
File systems were invented decades ago when the biggest hard drives could only hold a few thousand files at most, and drives were also so expensive that most people only had one of them. Traditional file systems are antiquated and need to be replaced with something better.
Self-host and use Photoview [1], it has face recognition. I use this since 2 years with 5TB of photos (on ZFS) and it works flawlessly. It is worth it - everything under your control and you decide when to migrate. My phone's photos are also uploaded automatically to a Nextcloud folder, which is read-mounted to Photoview and scanned automatically.
* A nextcloud+recognize (for face recog) instance running on Hetzner. $20/month (I co-host other services on the VM). This is attached to a Hetzner storage box.
* Syncthing pushes all of nextcloud to a zfs.rent (great service BTW) machine, I purchased my disks up front, so it's $cheap/month. ZFS snapshots are taken.
* Local RPi NAS. Syncthing up to the other two.
About $40/month in total. How valuable are 240k photos to you?
Uploading photos is a matter of sending them to Nextcloud. Syncthing does the rest.
I'm curious if you've asked your family members whether they're ok with having photos of themselves being used to train Amazon's facial recognition model? I know it's a common practice and Amazon isn't alone, yet I feel people should be more aware of what's happening.
Digikam is less crashy than it used to be. Its slowly going through my 650,000+ photos and finding faces. I started this past Sunday. I expect to be done in a week or two.
I'm hoping that it's just a stale cache or microservice that hit an issue building its view of the metadata or something. They use S3 to store your data (according to this blog[0]) so my fingers are crossed that the data is still there, they just need to get an engineer onto it.
- Synology NAS is accessible on LAN & phones sync photos when at home
- Nightly rsync from internal NAS storage to 1 external (non-raid) hard drive
The nightly rsync does not delete files, so the external hard drive is pretty close to an off-site backup even though it's still attached. The only better thing I've thought of doing is mounting it ro after backups + re-mounting before backups.
At the end of the day we get 3 copies of data: the photo on your phone, the photo on the shared NAS, and the photo on an external hard drive that spends 99% of its life spun down.
IMO the biggest remaining risk to this is a house fire at night while you are at home. You wake up to alarms and smoke and run out the door without your phone. All three copies of your photos will burn at once. This is where an additional off-site would be valuable. That might be a second external hard drive that you store off-site and re-sync every few months, or maybe an encrypted cloud backup using restic.
I think that as somebody's already commented - having people send email to both legal@1password.com as well as to support@1password.com may be the best option here.
This represents an involuntary migration of customers who purchased a license to 1Password6 to REQUIRE them to upgrade to 1Passwords SaaS offering -- evidently through no fault of 1Password, since it's actually on Google for changing how the signing of the binaries changed.
I've added my email to the list of messages sent over to 1password.
DON'T BE RUDE in your email to them - I'm sure that they're frustrated with the change as well, but you also have to admit that they have a business to run.
a price break, and this unfortunate turn of events, might just be that final tipping point where I finally upgrade to get features that I don't want. :(
Can anybody reveal WHERE to buy the standalone / non-subscription version of 1Password? It seems like it's vanished off their site (it used to be that if you hunted for long enough, you could find it)
Thanks Apple