Looking for a LAN file sync solution
So a thing happened that statistically (in my mind anyway) should never have happened. Three hard drives in my RAID-6 array shit themselves in a very short amount of time, corrupting its NTFS partition and (yesterday) fully trashing the array. Fortunately it wasn't short enough that I lost everything - the third drive took just long enough to give me enough time to salvage some of my important stuff (I'm still sorting through that job).
I'm planning to build up a second array on another computer over the next several months for backup/sync purposes.
I've been looking at some solutions to prevent a similar situation but so far haven't come across . What I'd like is something that will do a file sync similar to Dropbox, but on my local network, and only for a specific set of folders in various locations on each client system. I've considered using Dropbox, but I don't like the idea of having to keep my files in the Dropbox directory structure on my hard drive. Linking directories (Windows-style linking anyway) doesn't seem to work. I'd settle for a decent incremental daily backup solution or one that happens when a computer connects to the network. Every computer in my house is on some flavor of Windows (I have an MSDN license thanks to work), but I'm not opposed to throwing Linux on something that isn't my gaming computer.
And for anyone who might be curious about DIY data recovery, I've been using Restorer Ultimate for years. It's a great piece of software and dirt cheap, especially when compared to the costs of professional recovery. The Pro version does RAID arrays.
Comments
Lots of us here use CrashPlan. It's only $5.00/month for a single computer and it's free if you backup your files to another user. Brian and I have been backing up to each others PCs for a few years now. They offer Windows and Linux versions.
I'm not so sure this is a good solution for me, especially not if another user is involved. My array is 12x3TB drives and it's at about 75% capacity. I'll take a look though.
Crashplan is also free between local pcs. I think it fits what you are looking for.
It's what we use to back things up to the NAS
I stand corrected. Thanks to both of you for the recommendation.
If you wanted to do it manually, Robocopy is a utility on Windows that allows file transfers, and can be thrown into a batch file that's tied to a Task Scheduler event set to happen at, say 3:00 AM every day. Something like:
robocopy \computer1name\c$\Data1 \computer2name\c$\Backups\Data1 /e /mir /np /log:backup_log.txt
Would get the job done.
/e = copy all sub-folders, even empty ones
/mir = mirror (check the files in the destination, and only copy newer files)
/np = no progress counter
/log: = create a logfile
Not a fan of CrashPlan personally. Never worked all that well in my experience (and I hate java apps).
I'd suggest looking at either Syncthing (and it's companion GUI: SyncThing-GTK). It can be a little annoying to set up (you have to exchange keys between the devices that will be syncing), but once it's set up it just works.
Alternately, if you want a more centralized storage with better versioning, take a look at Seafile.
I use both solutions for different things and they both work really well.
Just for certainty's sake, the /mir flag doesn't work only like that - it also has the ability to remove content from your source directories if it's been removed from the target directory. Gotta be careful.
@Snarkasm You do need to be careful with Robocopy, because it does exactly what you tell it to do. It can delete some files and folders from the target while adding others, but it does not modify the source directory. When using Robocopy, I always put the command in a batch file and include the /L [list] and /LOG switches first so it will show me what it is going to do before running it for real.
Any reason not to use Bittorrent Sync? I've been using it for almost two years now to keep files synced between my desktop and laptops. Also, I use it to mirror the pictures folder from all our phones and tablets to our desktop. So far, it's been pretty solid. That said, I still do use Dropbox for other use cases.
I like btsync from a functionality standpoint, but I dislike the fact that they don't seem to be trustworthy when it comes to keeping their word. Also, that they have shown their willingness to strip functionality out of the free product in order to make money.
SyncThing does basically the same thing as btsync but without the commercial entity trying to monetize it and without the reliance on a central service for brokering connections.