About Archive Tags RSS Feed

 

Random tools I would use - and pictures

9 February 2011 21:50

So previously I talked about pictures. Having spent a while playing with some new lighting ideas I'll present three recent images:

Now that's out of the way lets talk software. There are two specific things I think would be useful to me right now:

Opportunistic "cron"

I've got a few jobs that don't take long to run, and I schedule for once a day when my systems are idle.

It'd be nice to have some long-running daemon which could trigger these odd-jobs (which do things like update my list of mutt mailboxes) when the system has been idle for >30 minutes, or some similar criterion.

Ideally I'd say "Run this job when the system is idle, but if the system is never idle run it once a day, or once an hour regardless".

Content-aware tracking software

I use rsync to archive images across a number of systems.

Imagine I rename a few images locally, I have to re-rsync even though ideally all I need to do is rename the contents remotely. The issue is remembering what I've changed.

So given:

~/Images/Stolen-Souls/2011.02/
~/Images/Stolen-Souls/2011.02/02-Borcsa
~/Images/Stolen-Souls/2011.02/02-Camilla
~/Images/Stolen-Souls/2011.02/06-Indigo
~/Images/Stolen-Souls/2011.02/07-Rosa

If I rename "06-Indigo" then it's trivial to ssh to the remote host and do the same job. But what if I rename a bunch of individual files?

Ideally I'd be able to run something like:

~$ tool --snapshot ~/Images/
~$ mv Images/old.jpg Images/new.jpg
~$ tool --show-changes ~/Images
mv Images/old.jpg Images/new.jpg

I half believe git can do something like this, but it seems like you should be able to index the filename+SHA1+size to ~/.snapshot, then later do the same thing and output a list of "mv" commands.

Anyway; I'm drunk. I'm probably not making sense, and I need to sleep. Goodnight.

ObQuote: "Hello Mr. Svenning how have you been?" - Mallrats

Tags: software which should exist, stolen-souls | 13 comments

 

Comments on this entry

icon Claudius at 22:25 on 9 February 2011
http://chubig.net

I would assume that »rsync --fuzzy« does something at least similiar to what you want to achieve with ‘content-aware tracking software’, however, I doubt that it’ll work fine with large/widespread directories and never tested it myself.

icon Anonymous at 22:53 on 9 February 2011

Sounds like you want git-annex.

icon Foo at 23:14 on 9 February 2011

Fcron might be what you are searching for - unfortunately upstream seems dead and the debian package is in really bad shape (lots of bugs including security issues). Maybe you want to have a look at it?

icon Koterpillar at 23:24 on 9 February 2011

For the "Content-aware tracking software", try unison.

icon RoboTux at 00:52 on 10 February 2011

I think unison can detect renames in files. Often in synchronization it does there is some fast copy from a local file. I think it does that when it detect a rename to avoid any network transfer.

icon gregoa at 01:03 on 10 February 2011

I like unison for syncing between hosts, and it seems quite good at handling this renaming stuff (by "shortcutting" to already available files with using some hashes).

icon Jason Weathered at 05:05 on 10 February 2011
http://jasoncodes.com/

`rsync --archive --fuzzy --delete-after` may be good enough for what you need. It will pick up any rename in the same directory if the content hasn't changed.

If the content has changed a bit (i.e. metadata) as well as a rename, it's still usually pretty good. The man file says: "The current algorithm looks in the same directory as the destination file for either a file that has an identical size and modified-time, or a similarly-named file."

icon foo at 05:19 on 10 February 2011

For the second one you want git-annex by the awesome Joey Hess.

icon Anonymous at 06:31 on 10 February 2011

Opportunistic "cron": Did you consider atd's batch command? You could add batch jobs (run once by atd when the system is idle) using cron. Not a perfect solution, but it might work.

Content-aware tracking software Did you consider Joey Hess' git-annex?

icon Anonymous at 06:31 on 10 February 2011

Opportunistic "cron": Did you consider atd's batch command? You could add batch jobs (run once by atd when the system is idle) using cron. Not a perfect solution, but it might work.

Content-aware tracking software Did you consider Joey Hess' git-annex?

icon Simon Richter at 06:47 on 10 February 2011

rsync has an option for that: "-y" or "--fuzzy", to make it look for "similar" files that the recipient already has. That usually works quite well for simple renames.

icon mp at 08:25 on 10 February 2011

Tool 1: fcron seems to almost allow it with its load average options

icon Steve Kemp at 08:37 on 10 February 2011
http://www.steve.org.uk/

Wow thanks for all the overnight comments.

Unison I think I don't need, and rsync's fuzzy option isn't sufficient for the general case, as I frequently rename directories and move contents around. From the help :

"The current algorithm looks in the same directory as the destination file .."

On the other hand git-annex looks ideal for me :)

 

Created by ephemeris.

© Steve Kemp