Random tools I would use - and pictures

Wednesday, 9 February 2011

So previously I talked about pictures. Having spent a while playing with some new lighting ideas I'll present three recent images:

Now that's out of the way lets talk software. There are two specific things I think would be useful to me right now:

Opportunistic "cron"

I've got a few jobs that don't take long to run, and I schedule for once a day when my systems are idle.

It'd be nice to have some long-running daemon which could trigger these odd-jobs (which do things like update my list of mutt mailboxes) when the system has been idle for >30 minutes, or some similar criterion.

Ideally I'd say "Run this job when the system is idle, but if the system is never idle run it once a day, or once an hour regardless".

Content-aware tracking software

I use rsync to archive images across a number of systems.

Imagine I rename a few images locally, I have to re-rsync even though ideally all I need to do is rename the contents remotely. The issue is remembering what I've changed.

So given:

~/Images/Stolen-Souls/2011.02/
~/Images/Stolen-Souls/2011.02/02-Borcsa
~/Images/Stolen-Souls/2011.02/02-Camilla
~/Images/Stolen-Souls/2011.02/06-Indigo
~/Images/Stolen-Souls/2011.02/07-Rosa

If I rename "06-Indigo" then it's trivial to ssh to the remote host and do the same job. But what if I rename a bunch of individual files?

Ideally I'd be able to run something like:

~$ tool --snapshot ~/Images/
~$ mv Images/old.jpg Images/new.jpg
~$ tool --show-changes ~/Images
mv Images/old.jpg Images/new.jpg

I half believe git can do something like this, but it seems like you should be able to index the filename+SHA1+size to ~/.snapshot, then later do the same thing and output a list of "mv" commands.

Anyway; I'm drunk. I'm probably not making sense, and I need to sleep. Goodnight.

ObQuote: "Hello Mr. Svenning how have you been?" - Mallrats

| 13 comments.

 

Comments On This Entry

[gravitar] Claudius

Submitted at 22:25:28 on 9 february 2011

I would assume that »rsync --fuzzy« does something at least similiar to what you want to achieve with ‘content-aware tracking software’, however, I doubt that it’ll work fine with large/widespread directories and never tested it myself.

[gravitar] Anonymous

Submitted at 22:53:52 on 9 february 2011

Sounds like you want git-annex.

[gravitar] Foo

Submitted at 23:14:34 on 9 february 2011

Fcron might be what you are searching for - unfortunately upstream seems dead and the debian package is in really bad shape (lots of bugs including security issues). Maybe you want to have a look at it?

[gravitar] Koterpillar

Submitted at 23:24:02 on 9 february 2011

For the "Content-aware tracking software", try unison.

[gravitar] RoboTux

Submitted at 00:52:33 on 10 february 2011

I think unison can detect renames in files. Often in synchronization it does there is some fast copy from a local file. I think it does that when it detect a rename to avoid any network transfer.

[gravitar] gregoa

Submitted at 01:03:15 on 10 february 2011

I like unison for syncing between hosts, and it seems quite good at handling this renaming stuff (by "shortcutting" to already available files with using some hashes).

[gravitar] Jason Weathered

Submitted at 05:05:34 on 10 february 2011

`rsync --archive --fuzzy --delete-after` may be good enough for what you need. It will pick up any rename in the same directory if the content hasn't changed.

If the content has changed a bit (i.e. metadata) as well as a rename, it's still usually pretty good. The man file says: "The current algorithm looks in the same directory as the destination file for either a file that has an identical size and modified-time, or a similarly-named file."

[gravitar] foo

Submitted at 05:19:42 on 10 february 2011

For the second one you want git-annex by the awesome Joey Hess.

[gravitar] Anonymous

Submitted at 06:31:25 on 10 february 2011

Opportunistic "cron": Did you consider atd's batch command? You could add batch jobs (run once by atd when the system is idle) using cron. Not a perfect solution, but it might work.

Content-aware tracking software Did you consider Joey Hess' git-annex?

[gravitar] Anonymous

Submitted at 06:31:53 on 10 february 2011

Opportunistic "cron": Did you consider atd's batch command? You could add batch jobs (run once by atd when the system is idle) using cron. Not a perfect solution, but it might work.

Content-aware tracking software Did you consider Joey Hess' git-annex?

[gravitar] Simon Richter

Submitted at 06:47:50 on 10 february 2011

rsync has an option for that: "-y" or "--fuzzy", to make it look for "similar" files that the recipient already has. That usually works quite well for simple renames.

[gravitar] mp

Submitted at 08:25:22 on 10 february 2011

Tool 1: fcron seems to almost allow it with its load average options

[author] Steve Kemp

Submitted at 08:37:19 on 10 february 2011

Wow thanks for all the overnight comments.

Unison I think I don't need, and rsync's fuzzy option isn't sufficient for the general case, as I frequently rename directories and move contents around. From the help :

"The current algorithm looks in the same directory as the destination file .."

On the other hand git-annex looks ideal for me :)

 

Comments are closed on posts which are more than ten days old.

Recent Posts

Recent Tags