A poor man's caching local file system, or "wide area file system" (wafs)

What I wanted was a custom file system that mirrored a remote file system such that

• all the directory entries of the remote file system were cached locally, so you could do immediate shell wildcarding, filename expansion, etc
  
• an on-demand copy such that if I opened one of the files, the system would copy from the remote system to the local system, and give you a handle to the file on the local system.
  
• When you updated the file, the system would use rsync to propagate the update to the remote system
  
• Ideally, I wanted all this implemented via a Linux userland file system
  

I did not require robustness in the face of multiple people updating the file, nor did I care whether remote updates were propagated locally. I assumed that my side was the "master" and the other side the "slave" and any changes I made I'd do on my side.

As a prototype, I skipped the part about on-demand copying from the remote system and just copied the files I wanted to play with up front. I then wrote a short shell script that scans for changes once a second and propagates them to the remote system via rsync.

This seems to work for the one use-case I've tried so far -- editing a script and seeing if the changes propagated quickly and accurately (they did). What I'm not sure is if the plan to ignore updates on the remote end will pan out.

Let me expand a bit on why I want this: I want to use my Lispworks editor for as much of my editing as possible. I also want to use Lispworks itself for as much scripting as possible, including scripting for work. Given that the work machine is two hops away (one hop to my work laptop, another hop (over a vpn) to the system itself), I needed a way to make the filesystem accesses a lot faster. (If I had a faster network I'd just use sshfs, as I briefly discussed earlier.)

So now, with a bit of preparation (the up front copying I mentioned), I can edit files locally (or process them with Lisp) and have saved changes propagate in 1-2 seconds.

Now that I have that, I wonder: will it do me any good? That is, it occurs to me: if I'm editing locally, and scripting locally, why do I care if the updates back to the work server happen quickly? I could just copy over here in the morning, and copy back in the evening. We'll see. :)

tags: sshfs  rsync  wafs  wide area file system  

links: digg this    del.icio.us    technorati    reddit