[tbnl-devel] uploading a huge file / ajax progress bar

[Mac Chan]

Hi,

I'm trying to see if it's feasible to use lisp + tbnl to build an
internal documentation system that allows user to upload huge files.

Typically users might upload files with size ranges from 20mb to 600mb.

I'm using lispworks as the development platform, but intend to deploy
the site in freebsd/linux + sbcl/cmucl.

I notice that by uploading a 600mb file on win32 + lispworks + tbnl
mod-lisp, the memory usage remains constant (ie. Lispworks is
streaming the content directly to disk).

However, with either freebsd/cmucl or linux/sbcl, top shows that all
of the uploaded contents are first stored in memory and then get
written to disk. This is a major scalability problem, I'm not sure if
this is a design issue that can be fixed.

Since uploading huge files take a bit of time, displaying a progress
is a must; otherwise user might think that the browser hangs. It is
also good to let them know how long will it take to upload the file so
they can check back in couple minutes instead of staring at a stalled
screen.

There exist some open source implementations for php and ruby.
http://www.raditha.com/php/progress.php
http://sean.treadway.info/demo/upload/

I quickly went through the tbnl request post initialization routine
and the rfc2388 module. It looks like what we need to do is to change
rfc2388 to continuously write a report of the current upload progress
to a file, and then add an entry point in tbnl to return a html page
showing the current progress. This page can be launched by javascript
when the upload form is submitted.

Am I missing any important piece of information in this picture?
If anyone can give me more pointers I would try my hands on this as an
exercise. I'm not sure if anyone would find this useful though.

Thanks in advance.
-- Mac