File I/O Metrics

[Garrett Dangerfield]

I tried changing (make-array buffer-size :element-type 'character)
to
(make-array buffer-size :element-type 'byte)
and I got additional warnings and it took 70 seconds instead of 20.

Thanks,
Garrett.

On Fri, Oct 21, 2022 at 1:47 PM Robert Goldman <rpgoldman at sift.net> wrote:

> I don't know what data you are reading but is there any chance that the
> difference is that when you read text in lisp as ISO-8859-1 lisp is
> actually processing the text as unicode, but when you are reading it in
> Java you are just slamming raw bytes into memory?
>
> Maybe this is relevant?
> https://stackoverflow.com/questions/979932/read-unicode-text-files-with-java
>
> I don't use Java myself, so I can't say, and I don't have access to your
> data, but it does seem like the Java code is doing something simpler than
> the Lisp code.
>
> What happens if you change your Lisp code to read-sequence of type byte
> instead of character?
>
> On 21 Oct 2022, at 13:43, Garrett Dangerfield wrote:
>
> I don't want to cause a firestore here but I was doing some simple
> benchmarks on file i/o between Java, ABCL, and SBCL and I'm a bit shocked,
> honestly.
>
> Reading a 2.5M file in 16M chunks in (using iso-8859-1):
> - abcl takes a tad over 1 second
> - sbcl takes 0.04 seconds
>
> Reading a 5.8G file in 16M chunks in (using iso-8859-1 for Lisp, for Java
> it's just bytes):
> - abcl takes...too long, I gave up
> - sbcl takes between 20 and 21 seconds
> - Java takes 1.5 seconds
>
> These are all run on the same computer using the same files, etc.
>
> What's up with this? Thoughts? I'd heard that SBCL should be as fast as C
> under at least some circumstances. I'd wager that C is at least as fast as
> Java (probably faster).
>
> Thanks,
> Garrett Dangerfield. (he/him/his)
>
> P.S. Don't get me wrong, I *LOVE* Lisp, I'm trying to get away from Java
> as
> fast as I can (the syntax is killing me slowly). I've used ABCL in
> projects before (it was wonderful, Java doesn't handle XML well).
>
> Lisp code:
> (with-open-file (stream "/media/danger/OS/temp/jars.txt" :external-format
> :iso-8859-1) ; great_expectations.iso
> (let ((size (file-length stream))
> (buffer-size (* 16 1024 1024)) ; 16M
> )
> (time
> (loop with buffer = (make-array buffer-size :element-type 'character)
> for n-characters = (read-sequence buffer stream)
> while (< 0 n-characters)))
> )))
>
> Java code:
> private static final int BUFFER_SIZE = 16 * 1024 * 1024;
> try (InputStream in = new
> FileInputStream("/media/danger/OS/temp/great_expectations.iso"); ) {
> byte[] buff = new byte[BUFFER_SIZE];
> int chunkLen = -1;
> long start = System.currentTimeMillis();
> while ((chunkLen = in.read(buff)) != -1) {
> System.out.println("chunkLen = " + chunkLen);
> }
> double duration = System.currentTimeMillis() - start;
> duration /= 1000;
> System.out.println(String.format("it took %,2f secs", duration));
> } catch (Exception e) {
> e.printStackTrace(System.out);
> } finally {
> System.out.println("Done.");
> }
>
> Robert P. Goldman
> Research Fellow
> Smart Information Flow Technologies (d/b/a SIFT, LLC)
>
> 319 N. First Ave., Suite 400
> Minneapolis, MN 55401
>
> Voice: (612) 326-3934
> Email: rpgoldman at SIFT.net
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/armedbear-devel/attachments/20221021/df47a79d/attachment.html>