[cdr-discuss] Proposed :TRUNCATE keyword for OPEN

[Richard M Kreuter]

Hello,

I'd like to submit the following as a CDR document.  Any comments?

Thank you,
RmK

Issue: OPEN-TRUNCATE

References: CLtL1 pp 418 through 422,
            dpANS3 pp 21-34 through 21-36,
	    accessible online at 
            http://www.lisp.org/HyperSpec/Body/fun_open.html

Related Issues: None.

Category: Addition.

Problem description: Common Lisp offers no explicit way to truncate an
existing file when the file is opened.  Interoperability of conforming
Common Lisp programs with software systems that prefer or require
truncating openings is hindered by the lack of an explicit way to
request such openings.

Proposal (OPEN-TRUNCATE:NEW-IF-EXISTS-ACTION):

Amend the text of OPEN (Function) in the Arguments and Values section to
replace the IF-EXISTS description with the text,

  IF-EXISTS--one of :error, :new-version, :rename, :rename-and-delete,
  :overwrite, :append, :truncate, :supersede, or ‘nil’.  The default
  is :new-version if the version component of FILESPEC is :newest, or
  :error otherwise.

and to add the following text in the Description section between the
descriptions of :APPEND and :SUPERSEDE

    :truncate
          Output operations on the stream destructively modify the
          existing file.  If DIRECTION is :io the file is opened in a
          bidirectional mode that allows both reading and writing.
          The file pointer is initially positioned at the beginning of
          the file and the file is truncated back to length zero when
          it is opened.

Examples:

;; Recall that the consequences of calls to OPEN permitted to vary
;; considerably.  In the following comments, "same file" and "different
;; file" assume a notion of file identity independent of file contents
;; or file names, which may not be applicable on all file systems, and
;; which cannot be distintinguished in conforming Common Lisp programs,
;; but which can nonetheless affect the correct operation of programs.

(with-open-file (f "test" :direction :output)
  (write-line "some text" f))
"some text"

;; A successful :OVERWRITE opening returns a stream associated with the
;; file named by the defaulted argument to OPEN, and so after the stream
;; is closed, the file named by the defaulted argument to OPEN is the
;; "same file" as the file so named before the opening.
(with-open-file (f "test" :direction :output :if-exists :overwrite)
  (values (truename f)
          (file-length f)))
=> #P"/home/me/test"
=> 9

;; Under this proposal, a successful :TRUNCATE opening will return a
;; stream associated with the file named by the defaulted argument to
;; OPEN and empty the file during the opening operation.  Consequently,
;; when the stream is closed, the file named by the defaulted argument
;; to OPEN will be the "same file" as the file so named before the
;; opening.
(with-open-file (f "test" :direction :output :if-exists :truncate)
  (values (truename f)
	  (file-length f))))
=> #P"/home/me/test"
=> 0

;; A successful :SUPERSEDE opening returns a stream that might or might
;; not be associated with a file named by the defaulted argument to
;; OPEN, and in the former case, the file associated with the stream
;; might be created and arranged to replace the existing file at the
;; time the stream is created, and so not be the "same file" as the one
;; that existed before OPEN was called.
(with-open-file (f "test" :direction :output :if-exists :supersede)
  (values (truename f)
          (file-length f)))
=> #P"/home/me/test"
=> 0
or
=> #P"/home/me/test-temp-12345"
=> 0

Rationale:

On some file systems, the customary way to open an existing file for
output or input and output preserves the association between the name
used to open the file and the identity of the file, but permits
destructive modification of the file's contents.  Common Lisp specifies
two IF-EXISTS actions (:OVERWRITE and :APPEND) whose specification map
straightforwardly to the identity preserving file openings on such file
systems, but lacks an IF-EXISTS action that calls for truncation of an
existing file's contents at the time the file is opened.

Additionally, on some file systems, there are several distinctions
between opening a file in a truncating manner and replacing an existing
file with a new one, which can make a replacing opening inadequate for
certain programs:

(1) Under Unix, replacing one file with another "breaks" any hard
    links to the replaced file, while a truncating open maintains the
    association between the file and its names.  (It is unknown
    whether other file systems have supported hard links with these
    particular semantics.)

(2) Again under Unix, the circumstances under which a program may
    replace one file with another are different from those where it
    may destructively modify an existing file (permissions on the
    directory versus permissions on the file).  Other systems may
    exhibit similar access control semantics.

(3) Under Unix, but probably other systems as well, it can be
    impractical or impossible for a program to ensure that a new
    file's metadata is equivalent to that of the file to be replaced.
    It's unclear how much work the Lisp implementation may be expected
    to carry out to ensure that a new file's metadata resembles that
    of the file to be replaced.

While hard links, access controls and file metadata are file system
details outside the scope of ANSI Common Lisp, programs must sometimes
interoperate with software that involve these details, and so prefer or
require expressly truncating openings.

Current practice:

No current (ca. 2007) implementation of which this author is aware
supports a :TRUNCATE IF-EXISTS action.

The MIT Lisp Machine manual circa 1984 documents a :TRUNCATE argument
for IF-EXISTS, so presumably truncating openings were implementable
on many file systems then (the MIT Lisp Machine at that time accessed
files on LMFS, ITS, TOPS-20, VMS, Multics and Unix).

http://common-lisp.net/project/bknr/static/lmman/files.xml#Opening%20and%20Closing%20File%20Streams-section

Cost to implementors:

For implementations that already perform a truncating opening under
some IF-EXISTS keyword, probably trivial.

For implementations that don't yet offer any way to truncate a file at
open time, the code for a truncating opening will probably resemble
that of :OVERWRITE or :APPEND.

Cost to Users:

None.  This is an upwardly compatible change.

Cost of non-adoption:

Programs that intentionally or accidentally rely on some
implementations' truncating openings will remain non-portable to other
implementations.

Performance impact:

Probably none.

Benefits:

Programs that need file truncation will be able to request it
explicitly.

Aesthetics:

Some users might employ one or another IF-EXISTS action to effect
tuncation on some implementations; under this proposal, such users will
be able to say what they mean.

Notes:

Implementations of the interface described in this document are
encouraged to ensure that the symbol :CDR-4 occurs in the *FEATURES*
list when the Lisp environment is configured to behave as described in
this document.

As implementors already have license to vary from the specification of
any OPEN option, and to signal an error if the implementation cannot
handle some OPEN option as specified, adding a new IF-EXISTS action
doesn't formally limit conforming implementations.