[movitz-devel] Suggested "filesystem" (was RE: Topic of list)

[Chris Capel]

-- Frode Vatvedt Fjeld wrote --

>> Would enforcing a strict barrier between packages be enough to
>> solve package conflicts and problems, and what would this barrier
>> look like?
>
> I believe there is only one real way to enforce safety and security
> barriers, and that's using the (page-based) memory access and
> privileges. Packages is mostly a naming mechanism, I think, so I
> don't think it's where one can deal with security.

I didn't mean to confuse the packaging question with the security
question--I understand that basic security must go much deeper than a
package system. Still, I think having the package system being
responsible for much of the higher-level security wouldn't be a bad
thing (see last paragraph). Namespaces can be a powerful method of
organization, with more possible applications than just naming things.

I don't see anything wrong with the normal kernel ideas of page-based
memory access privileges, but this isn't what the system administrator
sees. I guess my question was more oriented toward user-level
security, as in whether a multi-user OS applies to Lisp-OS's, whether
it's a good idea at all.

> Personally I'd be willing to consider tossing out the traditional
> file-system, but I don't really have any great ideas as to what
> exactly the alternative would be for general purpose storage.

I think the technical barriers to creating a true POS system might be
too great to try to provide such a thing right away. As James said,
it's not a well-understood problem. However, I think there are a lot
of features of a less-hierarchically-oriented system that could be
included without too much fuss.

Here's what I was thinking. I'm a long-time Windows user. I write C#
software for a living that runs MS SQL Server and depends on Active
Directory for security. So I know Windows pretty well. I've been using
Linux for about three years now, and I've tried several distributions,
so I think my impressions of Linux are fairly unbiased toward Linux
itself, as opposed to some part of Linux. What I like about Windows is
that when you install things, you don't *have* to configure them.
Things get put where they should be, automatically. As far as I
understand, this is impossible in Unix-based OS's because there are no
standards to dictate where things ought to go, so a lot of effort is
put into figuring out where to put things and then telling everything
where everything else is. Correct me if I'm wrong, but this is what
makes it impossible to write installers for anything less specific
than a particular proprietary Unix or Linux distro running on a
particular hardware platform, in the best case.

Let me fantasize about a better OS for a second. Let me call this OS
SOS, for "Super" OS. In SOS, anything you install has a name given to
it by its authors. For instance, cl-ppcre, the CL regex package. If
you wanted this package, you'd tell SOS the URL of the project, or
you'd give it a short name that it matched from a list of package
repository sites, and SOS downloads it and installs it, like
asdf-install, except that the package's files aren't put into some
directory somewhere, but instead into a big list of the system's
packages, maintained by the OS.

The SOS filesystem (yes, it's a filesystem) is not /basically/
hierarchical, but flat. Each package has its own base directory, which
can have sub-directories, and each directory is given a number of
attributes that specify what it is. The base directory for a package
is given the attr-package attribute (maybe a psuedo-attribute, because
it shouldn't be neccesary to actually store this), and possibly the
attr-home-directory attribute; The user's files should not be mixed
with other packages. The directory where the binaries are stored
(located under the package's base directory) are given an attribute
that lets SOS know to put the files contained (or perhaps the exports
defined in the files contained) into its search path for commands. Or
this attribute could be applied to individual files in the package.
Any logs (say, for a web server package) would be in a directory
marked by the attr-log attribute, and so on.

Naming conflicts would be resolved in this manner: each directory
/must/ be in a particular package, and each package has both a short
name, a description, and a GUID. The GUID is used by all packages
referencing the other package, and used consistently by the system,
internally. The short name of a package can be changed by the user, so
if they wanted to install two packages called "firebird" they could
refer to one as "firebird-browser" and the other as "firebird-db".
They could rename any package at any time without breaking anything.
Or, they could leave two packages names the same. The description is
used by the user to disambiguate if more than one short name is the
same, and the GUID by the system.

The version of a package would be separate from the GUID, but part of
the dependency resolving process. Multiple versions of a package could
be installed side by side, and applications that depend on a package
could either specify the exact version they require, a minimum
version, a list of versions, or something else in that vein. If the
required version isn't installed, it would be downloaded and installed
automatically (with the user's permission, of course). The user could
override a package's dependency resolution rules in case of a failure
of a certain package provider to provide backwards compatibility, or
any number of other exceptional conditions.

Navigation through the filesystem/package-system is sort of like
querying a relational DB. As complicated as that may first impress one
as being, I think for the end user it's actually less complicated.
They can select the package they're trying to find, the attributes of
the files they want, the short names of the files, or they can search
descriptions or other file or package meta-data. This query could be
in the form of a URI. A user wanting to find all logs in the system
could specify something like "file://attr:logs/apache2/access_log", or
"file://pkg:apache2/logs/access_log" to get to the same file.

I think the biggest win of this sort of system (given there aren't
problems I don't immediately see with it) is that every application
knows exactly where to find every library it needs, and it *knows*
that it will always be there. When one asks the question "Is XXX
installed on this system?" the answer will always be yes or no, and
never "Yes, but in /home/yyy" or "Yes, but I'm not sure where" or
"Yes, but its configuration files have been moved to zzz" or any other
of a million-kajillion things.

/Complete/ uninstallation (and, hopefully, installation) of packages
with a single command is possible.

A unified documentation system is possible. A file that contains the
index, TOC, and other meta-data of the documentation for a package can
be marked with an attr-documentation, and can be merged with the help
for all the packages on the system.

Also, a common menu system is possible. Independent of a window manager
or a console program, there could be a menu of all applications,
configuration options, peripherals, and so on, on the system.

Zero configuration required. That's the goal. You install something,
and you don't have to tell it where to find things, or create shortcuts
to it, or integrate it with your configuration system. All integration
is done automatically. You might have to tell it some stuff about how
to behave ("Which port should I listen on?") but that's part of the
installation process. Sensible, secure, usable defaults.

The biggest concern with the OS dictating things like the
documentation system, the menu system, and the configuration system is
that while these things probably need to be unified and standardized,
this doesn't give a lot of room for improvement--it's a big commitment--
so you /really/ need to get it right the first time. It also needs to
be designed so that administrators still have complete control over
everything they /need/ to have control over, and no more. Unix
definitely goes the wrong direction with this, and Windows goes too far.
But I think the benefits of standardizing everything you can about a
system without removing too much flexibility are more than great
enough to justify it.

While Microsoft's operating systems don't exactly get this done
elegantly, the fact that things can be installed on them with
/no trouble/ and /no hassle/ is a result of the fact that applications
know where to find things. There's an API (or a registry location, I'm
not sure) that tells you where the Program Files directory is, where
the My Documents directory is, where the Java Runtime is (either Sun's
or MS's, it doesn't matter--it gets it right). Shared libraries always
have the same name on every machine, and are always located in the
same place, %windir%\system32. Most of the ideas for SOS, however,
came from MS's .NET assembly versioning and the GAC.

I wholeheartedly believe that SOS's filesystem lacks only the
important property of automatic, transparent serialization
deserialization of Lisp data structures before it's a true POS. And
this could be incrementally added later. It fits right in with the
packages-as-basis-of-organization idea, too. The difference, to a
package, of "file" vs. "data structure" doesn't /need/ to be
clear-cut. It's all part of the same package, right?

One more thought. Instead of user-based security, resources could be
secured from packages. The web server package would have access to
network resources and web content, the user's package would have
access to his or her documents and user programs, and most packages would
have no access to files in other packages except for dependencies.

> [...] Also, Movitz is still so immature that any substantial project
> using it would be likely to influence Movitz as a platform also.

I'm afraid that, more than the security question, the question of
filesystems and package systems are irrelevant to Movitz proper, but
you may know better than me about the implementation requirements, Frode.


Chris Capel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.common-lisp.net/pipermail/movitz-devel/attachments/20040411/2a65fc19/attachment.html>