[xcvb-devel] Initial release of desire, a module transport/knowledge layer.

Fri Oct 9 01:54:04 UTC 2009

Good day folks,

I've finally took my time to document desire and scratch off
some rough edges.

Let the preliminary documentation speak for itself:

	http://www.feelingofgreen.ru/shared/git/desire/doc/overview.txt

It is also attached, for the sake of completeness.

An accompanying bootstrap script:

	http://www.feelingofgreen.ru/shared/git/desire/climb.sh

regards,
  Samium Gromoff
--
                                 _deepfire-at-feelingofgreen.ru
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org

---- 8< ----

Overview of desire, a software knowledge and distribution system.
                  October 9, 2009, 05:43

0.  Index
0.1   About this document
1.  What is desire (supposed to be)?
2.  Overview of terms
2.1   Distributors
2.2   Remotes
2.3   Modules
2.4   Systems
2.5   Applications
2.6   Localities
3.  API (aka end-user interface)
3.1   Querying the knowledge base: user level
3.2   Making wishes
3.2.1   Reader macros for add-module
4.  Workflow
4.1  Streamlined bootstrap
4.2  Advanced use
4.2.1 User mode
4.2.2 Wishmaster mode
4.2.3 The real truth about wishmaster and user modes
5.  Shortcomings

0.1 About this document

This document has a canonical location:

	http://www.feelingofgreen.ru/shared/git/desire/doc/overview.txt

1. What is desire (supposed to be)?

The domain of desire includes software distribution points, software
modules and applications. All these terms are represented internally
in a detailed, comprehensive manner, and full knowledge about them is
maintained in a PRINT/READ-able form, additionally being easily
transportable[1] between desire nodes.

There are two sides of desire[2]:
      - maximally streamlined software retrieval, and
      - redistribution.

--- footnotes ---
1. Not yet, but the client mode depends on it, so at least partially it works.
2. Although, by nature of being a full-knowledge system the applications might easily go beyond that.

2. Overview of terms

2.1 Distributors

The largest unit of software granularity is a 'distributor', which
corresponds to an internet domain name[1]. Currently, they don't carry
much information beyond just that.

'Wishmasters' represent instances of desire acting as distributors which
are in business of making ordinary desire users' lives easier by
repackaging modules from their respective VCS of choice into git,
thereby relieving burden from the user and reducing the amount of points
of failure.

2.2 Remotes

Distributors contain one or more 'remotes', which combine following
concepts:
      - version control system type (git, darcs, cvs or svn)
      - transport type (native, http or rsync)
      - simple pattern path on the distributor
      - an internet port number

The concept of remote serves (or actually, the more general type of
location does) as a point of distribution for a group of modules.

2.3 Modules

Modules represent units of software, as released by the distributor,
and contain the information remaining to complete the information
provided by less granular concepts to construct a module address.

Modules can be provided by several different remotes of different
distributors.

2.4 Systems

Descending further down we meet systems.
Systems are objects only meant to be relevant in the domain of Common Lisp
software, and more precisely -- to backend system definition facilities,
such as ASDF, XCVB, Mudballs or others[2].

The concept of system introduces inter-system dependencies, which cross
module boundaries, producing inter-module dependencies.

Evidently, there can be several systems per module, and also those
can be obscured from the end-user, either intentionally or by unfortunate
accident[3].

Desire handles all these complications and operates on the full
inter-module dependency graph. It also doesn't store that graph anywhere,
recomputing it instead every time a request for module is performed.

2.5 Applications

Applications are simple extensions of systems, providing some very
preliminary support for launching applications, potentially simplifiable
to the point of the end-user being able to say "launch mcclim" and
have everything downloaded, required systems loaded and the application
up and running as a result.

2.6 Localities

The more general type of 'location' besides remotes also includes
'localities', which serve the same purpose as remotes, but on the local
machine. Master localities are canonical locations for module repositories
managed by a specific VCS type.
The master git locality is a canonical location for module redistribution.

--- footnotes ---

1. Actually, sometimes a group of domain names, like in case of sourceforge.
2. Currently, the only backend system implemented is ASDF.
3. Recovering such hidden systems complicates construction of full dependency graph in case of ASDF.

3. API (aka end-user interface)

3.1 Querying the knowledge base: user level

distributor name &key (if-does-not-exist :error) => distributor
remote name &key (if-does-not-exist :error) => remote
module name &key (if-does-not-exist :error) => module
system name &key (if-does-not-exist :error) => system
app name &key (if-does-not-exist :error) => app
locality name &key (if-does-not-exist :error) => locality
master name &key (if-does-not-exist :error) => locality

name object => symbol

url remote-designator &optional module-specifier => string

apropos-desr string-designator &optional set-designator => <no values>
apropos-desr-list string-designator &optional set-designator => desirables
list-modules => <no values>

module-present-p module-designator module &optional (locality (master 'git)) check-when-present-p (check-when-missing-p t) => boolean
system-loadable-p system-designator &optional (locality (master 'git)) => generalised-boolean

3.2 Making wishes

init path &key as (default-wishmasters (list *default-wishmaster*)) => boolean
lust &rest module-names => boolean
add-module url &optional module-name &key systemlessp (system-type *default-system-type*) (lust *auto-lust*) => module

3.2.1 Reader macros for add-module

Following reader macro is enabled by install-add-module-reader
#@"u://r.l"
#@("u://r.l" &optional module-name &key (lust *auto-lust*))

3.x Querying the knowledge base: advanced level

system-definition system repository-path &key (if-does-not-exist :error) => pathname

4. Workflow

4.1 Streamlined bootstrap

To attain maximum ease, desire includes a booststrap script, whose
canonical permanent location is:

	http://www.feelingofgreen.ru/shared/git/desire/climb.sh

The first argument should be an empty, writable filesystem location.
That is, the exact filename must not exist, but its parent must exist,
and be writable.

This location will be registered as root directory for per-VCS localities,
serving to satisfy storage needs arising from user module retrieval requests.

The path to the 'git/.asdf-registry/' subdirectory of this root directory
should be pushed into ASDF:*CENTRAL-REGISTRY*, if ASDF is to have access
to modules in the desire's pool.

This script when invoked performs following operations:
  - use git to download modules desire depends on, placing them in /tmp,
  - load desire,
  - the most important part, cheer up the user!

4.2 Advanced use

Depending on the way you intend to run desire, you issue either:

	(init "/path/to/root/"), for user mode
or:
	(init "/path/to/root/" :as 'well-known-distributor), for wishmaster mode

---- Note: ----
This path provided to INIT will henceforth be referred to as $ROOT.
---------------

The modes provide the end-user with a tradeoff between hassle and variety
of module available.

This stems from the difference between operation of wishmaster and user modes.

Wishmasters use a variety of tools to access the whole set of modules retrievable
using desire, but from this power comes fragility and, because wishmasters also
serve the role of simplifier gateways, time consuming whatever->git repository
conversion process.

User mode is the chief reason for that complications, as the only thing the user
needs to have is git, and boy is that fast[1].

The price, though, is relying on the set of modules available from wishmasters.

4.2.1 User mode

During INIT a default wishmaster (currently 'git.feelingofgreen.ru) will be
chosen, its module export set will be retrieved and that, with the notable
addition of modules present in $ROOT/git/, will constitute the whole set
available to the end user.

Who is, then, free to use LUST, APROPOS-DESR, LIST-MODULES and other
such functions listed in section 3.

4.2.2 Wishmaster mode

This mode requires that the $ROOT/git/.meta/DEFINITIONS file contains
a definition for the distributor your desire node will act as,
and whose name you will pass using the :AS keyword to the INIT function.

The INIT function:
    - scans for modules in the canonical export location $ROOT/git/
    - checks that this set includes every module that is claimed to be
      "well known" to be distributed by our distributor[2], otherwise
      signalling an error
    - publishes the informations about non-"well known", converted modules
    - calls it a day

ADD-MODULE and the accompanying reader macro #@"u://r.l/" can be used
to populate the set of known modules by hand.

--- footnotes ---

1. Not faster than downloading and extracting release tarballs, though.
2. This is tied to the concept of well known release locations and differs
   from the set of modules converted and reexported in the wishmaster
   process.

5. Shortcomings

* SBCL-only
* ASDF-only
* has a non-trivial amount of CL library dependencies, half of them
  not exactly being common
* calls out to an obscene amount of external executables, thereby only
  being able to guess about failure reasons
* the master mode is git-centric
* obviously has rough edges
* this documentation sucks
* the author wants sleep