From chris at iBankTech.NET Fri Apr 9 18:40:53 2004 From: chris at iBankTech.NET (Chris Capel) Date: Fri, 9 Apr 2004 13:40:53 -0500 Subject: [movitz-devel] Topic of list Message-ID: This seems to me like a good list to discuss operating system design issues specific to a hypothetical MoKA. This has been summarily brought up, but would this be a good place to get in depth about what the Right Thing is as far as a lisp OS goes? A few things that've been on my mind: Would a persistent object store instead of a filesystem be more approiate? (Say yes.) How would security be handled? Would enforcing a strict barrier between packages be enough to solve package conflicts and problems, and what would this barrier look like? This kind of questions aren't really strictly Movitz related, but the Movitz project does host a little kernel. Would a real OS kernel built (eventually) on Movitz be in a separate project? Until such a kernel materializes, should this kind of discussion take place here or somewhere else? Chris Capel From james at unlambda.com Fri Apr 9 22:57:35 2004 From: james at unlambda.com (James A. Crippen) Date: Fri, 09 Apr 2004 22:57:35 -0000 Subject: [movitz-devel] Topic of list In-Reply-To: References: Message-ID: "Chris Capel" writes: > This seems to me like a good list to discuss operating system design > issues specific to a hypothetical MoKA. This has been summarily brought > up, but would this be a good place to get in depth about what the Right > Thing is as far as a lisp OS goes? A few things that've been on my > mind: > > Would a persistent object store instead of a filesystem be more > approiate? (Say yes.) Sorry, no, not as a replacement. You would have trouble implementing the filesystem part of many existing applications, for instance. LispMs had both a persistent object store (an OODBMS, actually, Statice on Genera and PCLOS on Explorer) and a filesystem. They served different purposes. Filesystems aren't useful for organizing nonhierarchical data, but persistent object stores are more clumsy and slower than filesystems when it comes to clearly hierarchical or block data. There are also reinstantiation problems such as ensuring that restored persistent objects are still EQ to the originals before storage, which can be a major PITA. If you think a POS is a good idea then don't hesitate to implement it. But it's not a good idea to have a general purpose operating system without a filesystem for it. People are very used to file handling and will be very confused when confronted with a system that doesn't do so. Now, if you wanted to get creative you could implement a filesystem as a sort of POS. But the user interface would look just like any other filesystem and that's how people would tend to use it, hierarchical or not. But filesystems have significant wins when it comes to linear data and hierarchical organization, and there are many well known designs and optimizations available for handling such data, whereas making a fast and efficient POS is still not a well understood problem. If you're looking towards a special purpose system or an embedded system that relies only on a POS then of course that makes sense. But for general purpose, especially desktop programming use, a POS isn't a sufficient replacement for a filesystem. The two organize data in completely different ways, and have very different purposes. > How would security be handled? The original LispMs had an open architecture. Anything could look at and write to anything. The user could poke at random locations in memory using LOCF in the listener. So essentially on a LispM there's no 'security'. This is very appropriate for a developer's workstation, because security really gets in the way. Put it on a network and make the network secure, but trust the developer to Do The Right Thing on his own box. Don't run unauthorized and unreviewed foreign code. And you don't have the null pointer or random pointer problem in Lisp that you do in C so there's a lot less worry about heap corruption. However, security shouldn't be hard to handle if you implement multiprocessing like on a Unixy system where every process has its own address space. But that may not be a win for Lisp. IPC is a pain in the butt and multiprocessing might be better handled the old LispM way for Lisp, with a global shared address space and lots and lots of swap (usually 10 to 1 or more). Threading is also a royal PITA in many cases. It's a big win in some ways, but if you make your processes lightweight enough and have an open and shared address space then you get threads for free, because they're just like ordinary processes. Have your threads lock global data structures and bind their local data structures, and no problems. > Would enforcing a strict barrier between packages be enough to solve > package conflicts and problems, and what would this barrier look like? Package locking is already implemented in some Lisps. See CMUCL, which IIRC has such a beast. Package locking is widely regarded as a Good Thing. Bt of couse it's circumventable, but only if you really want to. > This kind of questions aren't really strictly Movitz related, but the > Movitz project does host a little kernel. Would a real OS kernel built > (eventually) on Movitz be in a separate project? Until such a kernel > materializes, should this kind of discussion take place here or > somewhere else? I don't think that 'LispOS' should be discussed here unless code is actually being produced. Without code the discussion is just noise here (and yes, I did just contribute to it). Design ideas should be hashed out elsewhere, on a new LispOS list (the old one is dead) or on cll. There's a great potential for LispOS discussion to turn into a huge thread and maybe a flamewar, and that would be totally inappropriate here. This list should be reserved for discussion of actual code being implemented in Movitz, and not mere ideation. 'james -- James A. Crippen Lambda Unlimited 61.2204N, 149.8964W Recursion 'R' Us Anchorage, Alaska, USA, Earth Y = \f.(\x.f(xx))(\x.f(xx)) From ffjeld at common-lisp.net Sun Apr 11 16:32:40 2004 From: ffjeld at common-lisp.net (Frode Vatvedt Fjeld) Date: Sun, 11 Apr 2004 18:32:40 +0200 Subject: [movitz-devel] Re: Topic of list References: Message-ID: <2hn05ibhlj.fsf@vserver.cs.uit.no> "Chris Capel" writes: > Would a persistent object store instead of a filesystem be more > approiate? (Say yes.) > > How would security be handled? Personally I'd be willing to consider tossing out the traditional file-system, but I don't really have any great ideas as to what exactly the alternative would be for general purpose storage. > Would enforcing a strict barrier between packages be enough to solve > package conflicts and problems, and what would this barrier look > like? I believe there is only one real way to enforce safety and security barriers, and that's using the (page-based) memory access and privileges. Packages is mostly a naming mechanism, I think, so I don't think it's where one can deal with security. > This kind of questions aren't really strictly Movitz related, but > the Movitz project does host a little kernel. Would a real OS > kernel built (eventually) on Movitz be in a separate project? Until > such a kernel materializes, should this kind of discussion take > place here or somewhere else? I'd say that so far there's so little traffic here, that this list would be appropriate for such discussions. Also, Movitz is still so immature that any substantial project using it would be likely to influence Movitz as a platform also. -- Frode Vatvedt Fjeld From ffjeld at common-lisp.net Sun Apr 11 17:16:47 2004 From: ffjeld at common-lisp.net (Frode Vatvedt Fjeld) Date: Sun, 11 Apr 2004 19:16:47 +0200 Subject: [movitz-devel] Re: Topic of list References: Message-ID: <2hisg6bfk0.fsf@vserver.cs.uit.no> james at unlambda.com (James A. Crippen) writes: > If you think a POS is a good idea then don't hesitate to implement > it. But it's not a good idea to have a general purpose operating > system without a filesystem for it. People are very used to file > handling and will be very confused when confronted with a system > that doesn't do so. But don't forget the confusion that actually results from the omnipresent file-system model. Not just the confusion that computer novices experience when first using a computer, but all the packaging, versioning, (sym-)linking, etc. issues that to at least some degree might be rooted in the way filesystems are. > If you're looking towards a special purpose system or an embedded > system that relies only on a POS then of course that makes > sense. But for general purpose, especially desktop programming use, > a POS isn't a sufficient replacement for a filesystem. The two > organize data in completely different ways, and have very different > purposes. I don't think I agree with this. A desktop system tries to present the analogy of a desktop, with drawers for the files one isn't working on at the moment. The POS won't have any problems providing this. And I don't think that e.g. the way Windows maps the desktop metaphor onto filesystems is in any way clearly the Right solution. > However, security shouldn't be hard to handle if you implement > multiprocessing like on a Unixy system where every process has its > own address space. Remember that you can have protection also in a single address-space system. -- Frode Vatvedt Fjeld From chris at iBankTech.NET Sun Apr 11 22:40:05 2004 From: chris at iBankTech.NET (Chris Capel) Date: Sun, 11 Apr 2004 17:40:05 -0500 Subject: [movitz-devel] Suggested "filesystem" (was RE: Topic of list) Message-ID: -- Frode Vatvedt Fjeld wrote -- >> Would enforcing a strict barrier between packages be enough to >> solve package conflicts and problems, and what would this barrier >> look like? > > I believe there is only one real way to enforce safety and security > barriers, and that's using the (page-based) memory access and > privileges. Packages is mostly a naming mechanism, I think, so I > don't think it's where one can deal with security. I didn't mean to confuse the packaging question with the security question--I understand that basic security must go much deeper than a package system. Still, I think having the package system being responsible for much of the higher-level security wouldn't be a bad thing (see last paragraph). Namespaces can be a powerful method of organization, with more possible applications than just naming things. I don't see anything wrong with the normal kernel ideas of page-based memory access privileges, but this isn't what the system administrator sees. I guess my question was more oriented toward user-level security, as in whether a multi-user OS applies to Lisp-OS's, whether it's a good idea at all. > Personally I'd be willing to consider tossing out the traditional > file-system, but I don't really have any great ideas as to what > exactly the alternative would be for general purpose storage. I think the technical barriers to creating a true POS system might be too great to try to provide such a thing right away. As James said, it's not a well-understood problem. However, I think there are a lot of features of a less-hierarchically-oriented system that could be included without too much fuss. Here's what I was thinking. I'm a long-time Windows user. I write C# software for a living that runs MS SQL Server and depends on Active Directory for security. So I know Windows pretty well. I've been using Linux for about three years now, and I've tried several distributions, so I think my impressions of Linux are fairly unbiased toward Linux itself, as opposed to some part of Linux. What I like about Windows is that when you install things, you don't *have* to configure them. Things get put where they should be, automatically. As far as I understand, this is impossible in Unix-based OS's because there are no standards to dictate where things ought to go, so a lot of effort is put into figuring out where to put things and then telling everything where everything else is. Correct me if I'm wrong, but this is what makes it impossible to write installers for anything less specific than a particular proprietary Unix or Linux distro running on a particular hardware platform, in the best case. Let me fantasize about a better OS for a second. Let me call this OS SOS, for "Super" OS. In SOS, anything you install has a name given to it by its authors. For instance, cl-ppcre, the CL regex package. If you wanted this package, you'd tell SOS the URL of the project, or you'd give it a short name that it matched from a list of package repository sites, and SOS downloads it and installs it, like asdf-install, except that the package's files aren't put into some directory somewhere, but instead into a big list of the system's packages, maintained by the OS. The SOS filesystem (yes, it's a filesystem) is not /basically/ hierarchical, but flat. Each package has its own base directory, which can have sub-directories, and each directory is given a number of attributes that specify what it is. The base directory for a package is given the attr-package attribute (maybe a psuedo-attribute, because it shouldn't be neccesary to actually store this), and possibly the attr-home-directory attribute; The user's files should not be mixed with other packages. The directory where the binaries are stored (located under the package's base directory) are given an attribute that lets SOS know to put the files contained (or perhaps the exports defined in the files contained) into its search path for commands. Or this attribute could be applied to individual files in the package. Any logs (say, for a web server package) would be in a directory marked by the attr-log attribute, and so on. Naming conflicts would be resolved in this manner: each directory /must/ be in a particular package, and each package has both a short name, a description, and a GUID. The GUID is used by all packages referencing the other package, and used consistently by the system, internally. The short name of a package can be changed by the user, so if they wanted to install two packages called "firebird" they could refer to one as "firebird-browser" and the other as "firebird-db". They could rename any package at any time without breaking anything. Or, they could leave two packages names the same. The description is used by the user to disambiguate if more than one short name is the same, and the GUID by the system. The version of a package would be separate from the GUID, but part of the dependency resolving process. Multiple versions of a package could be installed side by side, and applications that depend on a package could either specify the exact version they require, a minimum version, a list of versions, or something else in that vein. If the required version isn't installed, it would be downloaded and installed automatically (with the user's permission, of course). The user could override a package's dependency resolution rules in case of a failure of a certain package provider to provide backwards compatibility, or any number of other exceptional conditions. Navigation through the filesystem/package-system is sort of like querying a relational DB. As complicated as that may first impress one as being, I think for the end user it's actually less complicated. They can select the package they're trying to find, the attributes of the files they want, the short names of the files, or they can search descriptions or other file or package meta-data. This query could be in the form of a URI. A user wanting to find all logs in the system could specify something like "file://attr:logs/apache2/access_log", or "file://pkg:apache2/logs/access_log" to get to the same file. I think the biggest win of this sort of system (given there aren't problems I don't immediately see with it) is that every application knows exactly where to find every library it needs, and it *knows* that it will always be there. When one asks the question "Is XXX installed on this system?" the answer will always be yes or no, and never "Yes, but in /home/yyy" or "Yes, but I'm not sure where" or "Yes, but its configuration files have been moved to zzz" or any other of a million-kajillion things. /Complete/ uninstallation (and, hopefully, installation) of packages with a single command is possible. A unified documentation system is possible. A file that contains the index, TOC, and other meta-data of the documentation for a package can be marked with an attr-documentation, and can be merged with the help for all the packages on the system. Also, a common menu system is possible. Independent of a window manager or a console program, there could be a menu of all applications, configuration options, peripherals, and so on, on the system. Zero configuration required. That's the goal. You install something, and you don't have to tell it where to find things, or create shortcuts to it, or integrate it with your configuration system. All integration is done automatically. You might have to tell it some stuff about how to behave ("Which port should I listen on?") but that's part of the installation process. Sensible, secure, usable defaults. The biggest concern with the OS dictating things like the documentation system, the menu system, and the configuration system is that while these things probably need to be unified and standardized, this doesn't give a lot of room for improvement--it's a big commitment-- so you /really/ need to get it right the first time. It also needs to be designed so that administrators still have complete control over everything they /need/ to have control over, and no more. Unix definitely goes the wrong direction with this, and Windows goes too far. But I think the benefits of standardizing everything you can about a system without removing too much flexibility are more than great enough to justify it. While Microsoft's operating systems don't exactly get this done elegantly, the fact that things can be installed on them with /no trouble/ and /no hassle/ is a result of the fact that applications know where to find things. There's an API (or a registry location, I'm not sure) that tells you where the Program Files directory is, where the My Documents directory is, where the Java Runtime is (either Sun's or MS's, it doesn't matter--it gets it right). Shared libraries always have the same name on every machine, and are always located in the same place, %windir%\system32. Most of the ideas for SOS, however, came from MS's .NET assembly versioning and the GAC. I wholeheartedly believe that SOS's filesystem lacks only the important property of automatic, transparent serialization deserialization of Lisp data structures before it's a true POS. And this could be incrementally added later. It fits right in with the packages-as-basis-of-organization idea, too. The difference, to a package, of "file" vs. "data structure" doesn't /need/ to be clear-cut. It's all part of the same package, right? One more thought. Instead of user-based security, resources could be secured from packages. The web server package would have access to network resources and web content, the user's package would have access to his or her documents and user programs, and most packages would have no access to files in other packages except for dependencies. > [...] Also, Movitz is still so immature that any substantial project > using it would be likely to influence Movitz as a platform also. I'm afraid that, more than the security question, the question of filesystems and package systems are irrelevant to Movitz proper, but you may know better than me about the implementation requirements, Frode. Chris Capel -------------- next part -------------- An HTML attachment was scrubbed... URL: From ffjeld at common-lisp.net Tue Apr 13 09:37:17 2004 From: ffjeld at common-lisp.net (Frode Vatvedt Fjeld) Date: Tue, 13 Apr 2004 11:37:17 +0200 Subject: [movitz-devel] Re: Suggested "filesystem" References: Message-ID: <2h1xmsb4mq.fsf@vserver.cs.uit.no> It seems to me that for a general purpose OS, one really needs to think hard about the relationship between persistent and transient structures. I mean, the naive model for persitence in a lisp system is to say (dump-world-image), and boot or load that image later. (Much like the "hibernate" operation on laptops.) In contrast there's unix, with its millions of config-files whose contents determine the persistent system state, and windows which also has the registry as a slightly more principled approach. LispOS also needs more fine-grained control over what's persistent and what's not, i.e. someting akin to the registry. One needs to be able to say "load/save the system's _configuration_", as opposed to its entire state, and this perhaps implies that programmers need to be able to express that e.g. some entitiy is part of the system's configuration. For example, consider some method on a system-standard gf that provides some configuration-specific specialization. One might save the defmethod form, or the method meta-object, or something else. Well, I suppose many of these questions will sort themselves out as a system is in fact built. Which brings me to my second point, which is that if someone was to write a simple interface to the IDE disk drives (that primarily works with bochs), this would be an excellent starting-point for actually experimenting with these concepts. I don't want to come across as whining, but I have to say I'm slightly disappointed so far when it comes to the apparent disinterest in contributing code (or even just bug/feature-reports) here. Hopefully this might change when the platform becomes more mature, but obviously contributions from others would speed up this process also, and not to mention increase the chances of removing bad design decisions of mine before they are embedded to deep in the system. -- Frode Vatvedt Fjeld From ffjeld at common-lisp.net Tue Apr 13 09:48:59 2004 From: ffjeld at common-lisp.net (Frode Vatvedt Fjeld) Date: Tue, 13 Apr 2004 11:48:59 +0200 Subject: [movitz-devel] Movitz maturity Message-ID: <2hwu4k9pis.fsf@vserver.cs.uit.no> On a related note (to my previous message), I just came back from 6 days of easter holiday. Over this period I left a machine running the current los0 image and (ip4-test), which answers to ping messages and prints out some packet information about incoming (broadcast) packets (our LAN is rather noisy). Each incoming packet is freshly allocated as an array, and some cons-cells are allocated here and there, I believe. Everything is still running nicely, so I find this experiment to be something of a milestone with respect to the maturity of Movitz :-) -- Frode Vatvedt Fjeld From peter.minten at wanadoo.nl Sun Apr 18 17:30:05 2004 From: peter.minten at wanadoo.nl (Peter Minten) Date: Sun, 18 Apr 2004 19:30:05 +0200 Subject: HD driver (was: Re: [movitz-devel] Re: Suggested "filesystem") In-Reply-To: <2h1xmsb4mq.fsf@vserver.cs.uit.no> References: <2h1xmsb4mq.fsf@vserver.cs.uit.no> Message-ID: <4082BB1D.408@wanadoo.nl> Hi, Frode Vatvedt Fjeld wrote: > Which brings me to my second point, which is that if someone was to > write a simple interface to the IDE disk drives (that primarily works > with bochs), this would be an excellent starting-point for actually > experimenting with these concepts. I don't want to come across as > whining, but I have to say I'm slightly disappointed so far when it > comes to the apparent disinterest in contributing code (or even just > bug/feature-reports) here. Hopefully this might change when the > platform becomes more mature, but obviously contributions from others > would speed up this process also, and not to mention increase the > chances of removing bad design decisions of mine before they are > embedded to deep in the system. Ok, you asked for it :-). I wrote a little IDE driver for ATA-1 compatible LBA capable disks (AFAIK Bochs disks are of that type), it's at 'http://www.il.fontys.nl/~silvernerd/harddisk.lisp'. Currently async read and write of 1-256 sectors in one go is implemented. The code does not compile due to a strange error about a variable (data) not being defined. The code is this: (defstruct hd-read-sectors-task (start-sector 0 :type (unsigned-byte 28)) (count 1 :type (integer 1 256)) (data #() :type data-vector) (offset 0 :type (integer 0 *))) (defmethod hd-controller-handle-task-signal (hdc (task hd-read-sectors-task)) (with-slots (count data offset) task (let ((status (io-port (hd-controller-command-register hdc 'status) :unsigned-byte8)) (read-data (io-port (hd-controller-command-register hdc 'status) :unsigned-byte16))) ;; by now the drive is getting the next piece, if necessary, ;; so I hope this code is reentrant (if (= 0 (logand (power 2 (hd-controller-status 'error)) status)) (progn ;; read 512 bytes (dotimes (i 256) (setf (aref data offset) (logand read-data #xFF)) (setf (aref data (1+ offset)) (logand read-data #xFF00)) (incf offset 2)) (= offset (1- (* count 512)))) ;return value, are we done or not? (error "Harddrive read-sectors returned error. Controller nr ~A, HD number: ~A, error register: ~A." (hd-controller-number hdc) (hd-controller-active-hd hdc) (io-port (hd-controller-command-register hdc 'error) :unsigned-byte8)))))) I know that using with-slots on a structure is not exactly common (though as I understand the HyperSpec it's legal) so I tried replacing it with let for a moment to see if the error was in the with-slots. Let gave the same result: symbol 'data' undefined. When I removed data from the code altogether it started complaining about offset. Furthermore on my box there is a problem with binary-types: On CMUCL 18e, custom core. Binary-types and Movitz version: CVS 17-apr-2004. Slime error message: Execution of a form compiled with errors: (DEFMETHOD MAKE-LOAD-FORM ((BINARY-TYPES::OBJECT BINARY-TYPES::RECORD-SLOT) &OPTIONAL BINARY-TYPES::ENVIRONMENT) (DECLARE (IGNORE BINARY-TYPES::ENVIRONMENT)) (WITH-SLOTS (BINARY-TYPES::NAME TYPE BINARY-TYPES::MAP-WRITE BINARY-TYPES::MAP-READ BINARY-TYPES::MAP-READ-DELAYED BINARY-TYPES::HIDDEN-READ-SLOT) BINARY-TYPES::OBJECT `(BINARY-TYPES::MAKE-RECORD-SLOT :NAME ',BINARY-TYPES::NAME :TYPE ',TYPE :MAP-WRITE ,BINARY-TYPES::MAP-WRITE :MAP-READ ,BINARY-TYPES::MAP-READ :MAP-READ-DELAYED ,BINARY-TYPES::MAP-READ-DELAYED :HIDDEN-READ-SLOT ',BINARY-TYPES::HIDDEN-READ-SLOT))) [Condition of type KERNEL:SIMPLE-PROGRAM-ERROR] The error disappears when I abort and do (load "load") again. Greetings, Peter Minten From ffjeld at common-lisp.net Mon Apr 19 22:46:41 2004 From: ffjeld at common-lisp.net (Frode Vatvedt Fjeld) Date: Tue, 20 Apr 2004 00:46:41 +0200 Subject: [movitz-devel] Re: HD driver References: <2h1xmsb4mq.fsf@vserver.cs.uit.no> <4082BB1D.408@wanadoo.nl> Message-ID: <2hu0zfpovi.fsf@vserver.cs.uit.no> Peter Minten writes: > Ok, you asked for it :-). I wrote a little IDE driver for ATA-1 > compatible LBA capable disks (AFAIK Bochs disks are of that type), > it's at > 'http://www.il.fontys.nl/~silvernerd/harddisk.lisp'. Currently async > read and write of 1-256 sectors in one go is implemented. Great, it's nice to see someone accept a challenge :-) > The code does not compile due to a strange error about a variable > (data) not being defined. [..] > I know that using with-slots on a structure is not exactly common > (though as I understand the HyperSpec it's legal) so I tried > replacing it with let for a moment to see if the error was in the > with-slots. Let gave the same result: symbol 'data' undefined. When > I removed data from the code altogether it started complaining about > offset. The problem was simply that there was no with-slots macro in Movitz, so the compiler tried to compile it as a function call, which should explain the behavior you saw. I have added a with-slots macro now. Also, I have re-worked defstructs quite a bit, so that slot-value (and therefore with-slots) will actually work with structs. So I believe your code should compile fine now. I would however suggest to prefer defclass over defstruct in this case, and in general. > Furthermore on my box there is a problem with binary-types: [..] > > On CMUCL 18e, custom core. > Binary-types and Movitz version: CVS 17-apr-2004. > > Slime error message: > > Execution of a form compiled with errors: > (DEFMETHOD MAKE-LOAD-FORM > ((BINARY-TYPES::OBJECT BINARY-TYPES::RECORD-SLOT) &OPTIONAL > BINARY-TYPES::ENVIRONMENT) > (DECLARE (IGNORE BINARY-TYPES::ENVIRONMENT)) > (WITH-SLOTS > (BINARY-TYPES::NAME TYPE BINARY-TYPES::MAP-WRITE > BINARY-TYPES::MAP-READ BINARY-TYPES::MAP-READ-DELAYED > BINARY-TYPES::HIDDEN-READ-SLOT) > BINARY-TYPES::OBJECT > `(BINARY-TYPES::MAKE-RECORD-SLOT :NAME > ',BINARY-TYPES::NAME > :TYPE > ',TYPE > :MAP-WRITE > ,BINARY-TYPES::MAP-WRITE > :MAP-READ > ,BINARY-TYPES::MAP-READ > :MAP-READ-DELAYED > ,BINARY-TYPES::MAP-READ-DELAYED > :HIDDEN-READ-SLOT > ',BINARY-TYPES::HIDDEN-READ-SLOT))) > [Condition of type KERNEL:SIMPLE-PROGRAM-ERROR] > > The error disappears when I abort and do (load "load") again. I don't know what this is. When does it occur? During the first (load "load")? -- Frode Vatvedt Fjeld From ffjeld at common-lisp.net Mon Apr 19 23:10:40 2004 From: ffjeld at common-lisp.net (Frode Vatvedt Fjeld) Date: Tue, 20 Apr 2004 01:10:40 +0200 Subject: [movitz-devel] Re: HD driver References: <2h1xmsb4mq.fsf@vserver.cs.uit.no> <4082BB1D.408@wanadoo.nl> Message-ID: <2hpta3pnrj.fsf@vserver.cs.uit.no> I added harddisk.lisp to CVS as losp/tmp/harddisk.lisp. I thought I'd add the losp/tmp directory as a sort of scratch area for files like these, so they can be developed somewhat before being moved somewhere more sensible. I looked briefly at your code, and thought I'd mention something about interrupts. First, I changed the accessor (muerte:interrupt-handler ) to (muerte:exception-handler ), so you probably want to change to this. Also, remember that the IRQs are routed throught the PIC (well, really the APIC I suppose) to some exception number. The idt-init (x86-pc/interrupt.lisp) function does this, and currently in los0 it's routed such that e.g. IRQ 14 becomes exception number 32+14 = 46, if I'm not mistaken. Also you must make sure interrupts are enabled (i.e. with STI), and that the PIC mask is set up ok (accessor pic8259-irq-mask). In short, if at all possible, it's probably a good idea initially to have a stupidly-polling-but-guaranteed-to-work interface (but I don't know anything about IDE hardware interfacing issues, so this may not be possible). -- Frode Vatvedt Fjeld From ffjeld at common-lisp.net Wed Apr 21 07:31:20 2004 From: ffjeld at common-lisp.net (Frode Vatvedt Fjeld) Date: Wed, 21 Apr 2004 09:31:20 +0200 Subject: [movitz-devel] Re: HD driver References: <2h1xmsb4mq.fsf@vserver.cs.uit.no> <4082BB1D.408@wanadoo.nl> Message-ID: <2hk709okhj.fsf@vserver.cs.uit.no> Peter Minten writes: > Furthermore on my box there is a problem with binary-types: I checked in binary-types.lisp yesterday, iirc with a fix for this that I received from someone a long time ago, but forgot to check in. -- Frode Vatvedt Fjeld From peter.minten at wanadoo.nl Tue Apr 20 17:39:32 2004 From: peter.minten at wanadoo.nl (Peter Minten) Date: Tue, 20 Apr 2004 19:39:32 +0200 Subject: [movitz-devel] Re: HD driver In-Reply-To: <2hpta3pnrj.fsf@vserver.cs.uit.no> References: <2h1xmsb4mq.fsf@vserver.cs.uit.no> <4082BB1D.408@wanadoo.nl> <2hpta3pnrj.fsf@vserver.cs.uit.no> Message-ID: <40856054.6040709@wanadoo.nl> Frode Vatvedt Fjeld wrote: > In short, if at all possible, it's probably a good idea initially to > have a stupidly-polling-but-guaranteed-to-work interface (but I don't > know anything about IDE hardware interfacing issues, so this may not > be possible). The only thing is to insert a delay of about 500 nanoseconds. I tried using sleep for it but according to grep it wasn't implemented yet, so for the moment I'm using (loop for x from 1 to 1000 collecting x). Greetings, Peter Minten From ffjeld at common-lisp.net Wed Apr 21 21:34:59 2004 From: ffjeld at common-lisp.net (Frode Vatvedt Fjeld) Date: Wed, 21 Apr 2004 23:34:59 +0200 Subject: [movitz-devel] Re: HD driver References: <2h1xmsb4mq.fsf@vserver.cs.uit.no> <4082BB1D.408@wanadoo.nl> <2hpta3pnrj.fsf@vserver.cs.uit.no> <40856054.6040709@wanadoo.nl> Message-ID: <2hoeplm2v0.fsf@vserver.cs.uit.no> Peter Minten writes: > The only thing is to insert a delay of about 500 nanoseconds. I > tried using sleep for it but according to grep it wasn't implemented > yet, so for the moment I'm using (loop for x from 1 to 1000 > collecting x). Ok. First of all I'd suggest not to use consing as a delay mechanism. Secondly, if I'm not much mistaken, 500 nanoseconds is 500 cycles on a 1 GHz machine, so that loop expression is probably off by a factor of about 100 to 1000. Hm.. are you sure it's nanoseconds? I mean, 500 nanseconds is so little it's getting complicated to measure with any accuracy, not the least on older CPUs, or across the PCI bus. -- Frode Vatvedt Fjeld From peter.minten at wanadoo.nl Thu Apr 22 08:03:24 2004 From: peter.minten at wanadoo.nl (peter.minten at wanadoo.nl) Date: Thu, 22 Apr 2004 10:03:24 +0200 Subject: [movitz-devel] Re: HD driver In-Reply-To: <2hoeplm2v0.fsf@vserver.cs.uit.no> References: <2h1xmsb4mq.fsf@vserver.cs.uit.no> <4082BB1D.408@wanadoo.nl> <2hpta3pnrj.fsf@vserver.cs.uit.no> <40856054.6040709@wanadoo.nl> <2hoeplm2v0.fsf@vserver.cs.uit.no> Message-ID: <1082621004.40877c4c6c30c@webmail.wanadoo.nl> Citeren Frode Vatvedt Fjeld : > Peter Minten writes: > > > The only thing is to insert a delay of about 500 nanoseconds. I > > tried using sleep for it but according to grep it wasn't implemented > > yet, so for the moment I'm using (loop for x from 1 to 1000 > > collecting x). > > Ok. First of all I'd suggest not to use consing as a delay > mechanism. Secondly, if I'm not much mistaken, 500 nanoseconds is 500 > cycles on a 1 GHz machine, so that loop expression is probably off by > a factor of about 100 to 1000. > > Hm.. are you sure it's nanoseconds? I mean, 500 nanseconds is so > little it's getting complicated to measure with any accuracy, not the > least on older CPUs, or across the PCI bus. This is what the spec says: Upon receipt of a Class 1 command, the drive sets BSY within 400 nsec. But if it's only 400 cycles then it's likely enough that BSY is not read before it gets set (and possibly reset) in all cases (at least for testing). So I'll remove the loop. Greetings, Peter From ffjeld at common-lisp.net Thu Apr 22 14:40:16 2004 From: ffjeld at common-lisp.net (Frode Vatvedt Fjeld) Date: Thu, 22 Apr 2004 16:40:16 +0200 Subject: [movitz-devel] Re: HD driver References: <2h1xmsb4mq.fsf@vserver.cs.uit.no> <4082BB1D.408@wanadoo.nl> <2hpta3pnrj.fsf@vserver.cs.uit.no> <40856054.6040709@wanadoo.nl> <2hoeplm2v0.fsf@vserver.cs.uit.no> <1082621004.40877c4c6c30c@webmail.wanadoo.nl> Message-ID: <2hbrlkm5yn.fsf@vserver.cs.uit.no> peter.minten at wanadoo.nl writes: > This is what the spec says: > > Upon receipt of a Class 1 command, the drive sets BSY within 400 nsec. > > But if it's only 400 cycles then it's likely enough that BSY is not > read before it gets set (and possibly reset) in all cases (at least > for testing). So I'll remove the loop. Maybe it's possible to poll the drive's BSY register, so that the driver is truly synchronized with the drive? Anyway, I should probably construct some mechanism for small, calibrated delays such as this. But for now, I think to do nothing, or something simple like (dotimes (i 500)) will be good enough. Any success with reading or writing yet? -- Frode Vatvedt Fjeld From peter.minten at wanadoo.nl Thu Apr 22 17:04:44 2004 From: peter.minten at wanadoo.nl (Peter Minten) Date: Thu, 22 Apr 2004 19:04:44 +0200 Subject: [movitz-devel] Re: HD driver In-Reply-To: <2hk709okhj.fsf@vserver.cs.uit.no> References: <2h1xmsb4mq.fsf@vserver.cs.uit.no> <4082BB1D.408@wanadoo.nl> <2hk709okhj.fsf@vserver.cs.uit.no> Message-ID: <4087FB2C.9000502@wanadoo.nl> Frode Vatvedt Fjeld wrote: > Peter Minten writes: > > >>Furthermore on my box there is a problem with binary-types: > > > I checked in binary-types.lisp yesterday, iirc with a fix for this > that I received from someone a long time ago, but forgot to check in. > It works now, thanks. Greetings, Peter From peter.minten at wanadoo.nl Thu Apr 22 17:22:49 2004 From: peter.minten at wanadoo.nl (Peter Minten) Date: Thu, 22 Apr 2004 19:22:49 +0200 Subject: [movitz-devel] Re: HD driver In-Reply-To: <2hbrlkm5yn.fsf@vserver.cs.uit.no> References: <2h1xmsb4mq.fsf@vserver.cs.uit.no> <4082BB1D.408@wanadoo.nl> <2hpta3pnrj.fsf@vserver.cs.uit.no> <40856054.6040709@wanadoo.nl> <2hoeplm2v0.fsf@vserver.cs.uit.no> <1082621004.40877c4c6c30c@webmail.wanadoo.nl> <2hbrlkm5yn.fsf@vserver.cs.uit.no> Message-ID: <4087FF69.7060907@wanadoo.nl> Frode Vatvedt Fjeld wrote: > Maybe it's possible to poll the drive's BSY register, so that the > driver is truly synchronized with the drive? I'm currently doing busy waiting on BSY. But the problem is determining whether a zero means ready, or not yet set. > Any success with reading or writing yet? Well, in theory my current code should work. In practice I have to fix fairly trivial bugs first, which is time consuming given the lack of incremental compilation of the image and the long compile time (~10 minutes on my 1GHz box). Would it be possible to dump state at some point in compilation and start a new compilation from there instead of restarting from scratch? Even if that generates multi-meg images it would be a huge improvement in efficiency of compilation. Greetings, Peter From ffjeld at common-lisp.net Thu Apr 22 19:30:26 2004 From: ffjeld at common-lisp.net (Frode Vatvedt Fjeld) Date: Thu, 22 Apr 2004 21:30:26 +0200 Subject: [movitz-devel] Re: HD driver References: <2h1xmsb4mq.fsf@vserver.cs.uit.no> <4082BB1D.408@wanadoo.nl> <2hpta3pnrj.fsf@vserver.cs.uit.no> <40856054.6040709@wanadoo.nl> <2hoeplm2v0.fsf@vserver.cs.uit.no> <1082621004.40877c4c6c30c@webmail.wanadoo.nl> <2hbrlkm5yn.fsf@vserver.cs.uit.no> <4087FF69.7060907@wanadoo.nl> Message-ID: <2h3c6vn73h.fsf@vserver.cs.uit.no> Peter Minten writes: > Well, in theory my current code should work. In practice I have to > fix fairly trivial bugs first, which is time consuming given the > lack of incremental compilation of the image and the long compile > time (~10 minutes on my 1GHz box). But.. there is incremental compilation (!) For example, (movitz-compile-file "losp/tmp/harddisk.lisp") should do the right thing wrt. *image*, and this takes maybe one second on my machine. What movitz-mode.el does is e.g. upon M-C-x, to extract a top-level form into a temporary file and applying movitz-compile-file to it. Something similar should come into existence for Slime, but until then, at least one can use movitz-compile-file manually. -- Frode Vatvedt Fjeld From peter.minten at wanadoo.nl Fri Apr 23 17:33:47 2004 From: peter.minten at wanadoo.nl (Peter Minten) Date: Fri, 23 Apr 2004 19:33:47 +0200 Subject: [movitz-devel] Re: HD driver In-Reply-To: <2h3c6vn73h.fsf@vserver.cs.uit.no> References: <2h1xmsb4mq.fsf@vserver.cs.uit.no> <4082BB1D.408@wanadoo.nl> <2hpta3pnrj.fsf@vserver.cs.uit.no> <40856054.6040709@wanadoo.nl> <2hoeplm2v0.fsf@vserver.cs.uit.no> <1082621004.40877c4c6c30c@webmail.wanadoo.nl> <2hbrlkm5yn.fsf@vserver.cs.uit.no> <4087FF69.7060907@wanadoo.nl> <2h3c6vn73h.fsf@vserver.cs.uit.no> Message-ID: <4089537B.4060808@wanadoo.nl> Frode Vatvedt Fjeld wrote: > Peter Minten writes: > > >>Well, in theory my current code should work. In practice I have to >>fix fairly trivial bugs first, which is time consuming given the >>lack of incremental compilation of the image and the long compile >>time (~10 minutes on my 1GHz box). > > > But.. there is incremental compilation (!) > > For example, (movitz-compile-file "losp/tmp/harddisk.lisp") should do > the right thing wrt. *image*, and this takes maybe one second on my > machine. Thanks, now it works fast. > What movitz-mode.el does is e.g. upon M-C-x, to extract a top-level > form into a temporary file and applying movitz-compile-file to > it. Something similar should come into existence for Slime, but until > then, at least one can use movitz-compile-file manually. Or semi-manually: (defun l () (movitz::movitz-compile-file ...)) :-). Anyway I've uploaded a new version of the code. It compiles, it calls the harddisk and ... it fails. There are some issues I don't yet understand, like why it asks for a media change on a static HD. I'll implement identify-drive sometime these days, 128 bytes of info should be helpful :-). Greetings, Peter From a_bakic at yahoo.com Fri Apr 23 21:41:21 2004 From: a_bakic at yahoo.com (Aleksandar Bakic) Date: Fri, 23 Apr 2004 14:41:21 -0700 (PDT) Subject: [movitz-devel] Re: HD driver In-Reply-To: <4089537B.4060808@wanadoo.nl> Message-ID: <20040423214121.42042.qmail@web40604.mail.yahoo.com> > > What movitz-mode.el does is e.g. upon M-C-x, to extract a top-level > > form into a temporary file and applying movitz-compile-file to > > it. Something similar should come into existence for Slime, but until > > then, at least one can use movitz-compile-file manually. There is such a thing in slime-movitz.el. In case movitz-compile-defun is not bound to \M-\C-z\C-c, try evaluating (add-movitz-key-bindings). Please let me know if it does not work, I think it did work for me. Alex __________________________________ Do you Yahoo!? Yahoo! Photos: High-quality 4x6 digital prints for 25? http://photos.yahoo.com/ph/print_splash From ffjeld at common-lisp.net Fri Apr 23 22:10:12 2004 From: ffjeld at common-lisp.net (Frode Vatvedt Fjeld) Date: Sat, 24 Apr 2004 00:10:12 +0200 Subject: [movitz-devel] Re: HD driver References: <4089537B.4060808@wanadoo.nl> <20040423214121.42042.qmail@web40604.mail.yahoo.com> Message-ID: <2hy8oml517.fsf@vserver.cs.uit.no> Aleksandar Bakic writes: > There is such a thing in slime-movitz.el. In case > movitz-compile-defun is not bound to \M-\C-z\C-c, try evaluating > (add-movitz-key-bindings). Please let me know if it does not work, I > think it did work for me. Right, sorry.. I completely forgot about slime-movitz.el which of course people should use! And if there's problems, submit a bug-report or patch. -- Frode Vatvedt Fjeld From ffjeld at common-lisp.net Sat Apr 24 15:15:20 2004 From: ffjeld at common-lisp.net (Frode Vatvedt Fjeld) Date: Sat, 24 Apr 2004 17:15:20 +0200 Subject: [movitz-devel] Re: HD driver References: <2h1xmsb4mq.fsf@vserver.cs.uit.no> <4082BB1D.408@wanadoo.nl> <2hpta3pnrj.fsf@vserver.cs.uit.no> <40856054.6040709@wanadoo.nl> <2hoeplm2v0.fsf@vserver.cs.uit.no> <1082621004.40877c4c6c30c@webmail.wanadoo.nl> <2hbrlkm5yn.fsf@vserver.cs.uit.no> <4087FF69.7060907@wanadoo.nl> <2h3c6vn73h.fsf@vserver.cs.uit.no> <4089537B.4060808@wanadoo.nl> Message-ID: <2hu0z9l853.fsf@vserver.cs.uit.no> I checked in your newver version to CVS. One comment about this function from harddisk.lisp: (defun div (a b) "Floored integer division, the painful way." (let ((r 0) (x a)) (while (>= x 0) (decf x b) (incf r)) (1- r))) I suspect you can use the standard CL function truncate instead of this? -- Frode Vatvedt Fjeld From peter.minten at wanadoo.nl Sat Apr 24 16:16:03 2004 From: peter.minten at wanadoo.nl (Peter Minten) Date: Sat, 24 Apr 2004 18:16:03 +0200 Subject: [movitz-devel] Re: HD driver In-Reply-To: <2hu0z9l853.fsf@vserver.cs.uit.no> References: <2h1xmsb4mq.fsf@vserver.cs.uit.no> <4082BB1D.408@wanadoo.nl> <2hpta3pnrj.fsf@vserver.cs.uit.no> <40856054.6040709@wanadoo.nl> <2hoeplm2v0.fsf@vserver.cs.uit.no> <1082621004.40877c4c6c30c@webmail.wanadoo.nl> <2hbrlkm5yn.fsf@vserver.cs.uit.no> <4087FF69.7060907@wanadoo.nl> <2h3c6vn73h.fsf@vserver.cs.uit.no> <4089537B.4060808@wanadoo.nl> <2hu0z9l853.fsf@vserver.cs.uit.no> Message-ID: <408A92C3.2060801@wanadoo.nl> Frode Vatvedt Fjeld wrote: > I checked in your newver version to CVS. > > One comment about this function from harddisk.lisp: > > (defun div (a b) > "Floored integer division, the painful way." > (let ((r 0) > (x a)) > (while (>= x 0) > (decf x b) > (incf r)) > (1- r))) > > I suspect you can use the standard CL function truncate instead of > this? Yes, I forgot to grep for truncate (I did look at floor, but that wasn't implemented yet). Greetings, Peter From peter.minten at wanadoo.nl Tue Apr 27 17:25:53 2004 From: peter.minten at wanadoo.nl (Peter Minten) Date: Tue, 27 Apr 2004 19:25:53 +0200 Subject: [movitz-devel] Possible macroexpansion bug Message-ID: <408E97A1.9070101@wanadoo.nl> Hi, I've put up a new version of the harddisk code (http://www.il.fontys.nl/~silvernerd/harddisk.lisp), featuring a complete code rewrite (as usual :-). The problem is, it doesn't compile due to some weird macroexpansion that somehow turns the io-port code in define-register-accessors (invoked with clauses of (name offset type)) into (io-port (+ ... type)). I checked with CMUCL macroexpand-1 and it doesn't expand that way there. Note that I slightly modified the code after the last test, so there may be other errors coming first, but since those modifications are lower in the file (and consist only of renaming to the new naming scheme) I doubt it. (defmacro define-register-accessors (&rest clauses) (let ((gs-hdc (gensym "hdc-"))) `(progn ,@(mapcar #'(lambda (clause) `(defmacro ,(intern (concatenate 'string (string (first clause)) (string '-register))) (,gs-hdc) `(io-port (+ (slot-value ,,gs-hdc 'command-base) ,,(second clause)) ,,(third clause)))) clauses)))) Greetings, Peter From ffjeld at common-lisp.net Tue Apr 27 19:29:42 2004 From: ffjeld at common-lisp.net (Frode Vatvedt Fjeld) Date: Tue, 27 Apr 2004 21:29:42 +0200 Subject: [movitz-devel] Re: Possible macroexpansion bug References: <408E97A1.9070101@wanadoo.nl> Message-ID: <2hoepdi5i1.fsf@vserver.cs.uit.no> Peter Minten writes: > I've put up a new version of the harddisk code > (http://www.il.fontys.nl/~silvernerd/harddisk.lisp), featuring a > complete code rewrite (as usual :-). The problem is, it doesn't > compile due to some weird macroexpansion that somehow turns the > io-port code in define-register-accessors (invoked with clauses of > (name offset type)) into (io-port (+ ... type)). I checked with > CMUCL macroexpand-1 and it doesn't expand that way there. Yes, it's most likely a problem with my very messy and in-need-of-reimplementing backquote implementation. However, I'd suggest using an approach like the one I've adopted in x86-pc/dp8390.lisp for such register accessors. However, this needs some fixing because it assumes 8-bit registers exclusively right now. I'll look into it shortly. -- Frode Vatvedt Fjeld From james at unlambda.com Wed Apr 28 08:02:08 2004 From: james at unlambda.com (James A. Crippen) Date: Wed, 28 Apr 2004 08:02:08 -0000 Subject: [movitz-devel] Question about :sub-program Message-ID: I have a minor question about assembler syntax. What's the :sub-program frobnitz supposed to be doing here? (:jnz '(:sub-program () (:int 66))) Sometimes I see it with an argument: (:jl '(:sub-program (failed) (:int 112) (:halt) (:jmp 'failed))) It seems like it's an embedded subroutine. Am I right? I'm having a hard time working out its semantics from the reader. 'james -- James A. Crippen Lambda Unlimited 61.2204N, 149.8964W Recursion 'R' Us Anchorage, Alaska, USA, Earth Y = \f.(\x.f(xx))(\x.f(xx)) From ffjeld at common-lisp.net Wed Apr 28 09:01:32 2004 From: ffjeld at common-lisp.net (Frode Vatvedt Fjeld) Date: Wed, 28 Apr 2004 11:01:32 +0200 Subject: [movitz-devel] Re: Question about :sub-program References: Message-ID: <2h8yggiihf.fsf@vserver.cs.uit.no> james at unlambda.com (James A. Crippen) writes: > I have a minor question about assembler syntax. What's the > :sub-program frobnitz supposed to be doing here? The argument syntax is something like this: (quote (:sub-program (&optional (label (gensym))) &rest sub-program)) and this will insert the label and sub-program somewhere below that instruction in the program. That somewhere will have to be after e.g. a jmp or ret instruction. After the assembly reader is done, that argument is going to look the same as (quote