[clo-devel] cl-net upgrading update [2011-04-02]

[Erik Huelsmann]

Since this mail, I've been quietly working on setting up the VPS we
need anyway, while waiting for people to get learn about the move and
hopefully join the mailing list.

On Tue, Mar 29, 2011 at 11:35 PM, Erik Huelsmann <ehuels at gmail.com> wrote:
> The server running common-lisp.net has been running in the current
> configuration for a number of years now. Its Debian distribution is
> still Etch, which has been moved to the archives June 20th last year.
> As a consequence, we're locked out of even the most basic of
> maintenance needs: security updates.
>
> Bottom line: common-lisp.net as it is today can't be maintained
> anymore. We need to upgrade the system.
>
> We have two options for the system upgrade:
>
> 1. Copy the VPS and run dist-upgrade; then fix whatever falls over.
> 2. Create a new VPS, and make a clean start, meaning that we identify
> all services currently running on cl-net and we arrange for the same
> or equivalent services to be running on the new system.
>
>
> Stelian has indicated his preference is for option (2), as too much
> has been customized or is installed in custom locations to make a
> standard dist-upgrade work. I concur.

Well, the new VPS is up and running. The technical configuration isn't
quite to my liking, but we can definitely work with it.

> Another remark from Stelian is that we could migrate one service at a
> time, creating subdomains (svn.common-lisp.net, git.common-lisp.net,
> trac...., etc) which might help future migrations.

As far as the domains go: I've set up 'new.common-lisp.net' to point
to the new VPS; all the other domains will come when we're ready to go
live with the different services.

> So, from where we stand now, we need to:
>
> 1) Identify all services, users and data we need to migrate
> 2) Find people who want to help migrate the services
> 3) Per component, identify a migration plan - we can start migrating
> services with complete migration plans
> 4) Test the new clean setups as well as the combination of the new
> setup and the old data

While at it, I've aptitude installed most of the services mentioned below.

> Services that I know we're running [without looking at the box]:
>
>  * Mail [MTA]
- postfix, postfix-gld

>  * Web
- apache2

>  * Ticket tracker (RT - internal use)
- otrs2

>  * Mailing lists
- mailman

>  * Version control
- subversion, git, hg, darcs, cvs

>  * Repository browsing
>   - Subversion
>   - Git
>   - Darcs
>   - CVS (do we still want to support CVS ??)
>  * Trac
- trac, trac-mercurial, trac-xmlrpc, trac-git

Does anybody have experience with trac-mastertickets, trac-spamfilter,
trac-email2trac or trac-accountmanager? How would they fit with the
cl-net site setup? They all sound interesting.

>  * spam detection
- spamassasin; although postfix integration still needs setting up

>  * Git deamon
>  * rsync (what for?)
- rsync

>  * sshd
- sshd

>  * ftpd
--- Actually, no. The service is running but unused on current cl-net;
I think we can use http for downloads and use scp for uploads.

>  * lisppaste (on sbcl-0.9.7??)
-- to be done

>  * fast-cgi
- libapache2-mod-fastcgi

>  * custom administration scripts
-- to be done


With the status above, we're getting closer to the point where we
actually have to do something with the data and services on the
current machine.

Before we get that far, though, it would be nice if someone could
configure the OTRS instance I set up on the new box for cl-net use.
[I'll help, of course.]

I've been thinking about the next steps. The easy steps would be:

1. copy /project data to the new server
2. nfs-mount /project on the old box (if we use NFSv3, both Subversion
and CVS should be NFS safe, I understand)
3. migrate the anonymous read services for version control to the new box
4. copy and test the custom scripts, including installation of cron jobs etc
5. migrate trac.common-lisp.net to the new server

and from there it requires a bit more thought to get things right.
Remaining questions:
* how to migrate the users password and other secure data
* related: how to we bridge the gap between the time of the migration
of the user account data (/etc/passwd and /etc/group) and the
migration of the last service
* which services need account (login) data [I know about Subversion
commit, trac administration, cvs commit, website update; others?]

Well, more questions and ideas how to proceed are *very* welcome.

Bye,


Erik.