[clo-devel] 2011-04-15: cl-net migration progress

[Erik Huelsmann]

Hi Stelian,

Thanks for thinking with me on how to do this!

On Fri, Apr 15, 2011 at 11:44 PM, Stelian Ionescu <sionescu at cddr.org> wrote:
> On Fri, 2011-04-15 at 22:21 +0200, Erik Huelsmann wrote:
>> My work on cl-net migration got stuck a little on tech.coop internal
>> work this week. However, I'm fully back to the migration and I'm
>> making copies of /project and /home right now. I suppose I'll be
>> creating a copy of much inside /custom too, when the other two
>> transfers have finished.
>
> If you make a backup while the server is still accessible you'll lose
> data modified after the backup but before the restore and switch to the
> new server.

Absolutely agreed. So, copying the data from an lvm snapshot is not
the only action. However, in the past, I've been able to reduce
downtimes of servers to just minutes by copying the bulk of the data
in advance, followed by bringing down the server, doing a quick rsync
and bringing up the new server. It's the plan I intend to use this
time aswell.

> Here's how I'd do it:
> 1a) Start a new SSH server on old.cl.net(with a different RSA key) on a
> port other than 22 and move the current keys to new.cl.net
> 1b) Add an iptables redirect of port 22 to new.cl.net but don't start it
> on new.cl.net until the migration is done and kick out all people logged
> on the server
> 1c) Shutdown all services that might try to access /home and /project
> (apache, git, cvs, svn, etc...)
> 2a) Move /home and /project to new.cl.net
> 2b) Export them via a network FS to old.cl.net
> 3) Start the services and hope they work

My current status is that I copied 'hot' /project, /home and /custom.
I'll be able to switch cl-net off tonight, do the remaining copies and
turn old cl-net back on. That way, the data is on new-cl-net, but the
user accounts are still on old-cl-net.

This would allow us to migrate anonymous services such as Trac,
subversion, cvs and repository browsing.

Since the IP of the server will change, what do we gain by using the
same SSH key? I mean, ssh will tell the user the IP has changed, won't
it?




> If all goes well, all those services should be able to access the
> projects' data via network and we can start to migrate each service to
> new.cl.net whenever there's time
>
> BTW, do you think we should use something other than NFS, perhaps CIFS ?

Well, given the long history of *nix and NFS, I'd say they make a good
combination, at least for a small setup as this: we're going to phase
out the networked filesystem after the migration again; I'm not
anticipating adding further nodes into a grid or something alike.

>> At the same time, I'm trying to figure out how to best migrate
>> /etc/password- and /etc/group-. Anybody here done someting alike
>> before? I mean, I can't just copy it over, because the system accounts
>> shouldn't be overwritten. I suppose a 'grep -v' (for inverse matching)
>> should get us quite far filtering out system (uid<500)  accounts.
>
> Human users should maintain their passwd records(and implicitly, their
> uid). A little awk will do the trick

Ah. Thanks for pointing me to awk. It's definitely not a tool which is
top of mind for me. I've got a script which works for /etc/passwd and
can trivially be adapted to /etc/groups. All that remains is ofcourse
the migration of /etc/shadow and /etc/gshadow, which need to be
filtered for /etc/passwd and /etc/groups lines being migrated. This
should also be rather trivial.

>> Another issue I'm thinking about is that we're going to switch to
>> subdomains for hosting our services. However, the benefits of such a
>> change seem low if we don't switch to a structure where our data is
>> also grouped per service. Currently everything is in /project, which
>> is grouped around projects, requiring all services to run on a single
>> server because they need to access specific directories - one in each
>> project.
>> I'm thinking we should reverse the design: grouping data per service
>> and making /project a set of symbolic links into the service
>> directories. That way, all that's required is that the links point to
>> the right locations in the file system, possibly NFS mount points, if
>> the service were to run on a different server.
>
> This is a good idea, especially since it requires very little additional
> work, just a little care not to break users' existing setups

Right. However, short term steps (sorry, I started about it myself...) would be:

 * Migrate basic data, switching to NFS access [this I can do tonight]
 * Migrate the CRON jobs
 * Migrate the password and group files [after this step, people can
log in and test to see if svn, cvs, hg, darcs and git really work]
 * Switch ssh access to the new box
 * Migrate anonymous access services (www, subversion [svn://, add
http://], cvs, trac, git, others?]
 * Migrate mail service [Hans Huebner will take this up as soon as
he's got time]


Comments? Additions?

Erik.