crablfs: User Based Package Management System -- User's Concise Manual

Author: (Chowroc)
Date: 2006-10-25
Email: chowroc.z@gmail.com

(1) abstract:
crablfs: User Based Package Management System which is
written in python. Every package and its files belong to a system
normal user, and a batch script to rebuild or migrate your
personalized LFS/BLFS system by install all your packages
automatically all at once. The goal of "crablfs" is to exert more
control on packages that compiled from source, and reduce the costs
of system migration and backup.

The basic theory is: every package in the system has a corresponding
user it belongs to, the installation actions(copy, mkdir, etc) will
run by this user, thus all the files a package generated have the
mode of this user. At the same time, all those users belongs to a
'install' group, and the key directories in the system such as
/usr/local, /usr/local/bin, etc have g+s,o+t mode, and owned to the
'install' group, to make different packages create their own files
and directories in those directories.

There are many benefits to use this method. which can be looked up
from the "LFS hints":

More Control and Package Management using Package Users (v1.2)
http://www.linuxfromscratch.org/hints/downloads/files/more_control_and_pkg_man.txt

Based on this theory, the package manager "userpack" is a command
line interface like rpm and paco, for finishing some necessary
operations. It can install, remove, query a package, or list all
packages of current system, or find the package owner of a specified
file. the package name($pkgname-$version) will add to the list file
(/usr/src/packages.list for default).

Every package user has its own HOME dir, which is default to
/usr/src/$pkgname. It will stores compressed source archives, all
patches and supplementary files for installation, and installation
profiles which contains the information of the package, such as
user/group, source archive, patches, create time and those commands
to build this package.

By default, "userpack" use interactive mode, which will start a
shell like command line, in which you type "cmd $command" to execute
the corresponding command, and those commands will been recorded to
"installation profile"(described before). This profile can be used
to execute installation commands automatically in noninteractive
mode.

The script "crablfs" is a batch script write in python, it is
another ALFS instance in other words, which make use of the
automated ability of "userpack". It can use the list
file(/usr/src/packages.list) mentioned before, to operate the
compressed source archives and installation profiles you appointed
automatically. And you can also specified a directory that has the
same structure as $homepre(/usr/src) to build a ALFS system
automatically. This means that you can migrate from an old system to
new ones more easily -- and rebuild your personalized system more
quickly.

The essential differences among Linux distributions is the
differences of the package management ways. As a distribution, LFS
actually does not has a mature package manager itself, this of
course roots to the flexibility of LFS, but for daily use, a mature
package management system is necessary. It is recommended to use
User Based Management in the BLFS document, but the LFS hints only
give the basic principles which is very creative, but the shell
scripts is not enough to form a self-consistency system. So I write
one regardless of unveil my superficialness, a hope that my crude
remarks may draw forth by abler people.

Of course crablfs is not only used for LFS, it aims to those
packages that build from source code, so it can used on other
distributions, too.

It's now available from:
http://cheeseshop.python.org/packages/source/c/crablfs/

(2) install crablfs
To make sure that crablfs also under the control of package
management, to install by this way:
# export PYTHONPATH=/opt/lib/python2.4/site-packages
// sys.path will not be influenced by PYTHONPATH
# tar xfz pexpect-2.1.tar.gz
# cd pexpect-2.1
# python setup.py install --prefix=/opt
# cd ..

# tar xfz crablfs-0.1.tar.gz
# cd crablfs-0.1
# vi userpack.dirs
// define all key directory
// these dirs will belong to install groupand have g+s,o+t mode
/usr/local/
/usr/local/bin/
/usr/local/doc/
/usr/local/etc/
/usr/local/include/
/usr/local/info/
/usr/local/lib/
/usr/local/libexec/
/usr/local/man/
/usr/local/man/man1/
/usr/local/man/man2/
/usr/local/man/man3/
/usr/local/man/man4/
/usr/local/man/man5/
/usr/local/man/man6/
/usr/local/man/man7/
/usr/local/man/man8/
/usr/local/sbin/
/usr/local/share/
/usr/local/share/doc/
/usr/local/share/info/
/usr/local/share/locale/
/usr/local/share/man/
/usr/local/share/man/man1/
/usr/local/share/man/man2/
/usr/local/share/man/man3/
/usr/local/share/man/man4/
/usr/local/share/man/man5/
/usr/local/share/man/man6/
/usr/local/share/man/man7/
/usr/local/share/man/man8/
/usr/local/share/misc/
/usr/local/share/terminfo/
/usr/local/share/zoneinfo/
/usr/lib/python2.4/site-packages/
/usr/lib64/python2.4/site-packages/
// Adjust it for you own needs
# python setup.py install --prefix=/opt
# cd ..

# /opt/bin/userpack init
# /opt/bin/userpack install -f /tmp/crablfs-0.1.tar.gz crablfs-0.1
crablfs> cmd tar xfz crablfs-0.1.tar.gz
crablfs> cmd cd crablfs-0.1
crablfs> cmd python setup.py install --install-scripts=/usr/local/bin
crablfs> cmd cd ..
crablfs> cmd rm -rf crablfs-0.1
crablfs> commit

# userpack install -f /tmp/pexpect-2.1.tar.gz pexpect-2.1
crablfs> cmd tar xfz pexpect-2.1.tar.gz
crablfs> cmd cd pexpect-2.1
crablfs> cmd python setup.py install
crablfs> cmd cd ..
crablfs> cmd rm -rf pexpect-2.1
crablfs> commit

# unset PYTHONPATH

(3) package management:
manager: userpack

a. install a package:
# userpack install $pkgname-$version
for example:
# userpack install rxvt-2.7.10
you can classify:
# userpack install meida.mplayer-1.0pre8
because the system support '.' for user name.

***
NOTICE:
* It is not true for LFS system,the shadow package that LFS use
only support [-a-z0-9_] chars, so I wrote a shadow module to access
/etc/passwd, /etc/group, /etc/shadow, /etc/gshadow directly.

*. The classification function have not been tested strictly.
***

Under the conditions above, the default action will taken, that is:
the compressed source archive and patches are considered to have
been in the dir of HOME(/usr/src/$pkgname), other will you should
specify it with argument:
# userpack install -f /mnt/file/packages/rxvt-2.7.10.tar.gz rxvt-2.7.10
to make those archive and patches to be copied to HOME.

This command will start an interactive command line, it is simple
and only several command. In fact it is like a shell, you can type
any 'shell' command to install the package, only need add 'cmd'
before the command to make the 'userpack' records the commands.
These commands will be stored into HOME/.config(installation
profile).
crablfs> cmd tar xfz rxvt-2.7.10.tar.gz
crablfs> cmd cd rxvt-2.7.10
crablfs> cmd ./configure
crablfs> cmd make
crablfs> cmd make install
crablfs> cmd cd ..
crablfs> cmd rm -rf rxvt-2.7.10
crablfs> commit

You should known that the current user is the package user, not
root; and current working directory is the user's HOME, so there may
be ownership problem(What is the purpose of User Based Package
Management System).

When you type commands, you can use 'list' to see what you have typed:
crablfs> list
0, tar xfz rxvt-2.7.10.tar.gz
1, cd rxvt-2.7.10
2, ./configure
3, make
4, make install

Use 'rollback' to clean up these, and 'del N' to delete specified
command.

You must 'commit/cmt' at last, thus all the commands will be
recorded to the .config, the $pkgname-$version couple will been add
to list file(/usr/src/packages.list for default), to indicate that
the package has been installed. If there is a command error makes
the installation failed, and the fault can't been clean up by
yourself, just type 'quit' and the installation terminated, but the
archive and patches will still be copied to HOME.

If there is any other supplementary files:
# userpack install -f /mnt/file/packages/MPlayer-1.0per8.tar.bz2 \
    -p /mnt/file/packages/all-20060611.tar.bz2 \
    -p /mnt/file/packages/Blue-1.6.tar.bz2 mplayer-1.0pre8
// all is mplayer's codecsBlue is GUI Skin.

If you have the installation profile before, you can use -a|--auto
to install automatically:
# userpack install -a rxvt-2.7.10
This means the HOME/.config exists, otherwise use -c|--profile:
# userpack install -ac /mnt/packages/confiles/rxvt-2.7.10 rxvt-2.7.10

If you have burned /usr/src to DCD-R's sources dir, you can also
execute the automatically installation below:
# userpack install -as /mnt/dvdrom/sources rxvt-2.7.10

Moreover, the sources could also on the net such as ftp or ssh:
ssh://localhost/sources/mlterm-2.9.3.tar.gz
scp://localhost/sources/mlterm-2.9.3.tar.gz
the 2 above is same,
ftp://localhost/pub/sources/mlterm-2.9.3.tar.gz
for example:
# userpack install \
    -f ftp://192.168.0.1/pub/sources/mlterm-2.9.3.tar.gz mlterm-2.9.3

***
If the package name(mlterm-2.9.3) is not given, it can be parsed
from the archive name(mlterm-2.9.3.tar.gz) automatically by
'userpack'.
***

If a archive has problem and makes installation failed, the archive
still has been copied to HOME, and the next time 'userpack' will
check whether the archive exists in HOME, if true, copy will not
occur. To change this, use -C|--copy-force:
# userpack install -Caf libesmtp-1.0.3r1.tar.bz2

About the user/group ID, 'userpack' can define its own range,
default from 1000 to 20000, U can define in /etc/login.defs:
PACK_UID_MIN    10000
PACK_UID_MAX    25000
PACK_GID_MIN    10000
PACK_GID_MAX    25000

b. remove:
# userpack remove rxvt-2.7.10

c. list all packages under the control:
# userpack packs
rxvt-2.7.10
mplayer-1.0pre8
crablfs-0.1
pexpect-2.1
mlterm-2.9.3

d. query a file's package owner:
# userpack owner /usr/local/bin/mlterm
mlterm-2.9.3

e. list all files a package contains:
# userpack files pexpect-2.1
/usr/lib/python2.4/site-packages/fdpexpect.py
/usr/lib/python2.4/site-packages/pxssh.py
/usr/lib/python2.4/site-packages/pexpect.py
/usr/lib/python2.4/site-packages/pxssh.pyc
/usr/lib/python2.4/site-packages/pexpect.pyc
/usr/lib/python2.4/site-packages/fdpexpect.pyc

(4) About the pkgname & version rule:
This is the argument to
'userpack', the form is '$pkgname-$version', the python re pattern
is:
'^(?P<pkgname>(\w+-{0,1})+)(-(?P<version>(\d[\w]*[-\.])*\d[\w]*))'

That is '$pkgname' part is 'part1[-part2][-part3][...]', every part
could contains '[-a-zA-Z0-9_]', no rules for start char, but it is
recommended to use alphabet; when meet '-[0-9][a-zA-Z0-9_]\.'
pattern('-' follows a string start with digit, and follows a '.'),
this will be considered as start part of $version, $version can also
contains other '-', and when meet a string not start with digit,
$version ends. chars follows are ignored.

(5) ALFS
It is mentioned before that all commands will be recorded to
HOME/.config, and 'userpack' can also use this commands to run
automatically, so there could be a batch script to use packages.list
to install those packages all at once. Based on this, we can get an
instance of ALFS.

The target of the instance is extended to tranditional nALFS and
jhalfs, for more personalized environment, for example: The first
time, I build LFS/BLFS step by step, and the packages installed,
related with their own's information & operations are recorded to
the profiles, the next time, I can use those profiles to build the
software envrionment I need with a 'turn key' way, exactly as those
before. This means I can migrate many times, and others can also use
my profiles to build their own personalized system. The end target
also contains a platform independent operation.

This batch script is 'crablfs'.

***
So far, it is only tested under the condition that the LFS base
system has been built, to use 'crablfs' to build a bunch of BLFS
packages, so it is the future to make the LFS base system under the
control of package management system.
***

At first, adjust the tool chain:
# cd /blfs-sources
# tar xfj python-2.4.1.tar.bz2
# cd python-2.4.1
# patch -Np1 -i ../python-2.4.1-gdbm-1.patch
# ./configure --prefix=/opt --enable-shared
# make && make install
# cd ..
# rm -rf python-2.4.1
// U'd better delete it, otherwise may be problems

# export PYTHONPATH=/opt/lib/python2.4/site-packages
# tar xfz pexpect-2.1.tar.gz
# cd pexpect-2.1
# /opt/bin/python setup.py install --prefix=/opt
# cd ..
# rm -rf pexpect-2.1

# tar xfz crablfs-0.1.tar.gz
# cd crablfs-0.1
# cp userpack.dirs.blfs userpack.dirs
// Adjust for you own need and system
// Whenever U meet pemission problem, just adjust /etc/userpack.dirs
//  and run 'userpack init'
# /opt/bin/python setup.py install --prefix=/opt

# /opt/bin/userpack init
# /opt/bin/userpack install -f python-2.4.1.tar.bz2 -p
# python-2.4.1-gdbm-1.patch python-2.4.1
crablfs> cmd tar xfj python-2.4.1.tar.bz2
crablfs> cmd cd python-2.4.1
crablfs> cmd ./configure --prefix=/usr --enable-shared
crablfs> cmd make
crablfs> cmd make install
crablfs> cmd cd ..
crablfs> cmd rm -rf python-2.4.1
crablfs> commit

# /opt/bin/userpack install -f crablfs-0.1.tar.gz crablfs-0.1
crablfs> cmd tar xfz crablfs-0.1.tar.gz
crablfs> cmd cd crablfs-0.1
crablfs> cmd python setup.py install --install-scripts=/usr/local/bin
crablfs> cmd cd ..
crablfs> cmd rm -rf crablfs-0.1
crablfs> commit

# userpack install -f pexpect-2.1.tar.gz pexpect-2.1
crablfs> cmd tar xfz pexpect-2.1.tar.gz
crablfs> cmd cd pexpect-2.1
crablfs> cmd python setup.py install
crablfs> cmd cd ..
crablfs> cmd rm -rf pexpect-2.1
crablfs> commit

# unset PYTHONPATH
# userpack packs
python-2.4.1
crablfs-0.1
pexpect-2.1

And use the same way, I installed some packages:
# userpack packs
# cat /usr/src/packages.list
python-2.4.1
crablfs-0.1
pexpect-2.1
openssl-0.9.7g
cracklib-2.8.3
Linux-PAM-0.80
iptables-1.3.3
gnupg-1.4.1
pcre-6.1
libxml-1.8.17
libxml2-2.6.20
libxslt-1.1.14
gdbm-1.8.3
pkg-config-0.19
glib2-2.6.4
expat-1.95.8
libesmtp-1.0.3r1
lzo-2.01
libusb-0.1.10a
libjpeg-6b
libpng-1.2.8
which-2.16

# mv /usr/src/packages.list /blfs-sources/
# config.copy /usr/src
// This will generate a profiles dir in current dir,
// which contains all the profiles from $homepre(/usr/src)
// This shell script will not be installed,
// you can find it in the archive
# mv profiles /blfs-sources
# crablfs -t alfs \
    -C /blfs-sources/profiles/ \
    -F /blfs-sources/ \
    /blfs-sources/packages.list

If you have installed such a BLFS system and burned /usr/src all to
the DVD-R, you can do this:
# crablfs -t crablfs[/default] \
    -s /mnt/dvdrom/sources/ \
    /mnt/dvdrom/sources/packages.list

crablfs will record current successfully installed package name to
/var/log/crablfs/.mark. Because of the packages dependencies
problems, if one package failed, the 'crablfs' process terminated.
After you clean up the problem, you can continue from the
breakpoint.

So if .mark point to the last item of packages.list, 'crablfs' will
do nothing. Adjust it for you need.

Moreover, 'crablfs' will check packages.list, illegal package name
will terminate 'crablfs' and do nothing; if the name in .mark not in
packages.list, you will get a "x not in list" error report, you can
adjust corresponding, or just delete the .mark file.

So far, there are few permission problems, and all can avoid by
changing the installtion arguments. The wrappers for chmod, chgrp,
chown, install may be needed, but so far I have not see the
neccessary. To dicide this for further using and testing.

(6) The next plan
* BLFS crablfs tool chain adjusting shell script
* Add upgrade part
* Add gettext internationlized support
* Whole infomations show of a package
* PyUnit
* Enganced cmdline completion function
* Add code for cmdline to read .config history commands
* A command editor?
* File lock of packages.list
* Change .config to the form of '$pkgname-$version'
* Enganced platform independent support:
    Add var and env var to cmdline for different $ARCH & $LOCALE
* Support multi installation sources
* Enganced classification mechanism
* Intergrated checking
* Enganced configuration getting & setting: Plain Tree & XML
* Consider package dependencies problems
* Better documents: internal, design doc and manual

*********************************************************************
This is the first time I distribute my work, because the lack of
ability, I can't say I have done it commendably, I will try my best
to improve it in the future, so hope every can give me more
feedbacks.

I hope this program is useful for you, and I also hope that we can
get a more convenient LFS distribution from this way.

Thank you.
*********************************************************************

crablfs Copyright (c) 2006, (chowroc.z@gmail.com)

This file is part of crablfs.

crablfs is free software; you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation; either version 2 of the License, or
(at your option) any later version.

crablfs is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with Foobar; if not, write to the Free Software
Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
