Linux Containers seems to be the “new kid on the virtualization block” so I decided it was time to take the technology for a spin.
What are Linux Containers?
Linux Containers (LXC) are an operating system-level virtualization method for running multiple isolated server installs (containers) on a single control host. LXC does not provide a virtual machine, but rather provides a virtual environment that has its own process and network space. It is similar to a chroot, but offers much more isolation.
~ Dwight Schauer ~
Linux containers has several features / advantages:
Advantages:
- Better isolation as compared to a chroot (chroot jail).
- Low overhead. LXC uses minimal resources in terms of RAM and hard drive space without the overhead of installing a guest OS in a virtual machine ( VMWare / VirtualBox / KVM ) .
- Applications and services (servers) run at native speed.
- There is support for Linux containers in libvirt .
- Linux containers work well with btrfs .
- No special hardware is required, runs on 32 and 64 bit processors.
- Linux containers are Open source.
- Unlike XEN or OpenVZ , no patch is required to the kernel.
But there is also a downside:
- Linux containers run Linux processes on a Linux kernel. This means you can run Linux (Fedora container on an Ubuntu host) but not other operating systems (Not BSD / OSX / Windows).
- There are no GUI (graphical) interfaces to configure or manage the containers.
- There is a paucity of documentation on how to install and configure a container.
- Configuring a container requires a modest technical knowledge and skill (and a large grain of patience).
In this post I will show how to configure your host node using Fedora and Ubuntu as examples. This information is relatively generic and you should be able to adapt most of the information to other Linux hosts. Some distributions (Arch / SUSE) have specific reference pages listed at the end of this post.
Configuration of the Host takes 4 steps:
- Kernel.
- LXC tools.
- Cgroups.
- Bridge network card.
Kernel
The good news, Linux containers are part of the mainstream Linux kernel. This means there is no need to patch or compile a custom kernel.
To run Linux containers you need kernel >= 2.6.27. The stock or default kernels, 2.6.29 or greater, that ship with most Linux distributions work out of the box.
LXC tools
There are a set tools are a set of commands / scripts to create and manage containers.
lxc – These are the scripts used to manage containers. Home page .
bridge-utils – Although there are several options for networking, the consensus appears that the easiest ( ? best) option for networking is a bridge.
debootstrap / febootstrap – These tools allow one to install a minimal Debian/Ubuntu or Fedora root file system.
Fedora Host
su -c “yum install lxc bridge-utils febootstrap”
Ubuntu Host
sudo apt-get install lxc bridge-utils debootstrap
Alternately you may compile the most recent lxc package from source :
Get the source code (the lxc project also maintains a git repository for bleeding edgers). At the time of this post, lxc-0.6.5 was the most recent version and, since it was recently released, no updated git version was yet available.
lxc source code
wget http://lxc.sourceforge.net/download/lxc/lxc-0.6.5.tar.gz
tar xvf lxc-0.6.5.tar.gz
Install dependencies:
Feodra
su -c “yum install -y make gcc libcap libcap-devel”
Ubuntu
sudo apt-get -y install libcap-dev
Compile and install
cd lxc-0.6.5
./configure –prefix=/usr # two – - in front of “–prefix”
make
sudo make install # Ubuntu
su -c “make install” # Fedora
As you compile you may see a message about running lxc as a user (vs root). You may ignore this advice as I could not get lxc-setcap working (the majority of the lxc commands require root privileges even after enabling the “lxc-setcap”).
If you wish to have a non root user to use the lxc tools, you can add the needed capabilities to the tools by invoking the ‘lxc-setcap’ script. To remove the capabilities, use the ‘lxc-setcap -d’.
cgroups
AKA Control groups
cgroups are basically the configuration files that set and regulate (limit) resources available to the containers.
For additional information on cgroups see :
Linux Kernel Documentation – cgroups
Using CGroups with libvirt and LXC/KVM guests in Fedora 12
man lxc-cgroup
To configure cgroups, as root, run the following commands:
mkdir /cgroup
Using any editor, edit /etc/fstab to include:
none /cgroup cgroup defaults 0 0
/cgroups will now mount automatically when you boot your system. To mount it manually without rebooting:
mount /cgroup
Bridge network card
As outlined in the lxc.conf man page there are several options for networking. The consensus when I searched the documentation was to use a bridge, so that is what I will outline here. Feel free to try the other options (examples in lxc.conf man page).
Unfortunately, configuring a bridge must be done manually. Also Network Manager will not manage the bridge (nor will many graphical firewall tools).
Note: You can not bridge a wireless card.
Fedora
As root:
chkconfig NetworkManager off
chkconfig network on
service NetworkManager stop
service network start
Using any editor, edit /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
HWADDR=aa:bb:cc:dd:ee:ff:11:22 # DO NOT change your hardware mac
ONBOOT=yes
BRIDGE=br0
USERCTL=no
IPV6INIT=no
Make a new config file for your bridge
Using any editor, edit /etc/sysconfig/network-scripts/ifcfg-br0
To obtain an ip address automatically using DHCP:
DEVICE=br0
TYPE=Bridge
BOOTPROTO=dhcp
ONBOOT=yes
DELAY=0
To set a static IP:
HOSTNANE=fedora
DEVICE=br0
ONBOOT=yes
BOOTPROTO=static
DELAY=0
TYPE=Bridge
IPADDR=192.168.0.10
NETWORK=192.168.0.0
NETMASK=255.255.255.0
GATEWAY=192.168.0.1
MTU=1500
DNS1=192.168.0.1
IPV6INIT=no
USERCTL=no
Next, using any editor, edit /etc/sysconfig/network-scripts/ifup-post
Just above the “exit 0″ at the bottom add:
if [ $DEVNAME = "br0" ]; then
/usr/sbin/brctl setfd br0 0
fi
Restart your network:
service network restart
Note: You may need to manually edit /etc/resolv.conf and add your nameserver back in.
nameserver 192.168.0.1
Ubuntu
Turn network manager off or remove it.
sudo apt-get remove –purge network-manager network-manager-gnome
Stop your networking
service networking stop
Using any editor, edit /etc/network/interfaces adding lines for your bridge:
To obtain an ip address automatically using DHCP:
auto lo
iface lo inet loopback
auto br0
iface br0 inet dhcp
bridge_ports eth0
bridge_stp off
bridge_maxwait 5
post-up /usr/sbin/brctl setfd br0 0
To set a static IP:
auto lo
iface lo inet loopback
auto br0
iface br0 inet static
address 192.168.0.10
netmask 255.255.255.0
broadcast 192.168.0.255
gateway 192.168.0.1
bridge_ports eth0
bridge_stp off
bridge_maxwait 5
post-up /usr/sbin/brctl setfd br0 0
Using any editor, edit /etc/resolv.conf and add in your nameserver
nameserver 192.168.0.1
Managing containers
Containers, or guests, are built either with lxc tools (lxc-debian or lxc-fedora), debootstrap / febootstrap, or by converting openvz containers.
In Fedora, lxc-debian is not working. In Ubuntu neither lxc-fedora nor lxc-debian are working (and there is no lxc-ubuntu yet).
OpenVZ containers may be old / out of date and will require some modification.
Thus, of the 3 options I prefer bootstraping a minimal container.
In practice, creating a container takes some time and in my next blogs I will post detailed instructions for Fedora and Ubuntu.
In this post I will assume you have a working root file system, I call mine rootfs.fedora , rootfs.ubuntu, etc
Note: The commands to manage containers are run as root.
Create a container
To use LXC to provide VPS you need two things, a directory containing the file system, aka rootfs (see the above section), and a configuration file.
To create a config file for your container I suggest you either editing a generic template or better, read the man page and write one from scratch. See man lxc.conf for options
Basic file (very slightly modified from the default config file generated from lxc-fedora on a Fedora host):
lxc.utsname = container_name
lxc.tty = 4
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.name = eth0
lxc.network.mtu = 1500
lxc.network.ipv4 = your_network_here #for example 192.168.0.0/24
lxc.rootfs = /full/path/to/rootfs
lxc.cgroup.devices.deny = a
# /dev/null and zero
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
# consoles
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 4:0 rwm
lxc.cgroup.devices.allow = c 4:1 rwm
# /dev/{,u}random
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
# /dev/pts/* - pts namespaces are "coming soon"
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 5:2 rwm
# rtc
lxc.cgroup.devices.allow = c 254:0 rwm
Basically you will edit 3 lines (although you may use additional options, see man lxc.conf).
lxc.utsname = container_name
lxc.network.ipv4 = your_network_here (ie 192.168.0.0/24)
lxc.rootfs = /full/path/to/rootfs
The lines with “lxc.group …” define which resources are available to the container and configure confinement. The line “lxc.cgroup.devices.deny = a” denies access to everything and exceptions are added in in the following lines as needed. Yes, the terminology is cryptic, and for a better understanding I will have to refer you to the cgroup references in this blog.
Once you have a rootfs and a config file, run lxc-create :
lxc-create -f /path_to/config.file -n name_of_container
For example:
lxc-create -f /root/config.fedora -n fedora
Start a container
The basic command is lxc-start -n name_of_container
lxc-start -n fedora
Starting this way will show you the boot messages you would normally get when booting your distro. Watch for error messages (you will need to fix them manually).
Once the container is working, you can start it with the -d (demonize) option.
lxc-start -d -n fedora
Note: Some containers may not start with the -d option, in that case, IMO, better to use screen:
screen -dmS lxc_container_name lxc-start -n lxc_container_name
For example:
screen -dmS ubuntu lxc-start -n ubuntu
Stop a container
lxc-stop -n name_of_container
For example:
lxc-stop -n fedora
Destroy a container
lxc-destroy -n name_of_container
For example:
lxc-destroy -n fedora
Manage resources
The resources available to your containers is managed via cgroups. You manage them with the lxc-cgroup command. The resources are listed as flat text files in /cgroup/name_of_container .
A detailed discussion of cgroup is beyond this post, but I may devote a separate blog to this topic after finishing instructions for Fedora and Ubuntu containers. See the links in the cgroup section above for helpful links to “get you started”.
Tips
A few comments on the structure of containers on the host file system.
1. /var/lib/lxc : Container configuration files are kept in /var/lib/lxc . It makes life easier to make a link or bind /var/lib/lxc to /lxc
mkdir /lxc
ln -s /var/lib/lxc /lxc
2. /var/cache/lxc : Used by lxc-debian and lxc-fedora as a cache for downloading and installing a rootfs .
3. /cgroup/name_of_container : Location of container cgroup files.
4. lxc-fedora
When using lxc-fedora, the root file system is downloaded to /var/cache/lxc/fedora and then copied to your current working directory. This makes it much faster the next time you invoke lxc-fedora as it will copy the cached rootfs to your current working directory without repeating the downloading and installation of the base packages.
So, for example, if you lxc-fedora create, you will be asked a few questions, including the name of your container and network configuration.
If you run the command from /root, the script will cache and install the packages in /var/cache/lxc/fedora, copy the /var/cache/lxc/fedora to /root/name_of_container, and generate a configuration file based on the options you give.
When you then create a container, with lxc-create, the configuration files are generated in /var/lib/lxc , and a link is generated for the rootfs to /root/fedora (assuming you names the container fedora).
This behavior is NOT emulated if you use bootstrap to make a rootfs manually or import (convert) an openvz template.
The down side of lxc-fedora is that the container still needs some modification and it is larger then if you use febootstrap.
See my blogs on generating a Fedora and Ubuntu container for detailed instructions.
5. IMHO, containment is very poor if you chroot into the container form the host node. Use lxc-console or ssh into the container.
References:
