LXC – Linux Containers

Linux Containers seems to be the “new kid on the virtualization block” so I decided it was time to take the technology for a spin.

What are Linux Containers?

Linux Containers (LXC) are an operating system-level virtualization method for running multiple isolated server installs (containers) on a single control host. LXC does not provide a virtual machine, but rather provides a virtual environment that has its own process and network space. It is similar to a chroot, but offers much more isolation.

~ Dwight Schauer ~

Linux containers has several features / advantages:

Advantages:

  • Better isolation as compared to a chroot (chroot jail).
  • Low overhead. LXC uses minimal resources in terms of RAM and hard drive space without the overhead of installing a guest OS in a virtual machine ( VMWare / VirtualBox / KVM ) .
  • Applications and services (servers) run at native speed.
  • There is support for Linux containers in libvirt .
  • Linux containers work well with btrfs .
  • No special hardware is required, runs on 32 and 64 bit processors.
  • Linux containers are Open source.
  • Unlike XEN or OpenVZ , no patch is required to the kernel.

But there is also a downside:

  • Linux containers run Linux processes on a Linux kernel. This means you can run Linux (Fedora container on an Ubuntu host) but not other operating systems (Not BSD / OSX / Windows).
  • There are no GUI (graphical) interfaces to configure or manage the containers.
  • There is a paucity of documentation on how to install and configure a container.
  • Configuring a container requires a modest technical knowledge and skill (and a large grain of patience).

In this post I will show how to configure your host node using Fedora and Ubuntu as examples. This information is relatively generic and you should be able to adapt most of the information to other Linux hosts. Some distributions (Arch / SUSE) have specific reference pages listed at the end of this post.

Configuration of the Host takes 4 steps:

  1. Kernel.
  2. LXC tools.
  3. Cgroups.
  4. Bridge network card.

Kernel

The good news, Linux containers are part of the mainstream Linux kernel. This means there is no need to patch or compile a custom kernel.

To run Linux containers you need kernel >= 2.6.27. The stock or default kernels, 2.6.29 or greater, that ship with most Linux distributions work out of the box.

LXC tools

There are a set tools are a set of commands / scripts to create and manage containers.

lxc – These are the scripts used to manage containers. Home page .

bridge-utils – Although there are several options for networking, the consensus appears that the easiest ( ? best) option for networking is a bridge.

debootstrap / febootstrap – These tools allow one to install a minimal Debian/Ubuntu or Fedora root file system.

Fedora Host

su -c “yum install lxc bridge-utils febootstrap”

Ubuntu Host

sudo apt-get install lxc bridge-utils debootstrap

There is also a LXC PPA.

Alternately you may compile the most recent lxc package from source :

Get the source code (the lxc project also maintains a git repository for bleeding edgers). At the time of this post, lxc-0.6.5 was the most recent version and, since it was recently released, no updated git version was yet available.

lxc source code

wget http://lxc.sourceforge.net/download/lxc/lxc-0.6.5.tar.gz
tar xvf lxc-0.6.5.tar.gz

Install dependencies:

Feodra

su -c “yum install -y make gcc libcap libcap-devel”

Ubuntu

sudo apt-get -y install libcap-dev

Compile and install

cd lxc-0.6.5
./configure --prefix=/usr
make
sudo make install # Ubuntu
su -c “make install” # Fedora

As you compile you may see a message about running lxc as a user (vs root). You may ignore this advice as I could not get lxc-setcap working (the majority of the lxc commands require root privileges even after enabling the “lxc-setcap”).

If you wish to have a non root user to use the lxc tools, you can add the needed capabilities to the tools by invoking the ‘lxc-setcap’ script. To remove the capabilities, use the ‘lxc-setcap -d’.

cgroups

AKA Control groups

cgroups are basically the configuration files that set and regulate (limit) resources available to the containers.

For additional information on cgroups see :

Linux Kernel Documentation – cgroups
Using CGroups with libvirt and LXC/KVM guests in Fedora 12
man lxc-cgroup

To configure cgroups, as root, run the following commands:

mkdir /cgroup

Using any editor, edit /etc/fstab to include:

none /cgroup cgroup defaults 0 0

/cgroups will now mount automatically when you boot your system. To mount it manually without rebooting:

mount /cgroup

Bridge network card

As outlined in the lxc.conf man page there are several options for networking. The consensus when I searched the documentation was to use a bridge, so that is what I will outline here. Feel free to try the other options (examples in lxc.conf man page).

Unfortunately, configuring a bridge must be done manually. Also Network Manager will not manage the bridge (nor will many graphical firewall tools).

Note: You can not bridge a wireless card.

Fedora

As root:

chkconfig NetworkManager off
chkconfig network on
service NetworkManager stop
service network start

Using any editor, edit /etc/sysconfig/network-scripts/ifcfg-eth0

DEVICE=eth0
TYPE=Ethernet
HWADDR=aa:bb:cc:dd:ee:ff:11:22 # DO NOT change your hardware mac
ONBOOT=yes
BRIDGE=br0
USERCTL=no
IPV6INIT=no

Make a new config file for your bridge

Using any editor, edit /etc/sysconfig/network-scripts/ifcfg-br0

To obtain an ip address automatically using DHCP:

DEVICE=br0
TYPE=Bridge
BOOTPROTO=dhcp
ONBOOT=yes
DELAY=0

To set a static IP:

HOSTNANE=fedora
DEVICE=br0
ONBOOT=yes
BOOTPROTO=static
DELAY=0
TYPE=Bridge
IPADDR=192.168.0.10
NETWORK=192.168.0.0
NETMASK=255.255.255.0
GATEWAY=192.168.0.1
MTU=1500
DNS1=192.168.0.1
IPV6INIT=no
USERCTL=no

Next, using any editor, edit /etc/sysconfig/network-scripts/ifup-post

Just above the “exit 0″ at the bottom add:

if [ $DEVNAME = "br0" ]; then
/usr/sbin/brctl setfd br0 0
fi

Restart your network:

service network restart

Note: You may need to manually edit /etc/resolv.conf and add your nameserver back in.

nameserver 192.168.0.1

Ubuntu

Turn network manager off or remove it.

sudo apt-get purge network-manager network-manager-gnome

Stop your networking

service networking stop

Using any editor, edit /etc/network/interfaces adding lines for your bridge:

To obtain an ip address automatically using DHCP:

auto lo
iface lo inet loopback

auto br0
iface br0 inet dhcp
bridge_ports eth0
bridge_stp off
bridge_maxwait 5
post-up /usr/sbin/brctl setfd br0 0

To set a static IP:

auto lo
iface lo inet loopback

auto br0
iface br0 inet static
address 192.168.0.10
netmask 255.255.255.0
broadcast 192.168.0.255
gateway 192.168.0.1
bridge_ports eth0
bridge_stp off
bridge_maxwait 5
post-up /usr/sbin/brctl setfd br0 0

Using any editor, edit /etc/resolv.conf and add in your nameserver

nameserver 192.168.0.1

Managing containers

Containers, or guests, are built either with lxc tools (lxc-debian or lxc-fedora), debootstrap / febootstrap, or by converting openvz containers.

In Fedora, lxc-debian is not working. In Ubuntu neither lxc-fedora nor lxc-debian are working (and there is no lxc-ubuntu yet).

OpenVZ containers may be old / out of date and will require some modification.

Thus, of the 3 options I prefer bootstraping a minimal container.

In practice, creating a container takes some time and in my next blogs I will post detailed instructions for Fedora and Ubuntu.

In this post I will assume you have a working root file system, I call mine rootfs.fedora , rootfs.ubuntu, etc

Note: The commands to manage containers are run as root.

Create a container

To use LXC to provide VPS you need two things, a directory containing the file system, aka rootfs (see the above section), and a configuration file.

To create a config file for your container I suggest you either editing a generic template or better, read the man page and write one from scratch. See man lxc.conf for options

Sample configuration file (very slightly modified from the default config file generated from lxc-fedora on a Fedora host):

lxc.utsname = container_name
lxc.tty = 4
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br0
lxc.network.name = eth0
lxc.network.mtu = 1500
lxc.network.ipv4 = your_network_here #for example 192.168.0.0/24
lxc.rootfs = /full/path/to/rootfs.container_name
lxc.mount = /full/path/to/fstab.container_name
lxc.cgroup.devices.deny = a
# /dev/null and zero
lxc.cgroup.devices.allow = c 1:3 rwm
lxc.cgroup.devices.allow = c 1:5 rwm
# consoles
lxc.cgroup.devices.allow = c 5:1 rwm
lxc.cgroup.devices.allow = c 5:0 rwm
lxc.cgroup.devices.allow = c 4:0 rwm
lxc.cgroup.devices.allow = c 4:1 rwm
# /dev/{,u}random
lxc.cgroup.devices.allow = c 1:9 rwm
lxc.cgroup.devices.allow = c 1:8 rwm
# /dev/pts/* - pts namespaces are "coming soon"
lxc.cgroup.devices.allow = c 136:* rwm
lxc.cgroup.devices.allow = c 5:2 rwm
# rtc
lxc.cgroup.devices.allow = c 254:0 rwm

Basically you will edit 4 lines (although you may use additional options, see man lxc.conf).

lxc.utsname = container_name
lxc.network.ipv4 = your_network_here (ie 192.168.0.0/24)
lxc.rootfs = /lxc/rootfs.container_name
lxc.fstab = /lxc/fstab.container_name

The lines with “lxc.group …” define which resources are available to the container and configure confinement. The line “lxc.cgroup.devices.deny = a” denies access to everything and exceptions are added in in the following lines as needed. Yes, the terminology is cryptic, and for a better understanding I will have to refer you to the cgroup references in this blog.

A fstab file is used to “pivot root” and defines your mounts. The full path is used in this file.

Sample fstab:

none /lxc/rootfs.lucid/dev/pts devpts defaults 0 0
none /lxc/rootfs.lucid/proc proc defaults 0 0
none /lxc/rootfs.lucid/sys sysfs defaults 0 0

Once you have a rootfs, fstab, and a config file, run lxc-create :

lxc-create -f /path_to/config.file -n name_of_container

For example:

lxc-create -f /root/config.fedora -n fedora

Start a container

The basic command is lxc-start -n name_of_container

lxc-start -n fedora

Starting this way will show you the boot messages you would normally get when booting your distro. Watch for error messages (you will need to fix them manually).

Once the container is working, you can start it with the -d (demonize) option.

lxc-start -d -n fedora

Stop a container

lxc-stop -n name_of_container

For example:

lxc-stop -n fedora

Destroy a container

lxc-destroy -n name_of_container

For example:

lxc-destroy -n fedora

Manage resources

The resources available to your containers is managed via cgroups. You manage them with the lxc-cgroup command. The resources are listed as flat text files in /cgroup/name_of_container .

A detailed discussion of cgroup is beyond this post, but I may devote a separate blog to this topic after finishing instructions for Fedora and Ubuntu containers. See the links in the cgroup section above for helpful links to “get you started”.

Tips

A few comments on the structure of containers on the host file system.

1. /var/lib/lxc : Container configuration files are kept in /var/lib/lxc . It makes life easier to make a link or bind /var/lib/lxc to /lxc

mkdir /lxc
ln -s /var/lib/lxc /lxc

2. /var/cache/lxc : Used by lxc-debian and lxc-fedora as a cache for downloading and installing a rootfs .

3. /cgroup/name_of_container : Location of container cgroup files.

4. lxc-fedora

When using lxc-fedora, the root file system is downloaded to /var/cache/lxc/fedora and then copied to your current working directory. This makes it much faster the next time you invoke lxc-fedora as it will copy the cached rootfs to your current working directory without repeating the downloading and installation of the base packages.

So, for example, if you lxc-fedora create, you will be asked a few questions, including the name of your container and network configuration.

If you run the command from /root, the script will cache and install the packages in /var/cache/lxc/fedora, copy the /var/cache/lxc/fedora to /root/name_of_container, and generate a configuration file based on the options you give.

When you then create a container, with lxc-create, the configuration files are generated in /var/lib/lxc , and a link is generated for the rootfs to /root/fedora (assuming you names the container fedora).

This behavior is NOT emulated if you use bootstrap to make a rootfs manually or import (convert) an openvz template.

The down side of lxc-fedora is that the container still needs some modification and it is larger then if you use febootstrap.

See my blogs on generating a Fedora and Ubuntu container for detailed instructions.

5. IMHO, containment is very poor if you chroot into the container form the host node. Use lxc-console or ssh into the container.

References:

This entry was posted in Linux and tagged , . Bookmark the permalink.

41 Responses to LXC – Linux Containers

  1. Guy Unix says:

    This is a good idea for light-weight server virtualization, but I hope that you can seriously improve how this whole thing works. This seems like a rather poor implementation of Solaris Containers (i.e. zones). Zone-awareness is integrated into Solaris at *all* levels, from installation, to filesystem management, to patching the system, networking, etc…instead of hacked on top of the “chroot” concept.

    “Note: Some containers may not start with the -d option, in that case, IMO, better to use screen:
    screen -dmS lxc_container_name lxc-start -n lxc_container_name” — You don’t know if a container will start up properly, so you run it out of screen?

  2. bodhi.zazen says:

    @ Guy Unix – yep, Linux Containers are under rapid development and I would not advise you run them in a production environment.

    Your observation that “some containers may not start” is accurate. At this time there are very few reliable scripts (lxc-fedora seem to work best at the moment) to install and configure a container, and if you wish to build your own be prepared to debug init scripts.

  3. Sean Clarke says:

    This looks excellent – I cannot believe the community is not making more noise over this technology, perhaps they are to busy coo’ing over Xen ??

    If this matures to a technology similiar to Solaris Zones it could be a major win for the Linux community. I have been looking at this space for a while, openvz was never in the main branch and from memory there were another 2 offerings that were also available.

    This is a great visualisation technology that has completly shaped the Solaris deployment and production landscape – I have seen it use end to end from development to test to production deployment as a natural task – create a zone, deploy app/db etc.

    Don’t get me wrong, Xen is good and has it’s place, but for Linux on Linux hosting – containers just makes more sense. I am an avid OpenSolaris/Linux user and this is one of the major advantages OpenSolaris has in this area – they are so easy and versatile to use – I can knock up a new zone or clone an existing zone (with ZFS) in seconds and VERY, VERY easily (interesting as Solaris is normally harder to configure than Linux).

    I’m sure as it matures, as better/easier tools become available and we some integration when BTRFS it will be a great technology – I’ll certainly be having a look in Ubuntu 9.10 an look forward to the improvements in 10.04.

    Keep up the excellent work!
    Sean

  4. From the description, this sounds somewhat similar to FreeBSD jails, which similarly are a “hack” on top of chroot. Have you ever encountered FreeBSD jails, and if so, would you care to compare/contrast FreeBSD jails, OpenVZ, and LXC? I think that would be interesting information for anybody interested in lightweight containerization.

    -ELG

  5. Ricky says:

    Wow!It’s excellent.Linux containers run Linux processes on a Linux kernel. This means you can run Linux but not other operating systems.

  6. bodhi.zazen says:

    Ricky: That is correct. Guests or containers are Linux only.

  7. bodhi.zazen says:

    Eirck : These tools are all very similar, although they vary somewhat in features.

    Here is a nice comparison:

    http://en.wikipedia.org/wiki/Operating_system-level_virtualization

    LXC is NOT as developed as BSD jails and Linux has the disadvantage in that, among other things there is a great deal of variability across distros and the various boot scripts where BSD is more uniform.

    LXC is, IMO, in early development and, again, IMHO, not yet ready for a production server (personally I have concerns about the ease of breaking out of the container, and again, IMO, the LXC containers are not as isolated from the main OS / kernel processes as I would like).

    For now, I will stay with openvz.

  8. Pingback: Shadows of epiphany » Blog Archive » LXC Configure Ubuntu Lucid Containers

  9. Pingback: Shadows of epiphany » Blog Archive » LXC Configure Debian Containers

  10. Pingback: Shadows of epiphany » Blog Archive » LXC Configure Fedora Containers

  11. Pingback: LXC: Linux container tools

  12. Pingback: NIMDAE - Virtualizing With lxc In Ubuntu – Done Right!

  13. Pingback: On Linux, what is the best way to safely run code that isn't trusted without virtualizing the whole OS? - Quora

  14. Phil Howard says:

    Even if “not yet ready for production” this seems like a usable way to do certain things that require isolation such as running pure 32, pure 64, and multilib systems together on the same hardware. I have used plain old chroot as a means to do alternate toolchains where problems have been encountered with a multi-toolchain system. Each chroot is a full system (but, BTW, with identical files cross hardlinked to save space) with a different installed toolchain. I would simply fire up the build by having it chroot in and run the script to run the build steps. This isn’t a security-needed scenario and true virtualization would be overkill since most of the time the actual kernel isn’t the issue (more often the library version is). But I do want to test my code in 32 bit, 64 bit, and multilib scenarios under different library versions after different compilers compile it.

    But LXC is making me think about more possibilities that I might otherwise have to use KVM for.

  15. bodhi.zazen says:

    Phil: LXC has promise, will be interesting to watch it mature.

  16. Sid says:

    As an avid fan of FreeBSD’s jails and ZFS on FreeBSD looking for a Linux equivalent to use, I’m happy that Linux containers are maturing, and that BTRFS is too. That will give me another option in the Linux OS sphere.

  17. Pingback: Containervirtualisierung in Linux: So funktioniert lxc - Seite 1 von 4 - Storage & Server | Tests & Technik | ZDNet.de

  18. Ulli Horlacher says:

    Ubuntu 10.04:
    apt-get remove -purge network-manager network-manager-gnome
    E: Command line option ‘p’ [from -purge] is not known.

    This is correct:
    apt-get purge network-manager network-manager-gnome

  19. bodhi.zazen says:

    Thank you Ulli Horlacher, I made the correction.

  20. Ulli Horlacher says:

    You have:
    sudo apt-get remove purge network-manager network-manager-gnome
    which is wrong syntax. Correct is:
    sudo apt-get remove network-manager network-manager-gnome
    or
    sudo apt-get purge network-manager network-manager-gnome

  21. bodhi.zazen says:

    @Ulli Horlacher

    I am fairly certain you can use both the remove and purge options (that is remove –purge), nonetheless I updated my post.

    IMO you need to purge NM to remove the configuration files.

  22. Ulli Horlacher says:

    No, “remove -purge” is incorrect syntax as it is ” remove purge”.
    Correct syntax is either “purge” or “remove –purge”, see “man apt-get”.
    UNIX programs are not like humans who can auto-correct small typos.

  23. Ulli Horlacher says:

    Once again the STUPID HTML form has corrupted my text.
    Insert yourself the missing hyphens whereever they are necessary.
    I give up. This is my last contribution in this braindead web forum.
    You can contact me via e-mail if you want:

    framstag@rus-uni-stuttgart.de

  24. bodhi.zazen says:

    @Ulli Horlacher :

    You can use
    sudo apt-get remove – -purge foo (two “-” in front of purge)

    or

    sudo apt-get purge foo

  25. Roberto Aloi says:

    On Ubuntu Lucid, the step:

    service networking stop

    will fail due to:

    Unkown instance:

    Possibly related bug:

    https://bugs.launchpad.net/ubuntu/+source/sysvinit/+bug/440179

    Any workaround?

  26. bodhi.zazen says:

    @Roberto Aloi :

    I personally switched to Debian for both OpenVZ and LXC as Ubuntu has in the past been too buggy.

    With Ubuntu 11.04 they were planning better support for LXC, but I have not yet tried it.

    Also, LXC is in fairly rapid development, and you might get better support if you post to the LXC mailing list.

    At a minimum I highly suggest you try 11.04.

  27. Bram says:

    Bodhi,

    How do you mean buggy? Did the host crash or what happened?

    I’m considering lucid as a Host OS with LXC and I don’t want it to go down. If there are issues with the containers themselves, can you elaborate?

    Thanks.

  28. bodhi.zazen says:

    @Bram – LXC is in rapid development and if you want to be “informed” you should join the mailing lists.

    Buggy meaning guest do not always work as expected, there is a lack of documentation, tons of user scripts to manage various features without a good quality central method of managing guests, and isolation of the guests is less then optimal.

    I would NOT give root access within a guest to an untrusted client.

    I am sure LXC will be the way of the future, but I still consider it not ready for prime time. YMMV.

  29. Enass says:

    lxc-start: no configuration file for ‘/sbin/init’ (may crash the host)

  30. Jim says:

    I have tried the nova-compute-lxc. Since VMs use the single kernel, one of them make the system crash may cause all of them to crash together. Is this a downside of it? Can we do something to prevent this from happening?

    Thanks.

  31. bodhi.zazen says:

    @Jim – I have not migrated to LXC as I also find isolation of guests is not as complete as I prefer. Personally I continue to use OpenVZ or KVM.

    I would advise you file a bug report or post to the LXC mailing list, the mailing list is quite active and you may find a better answer there.

  32. Linux Howtos says:

    I think this is among the most vital information for me. And i am happy reading your article. But should remark on some common issues, The site taste is great, the articles is in point of fact excellent : D. Good process, cheers

  33. Pingback: unable to boot into my Linux

  34. Schorschi says:

    Maybe update for LXC 0.7.5 or 0.8.0? And Fedora 17? For example, lxc-fedora template script seems to be missing when LXC 0.7.5 installed on Fedora 17? And febootstrap is implied as depreciated?

  35. bodhi.zazen says:

    @Schorschi – It is sort of on my to-do list. Part of the problem is LXC is in such rapid development and documentation lags.

    I highly advise you subscribe to the (LXC) mailing lists.

    febootstrap has been replaced by mock (FWIW – mock works very well)

    http://fedoraproject.org/wiki/Projects/Mock

    and some people use openvz templates for LXC as well.

    http://wiki.openvz.org/Download/template/precreated

    openvz templates are tar balls ;)

  36. Schorschi says:

    Thanks for the fast reply. I am on the mailing list, and I have been using OpenVZ off and on for about 20 months now. I am a bit surprised how slow LXC development is moving… which is not slow, but just was expecting it to be further along than it is, say core to RHEL 6, but not yet! I finally did compile and install 0.8.0-rc2, which did resolve a few issues I was having with 0.7.5 which is (core) Fedora 17 repository. Most familar with system containers, but have just started do more with application containers (supermins). Have a long history and background with VMware since 2004, Microsoft Virtual Server/Hyper-V since 2005, and of course KVM and RHEV since inception. So quite familiar with OS isolation virtualization (type 1 and 2 hypervisors), but see application isolation (based on type 2 hypervisors or container based models) as coming into its own in the next 18 months. Some might find this interesting illustrates my perspective, and why I would like to believe LXC development was more mature, even faster, than as fast as it is now…
    http://itknowledgeexchange.techtarget.com/server-virtualization/is-hypervisor-based-virtualization-doomed/

  37. Pingback: Experimenting with Linux Containers « Fragments

  38. Pingback: Sunday, 03/24/13 | Cody Doucette

  39. armand says:

    Hi. thanks for the great write up…i had a question:

    i know its possible to create a container using an older ubuntu version but is it possible to run a container running on an older kernel version.
    i.e spin up new LXC container and downgrade kernel for just this container?

    thanks

  40. bodhi.zazen says:

    @armand – No, LXC is part of the kernel, so the containers all run the same kernel, the same as the host. If you want a different kernel you need KVM, VirtualBox, VMWare, etc.

  41. Anonymous Paperclip says:

    Are you certain there’s no way to bridge a wireless card? I’ve seen people mentioning the use of some iptables and ebtables rules to make just that work. It certaintly would be nice to have a throwaway development container on a laptop while out of the office.

Leave a Reply

Your email address will not be published. Required fields are marked *

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>