Jail Primer


Important News: With the publication of RELEASE 10.0 the legacy rc.d jail method is depreciated and with RELEASE 11.0 support for legacy rc.d jail method is scheduled to be dropped all together. Further more, the /etc/rc.d/jail script has been changed to convert jail's defined in /etc/rc.conf to the jail(8) format and then start then using those jail(8) definitions.  Chapter 16.5 The Modern rc.d Method and Chapter 16.6 The Legacy rc.d Method are marked as depreciated. They are still valid for RELEASE 8.x and 9.x until they reach "end-of-life", and are considered valid for all 10.x RELEASES.

Also note: RELEASE 9.1 where jail(8) was first introduced and RELEASE 9.2 contain bugs in the jail(8) program that are NOT fixed until 10.0-RELEASE-p2.

If you desire a more user friendly approach to jail creation and management check out the sysutils/qjail port.



Table of Contents

16.1 Synopsis

16.2 Introduction
     16.2.1 What is a Jail Cell
     16.2.2 What is a Jail System
     16.2.3 The Three Jail Cell Definition Methods

16.3 How to Build the Jail System
     16.3.1 Building the Jail System Filesystems
     16.3.2 Creating & Deleting Jail Cell Filesystems
     16.3.3 Enabling Jail Cell SSH Support

16.4 The jail.conf Method
     16.4.1 jail.conf Definition Statements
     16.4.2 Creating & Deleting the jail.conf jail cell Definition File
     16.4.3 Starting & Stopping jail.conf jail cells
     16.4.4 Boot Starting & Shutdown Stopping of jail.conf jail cells

16.5 The Modern rc.d Method depreciated
     16.5.1 rc.conf Definition Statements
     16.5.2 Creating & Deleting the rc.conf jail cell Definition file
     16.5.3 Starting & Stopping the rc.conf jail cells
     16.5.4 Boot Starting & Shutdown Stopping of rc.conf jail cells

16.6 The Legacy rc.d Method depreciated
     16.6.1 Starting & Stopping of Legacy rc.d jail cells

16.7 Working With Jail Cells
16.8 Summary of the provided Scripts & Their Usage
16.9 Jail Cell Network Traffic Flow
16.10 Vnet/Vimage



16.1 Synopsis

Time is money, So to shorten the time used to deploy a jail system, this Jail documentation has been written using scripts as examples and includes supporting information where it's used instead of pointing the reader to other external references. New concepts and terminology are introduced to clarify how the incarceration process is achieved using the jail function.

A jail system is too complicated to explain by giving the manual commands to issue from the console command line. Many parts of the script logic have no equivalent console command line commands. The scripts are broken down into logical functions that follow the logical functional progression for first building the jail system filesystems, creating the jail cell filesystems, creating the modern rc.d or legacy rc.d or the jail.conf definition statements files, followed by the starting, stopping and deleting of jail cells defined for each of those jail cell definition types.

These scripts are not to be though of as the only way of implementing jail systems. They are intended to demonstrate to the reader the concepts expressed in this documentation. The reader may use them as they are or modify them to their own needs.

Previous to 9.1-RELEASE jail systems were implemented utilizing the rc.d script method. With 9.1-RELEASE an second method of jail configuration and control became available using the jail(8) program that introduces the /etc/jail.confconfiguration file. It includes new extensions to enable special functions on a per jail basis.

After reading this documentation, you will know:


Other sources of useful information about jails are:


16.2 Introduction

This introduction is not intended to be a historical  review of the FreeBSD jail concept. It's purpose is to expose the reader to the two legacy generations of how FreeBSD jails were commonly used based on the computer hardware available at the time and introduce the third generation jail system solution.

The original FreeBSD developers felt the need for a method to restrict a processes access to the host system resources so if it becomes compromised the host system is protected from also being compromised. They achieved this goal with the "chroot" command which was in the original 4.4BSD system, from which the current FreeBSD RELEASE is a direct descendant. This first generation "chroot" environment, made it look like the named directory was the "root" IE: starting point; of a operating system directory tree. Just like "/" is today in FreeBSD. The "chroot" environment shared the hosts network and disk space. This trait continues into today’s jail systems.

In this basic incarnation, the "chroot" directory tree would only contain copies of the host's operating system binaries that were necessary to form a very simple running environment for a single application such as a apache web server. It took a host administrator with advanced knowledge and understanding of the operating system modules and the userland application to configure and setup such a "chroot" environment. This is still true today if such a jail is desired. At that time the maximum disk drive size available was 300MB and maximum memory size was 64K and the maximum cpu speed of 8mhz. These limited resources caused very poor host "chroot" environment performance and made the "chroot" concept unpopular. As you can imagine, occupants of these basic "chroot’s" influenced host administrators to stay at the RELEASE version they were at because of the size of the task to redevelop their "chroot" environment under a new RELEASE versions binaries.

The rc.d/jail script appeared in FreeBSD 4.0. With this second generation "chroot" enhancement came the renaming of a "chroot" environment to a "jail", the ability to assign IP address to a jail, auto starting jails at boot time, and a general shift in thought about the occupant of the jail. The first generation simplistic "chrooted" directory tree apache web server that had no easy way to be configured, progressed into a complete clone of the host's operating system with all the customizing options one is familiar with on the host. The "make buildworld" method of creating the jail filesystem was the bases of the second generation and is commonly used even today. The major shortcoming of this type of jail system at that time was each jail had its own copy of the hosts operating system binaries. Do no confuse "operating system binaries" with kernel bootable binaries which jails do not use. FreeBSD reserves a limited number of control structures for storing files and directories based on available memory size called inodes. Creating a few jails consumes many of those valuable inodes, eventually preventing the creation of additional jails. Worse yet, each jail loads it's own copy of the complete operating systems binaries into memory, this causes thrashing on the swap device as memory pages are swapped in and out as the limited memory is shared between the host and the jails. At this time the maximum disk drive size available was 1GB and maximum memory size was 256K and the maximum cpu speed of 64mhz. Besides consuming large amounts of the limited resources and creating performance degradation, this also causes a major administration headache when wanting to update the host operating system, because the host and the jails have to be running the same version of the operating system binaries.

With RELEASE 5.4 the newly enhanced nullfs command added the ability for jails to share a single set of the operating system running libraries between themselves. This third generation solution solved the performance problems of the second generation, but had its own problems. Setting up a third generation jail system was an undocumented manual one. Even as of 9.1-RELEASE the documentation only shows the "make buildworld" method of creating the jail filesystem. The manual administration of this jail system became increasing difficult with each additional jail.

With today's top end computers of 8 or 16 cpu cores, 256gb of memory and 3tb disk drives, the limited resources of the past are gone forever. Computers running thousands of jail are possible. An first generation jail system with it's userland application is still available to any host administrator having the ability to create one, but it's impossible to find any documentation about it and for all practical purposes its a dead subject.

The third generation solution separates the second generation single jail filesystem into two filesystems. The "sharedfs filesystem" now contains all of the operating system's executable libraries as read-only files and is mounted as an "nullfs" that is shared between all the individual filesystem directory tree jail cells. The "jail cell filesystem" just contains the operating system configuration files plus the userland occupant, which can be a server application or any of the available ports. This design effectively secures all the executable files from being updated or deleted and also secures the directories containing the executable files from having new files inserted by any user running inside of the jail cell.

Jail systems are a very powerful security tool for system administrators. They provide a method to compartmentalize services that are exposed to the public internet so the host environment is protected from becoming compromised if the jail cell should become compromised due to exploiting by a public attacker. Jail systems are also useful in protecting the host from malicious actions on an Local Are Network. Their basic usage can also be useful for advanced users with special needs.



16.2.1 What is a Jail Cell

A jail cell is a single compartmentalized directory tree structure that for all purposes, acts and behaves just like a computer running the FreeBSD operating system. Viewed from the public Internet or from the Local Area Network it's almost impossible to distinguish any difference between the two by everyone except maybe an expert technical attacker. A jail cell can be assigned a public routable IP address and receive unsolicited inbound requests for service, such as for a website being run by apache or a email server running postfix. If a jail cell were to become compromised by a public attacker taking advantage of  exploits  in the jail cell's running applications, most intruders would by unaware they are confined in a jail cell and any damage to the jail cells changeable configuration files content would have no effect on the host running the jail cell. The jail system shares the hosts disk space and the hosts network stack. All IP traffic passes through the host's network stack first and only the host's firewall can interrogate the traffic. This means that jail cells can't have their own firewalls. The only requirement of the jail system, is the running operating system executable libraries shared with the jail cells must be the same level as what the host is running.

The jail cell's ability to mimic a computer running the FreeBSD operating system is it's core advantage. This ability has many uses for an creative system administrator other than the security of public accessible services.



16.2.2 What is a Jail System

Multiple compartmentalized filesystem directory tree structured jail cells all sharing a single read only nullfs mounted filesystem containing a single copy of the operating system executable libraries is the primary design concept of the jail system documented here and comprises the  third generation jail system solution.

The jail system is comprised of two components:

1. The building of the jail system's  physical directory tree filesystems that forms the bedrock of this third generation solution,  which is defined by;

2. The jail cell control component which comprises the creation of the jail cell definition statements, starting and stopping of the defined jail cells by the host system.



16.2.3 The Three Jail Cell Definition Methods

Previous to 9.1-RELEASE all jail cells were based on jail cell definition statements being placed in the host's /etc/rc.conf file and manual start / stop control accomplished using /etc/rc.d scripts launched by the "service jail" command, with the option of auto boot time startup. This will be referred to as the legacy rc.d method.

With 9.1-RELEASE a second method became available, the jail.conf file method. It's based on the /etc/jail.conf file. The syntax of the jail cell definition statements in the jail.conf file is totally different from those used in the rc.conf file of the legacy rc.d method and the method of jail cell start / stop control  is now accomplished by the jail(8) command instead of using /etc/rc.d scripts launched by the "service jail" command. This new method allows far greater flexible in controlling the location and usage of the jail cells definition files and provides many parameters that can be applied on an per-jail cell basics.

The modern rc.d method has divorced it self from relying on the "service jail" command for starting and stopping and no longer do the rc.conf jail cell definition statements have to reside in the /etc/rc.conf file. They can now reside anywhere on the host's filesystem. This is basically the same freedom the jail.conf file method provides.

These three jail cell definition methods can be used together or separately because the jail system scripts have been designed to allow it. The jail system organizational design produces the Third Generation jail solution and these three jail cell definition methods all share that jail system.



16.3 How to Build the Jail System

The jail(8) manual page plus all the other jail documentation found by an Google search make the incorrect assumption that all users of the FreeBSD Operating System, installed the system sources and use the "make buildworld" method of maintaining their host systems. As a result, potential jail system users were forced to install the system sources and perform the time consuming compiles the "make buildworld" process requires just to be able to populate their jail system.

A faster method is available which does not rely on system source files. All RELEASE versions of FreeBSD have compressed install files available for download. The "pristine" method  is based on using one of those official compressed install files as source for populating the jail system with an pristine copy of the operating system. This method provides an added level of security over the "make buildworld" process, where intentionally or un-intentionally modified executables could populate the jail system. This potential security situation is avoided by populating the jail system with an pristine copy of the operating system.

It's important to draw your attention to this basic fact about jail systems. The only requirement is the operating system executables contained in the jail system must be at the same level as what the host is running. This is the only way to guarantee the integrity of your production jail system.

Were going to cover both the "make buildworld" method and the "pristine" method  ways of populating the template filesystem with an complete version of the operating system. The instructions covered here is the first step in creating the jail system.

The "make buildworld" method requires the host to have the complete operating system source located at /usr/src. The following commands are going to take some time to compile the source to populate the template filesystem. Once completed you move on to the 16.3.1 Building the Jail System Filesystems section to continue.

     mkdir -p /usr/jail/template
     cd /usr/src
     make world DESTDIR=/usr/jail/template
     make distribution DESTDIR=/usr/jail/template


The "pristine method" uses one of the "RELEASE" compressed install files as source for populating the jail system with an pristine copy of the operating system. Take note: the format of the "RELEASE" compressed install files changed between 8.x and 9.x, the "jail.pristine.fetch" script uses the 9.x format only. Once completed you move on to the 16.3.1 Building the Jail System Filesystems section to continue.

The script named "jail.pristine.fetch " has been provided. The script has comments which self document what it's doing. Please notice that "/usr/jail" is the location where this script will install the jail system filesystems. You can easily edit this script for any path location you desire, and then also make the same changes to all the other provided scripts shown later.

Issue this command from the command line
jail.pristine.fetch
When it completes the "template filesystem" is created.

Take Note: Executing the "make buildworld" method or "pristine" method is normally only done to create your original Jail system. But it can also be re-used to update your Jail system when the host system operating system is updated and crosses a major release level, IE; 8.2-RELEASE to 9.0-RELEASE.



16.3.1 Building the Jail System Filesystems

Please NOTE that the jail.pristine.fetch script or the "make buildworld" method from the previous section MUST have been completed before continuing with the script shown here.

To verify you are ready for this step, you can issue "ls -l /usr/jail" and you should see the template filesystem with the date and time that it was created and populated. Issuing "ls -l /usr/jail/template"and you should see what looks like an complete FreeBSD filesystem. If you see all this, then your ready for this step.

The following script named "jail.install.system " has been provided. The script has comments which self document what it's doing. Please notice that "/usr/jail" is the location where this script will install it's jail system filesystem. You can easily edit this script for any path location you desire, and then also make the same changes to all the other provided scripts shown later. Take note; this is the largest script of the whole series.

Upon the completion of running the "jail.install.system" script you will have an Third Generation jail system. Technically when ever the host updates the system source and does a make buildworld, or the "freebsd-update upgrade" command is used to upgrade the host system to a newer RELEASE level, this jail system filesystem  must be rebuilt to match the RELEASE level of the host. This is mandatory for production jail systems that have jail cells accessible from the public internet. The jail.install.system script contains code to remove the sharedfs filesystem when ever it's executed so nothing special has to be done before running this script  to re-build the jail system over an existing jail system.

Issue this command from the command line
jail.install.system
When it completes the "jail system filesystems" are created.

Take Note: Executing  the jail.install.system  is normally only done to create your original Jail system. But it can also be re-used to update your Jail system when the host system operating system is updated and crosses a major release level, IE; 8.2-RELEASE to 9.0-RELEASE.



16.3.2 Creating & Deleting Jail Cell Filesystems

No matter which jail cell definition method you chose to use, "legacy rc.d method" or "modern rc.d method" or "jail.conf method" they all have one thing in common, each individual jail cell needs it's own physical filesystem to run in. The creating and deleting of individual jail cell filesystems is an separate function from the creating and deleting of individual jail cell definition statement files based on one of the three definition methods.

The following two scripts named "jail.create.jailcell" and "jail.delete.jailcell" have been provided. These scripts have comments which self document what it's doing. Please notice that "/usr/jail" is the location where these scripts will create and delete the individual jail cell filesystems. You can easily edit the script for any path location you desire, but for uniformity it should be the same location the jail system is installed at.

The "jail.create.jailcell" script requires a single parameter to function. It needs a "jail cell name". Issued from the console command line it would look something like this   jail.create.jailcell dir0   where dir0 is the jail cell name.

There is a jail cell naming standard which is designed to alleviate technical problems when trying to start jail cells. Only underscore, dash and alphanumeric characters are valid in the jail cell name, and jail cell names that are all numeric are invalid. It's up to the user to keep manual records of what jail cell names have already been assigned.

Issue this command from the command line
jail.create.jailcell jailcellname

Take Note: You have to execute  jail.create.jailcell  for every jail cell you plan on having no matter which of three definition methods you choose to use to create the individual jail cell definition statement file.


The jail.delete.jailcell script:
This will delete the jail cell's physical filesystem. This script requires a single parameters to function. It needs the jail cell name to delete.

Issue this command from the command line
jail.delete.jailcell jailcellname



16.3.3 Enabling Jail Cell SSH Support

The jail cell's filesystem has to be updated to enable SSH support. The "jail.ssh.jailcell" script does this function. On the first start of that jail cell after running this script, SSH will be enabled from that point on. After this jail cell is started you have to create user accounts in that jail cell for users to login to.

To create user accounts in the jail cell you will have to open a jail cell root session using the host's "jexec" command and create the new user accounts you want using the pw command. Then restart the jail cell and you will be able to do SSH logins using the jail cell's IP address.

You would code this command on the host's command line this way      jexec jailcellname tcsh
Once your talking to the jail cell, you have all the normal abilities you would expect as a root user on the host. Entering exit will close down your session with the jail cell.

       pw adduser login.name -p 12-12-12 -c jail user -m -g wheel -s csh -w yes
This format of the pw command will assign the login password to be the same as the login.name and the -p flag will force the user to change that password on first login.

The jail.ssh.jailcell script:
This script requires a single parameter to function. It needs the jail cell name you want to target.

Issue this command from the command line
jail.ssh.jailcell jailcellname



16.4 The jail.conf Method

The jail.conf method became available to the general public with the publication of the 9.1-RELEASE. It defaults to using the /etc/jail.conf file as the depository of all the jail cell definition statements under it's control. This is not a mandatory requirement. Jail cell definition statements can also be separated into individual files for easy control of starting / stopping individual jail cells. That is how the jail cell definition statements are organized and utilized by the provided scripts. The /usr/sbin/jail program provides the start / stop control of jail cells defined utilizing the jail.conf definition syntax. The syntax provides for enabling parameters on a per jail cell basis.

After reading this section, you will:

During the development of the jail.conf file method documented here, a few bugs came to light with the jail(8) program which the author has fixed in 10.0-RELEASE-p2, but which are not included in the 9.1-RELEASE and 9.2-RELEASE.

If your running 9.1-RELEASE or 9.2-RELEASE, I recommend you use the modern rc.d method for your jail cell definition as documented here 16.5 The Modern rc.d Method.



16.4.1 jail.conf Definition Statements

This is the jail.conf  jail definition statements created by the jail.jailconf.def.create script. It's contains the definition parameters necessary for a standard jail. The syntax has some requirements which are similar to defining a function in a "sh" script. Notice the position of the opening and closing { }curly bracket. Values for the parameter must be enclosed in quotes "  " and terminated with a ; semicolon.

dir00 {
host.hostname  = "dir00";
ip4.addr           = "10.0.10.30";
interface           = "rl0"
path                  = "/usr/jail/dir00";
mount.fstab      = "/usr/local/etc/jail.conf.fstab/dir00 ";
exec.start         = "/bin/sh /etc/rc";
exec.stop         = "/bin/sh /etc/rc.shutdown";
exec.consolelog = "/var/log/jail.dir00.console.log";
mount.devfs;
}


dir00   Is the jail cell name and is inserted in 5 different jail cell definition parameters. Substitute your jail cell name for dir00.

host.hostname    This is the jail cell name. It becomes the value for the hostname command visible only from within the running jail cell. It has nothing to do with DNS access to the jail cell and does not have to be a fully qualified domain name like on the host system.

ip4.addr   This is the IP4 address / addresses to be assigned to the jail cell. The jail cell's primary IP address is used to access the jail over the network. Multiple IP addresses are allowed if each IP address in the list is separated by a "," comma. If multiple IP addresses are coded this parameter can not be enclosed in quotes.

interface   This is the NIC device name which to attach the IP address / addresses to. An alias will be automatically created on jail cell start and automatically removed on jail cell stop.

path   This is the path to the jail cell's filesystem mount point.

mount.fstab   This is the path to the file containing the fstab format type of record defining the mounting of the shared sharedfs as an read only nullfs filesystem. This path /usr/local/etc/jail.conf.fstab/ is the location used by the provided scripts, do not change it.

exec.start   This is the normal script used to start the jail cell.

exec.stop   This is the normal script used to stop the jail cell.

exec.consolelog   This creates the jail cells console log on first jail cell start and appends all following jail cell console messages to the bottom of this file.

mount.devfs   This enables the ability for the jail to auto mount the dev file system and apply rule number 4 as the default which restricts the devices visible inside the running jail cell.



There are many jail(8) parameters available, most are special purpose for the advanced user. But there are some which the general user may find interesting. To enable any of the jail(8) parameters listed here, you have to manually edit /usr/local/etc/jail.conf/jailname and add the desired parameters.


ip6.addr = IPv6 address /addresses to be assigned to the jail cell. It's the counterpart to ip4.addr.

securelevel = -1, 0, 1, 2, 3 values as defined in "man security". This does not really apply to the third generation jail system described here because all of the system executables are in an read only nullfs mounted filesystem which makes them impossible from being changed. The third generation jail system is by far a stronger form of jail security than the securelevel parameter can provide on an per-jail basis.

allow.mount.nullfs on a per-jail basis. This is restricted to directories inside of the jail. It has nothing to do with mounting nullfs from the host to the jail filesystem which is always available to do. Any mount_nullfs commands issued from within the jail are only in effect for the duration the jail cell is running. When the jail cell is stopped the established nullfs mount is neutralized. The exec.prestart or exec.poststart parameters may be used to automate the issuing the desired mount_nullfs commands.

allow.mount.zfs on a per-jail basis. This parameter has mandatory host requirements before it's useful. The host must have all or some part of it's hard drive space defined to zfs and actively using it. See zfs(8) for information on how to configure the zfs filesystem to operate from inside a jail. The exec.prestart or exec.poststart parameters may be used to automate the issuing of the desired zfs commands.

allow.quotas on a per-jail cell basis. This parameter has mandatory host requirements before it's useful. The host  must have quota compiled into it's kernel before this parameter has any effect on the started jail cell.

allow.sysvipc on a per-jail basis. This parameter breaks the security of the jail concept. It should never by enabled on a jail cell exposed to the public internet. This allows a process within a jail cell access to the System V IPC primitives. In the current jail(8) implementation, System V IPC primitives share a single namespace across the host and all jail(8) environments, meaning that processes within a jail cell would be able to communicate with (and potentially interfere with) processes outside of the jail cell, (the host and other jail cells).

devfs_ruleset on a per-jail basis. If for what ever reason you would need to have a special rule, you would add it to /etc/defaults/devfs.rules and code this parameter like this devfs_ruleset = "7" to assign that rule number to this jail cell only. See devfs(8) for more details.

allow.raw_sockets on a per-jail basis. This parameter breaks the security of the jail concept. It should never by enabled on a jail exposed to the public internet. Normally the ping command will get "Operation not permitted." error when issued from inside of a jail cell. This is a security feature embedded in the basic design of the FreeBSD jail environment. This default does not allow users or jail cell applications to create raw sockets. With raw sockets enabled, a jail cell user could use perl or python or some other port utility to create raw sockets and launch attacks on the host or the public network. If the jail cell has public internet access, an public attacker may compromise the jail cell and launch attacks on the host or the public network. The whois or dig commands can be used for the same purpose of determining if the jail cell has public internet access. Consideration of the security risk verses the convenience of using the ping command from inside the jail is of the highest order.

vnet on a per-jail basis. This parameter requires the experimental vimage to be compiled into the hosts kernel before enabling this parameter. If vimage is not compiled into the host's kernel and this parameter is included in the jail cells definition, it will cause the starting of the jail to fail with "unknown parameter vnet".

vnet.interface  This is the companion with the vnet parameter. Required when vnet parameter is coded. You populate it with the NIC device name of the interface you want vnet to use.

exec.fib on a per-jail cell basis. This parameter is used only under very special conditions by technically advanced users. It deals with alternate routing tables. It requires either a new kernel (with "options ROUTETABLES=2" or however many you want), or a boot-time setting with "net.fibs=2" in /boot/loader.conf (requiring a reboot).
setfib 1 route add default 198.192.64.21 creates routing table number 1 with that IP address. In this example exec.fib="1" would be coded. See setfib(8) and setfib(2) for details.

cpuset.id on a per-jail basis. This parameter is used to limit the number of CPUs the jail cell may use of the total CPUs available on the machine. Issuing "cpuset -g" command on the host will list the CPU identification number of each CPU available. Assigning a jail cell to an single CPU does not give that jail cell exclusive usage of the CPU or does it exclude other host processes from using that CPU. If  the "cpuset -g" command listed 0, 1, 2, 3, 4, 5, 6, 7 that means this computer has 8 CPU's. Coding cpuset.id=0,1,2 means CPU 0, 1, and 2 out of the 8 available CPUs are being assigned to this jail restricting that jail from using CPUs 3, 4, 5, 6, and 7.

 See rctl(8) for true resource control. Also this wiki article does a good job explaining what rctl does and how it works. https://wiki.freebsd.org/Hierarchical_Resource_Limits .



16.4.2 Creating & Deleting the jail.conf jail cell Definition File

The following two scripts named "jail.jailconf.def.create" and  "jail.jailconf.def.delete" have been provided. These scripts have comments which self document what it's doing. Please notice that "/usr/jail" is the location where these scripts will create and delete the individual jail cells. You can easily edit the script for any path location you desire, but for uniformity if should be that same location the jail system is installed at.

The "jail.jailconf.def.create" script requires 2 parameters to function and has 1 optional parameter. It needs a jailcellname, IP-address, and in most cases the NIC device name that ip-address is to be aliased to. Issued from the console command line it would look something like this jail.jailconf.def.create dir0 10.0.10.20 rl0

It's up to the user to keep manual records of what ip-address have already been assigned so no 2 jail cells have the same ip-address. It's also up to the user to keep manual records of what jail cell names have already been assigned.

The "jail.jailconf.def.create" script checks for the existence of a jail cell filesystem having the jail cell name entered with this script at execution. The jail.create.jailcel script must be run before this script to create the jail cell filesystem first.

Issue this command from the command line
jail.jailconf.def.create jailcellname IP-address nic


jail.jailconf.def.delete script:
This script requires a single parameters to function. It needs the jail cell name to delete.
Use "ls /usr/local/etc/jail.conf to list all your jail cell names. After this script completes you need to run "jail.delete.jailcell" script to delete the jail cell's filesystem.

Issue this command from the command line
jail.jailconf.def.delete jailcellname



16.4.3 Starting & Stopping jail.conf jail cells

The following two scripts named "jail.jailconf.start" and "jail.jailconf.stop" have been provided. These scripts have comments which self document what it's doing.

jail.jailconf.start script:
This script requires a single parameter to function. It needs the jailcellname of the jail cell you want started.

Issue this command from the command line
jail.jailconf.start jailcellname


jail.jailconf.stop script:
This script requires a single parameter to function. It needs the jailcellname of the jail cell you want stopped.

Issue this command from the command line
jail.jailconf.stop jailcellname



16.4.4 Boot Starting & Shutdown Stopping of jail.conf jail cells

The following script named "jail.jailconf.bootime" has been provided. This script has comments which self document what it's doing.

This is a rc.d-method type script because the rc.d system is the only way to implement boot time access to the jail system. To enable boot time start up of all the jail system's jail.conf type of jail cells, the host's /etc/rc.conf file needs this statement added, jailconf_enable="YES" and the jail.jailconf.bootime file has to be in the hosts /usr/local/etc/rc.d directory with read/execute file permissions.

Note: For orderly stopping of the jailconf jail cells the host system must issue the "shutdown -p now" command. Using the reboot or halt commands just terminates the running jail cells. This may damage data bases running inside the jail cells.

Instructions for using this script.
This script must be placed in /usr/local/etc/rc.d/jail.jailconf.bootime
You never directly execute this script.



16.5 The Modern rc.d Method depreciated

Since FreeBSD 4.0 when the /etc/rc.d/jail script first became available, all jail cells were based on jail cell definition statements being placed in the host's /etc/rc.conf file and the start / stop control accomplished using /etc/rc.d scripts launched by the "service jail" command. In this document this is now called the legacy rc.d method. What is documented here will work with all RELEASEs back to 4.0-RELEASE.

The modern rc.d method explained here has divorced it self from relying on the "service jail" command for starting and stopping jail cells and no longer do the rc.conf jail cell definition statements have to reside in the host's /etc/rc.conf file. They can now reside anywhere on the host's filesystem. This is basically the same freedom the jail.conf file method provides.

The modern rc.d method has been available since FreeBSD 4.0, but due to the complexity of the rc.d environment and how the /etc/rc.d/jail script was implemented, only a few people had the knowledge and/or the understanding to make the rc.d jail environment more flexible and user friendly. This method has never been documented before and is presented here as an alternative to the current legacy rc.d method.

The modern rc.d method uses the third generation jail system which has been designed with two directories for grouping the fstab records and the rc.conf type of definition statements into individual files. Only the organization of the location for storing the files has changed, plus the direct execution of the /etc/rc.d/jail script. The deployment of this modern rc.d method is simple and easy. Current legacy rc.d method jail users will feel right at home using the modern rc.d method.

After reading this section, you will:



16.5.1 rc.conf Definition Statements

The following group of rc.conf jail cell definition statements are what is required for a standard jail cell.

This is the rc.conf jail definition statements created by the jail.rcconf.def.create script. It contains the definition parameters necessary for a standard jail. The syntax has a single requirement, the values for the parameters must be enclosed in quotes " ".


jail_rc-dir0_hostname="rc-dir0"
jail_rc-dir0_rootdir="/usr/jail/rc-dir0"
jail_rc-dir0_fstab="/usr/local/etc/jail.rcconf.fstab/rc-dir0"
jail_rc-dir0_ip="10.0.10.20"   
jail_rc-dir0_interface="rl0"
jail_rc-dir0_mount_enable="YES"
jail_rc-dir0_devfs_enable="YES"
jail_rc-dir0_flags="-n rc-dir0 -l -U root"
jail_rc-dir0_exec_start="/bin/sh /etc/rc"
jail_rc-dir0_exec_stop="/bin/sh /etc/rc.shutdown"


rc-dir0 This is the jail cell name. It gets inserted 14 times into the standard rc.conf definition statements. Substitute your jail cell name for rc-dir00.

jail_rc-dir0_hostname  This is the jail cell name. It becomes the value for the hostname command visible only from within the running jail cell. It has nothing to do with DNS access to the jail cell.

jail_rc-dir0_rootdir  This is the path of the mount point of the jail cell. It's the top of the jails cells filesystem.

jail_rc-dir0_fstab This is the path to the file containing the fstab format type of record defining the mounting of the shared sharedfs as a read only nullfs filesystem. This path /usr/local/etc/jail.rcconf.fstab/ is the location used by the provided scripts, do not change it. .

jail_rc-dir0_ip Set to the primary IPv4 and/or the IPv6 addresses assigned to the jail cell. May be a sole address or a comma separated list of IP addresses. The jail cell's primary IP address is used to access the jail cell over the network.

jail_rc-dir0_interface   This is the NIC device name of the interface to be used when setting IP address alias. The alias is automatically created on jail cell startup and removed on jail cell stopping.

jail_rc-dir0_mount_enable  This causes the auto mount of all the entries pointed to by the jail_rc-dir0_fstab parameter on jail cell start up and auto dismounted at jail cell stop time.

jail_rc-dir0_devfs_enable  Auto mount the device file system inside the jail cell and defaults to using rule set number 4 which limits what devices a jail cell has access to.

jail_rc-dir0_flags This forces the internal population of the jail cell name variable so the "jls name" command will have the jail cell name to list instead of being forced  to list the JIDs which is unwanted.

jail_rc-dir0_exec_start and jail_rc-dir0_exec_stop  These are the default values. These statements are not required, but it's a good practice to always include them so any one coming along later will know for certain what the jail cell definition consists of.

Note: With 9.2-RELEASE the "jail_jailname_parameters=" definition parameter became available. You would populate this parameter with one of the per jail extended parameters documented in jail(8).
Coded this way jail_rc-dir0_parameters="allow.raw_sockets=1"

Other sources of useful information about the rc.conf definition statements are:


16.5.2 Creating & Deleting the rc.conf jail cell Definition file

The following two scripts named "jail.rcconf.def.create" and  "jail.rcconf.def.delete" have been provided. These scripts have comments which self document what it's doing. Please notice that "/usr/jail" is the location where these scripts will create and delete the individual jail cells. You can easily edit the script for any path location you desire, but for uniformity if should be that same location the jail system is installed at.

The "jail.rcconf.def.create" script requires 2 parameters to function and 1 option parameter. It needs a jailcellname, ip-address, and in most cases the NIC device name that ip-address is to be aliased to. The ip-address may contain multiple IPv4 and IPv6 address separated with commas ",". Issued from the console command line it would look something like this jail.rcconf.def.create dir0 10.0.10.20 rl0

It's up to the user to keep manual records of what ip-address have already been assigned so no 2 jail cells have the same ip-address. It's also up to the user to keep manual records of what jail cell names have already been assigned.

The "jail.rcconf.def.create" script checks for the existence of a jail cell filesystem having the jail cell name entered with this script at execution. The jail.create.jailcel script must be run before this script to create the jail cell filesystem first.

Issue this command from the command line
jail.rcconf.def.create jailcellname


jail.rcconf.def.delete script:
This script requires a single parameters to function. It needs the jail cell name to delete. Use "ls /usr/local/etc/jail.conf to list all your jail cell names. After this script completes you need to run "jail.delete.jailcell " script to delete the jail cell's filesystem.

Issue this command from the command line
jail.rcconf.def.delete jailcellname



16.5.3 Starting & Stopping the rc.conf jail cells

The following two scripts named "jail.rcconf.start" and "jail.rcconf.stop" have been provided. These scripts have comments which self document what it's doing.

jail.rcconf.start script:
This script requires a single parameter to function. It needs the jail cell name of the jail cell you want started.

Issue this command from the command line
jail.rcconf.start jailcellname


jail.rcconf.stop script:
This script requires a single parameter to function. It needs the jailcellname of the jail cell you want started.

Issue this command from the command line
jail.rcconf.stop jailcellname



16.5.4 Boot Starting & Shutdown Stopping of rc.conf jail cells

The following script named "jail.rcconf.bootime " has been provided. This script has comments which self document what it's doing.

This is a rc.d-method type script because the rc.d system is the only way to implement boot time access to the jail system. To enable boot time start up of all the jail system's rcconf type of jail cells the host's /etc/rc.conf file needs this statement added, rcconf_enable="YES" and the jail.rcconf.bootime script has to be in the host's /usr/local/etc/rc.d directory with read/execute file permissions.

Note: For orderly stopping of the rcconf jail cells the host system must issue the "shutdown -p now" command. Using the reboot or halt commands just terminates the running jail cells. This may damage data bases running inside of jail cells.

Instructions for using this script.
This script must be located in /usr/local/etc/rc.d/jail.rcconf.bootime
You never directly execute this script.



16.6 The Legacy rc.d Method depreciated

Since FreeBSD 4.0 when the /etc/rc.d/jail script first became available, all jail cells were based on jail cell definition statements being placed in the host's /etc/rc.conf file and the jail cell start / stop control accomplished by using /etc/rc.d scripts launched by the "service jail" command. This is now called the legacy rc.d method. What is documented here will work with all RELEASEs back to 4.0-RELEASE.

Other sources of useful information about the rc.conf definition statements are:
The rc.conf definition statements are covered in the 16.4.1 rc.conf Definition Statements section. In addition to those standard rc.conf definition statements, the jail_list="jailcellname...." statement is required one time and must be in front of all the jail cell definition statements that follow. In between the quotes you list the jail cell names separated by a space for all of the jail cell definitions.

You must hand edit the /etc/rc.conf file to insert your jail cell definition statements. Each jail cell defined this way must have a jail cell filesystem. You would use the jail.create.jailcell script to do that.

Adding the jail_enable="YES" statement to hosts /etc/rc.conf enables the starting of all jail cells defined in the host's /etc/rc.conf file at boot time and perform a orderly stopping of all running jail cells when the "shutdown -p now" command is issued from the host. Issuing the halt or reboot command on the host will just terminate the jail cell which may cause problems for database type applications running inside of the jail cell.



16.6.1 Starting & Stopping of Legacy rc.d jail cells

You issue the service(8) command on the host to start jail cells defined in the hosts /etc/rc.conf file. If the hosts /etc/rc.conf file contains the jail_enable="YES" statement then this format of the service command is available;

service jail start  means to start all jails defined if rc.conf
service jail start jailname   means to start only that jail
service jail start jailname jailname jailname  means to start only the jail names listed
service jail stop   means to start all jails defined if rc.conf
service jail stop jailname   means to start only that jail
service jail stop jailname jailname jailname  means to start only the jail names listed

If the jail_enable="YES" statement is commented out or missing from the hosts /etc/rc.conf file then this format of the service command is available

service jail onestart   means to start all jails defined if rc.conf
service jail onestart jailname   means to start only that jail
service jail onestart jailname jailname jailname  means to start only the jail names listed
service jail onestop   means to start all jails defined if rc.conf
service jail onestop jailname   means to start only that jail
service jail onestop jailname jailname jailname  means to start only the jail names listed

There is also the restart and onerestart command which first stops the jail cell and then starts the jail cell.

If the hosts /etc/rc.conf file contains the jail_enable="YES" statement then all jail cell definitions in /etc/rc.conf will be auto started at boot time and auto stopped in an controlled manner only if the shutdown command is issued on the host system. Commenting out the definition statements in the hosts /etc/rc.conf file for a selected jail cell is the only way of disabling that jail cell from being started at boot time, other that commenting out the jail_enable="YES" statement which disables all jail cells defined  in the hosts /etc/rc.conf files.



16.7 Working With Jail Cells

There are two jail cell related commands that are provided in the base system. The are;

jls
This command only runs on the host and lists all the running jail cells. It displays the jail cell name, the jail cells JID number, and the path to the jail cells mount point. The JID is a sequential number that gets assigned to each jail cell as it starts.  See jls(8) for details.

jexec
This command only runs on the host and is only used after the jail cell is started, it will open a root account session with the jail cell. Once your talking to the jail cell, you have all the normal abilities you would expect as the root user on the host. Entering exit will close down your session with the jail cell. You would code this command on the host command line this way jexec jailcellname tcsh  You can also use this host command to run commands in the jail cell, IE; jexec jailcellname ifconfig  See jexec(8) for details.


Installing applications inside of a jail.
To do so you need to be logged into the running jail cell. There is 3 ways this can be accomplished. From the host issue jexec jailcellname tcsh or from a remote computer using SSH or telnet. To do SSH, the jail cells Filesystem needs to have SSH enabled by running the the jail.ssh.jailcell script which is described here 16.3.3 Enabling Jail Cell SSH Support

Once you are logged in to the jail cell you can use the package system or the ports system to install applications in the normal way. By the very nature of the jail system, jail cells only need access to a ports filesystem during the population of the jail cell's applications.

When the jail cell needs to install an port version it's very simple to move the host's /usr/port filesystem to the jail cell like this;

mv /usr/ports /usr/jail/jailcellname/usr/

After which the port filesystem will be usable from within that jail cell after the jail cell has been started. When that jail cell no longer needs it, or the host system needs to use it, it's just as simple to move it back to the host.

mv /usr/jail/jailcellname/usr/ports /usr/


Using /usr/src inside of a jail cell.
The host's /usr/src filesystem can be moved back and forth between the host and the jail cell just as easily as the /usr/port Filesystem can be.


Updating the applications running in the jail cell.
As long as the host and the sharedfs stay in sync, and the update is from one sub-RELEASE to the next sub-RELEASE,  IE; 9.0-RELEASE to 9.1-RELEASE the installed ports/packages do not need reinstalling. If the host updates from a major RELEASE to the next major RELEASE, IE; 8.3-RELEASE to 9.0-RELEASE then all the ports/packages installed in the jail cells need to be updated also, just like they have to be on the host.



16.8 Summary of the provided Scripts & Their Usage

jail.pristine.fetch                Populate template with pristine copy of FreeBSD
jail.install.system               Build the third generation jail system filesystems

jail.create.jailcell               Create the jail cell filesystem.
jail.delete.jailcell                Delete the jail cell filesystem

jail.jailconf.def.create        Create the jail.conf definition statements file
jail.jailconf.def.delete         Delete the jail.conf definition statements file
jail.jailconf.start                 Start a jail.conf jail cell
jail.jailconf.stop                 Stop a jail.conf jail cell
jail.jailconf.bootime           Boot time auto start & shutdown auto stop for jail.conf jail cells

jail.rcconf.def.create          Create the modern rc.conf definition statements file
jail rcconf.def.delete           Delete the modern rc.conf definition statements file
jail rcconf.start                   Start a modern rc.conf jail cell
jail.rcconf.stop                   Stop a modern rc.conf jail cell
jail.rcconf.bootime             Boot time auto start & shutdown auto stop for modern rc.conf jail cells



16.9 Jail Cell Network Traffic Flow

Things which effect how individual jail cells are going to function. The basic network configurations described here are not the only ways of configuring jail system networks, but they are simple, straight forward and do not rely on firewall traffic manipulation.

Viewed from the NIC facing the public internet;

  1. Host Firewall;
    All traffic is processed first by the host's firewall.

  2. Public Routable IP Address;
    Under public network conditions the normal way for unsolicited service requests to reach your NIC facing the public internet is through the use of a registered domain name. Each registered domain name has a associated IP address that gets routed by your ISP to your NIC that's connected to your ISP. This is how the public internet works and is out side of the control of FreeBSD jail systems.

    Most home users don't have the luxury of purchasing static IP addresses from their ISP's. Static IP addresses are permanent, they will never change unlike dynamic IP addresses. Home service comes with a single dynamic IP address. The ISP may at any time change the dynamic IP address they have assigned you. Your operating system can automatically handle this on the fly without you even being aware it happened. If the home user has a registered domain name using the dynamic IP address to direct unsolicited traffic to their host, it will stop working every time the dynamic IP address changes. Some domain registrar's have a service where you run a program on your host that watches for the dynamic IP address to change and when it does, it updates the domain name to use the new dynamic IP address for routing traffic.

    All the traffic for that single public routable dynamic IP address is going to be processed by the host. If you assigned that same public routable IP address to 3 jail cells, Then all traffic to that single IP address will be processed first by the host and then by each of those 3 jail cells. This maybe a problem. This is when the port number comes into play. Lets say jail cell #1 is a web server and as such is listening on port 80. As long as the host, jail cell #2 and jail cell #3 have no applications listening on port 80, then jail cell #1 will be the only jail cell processing the port 80 traffic for that IP address. This concept is true for all port numbers. In this situation you would create the jail cell definition assigning that single public routable dynamic IP address to all 3 jail cell and exclude using the "interface" parameter. Some alternate solutions are. Some domain registrar's have a port forwarding service where you can define inbound port number to be reassigned as a different outbound port number, IE 22->6022. Lets say you have 3 registered domain names and you defined port forwarding this way, domain name1 22->60221, domain name2 22->60222, and domain name3 22->60223. In jail cell #1 you want to process domain name1 port 60221 you edit the jails /etc/services for SSH changing the config parameter from 22 to 60223. Configuring jail cell #2 and jail cell #3 using the same concept will permanently target the correct jail cell by domain name. You could use the same concept for port 80 and configure Apache in each jail cell to service the correct desired port number. You could also accomplish the same thing by configuring each jail cells apache to only service the desired domain name.

    Now lets say you have purchased 3 static public routable IP addresses, you assign one for the host and the other two have registered domain names that are assigned one of the remaining 2 static public routable IP addresses. In this case the host/jail cell processing flow is different. In this situation the IP addresses with separate domains can be assigned to jail cells and then inbound traffic will first be processed by the hosts firewall and then passed directly to the assigned jail cell without the host processing the traffic. Each of those two jail cells will have access to all the port numbers without the host or other jail cells being aware of that traffic. The host and the two jail cells can be running a web server without any interference from each other. This is the intended way that jail cells were designed to function. In this situation you would create each jail cell definition assigning one of the static IP addresses to the jail cell and include the "interface" parameter with the interface device name of the NIC facing the public internet.

    Using Domain names or Domain names with domain registrar's port forwarding service provides a method to target the desired jail cell with out needing to use the host's firewall NAT or port forwarding functions. Of course your firewall still needs stateful rules on those inbound custom port numbers.

    Another common jail configuration,  you have an domain name driven Apache web application in a jail cell and it reads and writes Database information from another jail on the same host which you don't want direct public internet access to. In this situation you would use the loopback internal interface. Edit your hosts /etc/rc.conf file and add this parameter cloned_interfaces="lo1" and when you create the jail cell definition file use lo1 as the "interface" parameter value and a non-routable IP address from those reserved by RFC 1918. 10.0.0.0-10.255.255.255 or 172.16.0.0-172.31.255.255 or 192.168.0.0-192.168.255.255. What ever non-routable IP address you assigned to the database jail cell is the IP address the Apache web application will use to communicate with the database jail.

  3. LAN Routable IP Address;
    Always use non-routable IP addresses from those reserved by RFC 1918. 10.0.0.0-10.255.255.255 or 172.16.0.0-172.31.255.255 or 192.168.0.0-192.168.255.255 when configuring your local area network. There are 2 ways a jail cell can be utilized on the Local area network, they are; local internal LAN access only, or local internal LAN access with access to the public internet.

    For local internal LAN access only, the jail cell might contain a web server application or the internal email server, could also be jail cells that LAN users login into with SSH or telnet to have their own FreeBSD operating system to play with. In this situation you would create each jail cell definition assigning one of the private non-public routable IP address to the jail cell and include the "interface" parameter with the interface device name of the NIC facing the internal LAN.

    For local internal LAN access with access to the public internet. The jail cell can be accessible by all work stations on the LAN as a email server or a jail cell under development requiring internet access to install ports or packages. In this situation you would create each jail cell definition assigning one of the private non-public routable IP address to the jail cell and include the "interface" parameter with the interface device name of the NIC facing the public internet. In addition, the IP address traffic needs to be NATed, (IE; network address translation) before it can be routed to the public internet. This is easily accomplished using a firewall NAT service. Doing so will enable jail cells with private non-public routable IP address to gain outbound access to the public internet. Of the three firewall solutions provided in the base system of the host, I find IPFILTER has the easiest NAT service to configure. In this situation you would create each jail cell definition assigning one of the private non-public routable IP address to the jail cell and include the "interface" parameter with the interface device name of the NIC facing the public internet.

  4. Port numbers;
    For hosts with a single public routable IP address, the general rule is, the host processes all network traffic so the jail system administrator has to insure the selected port number is only processed by the host or a jail cell but not both. This means the host can not be running a apache server listening on port 80  and some jail cell doing the same thing. The host will process all the port 80 traffic and the jail cells will never see any of the traffic.

  5. NIC Device Name;
    The jail.jailconf.def.create and jail.rcconf.def.create scripts create the 2 different types of jail cell definition statements. Both scripts require the jail cell name and the IP address to assign to the jail cell. The NIC device name is optional, but it's use is necessary for all situation except when multiple jail cells are being assigned the same single host public routable dynamic IP address. When the NIC device name is entered, the jail cell definition "interface" parameter gets populated with the entered value. When the jail cell is started an alias will automatically be created for the jail cell's IP address on that NIC device name and automatically removed when the jail cell is stopped. In most situations this is required to direct traffic to the jail cell.

  6. DHCP  LAN IP Addresses;
    The jail system administrator has to be aware of the LAN IP addresses under the control of the host's DHCP server. If a DHCP LAN IP address is assigned to a jail cell, when the jail cell starts the LAN PC using that IP address will lose it's connection.


16.10 Vnet/Vimage

As of 10.0-RELEASE, Vimage is still a highly experimental software extension for the jail facility. It provides the ability for a jail to have it's own network stack unlike a normal jail which shares the host network stack. This means a vimage jail can have it's own firewall and multiple network Nic's defined. Host system freezes and page faults are common occurrences when running a kernel with vimage compiled into the kernel. Basically use vimage at your own risk.

The major problem with vimage jails is the loss of memory when stopping a vimage jail. This is covered by PR # 164763. When it comes to running a firewall in a vimage jail your limited to IPFW and even then IPFW has limitations, dummynet and in kernel NAT cause system freezes. IPFILTER causes page fault at boot time. PF will run on the host but not run in the vimage jail. Following is a list of vimage PR numbers dealing with vimage firewall problems, 143621, 176112, 161094, 176992, 143808, 148155, 165252, 178480, 178482.

To enable vimage you need to add the "options vimage" statement to your kernel source and recompile your kernel and then boot that kernel.

There are 2 networking methods available for creating vnet/vimage jail networks, if_bridge/epair and netgraph. The if_bridge/epair method is far simpler to configure and use then the netgraph method. As of 9.2-RELEASE the sysutils/qjail port is the only utility available that automates vnet/vimage jail creation.