Shared webhosting

This document describes creating a "secure shared web hosting service" on HN (Hardware Node). It is NOT about per container shared web hosting.

Contents

The problem

One of the problems with shared web hosting (i.e. different people with each his/her own webpages) is that modern script languages such as PHP, Python, or Perl are too powerful. For example take the following PHP script:

<?php

function get_content($filename) {
  $handle = fopen($filename, 'r');
  echo fread($handle, filesize($filename));
  fclose($handle);
}

get_content('/home/ppuk34/www/forum/config.inc.php');

?>

With PHP you could use open_basedir to prevent this, but there are more ways. For example PHP Shell, a script that is often mis-used by people with not-so-good intentions. Or think about the Santy-worm which mis-used phpBB. Again there is a solution in the form of safe_mode, but lots of PHP scripts break if you enable this. For Python, Perl, or CGI-scripts there are no easy ways and you have to use wrappers or other tricks to chroot these.

The solution

You can waste hours of time in securing all the possible things you don't want in your shared webhosting environment. And unless you are very familiar with all the things modern scripting languages can do, you probably miss dozens of alternative routes. In this process you frustrate your clients, because security always means that legitimate things break. As a side effect of your hard work, you can waste hours of extra time in educating your users. But in the end most users don't care about security, unless they are themselves victims of a compromised host. Learning the hard way is by far the most effective method. One possible solution is dedicated webhosting, but most users don't have the experience to maintain a server or it is way to expensive for them.

The main problem with shared webhosting is that by its very nature all files which are served through the web are public. Apache for example uses only one account to read all files. As said, you can use tricks with CGI wrappers to execute the scripting languages under its own credentials. However this kind of security depends on the wrappers ability to securely separate the users. We all know that if this is broken — and most often it will be broken — the result is a higher clearance on the underlying filesystem. For most systems you need more than one wrapper, so the number of possible security problems grow. The ultimate user separation is in the kernel and you can view the modifications OpenVZ has done in this light. Instead of CGI wrappers we go one step higher and give every user its own minimal server. In the rest of this article we describe how shared webhosting with OpenVZ could be implemented.

Minimal server

Create an CTx with your favorite distro. Give it an internal IP-address in one of the ranges 10.0.0.0/8, 172.16.0.0/12 or 192.168.0.0/16. Then strip away all unnecessary init.d scripts so only the bare minimum is started. That means as a minimum syslogd and ssh so the account holder can upload his/her files through SCP/SFTP in his/her own minimal server. For this to work you need to set up destination NAT on CT0 from high numbered ports to port 22 on the given private IP address:

dnat="-j DNAT --to-destination"

iptables -t nat -P PREROUTING ACCEPT
iptables -t nat -A PREROUTING -p TCP --dport 10122 $dnat 192.168.13.101:22
iptables -t nat -A PREROUTING -p TCP --dport 10222 $dnat 192.168.13.102:22
...

The other thing you want for webhosting is of course a webserver as well. To minimize the amount of needed memory, we choose Lighttpd instead of the common Apache. Then configure the scripting language of your choice to run under this webserver. It is possible to use different languages/setups for different accounts as well. Also problematic CGI-scripts are not problematic anymore...

MySQL server

Most webhosting accounts use MySQL, but if you prefer another database server, go ahead. Create a new CTx with a lot more resources and again an internal IP-address. Now configure the accounts. As an extra security measure you can use the internal IP-address as well.

MySQL socket sharing

You can also share the socket of an CTx running MySQL, which is alot faster than TCP/IP. e.g.:

ln /var/lib/vz/private/101/var/run/mysqld/mysqld.sock /var/lib/vz/private/102/var/run/mysqld/mysqld.sock

Inside of 102:

$ mysql -u root -p -S /var/run/mysqld/mysqld.sock
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 9 to server version: 5.0.32-Debian_7etch1-log

Refreshing links to MySQL socket

Sharing the MySQL socket works really well until the MySQL database is restarted or the container running MySQL is restarted. When this happens, the socket file is removed and recreated. In most cases, a different inode will be used, causing existing hard links to the mysql.sock file to no longer work. The solution is to relink these sockets.

There are more elegant solutions to this problem, but the following script is a decent hack that can be run via a cron job every minute or two. It will loop through all containers between START_CTID and STOP_CTID and make sure that the links point to the correct socket. If they do not, the link will be recreated.

#!/bin/sh

# Created by Tauren Mills (tauren at tauren dot com) 2007-11-15

###################################
# Start of Configuration settings #
###################################

# Location of private containers:
PRIVATE=/vz/private

# Starting CT ID.  CTIDs with this ID or greater will have mysql.sock link created
START_CTID=1001

# Stopping CT ID.  CTIDs with this ID or less will have mysql.sock link created
STOP_CTID=2000

# Shared Mysql CT ID:
MYSQL_CTID=201

# Location of mysql socket file
MYSQL_SOCK_DIR=/var/lib/mysql

# Mysql socket file name
MYSQL_SOCK=mysql.sock

#################################
# End of Configuration settings #
#################################

# Display output if quiet is 0
QUIET=0

if [ $# -eq 1 -a "$1" = "--quiet" ]; then
        QUIET=1
fi

# Full path to socket
MYSQL_SOCK_FILE=${PRIVATE}/${MYSQL_CTID}${MYSQL_SOCK_DIR}/${MYSQL_SOCK}

[ $QUIET -eq 0 ] && echo
[ $QUIET -eq 0 ] && echo "Relinking process starting..."

# Get current location so we can set it back later
oldDirectory=`pwd`

# Check to see if MySQL container socket exists
if [ -S "${MYSQL_SOCK_FILE}" ]; then

        # Get inode of MySQL container socket
        mysql_inode=`ls -i ${MYSQL_SOCK_FILE} | awk '{ print $1;}'`

        # Search through containers
        cd $PRIVATE
        for i in * ; do
                # The current container to process
                veid=$i

                # Check if container should be processed
                if [ $veid -ne $MYSQL_CTID -a $veid -ge $START_CTID -a $veid -le $STOP_CTID ]; then

                        # Get this container's socket
                        vesock=${PRIVATE}/${veid}${MYSQL_SOCK_DIR}/${MYSQL_SOCK}

                        # Make sure folder exists
                        mkdir -p ${PRIVATE}/${veid}${MYSQL_SOCK_DIR}

                        # Check to see if this container has a socket already
                        if [ -S "${vesock}" ]; then
                                # Get inode of this container socket
                                ve_inode=`ls -i ${vesock} | awk '{ print $1;}'`

                                # Test if sockets are the same
                                if [ $mysql_inode -eq $ve_inode ]; then
                                        # No action required
                                        [ $QUIET -eq 0 ] && echo "$veid VALID:  socket ${vesock}"
                                else
                                        # Remove existing file if any
                                        if [ -a "${vesock}" ]; then
                                                rm ${vesock}
                                        fi

                                        # Create hardlink to mysql socket file
                                        ln ${MYSQL_SOCK_FILE} ${vesock}

                                        [ $QUIET -eq 0 ] && echo "$veid FIXED:  socket ${vesock}"
                                fi
                        else
                                # Socket didn't exist or file wasn't a socket

                                # Remove existing file if any
                                if [ -a "${vesock}" ]; then
                                        rm ${vesock}
                                fi

                                # Create hardlink to mysql socket file
                                ln ${MYSQL_SOCK_FILE} ${vesock}

                                [ $QUIET -eq 0 ] && echo "$veid FIXED:  socket ${vesock}"
                        fi
                else
                        [ $QUIET -eq 0 ] && echo "$veid SKIPPED"
                fi
        done
else
        [ $QUIET -eq 0 ] && echo "${MYSQL_SOCK_FILE} does not exist. Is MySQL running?"
fi

cd $oldDirectory

[ $QUIET -eq 0 ] && echo "Relinking process complete."
[ $QUIET -eq 0 ] && echo

The following post on the OpenVZ forum has some other suggestions on how to deal with this issue: Shared webhosting - problem with mysql socket

Proxy webserver

Because we have only one public IP-address, we need an trick to access every minimal server based on the hostname in the HTTP request. For SSH we used different ports, but that is not an option for websites. Again we create an CTx with an internal IP-address. On this server we install Lighttpd as well, because the proxying is very simple. First we must forward port 80 to this server:

dnat="-j DNAT --to-destination"

iptables -t nat -P PREROUTING ACCEPT
iptables -t nat -A PREROUTING -p TCP -d <external IP-address> --dport 80 $dnat 192.168.13.11:80
iptables -t nat -A PREROUTING -p TCP --dport 10122 $dnat 192.168.13.101:22
iptables -t nat -A PREROUTING -p TCP --dport 10222 $dnat 192.168.13.102:22
...

Then we create for every website an section in /etc/lighttpd/lighttpd.conf as follows:

$HTTP["host"] == "ve101.armorica.tk" {
  proxy.server  = ( "" => ( ( "host" => "192.168.13.101" ) ) )
}

You can map more names to the same IP-address if needed. The last step is to add mod_proxy to the server.modules section.

For apache add a VirtualHost directive

<VirtualHost external-IP-address:80>
        ServerName mydomainnameishere.com
        RewriteEngine     On
        RewriteRule       ^(.*)$        http://192.168.2.101$1  [P]
        RewriteRule       ^(.*)$        http://mydomainnameishere.com$1   [P]
</VirtualHost>

<VirtualHost external-IP-address:80>
        ServerName mydomainnameishere.com
        RewriteEngine     On
        RewriteRule       ^(.*)$        http://192.168.2.101$1  [P]
        RewriteRule       ^(.*)$        http://www.mydomainnameishere.com$1   [P]
</VirtualHost>

Other applications

Create for other applications as mail, make sure that the minimal servers use this one for sending mail from webpages, DNS etc. CTx as needed. The resulting server is shown in the figure above.

OpenVZ can easily be installed on most Dedicated Server Hosting Services