Sans Pareil Technologies, Inc.

Key To Your Business

Minnal


C++ Servlet Container



Poco C++ library includes a light-weight scalable HTTP Server that can be used to create high performance custom web servers. The only downside to the approach is that each HTTP request handler needs to be mapped statically in code in a HTTPRequestHandlerFactory. The server implementation would be more flexible and easier to maintain if there was a way to declaratively map the request handler to request URI’s. JEE servlet containers support a very elegant and flexible request dispatching system based on servlets and paths mapped in a JEE web.xml file.

SPT has developed the Minnal (lightning in Malayalam) servlet container that uses a standard web.xml file to map the request handlers to request URI’s. SPT made use of the CERN Reflex framework to load and create servlet instances based on mapped class names. Relfex uses GCC_XML to parse class header files and generate C++ classes that add compile time support for introspecting C++ classes.

Note:



Since version 1.4, Minnal does not use Reflex. We moved to a simpler macro based static registration system to register servlets by class name, since we do not use any other features of reflection such as dynamic method invocation.

Minnal Servlet Container

ServletContainer

ServletContainer is a standard Poco::Util::ServerApplication sub-class. It initialises a RequestDispatcher instance, which takes over request handling. The following code sets up our request dispatcher using the standard pattern for a Poco::Util::ServerApplication.

  ServerSocket svs( port );
  HTTPServer srv( new RequestDispatcher( configDirectory, xmlConfiguration, config() ), svs, params );
  srv.start();
  waitForTerminationRequest();
  srv.stop();

Note that it is possible to build a different version of ServletContainer that directly creates a VirtualHostServer if the container does not need to support virtual hosts.

A simple XML file is used to configure the container. The configuration file is parsed using a Poco::Util::XMLConfiguration, and the parsed configuration instance is passed to the RequestDispatcher instance. The RequestDispatcher uses the configuration information to instantiate the required VirtualHostServer instances - one for each virtual host configured. A sample minnal.xml configuration file is shown in the Configuration section.

RequestDispatcher

RequestDispatcher is an implementation of Poco::Net::HTTPRequestHandlerFactory. RequestDispatcher parses all virtualHost elements defined in the minnal.xml container configuration file and creates instances of VirtualHostServer for each virtual host configured. The createRequestHandler method merely matches the appropriate VirtualHostServer instance based on the Host HTTP header and delegates to the createRequestHandler method of the VirtualHostServer. If no Host header is present, delegates to the first virtual host configured in the configuration file (see Configuration section for sample minnal.xml configuration file).

Each virtualHost element is bound to a specific domain. Additional domain aliases may be specified as alias child elements. A map is built of all the domain and alias names as keys and the VirtualHostServer instance as the values. Access file if enabled for a virtual host is initialised and set for each virtual host instance.

VirtualHostServer

VirtualHostServer is our implementation of Poco::Net::HTTPRequestHandlerFactory. The web.xml file is parsed and the servlet name to servlet configuration (path, initialisation parameters, context parameters etc) mapping is stored. An additional map is maintained for servlet path to servlet name mapping and is used in servletForPath( const string & ) method. Using reflection to create the servlet instance once the servlet class name (fully qualified with namespace) is as simple as:

Reflex::Type t = Reflex::Type::ByName( servletClassName );
Reflex::Object o = t.Construct();
Servlet *servlet = static_cast<Servlet *>( o.Address() );



Note that for performance reasons we cache the Type::ByName look up. The Reflex code generator cannot handle references to Reflex classes in header files they process. Hence we use a Poco::Any to store the Type instances created. The following code shows how the request dispatcher users the servlet-path mapping to load the configured servlet.

    1: HTTPRequestHandler* RequestDispatcher::createRequestHandler(
    2:     const HTTPServerRequest& request )
    3: {
    4:   string servlet;
    5:
    6:   PathToServlet::Iterator it = pathToServlet.find( request.getURI() );
    7:   if ( it != pathToServlet.end() )
    8:   {
    9:     servlet = it->second;
   10:   }
   11:   else
   12:   {
   13:     servlet = servletForPath( request.getURI() );
   14:   }
   15:
   16:   return createServlet( servlet );
   17: }
   18:
   19: HTTPRequestHandler* RequestDispatcher::createServlet( const string &name )
   20: {
   21:   string cls = ( "/" == name ) ? DOCROOT_HANDLER : name;
   22:
   23:   Servlets::Iterator it = servlets.find( cls );
   24:   if ( it == servlets.end() ) return NULL;
   25:
   26:   SharedServletConfig sconfig = it->second;
   27:
   28:   TypeMap::Iterator tmi = typeMap.find( cls );
   29:   Type t = ( tmi == typeMap.end() ) ?
   30:         Type::ByName( sconfig->getServletClass() ) :
   31:         AnyCast<Type>( tmi->second );
   32:   if ( ! t ) return NULL;
   33:
   34:   if ( tmi == typeMap.end() )
   35:   {
   36:     Any any( t );
   37:     typeMap.insert( TypeMap::ValueType( cls, any ) );
   38:   }
   39:
   40:   Object o = t.Construct();
   41:   Servlet *handler = static_cast<Servlet *>( o.Address() );
   42:   handler->setConfig( sconfig );
   43:   handler->setLayeredConfig( &config );
   44:
   45:   return handler;
   46: }

Servlet

The Servlet class implements Poco::Net::AbstractHTTPRequestHandler. The base Servlet class implements the run() method and delegates to the appropriate doXxx method as JEE HTTPServlet does. The default implementation of the doXxx methods returns a HTTP_METHOD_NOT_IMPLEMENTED error. Sub-classes implement the appropriate doXxx methods.

    1: void Servlet::run()
    2: {
    3:   if ( "GET" == request().getMethod() )
    4:   {
    5:     doGet();
    6:   }
    7:   else if ( "POST" == request().getMethod() )
    8:   {
    9:     doPost();
   10:   }
   11:   else if ( "HEAD" == request().getMethod() )
   12:   {
   13:     doHead();
   14:   }
   15:   else if ( "OPTIONS" == request().getMethod() )
   16:   {
   17:     doOptions();
   18:   }
   19:   else if ( "TRACE" == request().getMethod() )
   20:   {
   21:     doTrace();
   22:   }
   23:   else if ( "PUT" == request().getMethod() )
   24:   {
   25:     doPut();
   26:   }
   27:   else if ( "DELETE" == request().getMethod() )
   28:   {
   29:     doDelete();
   30:   }
   31:   else
   32:   {
   33:     unsupported();
   34:   }
   35:
   36:   if ( accessLogger )
   37:   {
   38:     AccessLogRecord::log( getRequest(), getResponse(), *accessLogger );
   39:   }
   40: }
   41:

The base servlet class also provides common methods such as setting content-type for response based on file extension, sending appropriate error-page based on status code, checking request URI for validity etc.

Building

We need to run the genreflex script included with the Reflex distribution to generate the class files necessary to support reflection and include that into our project. We use QT Creator as our IDE of choice, and the easiest way to include this pre-compilation phase was through a shell script.

    1: #!/bin/ksh
    2:
    3: PATH=$PATH:/usr/local/reflex/bin:/usr/local/gccxml/bin
    4: DIR=`dirname $0`/..
    5: OUT_DIR=$DIR/src/test/reflection
    6:
    7: if [ ! -d $OUT_DIR ]
    8: then
    9:   mkdir -p $OUT_DIR
   10: fi
   11:
   12: set -x
   13: rm -f $OUT_DIR/*.*
   14: set +x
   15:
   16: for i in `find $DIR/src/api -type f -name "*.h"`
   17: do
   18:   set -x
   19:   genreflex $i -s $DIR/data/selection.xml -o $OUT_DIR \
   20:     -I/usr/local/poco/include \
   21:     -I$DIR/src/api -I$DIR/src/test
   22:   set +x
   23: done



This build step is executed only when a header file for an affected class is modified. The class files generated under src/test/reflection are added to the qmake project and included in the application executable.

Note:


We have since moved over to a Makefile based system to automatically regenerate the reflection sources when the associated header file changes. The make file is enabled as a pre-build phase in Qt Creator as earlier.

Configuration

Configuration of the servlet container is through a minnal.xml file. Each virtual host configured in the container is driven by its own web.xml compatible file. Each Servlet instance has a dedicated Poco::Logger instance associated with it. We use a servlet init-param to specify the Logger priority level. The priority levels are specified as the names of the Poco::Message::Priority enumeration values.

minnal.xml



    1: <?xml version="1.0" encoding="UTF-8"?>
    2: <minnal>
    3:   <server port='80' keepAlive='true' maxQueued='500' maxThreads='100'
    4:     user='minnal' group='minnal' mimeTypes='../etc/mimetypes.properties'>
    5:
    6:     <logging logFile='../var/logs/minnal.log' rotation='daily'
    7:       archive='timestamp' compress='true' purgeCount='15' times='local'
    8:       pattern='%Y-%m-%d %H:%M:%S.%c [%P]:%s:%q:%t' />
    9:
   10:     <virtualHost domain='sptci.com' webXml='sptci.xml' users='users.xml'>
   11:       <alias>www.sptci.com</alias>
   12:       <access logFile='../var/logs/sptci.log' rotation='daily'
   13:         archive='timestamp' compress='true' purgeCount='15' times='local'
   14:         enabled='true' />
   15:     </virtualHost>
   16:
   17:     <virtualHost domain='rakeshv.org' webXml='rakeshv.xml' users='users.xml'>
   18:       <alias>www.rakeshv.org</alias>
   19:       <alias>books.rakeshv.org</alias>
   20:       <access logFile='../var/logs/rakeshv.log' rotation='daily'
   21:         archive='timestamp' compress='true' purgeCount='15' times='local'
   22:         enabled='true' />
   23:     </virtualHost>
   24:
   25:   </server>
   26: </minnal>



web.xml



    1: <?xml version="1.0" encoding="UTF-8"?>
    2: <web-app id="Minnal-sptci" version="2.4" xmlns="http://java.sun.com/xml/ns/j2ee"
    3:         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    4:         xsi:schemaLocation="http://java.sun.com/xml/ns/j2ee http://java.sun.com/xml/ns/j2ee/web-app_2_4.xsd">
    5:
    6:   <display-name>Minnal/1.0 Server for sptci.com</display-name>
    7:
    8:   <context-param>
    9:     <description>The location of the document root directory.</description>
   10:     <param-name>documentRoot</param-name>
   11:     <param-value>../docroot/sptci</param-value>
   12:   </context-param>
   13:
   14:   <servlet>
   15:     <servlet-name>servlet</servlet-name>
   16:     <servlet-class>spt::servlet::Servlet</servlet-class>
   17:     <init-param>
   18:       <param-name>logLevel</param-name>
   19:       <param-value>PRIO_INFORMATION</param-value>
   20:     </init-param>
   21:   </servlet>
   22:   <servlet>
   23:     <servlet-name>docroot</servlet-name>
   24:     <servlet-class>spt::servlet::DocRootServlet</servlet-class>
   25:     <init-param>
   26:       <description>The default log level for the logger for the DocRootServlet.</description>
   27:       <param-name>logLevel</param-name>
   28:       <param-value>PRIO_INFORMATION</param-value>
   29:     </init-param>
   30:     <init-param>
   31:       <description>Flag that indicates that compressed file responses should be cached for efficiency.</description>
   32:       <param-name>cacheCompressedFiles</param-name>
   33:       <param-value>true</param-value>
   34:     </init-param>
   35:     <init-param>
   36:       <description>The location of the root directory under which cached compressed responses are stored.</description>
   37:       <param-name>documentRootCache</param-name>
   38:       <param-value>../var/cache/docroot/sptci</param-value>
   39:     </init-param>
   40:     <init-param>
   41:       <description>Flag to indicate that requests for URI with query strings are to be rejected.</description>
   42:       <param-name>denyQueryStrings</param-name>
   43:       <param-value>true</param-value>
   44:     </init-param>
   45:   </servlet>
   46:
   47:   <servlet-mapping>
   48:     <servlet-name>servlet</servlet-name>
   49:     <url-pattern>/servlet.html</url-pattern>
   50:   </servlet-mapping>
   51:   <servlet-mapping>
   52:     <servlet-name>servlet</servlet-name>
   53:     <url-pattern>/index.jsp</url-pattern>
   54:   </servlet-mapping>
   55:   <servlet-mapping>
   56:     <servlet-name>docroot</servlet-name>
   57:     <url-pattern>/*</url-pattern>
   58:   </servlet-mapping>
   59:
   60:   <welcome-file-list>
   61:     <welcome-file>index.html</welcome-file>
   62:   </welcome-file-list>
   63:
   64:   <error-page>
   65:     <error-code>404</error-code>
   66:     <location>/errors/notfound.html</location>
   67:   </error-page>
   68:   <error-page>
   69:     <error-code>500</error-code>
   70:     <location>/errors/error.html</location>
   71:   </error-page>
   72:
   73:   <security-constraint>
   74:     <display-name>Restricted GET To Employees</display-name>
   75:     <web-resource-collection>
   76:       <web-resource-name>Restricted Access - Get Only</web-resource-name>
   77:       <url-pattern>/restricted/employee/*</url-pattern>
   78:       <http-method>GET</http-method>
   79:     </web-resource-collection>
   80:     <auth-constraint>
   81:       <role-name>admin</role-name>
   82:     </auth-constraint>
   83:     <user-data-constraint>
   84:       <transport-guarantee>NONE</transport-guarantee>
   85:     </user-data-constraint>
   86:   </security-constraint>
   87:
   88:   <login-config>
   89:     <auth-method>BASIC</auth-method>
   90:     <realm-name>SPT Servlet Container Protected Area</realm-name>
   91:   </login-config>
   92:
   93: </web-app>
   94:

System Security

On UNIX platforms, access to reserved ports (ports with number < 1024) is restricted to the superuser user/role. Services however should never be run as root. The standard design pattern is to start a process as the root user and open the socket on the reserved port (80 for HTTP), and then switch the real and effective user/group for the process to a standard unprivileged user. We follow the same pattern for the Minnal server. The methods are implemented in ServletContainer and embedded in a #ifdef/#endif block to prevent them being compiled in unless necessary. We then add the DEFINE as appropriate in the qmake project file.

unix {
  INCLUDEPATH += \
      /usr/local/reflex/include \
      /usr/local/poco/include \
      src/api

  LIBS += \
        -L/usr/local/reflex/lib -lReflex \
        -L/usr/local/poco/lib -lPocoFoundation -lPocoNet -lPocoXML -lPocoUtil

  solaris* {
    DEFINES += SWITCH_USER_ID
    LIBS += -lsocket
  }
}



In ServletContainer.h we have the following additional sections:

#ifdef SWITCH_USER_ID
#include <grp.h>
#include <pwd.h>
#endif



#ifdef SWITCH_USER_ID
      void switchUser();
      void setGroup();
      void setUser();
      void changeLogOwner();
      void changeLogOwner( const char* file );
      struct group* getGroup();
      struct passwd* getUser();
#endif



The switchUser method is the main method that is invoked as appropriate. This method sets the real and effective user and group id values based on the user and group names configured in the server properties file. We also change the server log file ownership to this user:group to ensure that the server may continue to log after the process ownership has been switched.

Following is the conditional invocation of the method as implemented in ServletContainer.cpp

    1:   ServerSocket svs( port );
    2:   HTTPServer srv( new RequestDispatcher( configDirectory, xmlConfiguration, config() ), svs, params );
    3:   srv.start();
    4:
    5:   {
    6:     logger->information( QString( "Started %1 server with process id: %2" ).
    7:         arg( SERVER_IDENTIFIER ).arg( Process::id() ).toStdString() );
    8:   }
    9:
   10: #ifdef SWITCH_USER_ID
   11:   // Switch to un-privileged user/group if configured
   12:   switchUser();
   13: #endif
   14:
   15:   // wait for CTRL-C or kill
   16:   waitForTerminationRequest();



Additionally, we ensure that only the var directory for the server is writable by the user running the server. All other directories (including the docroot where static content is stored) are owned by root or other user and not writable by the user/group running the server. Figure shows the directory structure for the installed server.

MinnalStructure

Application Security

Application level security is provided at the Servlet level, and is based on standard security-constraint, and login-config elements in the virtual host level web.xml file. The base Servlet class provides a virtual authenticate method, that by default checks the configuration in the web.xml file and sends an appropriate HTTP authentication challenge response. If the client responds with proper credentials, it attempts to authenticate the supplied credentials.

<security-constraint>
  <display-name>Restricted GET To Employees</display-name>
  <web-resource-collection>
    <web-resource-name>Restricted Access - Get Only</web-resource-name>
    <url-pattern>/restricted/employee/*</url-pattern>
    <http-method>GET</http-method>
  </web-resource-collection>
  <auth-constraint>
    <role-name>admin</role-name>
  </auth-constraint>
  <user-data-constraint>
    <transport-guarantee>NONE</transport-guarantee>
  </user-data-constraint>
</security-constraint>
<login-config>
  <auth-method>BASIC</auth-method>
  <realm-name>SPT Servlet Container Protected Area</realm-name>
</login-config>



At present only HTTP Basic authentication is supported. A user contributed HTTP Digest authentication scheme is available for Poco, however, SPT has decided to wait until the Poco development team incorporates the contributed code before using that feature.

The default authentication implemented in Servlet uses a users.xml file (similar in structure to a tomcat-users.xml) file. Note that at present we do not process the role elements. A sample users.xml file is as shown:

<users>
  <role rolename="user"/>
  <role rolename="admin"/>
  <user username="nonadmin" password="password" roles="user"/>
  <user username="admin" password="password" roles="admin,user"/>
</users>



Servlet sub-classes that actually handle requests may over-ride the authenticate method with a more robust and secure authentication scheme, or any custom authentication scheme depending upon business requirements.

Performance

The following are test results for creating and destroying 100,000 servlet instances using Reflex and directly using new/delete. The first set of results were for a Servlet instance with very little initialisation code. Creating and deleting objects using reflection is on average 8.5 times slower without type caching, and 5 times slower with type caching. However, since it is unlikely that the web server will need to process 400,000 requests per second, this performance penalty should not prove significant (after all, we are looking at 2 microseconds to create/delete a Servlet instance without type caching, or 1.15 microseconds to create/delete a Servlet instance with type caching versus 0.23 microseconds to create/delete an instance using new/delete).

2011-10-17 19:46:32.7 [78858]:spt.servlet.servlet:I:Measuring time to create 100000 instances using reflex without type caching
2011-10-17 19:46:32.9 [78858]:spt.servlet.servlet:I:Reflex without type caching took 201985 microseconds to create 100000 instances
2011-10-17 19:46:32.9 [78858]:spt.servlet.servlet:I:Measuring time to create 100000 instances using reflex with type caching
2011-10-17 19:46:33.0 [78858]:spt.servlet.servlet:I:Reflex with type caching took 114394 microseconds to create 100000 instances
2011-10-17 19:46:33.0 [78858]:spt.servlet.servlet:I:Measuring time to create 100000 instances using new/delete
2011-10-17 19:46:33.1 [78858]:spt.servlet.servlet:I:New/Delete took 23622 microseconds to create 100000 instances


The next set of results are for a servlet instance with heavier initialisation code. As you can see creating/deleting instances using new/delete is now only 4.17x faster than Reflex without type caching and 2.49x faster with type caching. As the cost of instantiating an object increases, the performance differential between using Reflex and direct creation decreases.

2011-10-26 14:41:13.3 [28141]:spt.servlet.DocRootServlet:I:Measuring time to create 100000 instances using reflex without type caching
2011-10-26 14:41:13.6 [28141]:spt.servlet.DocRootServlet:I:Reflex without type caching took 309430 microseconds to create 100000 instances
2011-10-26 14:41:13.6 [28141]:spt.servlet.DocRootServlet:I:Measuring time to create 100000 instances using reflex with type caching
2011-10-26 14:41:13.8 [28141]:spt.servlet.DocRootServlet:I:Reflex with type caching took 184916 microseconds to create 100000 instances
2011-10-26 14:41:13.8 [28141]:spt.servlet.DocRootServlet:I:Measuring time to create 100000 instances using new/delete
2011-10-26 14:41:13.9 [28141]:spt.servlet.DocRootServlet:I:New/Delete took 74266 microseconds to create 100000 instances



Tests were executed on a MacBookPro with a 2.2 GHz Intel Core i7 processor.

For extreme high performance environments, it may be worthwhile using singleton servlet instances from a HTTPRequestHandler instead of having a Servlet be a sub-class of HTTPRequestHandler. This is the approach taken by JEE servlet containers, with individual request threads invoking a stateless servlet singleton instance.

Solaris SMF Scripts

The following Service Management Facility (SMF) files may be used to configure the Minnal server as a service in [Open]Solaris.

The service method script (which in our case is stored as /var/svc/method/minnal)

#!/sbin/sh
#
# ident    "@(#)minnal    1.0    2011/11/11 SMI"
#

. /lib/svc/share/smf_include.sh

MINNAL_HOME=/usr/local/minnal
PID_FILE=$MINNAL_HOME/var/minnal.pid

case "$1" in
start)
    cd ${MINNAL_HOME}/bin
    ./Minnal --config=../etc --pidfile=${PID_FILE} &
    ;;
restart)
    $0 stop
    $0 start
    ;;
stop)
    kill -15 `cat ${PID_FILE}`
    ;;
*)
    echo "Usage: $0 {start|stop|restart}"
    exit 1
    ;;
esac



The manifest file is stored under /var/svc/manifest/site/minnal.xml and imported using svccfg.

We have illustrated how simple it is to create high performance cross-platform servlet containers using C++ and Poco. We can easily use these concepts to create light-weight embedded web servers for a variety of platforms including mobile devices. We chose to use a standard JEE web.xml file as the primary configuration file for each virtual host configured for the system, since these are well documented and understood.

For the curious (Minnal is neither open source nor distributed), the API documentation for Minnal may be viewed in: HTML, PDF

We have not tried compiling Reflex for iOS yet, but Poco works perfectly on iOS (the core of our UMA framework is built around Poco). C++ is at present a more cross-platform development language than Java, since Java does not run on iOS, while C++ can be used on all popular platforms (except Blackberry phones, however the Blackberry tablets do support C++).