PsEPR Documentation: Server Guide

Contents


1 Overview

The PsEPRServer was origionally built as a framework to experiement with different types of event distribution and routing. While the collection of classes that make up the server can still be used to do experiementation, the only protocol that was built and tested was a publish/subscribe system. In this system (called the "Lease" system) a client makes an XMPP-like connection to a PsEPRServer, sends events addressed to "channels" and subscribes (or "creates leases") for events that appear on channels. The client side of the protocol is explained in the Protocol Guide.

To support the connections from clients and the routing of messages to other clients with leases, multiple PsEPRServers run and they communicate with each other. Their inter-server protocol is conceptually much like the clients -- the servers subscribe to channels on other servers for their clients.

This guide gives the internal architecture and operation of the PsEPRServer. Detailed interface specifications are in the javadoc API documentation.

2 Installation and Configuration

2.1 Building PsEPRServer

In the subversion tree for the PsEPR project on SourceForge, the sub directory org.psepr.PsEPRServer/build contains a Makefile that will take a jar file and create the PsEPRServer rpm. The way the server jar is usually created is by exporting it from Eclipse. There are beginnings of a build.xml for an Ant build, but that is not completed.

The version of the server is built from the Subversion version of the source file for org.psepr.PsEPRServer.PsEPRServer. The Makefile extracts that version number and builds the rest of the RPM filename (and thus the version) from the date.

2.2 Installing PsEPRServer

Building creates an RPM file of the PsEPR server but there are a few scripts and related files that must be installed.

In the subversion tree for the PsEPR project on SourceForge, the sub directory org.psepr.PsEPRServer/install includes the script pseprInstall which pushes the required RPMs to a PlanetLab sliver and installs same.

If you look in the script, you will see it looks for RPMs in a subdirectory. The RPMs it requires are:

As of January 20, 2008, these all can be fetched from http://psepr.org/download/rpms/

Additionally, the script installs a server configuration file which replaces whatever was installed by the RPM. So, you must also place the following files in the installation directory:

As of January 20, 2008, these all can be fetched from http://psepr.org/PsEPRServer/

The installation process is to cd into the install directory as checked out from subverion, create the rpm subdirectory, populate the directory with the RPMs listed above, add the two configuration files mentioned above into the install directory and then to execute:

              ./pseprInstall SLICENAME HOSTNAME
              
where SLICENAME is the name of the slice and HOSTNAME is the host to install the PsEPRServer.

2.3 Server Configuration

There are zillions of configurable parameters for the server. These are all listed in the module org.psepr.PsEPRServer.ParamServer. Each parameter name is slash seperated set of words. So, for instance, /PsEPRServer/peer/leasepeer/outboundQueueMax is the name of the output queue max size for the LeasePeer version of peer (server to server) connections.

The parameters all have default values which are also listed in the org.psepr.PsEPRServer.ParamServer. module. The values can be changed for a run of the server by specifying them in the file PsEPRServer.xml. This file is looked for at

  • /usr/local/psepr/etc/psepr/PsEPRServer.xml
  • $HOME/.psepr/PsEPRServer.xml
  • \Documents and Settings\$USER\.psepr\PsEPRServer.xml
  • ./PsEPRServer.xml

The format of the PsEPRServer.xml file is XML. The name of a parameter acts likes a poor man's XPath expression to select the value in the file. The default configuration file looks like:

<PsEPRServer>
    <client>
        <XMPP>
          <maximumConnections>400</maximumConnections>
      </XMPP>
    </client>
    <peer>
        <leasePeer>
            <!-- URL of central peer config file -->
            <configurationURL>http://psepr.org/PsEPRServer/config/LeasePeer.xml</configurationURL>
            <!-- number of seconds between fetches and checking of central peer config file -->
            <configurationCheckSeconds>300</configurationCheckSeconds>
            <!-- seconds that a peer makes a lease to another peer -->
            <leaseSeconds>600</leaseSeconds>
        </leasePeer>
    </peer>
    <service>
    	<statusReporter>
	    	  <!-- seconds between sending general status messages -->
	        <periodSeconds>300</periodSeconds>
	        <clientConnections> <include>no</include> </clientConnections>
	        <peerConnections> <include>no</include> </peerConnections>
	        <routeTable> <include>no</include> </routeTable>
	        <service> <include>yes</include> </service>
    	</statusReporter>
    </service>
    <debug>
        <comment>0xB072=SERVICES,PEER,GLEANER,LEASE,BADERROR,STATUSREPORT,IO</comment>
        <comment>0xB062=SERVICES,PEER,GLEANER,LEASE,BADERROR,IO</comment>
        <comment>0xB062=SERVICES,GLEANER,LEASE,BADERROR,IO</comment>
        <filterLevel>
            0xB062
        </filterLevel>
    </debug>
</PsEPRServer>
              

The default location for log files is /usr/local/psepr/logs/PsEPRServer. The log files are a rotated set of files (default of a maximum of 10 files of no more than 1 megabyte) so logs never need to be trimmed. The information that is logged is controlled by /PsEPRServer/debug/filterLevel who's bits turn on and off different log details. The bits are listed in the module org.psepr.PsEPRServer.DebugLogger. Several of the usual logging values are listed as comments in the sample configuration file.

The default ports use by the server are:

23340 unused
23341 LeasePeer incoming connections
23342
23343 OneEvent incoming connections
23344 WS-Notification incoming connections
23345 XMPP incoming connections
23346 unused
23347 unused

2.4 Multiple Server Configuration

When a PsEPRServer is running, the LeasePeer protocols requires each server to know all of the other servers to talk to. The server gets this information by fetching a configuration file from an URL. If the server cannot fetch this file, it will not receive any connections or do any work.

The default location for the inter-server configuration file is http://psepr.org/PsEPRServer/config/LeasePeer.xml and is specified in the individual server configuration file as discussed above. The configuration file is an XML file which specifies, for each server, which server is should connect to, a unique ID that the other server will identify itself with and an authentication code that is used for a peer to know it's talking to the correct peer.

The LeasePeer.xml file is most easily built with a script that is in the web site information in PsEPR's Subversion tree. In the subversion directory org.psepr.www/PsEPRServer/config the scripts buildLeasePeer and buildjClientxml takes the list of active servers from the file ActivePeers and builds LeasePeer.xml and a jClient.xml both of which are then put on a web server to be fetched by the servers and the clients.

3 Server Theory of Operation

The server's job is to act as the central hub in a publish/subscribe messaging system. The server receives subscriptions for events with certain attributes (channel, service, ...). Events are then received by the server from clients and a router modules, who has collected all the subscriptions, forwards the event to zero or more connections. This forwards the event to interested clients or other servers.

The server is organized into the following class instances:

  • RouteTable: table to route received events
  • InternalSender: process internally sent events
  • Connections: incoming and outgoing connections
    • Connections.Peer: creation and management of connections between peer servers
      • ConnectionManagerLeasePeer: connections between peer servers
        • ConnectionReaderPeer: peer message reader and sender
        • ...:
    • Connections.XMPP: XMPP connections from clients
      • ConnectionManagerXMPP: creation and management of XMPP client connections
        • ConnectionReaderXMPP: XMPP message reader and sender
        • ...:
      • Connections.XMPP: XMPP connections from clients
    • Connections.OneEvent: OneEvent connections from clients
      • ConnectionManagerOneEvent: creation and management of OneEvent client connections
        • ConnectionReaderOneEvent: OneEvent message reader and sender
        • ...:
      • Connections.XMPP: XMPP connections from clients
    • ...: other connections of different types
  • Processor: process per connection events (Leases)
    • LeaseProcessor: process XMPP client Lease events
    • PeerProcessor: process peer Lease events
  • Gleaner: delete old Connections
  • StatusReporter: sent events of server status
  • Discovery: sent events for client discovey of server configuration
  • Accounting: receive logon account events and manage authorizations to use this server
  • Commands: receive and perform server command events
  • InternalSender: process internally sent events

One concept used extensively throughout the server is that connections are self managing. By that we mean that objects that are created to manage a connection are in charge of managing themselves thereafter. So, for instance, a connection from a client could cause the ConnectionManagerXMPP to create a ConnectionReaderXMPP object but, thereafter, the ConnectionManager does not need to keep track and manage the ConnectionReader. This makes the PsEPRServer very robust because state is not kept in the managers.

3.1 Events

The server's job is moving events from sources to sinks. The basic structure is simple. org.psepr.ConnectionReaders receive connections from event sources, the events are packaged into org.psepr.ServerEvents and sent to the org.psepr.Route. The Router contains the destinations for events and the events are dispatched to the external connections.

3.2 Inner, Outter and ConnectionReaders

org.psepr.ConnectionReaders implement two interfaces: org.psepr.Inner and org.psepr.Outter. The Inner interface passes events from the ConnectionReader and the Outter interface receives events to go out the connection.

The queuing and threading model prevents any deadlocks by requiring three threads and two queues in each ConnectionReader:

  • input queue: where received events (from the socket or whatever) are placed once they are received;
  • outter queue: where events to be sent over the connection are placed;
  • reader thread: which listens to the input media (stream or socket), reads and assembles events and places the packaged events into the input queue;
  • router thread: which takes events off the input queue and calls the RouteTable for the event. This thread will do the search of the destinations in the RouteTable and call the Outter interfaces for each destination. The Outter interface just queues the event in the ConnectionReader's output queue;
  • output thread: which takes events off the output queue and sends the out the media connection (stream or socket).
There are these two queues and three threads for every connection.

If the internals of the PsEPRServer needs to generate and event, org.psepr.InternalSender implements an inbound queue with a router thread so that internal threads can't become tangled.

3.3 Router and Router Table

The org.psepr.Router calls org.psepr.RouteTable to get a list of org.psepr.Outter interfaces to send the event to. As different connections are made to the PsEPRServer and clients make leases they call RouteTable.addRoute to specify a potential destination for an event. Thus each entry in the RouteTable is for a destination: some connections request to hear about events that match certain criteria.

Each of the routes in RouteTable contain:

  • identifier: a code which uniquely identifies this route. This can be used to delete the route.
  • channel: the channel this lease is listening for
  • service: the particular service this lease is listening for. This can be NULL so say any service will match;
  • instance: the particular instance this lease is listening for. This can be NULL so say any instance will match;
  • expiration time: time when this route should be removed from the RouteTable. This makes routes self-managing in that whoever adds the route can add and forget since the entry will go away when it's expiration time is reached;
  • outter interface: where to send the event if the criteria is matched

3.4 ConnectionManagers

The class org.psepr.ConnectionManager is the super class for the controller of connections of different types. The ConnectionManager* classes manage all the connections of the different types.

3.4.1 ConnectionManagerXMPP

The first type of connection from the clients is an XMPP-like protocal. This ConnectionManager sits and waits on the XMPP port and waits for clients connections. For each client it creates a ConnectionReaderXMPP to read the events from the client.

3.4.2 ConnectionManagerOneEvent

A client that is only going to send one event and not create any leases to receive events, can connection to the OneEvent port and drop off that one event. This creates a org.psepr.OneEvent.ConnectionManagerOneEvent for each client connection.

3.4.3 ConnectionManagerLeasePeer

The PsEPRServer framework was origionally created to experiement with many different types of protocals but the first and only one created is the "Lease" or publish/subscribe to channel model. The interconnection between multiple PsEPRServers ("peers") is called "LeasePeer" because of the client leases that are made to subscribe to events. This ConnectionManager waits for and creates connections to the other peers. Refer to the server configuration section for the configuration files this manager uses to connect to the other peers.

The LeasePeer server configuration is a single level star configuration -- if there are 10 PsEPRServers, each of the 10 servers connect to the 9 other servers. When a client on a server requests a lease (subscription) on a channel, that server then subscribes to that channel on the other 9 servers. This simple configuration is not scalable but it does work up to some large numbers of clients. Successful installations have run 20 servers with up to 400 clients per server.

3.4.4 ConnectionManagerWSNotify

This is an uncompleted manager for WS-Notification events. It was noted that the channel model fits WS-Notification topics. This is an interesting idea that was never completed.

4 Configuration

5 Services

5.1 Status Reporter

5.2 Accounting

5.3 Discovery

The Discovery service periodically outputs on a channel information that comes from each of the ConnectionManagers. The purpose of this service is to allow clients to listen on the "discovery channel" and find all of the connection managers it can connect to. This is the only way for clients to find all of the servers for a particular type of connection.

The Discover service loops through all the ConnectionManagers calling ConnectionManager.xmlForDiscovery() to create a message. The format of the event payload is:

<payload xmlns="http://dsmt.org/schema/psepr/payload/PsEPRServer/discovery-1.0">
  <connections>
    <hostname>hostGeneratingDiscovery</hostname>
    <instance>instanceCodeForServer</instance>
    <serverVersion>serverVersionCode</serverVersion>
    <connection>
      <type>typeOfConnection</type>
      <hostname>hostToConnectTo</hostname>
      <port>portToConnectTo</port>
      <class>classnameOfConnectionManager</class>
    </connection>
  </connections>
</payload>
            
There will be multiple <connection> elements -- one for each ConnectionManager configured in this server.
typeOfConnection code for the connection. The most interesting ones are "XMPP" and "OneEvent".
hostToConnectTo A client should address this host to reach this ConnectionManager
portToConnectTo A client should address this port to reach this ConnectionManager
classnameOfConnectionManager the class name of the connection manager to match it up with other status services.

5.4 Commands

6 Thoughts on Future Work

p2p protocol improvement (xml stream)

peer connections (only one socket per peer

scalability. Not design goal for version 1. IRC strict tree with no duplicates. UDP has no duplicates and throws away. UUCP creates duplicates while routine but keeps message history. Multi-tier ideas to keep overhead low and not create duplicate messages.