PawnStorm

PawnStrom is a framework built on top of JGroups.  It is basically a client server based work distribution system that allows you to leverage personal computer resources as workers.  Google may have the resources to build out server room after server room of computers to do it's bidding, but all I have at my disposal are a couple Amazon EC2 micro instances and friends with internet connections.

There are really three types of nodes in PawnStorm.

Work Owner:  The Work Owners are responsible for submitting scheduled and ad hoc work to the Work Manager, and processing results.  The Work Owner submits schedule data and requests over JGroups, but results are processed by consuming a JMS queue.  Using JGroups for work scheduling give us more flexibility in the types of interactions we can have with the work Managers.  We can also use JGroups to get detailed work status from the work managers and all worker, that can be progress windows either in a management system or in the customer facing portal.  The system can have many work owners.

Work Manager: The Work Managers are responsible for assigning, routing, and tracking work.  There are two main types of work the Work Managers handle.  There is ad hoc work submitted by the Work Owners and work that has been scheduled by Work Owners.  Work Managers are also responsible for running JGroups Gossip routers, clustered Berkeley DBs, and clustered HornetQ messaging servers.  The Gossip routers are the backbone of the JGroups communication network, and are responsible for routing all messages between JGroup servers.  The clustered Berkeley DBs are used to persist data about work that is scheduled, running, and completed.  The clustered HornetQ server are responsible for storing and forwarding completed work back to Work Owners.

Worker:  A worker node is about as glorious as it sounds.  They do the work.  Worker also have the ability if they are working on a log running process to checkpoint their work back to the Work Manager.  This allows the Work Manager to assign their work to another worker in the even they crash or leave the group.  Worker have the ability to update themselves from the Work Managers, so reflect your every changing needs without requiring manual updates.