Message Oriented Middleware, Redux, Event Sourcing and CQRS at Serverboards

  Visit at GitHub

At Serverboards we wanted from the very beginning to use the top engineering techniques.

In this post I will explore some of the solutions we decided to use, their relationship and differences. This is two supporting patterns, Channels and CQRS, and two specific tools, Event Sourcing and Redux.

Message Oriented Middleware

Serverboards is at its heart a powerful Message Oriented Middleware. It connects clients (Web, Command Line) to plugins, all that using JSON RPC.

A Message Oriented Middleware is used to communicate between several components using message channels. There are different kinds of channels, but basically all of them offer a subscription to the channel, and whenever there is a message some or all of the subscribers can process the message. It is a one way communication, from producer to consumer; there is no communication from consumer back to the producer.

This is used in Serverboards, for example, so clients can subscribe to a channel and get notified when there is a change in the application state, so that the UI can modify itself.

It is also used as the basis for the RPC communication where a Point to Point channel is used so that only one RPC consumer consumes the message, and then writes the answer into a reply channel, if its a method invocation (as opposed to an event notification). The call has the reply address, as well as all the RPC method call payload.

Normally a full blown MOM is not needed for RPC, but it give several advantages as using the same tooling for debugging, and a lot of code can be shared.

Serverboards adds some extras over it, as permission check for RPC calls in the form of guards, functions that given the calling context and the calling message can decide if the message is allowed to be delivered at that endpoint.

Redux

On the frontend we use Redux. With redux you create some event stores that register some reducers that can given some actions define the current state of the application. Users then can subscribe to the store and access the state to continue the processing. It is very functional oriented, as the state is reduced using pure functions and given the same set of actions it will always reduce the same state.

This allows for example time travelling with just storing the actions. Get a given known state (maybe the beginning of time), apply again the reducers and you get again the state. If you start from a checkpointed state, you can get again to the same state faster.

In redux-react the way to work is create an action function that will be called always that the user performs an action like press a button. This action will perform all required side effects and it will dispatch some events to the event store. The event store then reduces the new state that will be notified to redux-react and it will set a new state into React.

Event Sourcing

In event sourcing we add some commands to a Event Manager. The Event Manager knows how to perform the given commands and change state of the system. A good example is accounting where the accountant writes always new records, never modifies any, and just doing the sums you can get a final state. In event sourcing this state is then saved into the database. Commands always are described in an imperative way. Preconditions should be checked before sending the command.

The events can also be saved into a journal so they can be replayed later, allowing for example time travelling, perfect accountability of the commands and also change of requirements. If the original commands saved enough information, a change of requirements allow to create a new state in a way that would not be possible without event sourcing.

All commands should be emmited with some independent object representation, so that for example if an item in the database changes its primary id key, the commands can still be applied. A way to accomplish this is to generate an UUID that is part of the object creation command (create an item with this UUID).

Finally some events can be delivered indicating there is a change of state. Events are defined using verbs in the past tense, indicating the action is already done.

For example if you use Event Sourcing to manage the account details of a given user and its account is compromised and changed in a nefarious way, you can just replay the account modification commands until the date of the breach and everything will be ok. If you were not using Event Sourcing, its better you have good backups.

It also allows to process in a delayed way, using queues, so that the systems have better throughput. This is used in the LMAX architecture with amazing results.

CQRS

Command Query Response Segregation essentially separates the concerns of the quering of data and the changing of it. It makes a full separation so that in no case a change of data can be used to also query it. The rules can be broken in specific cases, but always if there is really a good concern.

This programming pattern can not benefit from any specific framework or library, just to be careful when you write your code.

For example in a stack push is a command, top and empty are queries, but pop is something in the middle that can be separated into a top and a forget_top command, but, depending on the use, it then losses the atomicity, so, if atomicity of pop is required, CQRS allows pop to both change state and return a value.

CQRS is used quite often together with Event Sourcing, creating a pipeline in which the commands are sent to the executors, this change the state as required, and generate events that define the new state, that both the query functions and interested parties can listen for changes. As all the queries go though the same functions, some caching can be performed, for example, to avoid hitting the database, as just listening for the change events is enough to know the state of the system.

How all fits together

So as we saw both Redux and Event Sourcing can be implemented in terms of channels, but the evil is in the details, and for example Redux requires a state that is modified (not really, it is immutable, but the result is a new state) and passed around, which a normal channel implementation does not allow. So a new implementation is required. Event sourcing on its own could be implemented in terms of channels.

Redux and Event Sourcing are quite similar, with Redux just the second part of what CQRS with event sourcing does; it generates a state from the events of the changes already done into the system.

This is cool in the frontend, as the real side effects should be stored into the backend, and they way it is articulated, specially with redux-react fits perfectly so that events on the UI provoque controlled changes in the state that in term changes the UI.

So to make frontend and backend also fit, backend should use event sourcing (optionally with CQRS), so the flow is:

  1. User creates an UI event, for example clicks a button
  2. This triggers an action that may emit a redux event (status change sent to server) and sends a command to the backend.
  3. The backend sends the command to the event sourcing engine,
  4. A change is performed in the state of the application, and stored in database, an event is emmitted
  5. This event is notified to all the interested frontend clients.
  6. The frontend captures this event and sends it as an event to the redux store.
  7. The redux store modifies the UI as necessary.

If you followed with attention you saw that the event was sent to all the clients interested, so that other clients connected could see in almost real time changes on their UI based on actions of another client. And all this perfectly articulated, without any hack and perfectly reproducible on any part of the application.


Did you like the article? Do you think something is wrong? Anything to precise? Please comment below!