Serverboards Stack

  Visit at GitHub

Serverboards started as a Django application in its first incarnation, but after finding roadblocks a swift change of stack was required. In this blog post I will describe the current stack and rationale behind it.

TL;DR

Roadblocks at Version Zero

The first version was a more or less typical Django application. Custom SASS, postgres as database and many one page renders server side. A LPND stack (Linux, Postgres, NGINX, Django).

Soon we saw the need for asynchronous communications. For example for notifications to users when there are no page reloads. We could do some polling or longpolls, but websockets sounded like a good solution. Se we had another server to setup. Anyway we needed it for background tasks, as grabbing data from external servers or running commands. But in Python this means or threads, or processes, or async programming. Python 3.4 has support for asyncio, using yield from, and 3.5 has await and async. We were using 3.4. To use it you need a full call chain of @asyncio.coroutine or similar, or it just dont work.. and the code started to be quite spaghetti code.

Fast forward with a big three months trip, improved knowledge and insights of our problem to solve, some desire to change the code… and some crazy ideas on how to achieve it, and we get a rethink of the stack.

Backend

Bullet points:

  • Plugins at heart
  • Process centric. Maybe thousands of independent processes. Each functionality will be a external command, possibly executed several times, in different programming languages.
  • Communicate with JSON RPC. We need an RPC Broker on steroids.
  • WebSocekts for the web client. Other clients.
  • Resilient
  • Secure

I checked various technologies and languages. There were serious thoughts on doing it using Go (goroutines!) or Rust (efficient as C++, hopefully less shotgun your own foot), or another try on Python 3.5 (bad experience if you need asynchonicity), or Nodejs (but its single threaded). Actually Java and Scala were not options in any moment.

I really liked what I knew about Erlang and OTP. But my last taste of Erlang let me clear it’s not a pretty language. On the other hand everybody talks great things about Elixir.. so I gave it a try. And I liked a lot what I saw. There are many things I still don’t like, but these are the less.

So hundreds of thousands of lightweight processes, resilient, let it crash and recover as motto, good tooling, OTP, Phoenix Web Framework… I’m sold.

Elixir has Ecto as the main database middleware. Its not an ORM, just some DSL over SQL so that you create real queries. It has Postgres and Sqlite backends and some more to come, but just now we stay just with Postgres support.

Frontend

On the frontend of course at the end we need HTML5+CSS+JavaScript. And nowadays the best way to get there is using some JavaScript framework.

I did some tests with Ractive (it’s similar to a Django template with a model behind, on which I was comfortable, but very limited), or Angular (1? 2? is even 2 spec finalished? Also it makes you think their way, not the other way around), or Ember (cool! but not really what we need). Finally a deeper look into ReactJS made me really think its a good candidate. What really sold me is that it really is growing and growing getting past the competition (so no left in the cold soon.. too many people uses it: herd protection), and tooling.

From here its just to check what are the best practices and tooling around React:

  • WebPack – Many plugins, easier to use, even CSS can be loaded on the fly. Documentation is horrible, but once you get a good tutorial everything makes sense. The key is that you have loaders that transform whatever in JS modules. Loading this modules does something, as loading the CSS and making it ready for the application, or returning an image URL. And loaders are quite flexible.
  • Redux – CQRS, event sourcing.. but disguised as… not sure what. Why don’t they say its just that? Ah! Flux is the fashion now! Some comments

Oh, and maybe in the future we can do some React Native client.

For the CSS I though of keep using SASS, and as WebPack has great support for it, I just kept using it. But some order was needed. from previous experience something like Semantic UI way of naming stuff is great. Bootstrat could be used too, but the names seem arbitrary. So Semantic UI + SASS.

  • Semantic UI
  • SASS – Of course.

Command Line Interface

And finally for testing, fast checking things, and extensibility: a command line interface. It just provides a prompt and calls into the server.

Actually it provides all functionality of the web client, but on the CLI, using JSON RPC, but on a TCP socket (only at localhost), not on a WebSockets.

For this I went easy and did it in Python. It could be fancier, with asynchronous operations, and such. But no need for that. It does as bash: it lets you press enter to check if there was an asynchronous operation.

Final thoughts

After the decisions were made, change is possible but painful. Just now I don’t think any decision was wrong.

Only change from the initial layout is that instead of using the full Phoenix Elixir Web framework, we are just using some pieces, and for web itself we use Erlang’s cowboy (which Phoenix uses underneath).

On React initially I didn’t like that you can mix the presentation layer and the logic behind it, but with things as Redux lots of the logic just disappears and the pieces fit together.

Semantic UI is really great but seems to have some overhead that right now is ok, but I fear the future. Also the way to include it in your project is not so nice as it creates a copy of itself in your project so you can tweak it, or you stay with the mainline one. Better would be to use the mainline one at nodes_modules and stack over your changes, or just a config file with your tweaks. Anyway its going to stay some more time.

If you need more details on any of the decisions, please use the comments box below.