I am considering this stack off the shelf in my next big project:
- uWSGI - performs better than gunicorn and has support for async apps using gevent
- nginix - front end server
- pyramid - web framework
- mongodb - database
- mongoengine - mongodb and python mapper
- zeromq - messaging and communication
- jinja2 - for template engine
- gevent - for async processing
- gevent-zeromq - to make zeromq non-blocking and gevent compatible
- socket-io - JS lib for realtime communication
I still need to develop robust session management. I considered various options and came to conclusion if I want something fast, truly distributed and not using sticky session I should come up with my own session manager demon hosted on each node. I would use ZeroMQ to communicate to it.
Thanks. Actually I did consider Beaker it didn't fulfill my requirement. I wanted to replicate sessions across nodes actively and async, and also I wanted to persist to MongoDB async.
So here is how I was considering to build:
- sessions updated and validated by a demon process per node.
- each validation and update will be one call via ZeroMQ's Req/Rep pattern. With each call I can validate session and reset timestamp.
- after each validation I asynchly will replicate session across other nodes via ZeroMQ's Pub/Sub ( I don't care about extra memory)
- sessions will also persist in MongoDB (async), just in case each node is restarted, thus preserving session.
Btw, I only want to valid/invalidate a session token and keep authorization information. Any other small values I could simply keep in the cookies encrypted, e.g. User ID. Though I could keep session information in cookie as well, but this allows sessions to live forever and it's not good if I want to kick some user out.
Primarily I wanted it to be really fast, be able to invalidate session, and be available on any node via active replication. If I used Beaker, I had to use memcached which is not replicated (by default setup), and each validation would have required 2 round trips to memchached (validation, update stamp).
It's easier to write my own with ZeroMQ and I can build with custom logic at demon level.
You don't have to use memcached, the beaker extensions let you use whichever backend storage mechanism you want. Redis Cluster can be replicated and would work fine for this, for example.
I don't have a good grasp of what you're trying to achieve and why, but my overengineering alarms are going off.
Awhile back I looked into Mongo for sessions, but it isn't really designed to do this. Sessions are temporal, and AFAIK Mongo doesn't have a built in mechanism for expiring them, plus session data is usually small -- there are other datastores better suited for this type of data.
Primarily sessions will live in RAM. And I was going to use MongoDB to persist sessions just in case the daemon dies or session is not found in RAM as a fallback. The reason for using MongoDB is because it's my primary database.
But your concern about growing table of sessions is valid, which I will handle by periodically archiving the old sessions so that index sizes remain small.
Initially I was, because I am coming from the Oracle world. But for the performance gain, I can give up ACID compliance, which is solvable with additional code complexity.
For extra reliability in MongoDB, I am thinking to use replica sets (active/passive) and journaling.
is the uWSGI choice a purely based on performance? I like gunicorn for the simplicity, uWSGI has tons of options thought I wonder if they make management more difficult.
Yes, it was primarily because of the performance. According to these benchmarks, unicorn totally choked: "... At the bottom we have Twisted and Gunicorn ..."
Look at the comment, the twisted code is wrong. By default the twisted reactor is multiplatform, but you can imporve performance for your specific platform. It's in the doc.
If you run Twisted under linux and you should, use the epoll reactor.
- uWSGI - performs better than gunicorn and has support for async apps using gevent
- nginix - front end server
- pyramid - web framework
- mongodb - database
- mongoengine - mongodb and python mapper
- zeromq - messaging and communication
- jinja2 - for template engine
- gevent - for async processing
- gevent-zeromq - to make zeromq non-blocking and gevent compatible
- socket-io - JS lib for realtime communication
I still need to develop robust session management. I considered various options and came to conclusion if I want something fast, truly distributed and not using sticky session I should come up with my own session manager demon hosted on each node. I would use ZeroMQ to communicate to it.