nstore

nstore is a flexible storage backend for just about anything you can throw at it. I started writing nstore while reading through Rolling your own data persistence. I've applied a number of the techniques described in that presentation, along with a few of my own ideas. At the moment, nstore provides the following classes:

Storage backends

FileStore

A simple file-backed storage dict supporting automatic creation and removal of subdirectories when a '/' character is used in a key.

MemoryStore

A non-persistent memory storage dict that provides reasonably good performance. This is meant to be used mostly for caching

HTTPStore

A frontend for the RESTful server provided by HTTPDict, but it could also be used as a general purpose HTTP/REST client.

Storage wrappers

PrefixDict

Prefixes every key with the argument passed to the constructor. This allows you to share a backend store's namespace with multiple applications using different prefixes.

HTTPDict

Provides a basic HTTP GET/POST/DELETE interface for the backend store. Implements a WSGI application and uses the builtin wsgiref server to serve requests in a daemon thread.

LoggingDict

Writes out a basic access log for any get, set, and delete actions that are performed on the dict. Useful for debugging.

ZipDict

Stolen directly from the pycon slides, ZipDict compresses data using the builtin zlib module.

CacheDict

Provides the ability to return cached responses for get calls. Set and delete calls are written to the primary store synchronously and caches can optionally be written asynchronously.

PickleSerializer

Serializes python objects into the backend store using cPickle. Works reasonably well for simple tasks but has some strange quirks.

B64KeyDict

Converts all keys to base64 strings in order to provide binary safety for storage backends that may not allow invalid characters in the namespace. Note that this is no longer an issue with FileStore.

I haven't made much of an effort to cleanup any of this code for release, but if you're interested, shoot me an email at synack neohippie.net and I'll send along a copy of the module.

Future plans

Eventually, I'd like to build a Distributed Hash Table (DHT) on top of nstore, creating a simple storage cloud that can be used in any application.

# nstore nstore is a flexible storage backend for just about anything you can throw at it. I started writing nstore while reading through [Rolling your own data persistence](http://blog.codejams.net/pycon/). I've applied a number of the techniques described in that presentation, along with a few of my own ideas. At the moment, nstore provides the following classes: ## Storage backends ### FileStore A simple file-backed storage dict supporting automatic creation and removal of subdirectories when a '/' character is used in a key. ### MemoryStore A non-persistent memory storage dict that provides reasonably good performance. This is meant to be used mostly for caching ### HTTPStore A frontend for the RESTful server provided by HTTPDict, but it could also be used as a general purpose HTTP/REST client. ## Storage wrappers ### PrefixDict Prefixes every key with the argument passed to the constructor. This allows you to share a backend store's namespace with multiple applications using different prefixes. ### HTTPDict Provides a basic HTTP GET/POST/DELETE interface for the backend store. Implements a WSGI application and uses the builtin wsgiref server to serve requests in a daemon thread. ### LoggingDict Writes out a basic access log for any get, set, and delete actions that are performed on the dict. Useful for debugging. ### ZipDict Stolen directly from the pycon slides, ZipDict compresses data using the builtin zlib module. ### CacheDict Provides the ability to return cached responses for get calls. Set and delete calls are written to the primary store synchronously and caches can optionally be written asynchronously. ### PickleSerializer Serializes python objects into the backend store using cPickle. Works reasonably well for simple tasks but has some strange quirks. ### B64KeyDict Converts all keys to base64 strings in order to provide binary safety for storage backends that may not allow invalid characters in the namespace. Note that this is no longer an issue with FileStore. I haven't made much of an effort to cleanup any of this code for release, but if you're interested, shoot me an email at synack <at> neohippie.net and I'll send along a copy of the module. ## Future plans Eventually, I'd like to build a Distributed Hash Table (DHT) on top of nstore, creating a simple storage cloud that can be used in any application.