OpenSSH key visualization

Just thought I'd throw out a quick post with some examples of OpenSSH 5.1's key visualizations. See below.


Generating public/private rsa1 key pair.
Your identification has been saved in /opt/ssh/etc/ssh_host_key.
Your public key has been saved in /opt/ssh/etc/ssh_host_key.pub.
The key fingerprint is:
64:2a:99:60:92:db:f0:ad:39:15:75:c6:72:50:f3:10 root@wintermute
The key's randomart image is:
+--[RSA1 2048]----+
|      o+E.       |
| .   ..oo+       |
|+ o .  oo .      |
| B o + +         |
|. o * . S        |
|   + .           |
|  +              |
|   .             |
|                 |
+-----------------+
Generating public/private dsa key pair.
Your identification has been saved in /opt/ssh/etc/ssh_host_dsa_key.
Your public key has been saved in /opt/ssh/etc/ssh_host_dsa_key.pub.
The key fingerprint is:
db:21:5a:98:3f:40:97:98:60:bd:43:e1:cf:73:02:bb root@wintermute
The key's randomart image is:
+--[ DSA 1024]----+
|    o...         |
|   . ooo .       |
|     .*.o        |
|     .oO         |
|      =.S o      |
|       * B .     |
|      E + .      |
|         .       |
|                 |
+-----------------+
Generating public/private rsa key pair.
Your identification has been saved in /opt/ssh/etc/ssh_host_rsa_key.
Your public key has been saved in /opt/ssh/etc/ssh_host_rsa_key.pub.
The key fingerprint is:
d8:5b:4c:4f:be:41:d9:ab:d0:a9:54:6c:86:9c:c2:7a root@wintermute
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|       . . + o   |
|        o = O .  |
|       + + X . . |
|      o E = * .  |
|       . + o +   |
|        . . o    |
|                 |
|                 |
+-----------------+

I'll come up with a full post soon.


More NikiWiki patches

My pet project, NikiWiki is starting to grow up. Since my last post, I've added a number of features and can cross a few items off the todo list. The most visible change is that NikiWiki can now be used for a basic blog.

I moved neohippie.net over to NikiWiki about a month ago and have been fairly impressed with how easily I can reshape the site with very little effort. Any subdirectory can become a "blog" by adding a .index file inside it with a comma delimited list of page names and unix timestamps. While this certainly isn't the most elegant solution, it works fairly well and doesn't get in the way too much.

NikiWiki now has PAM authentication using Chris Lee's incredibly simple pam.py module. You can implement a number of different authentication methods just by loading and unloading PAM modules. However, if you feel the need to add something a bit more custom, simply implement an authenticate(username, password) method that returns a True or False value indicating whether or not the user has been authenticated. For example, if you wanted to read from an htpasswd file:

from crypt import crypt
def authenticate(username, password): for line in file('/path/to/htpasswd'): line = line.split(':', 1) if line[0] == username and line[1] == crypt(password): return True else: return False
It's possible that some users won't want to use PAM for authentication but I think it makes sense to push users in that direction anyway. Standard authentication methods are a good thing. If you don't like it, change it.

NikiWiki now includes a module I've been working on called nstore. nstore provides a fairly clean and pythonic method for building complicated data storage systems for key->value pairs. The default NikiWiki build just uses a simple FileStore, but you can change this to give you all sorts of neat configurations. For example, I run neohippie.net on a set of three servers in two physical locations. I keep a central authoritative data store on one server, with local caches on the others. My NikiWiki data store looks like this:

self.data = nstore.CacheDict(
     nstore.HTTPStore('http://api.neohippie.net:9608/'),
     caches=[
       nstore.FileStore('/web/sites/neohippie/cache/')
     ]
   )
   
I've added the newlines for clarity. In a nutshell, this configuration performs reads and writes through a CacheDict that uses an HTTPStore as the primary data store and the local filesystem as a cache. The first time an element is read, it is read from the HTTPStore and saved in the local FileStore. On successive reads, the element is read directly out of the local FileStore.

There are a few obvious limitations to this approach, the most notable is that local caches are not updated after the first fetch and they never expire. This is fine for seldom updated sites where I just clear the caches after I change anything but may be a problem for more complicated sites.

I've got some interesting ideas for nstore (DHT anyone?) that should materialize in the near future. Until then, feel free to hack around with it. I've found that writing new UserDicts for nstore can be a lot of fun.

Back to NikiWiki. The other large change introduced recently was a WSGIApp class that completely negates the need for web.py. I was using very little of the functionality within web.py and decided that it would be nice to have one less dependency to worry about. Usage is fairly simple:

   server = WSGIServer(WSGIApp(urls)).run()
   
This is assuming that your WSGIServer takes a method reference as it's first argument. NikiWiki uses flup to provide a FastCGI interface that you can plug into your web server but any WSGI server should work.

For those not keeping track, the current list of dependencies looks like this:

All of the dependencies except for Python and nstore can be installed with easy_install.

Known bugs:

Feel free to bug me via email or jabber if you have problems with NikiWiki.


NikiWiki updates and ideas

I've been playing around with a toy wiki implementation for a few months and apparently I haven't really announced it. NikiWiki is a simple file-based wiki written in Python using web.py and markdown syntax to avoid having to create yet another wiki syntax. It started out as a fork of the code that runs my blog in an attempt to clean up and genericize the code a bit. However, when I took a step back, I realized that I wasn't very far off from having a functional wiki, so I decided to pursue that course with the intent of eventually turning it into a wiki-based blog. I'm still not exactly sure how that's all going to fit together, but I have a few ideas.

In my mind, the definition of a wiki is a website that can be edited by anyone, including the ability to add, change, or delete arbitrary pages and/or content. At such an early stage, nikiwiki just about meets those criteria and little else. nikiwiki currently does not include any form of version control, making it prone to defacement or editing mistakes. In my own deployment of nikiwiki, I setup the data directory as a git repository, giving me some basic control over the pages, but this methodology has many obvious problems, particularly that I have to manually run git commit when I want to save the state of the site.

Todo

Version control

This brings me to my list of ideas for future features... I think version control is an important part of a wiki and will need to be addressed fairly soon. I like the idea of keeping everything within git or svn or some other existing versioning system, rather than attempting to re-invent the wheel. For that reason, I'll probably end up just building a REST interface for manipulating the git repository storing the pages and trigger certain actions automatically when a page is updated; git commit, for instance.

Authorization

I have already started implementing user authentication with PAM. I feel that PAM is the right way to go because it avoids having to create a separate user store and trying to figure out how to get it to integrate with the rest of your environment. Just configure PAM to supply authentication using whatever method you want eg. LDAP, NIS, RADIUS, Kerberos, etc. At some point, I may decide to implement some form of authorization as well (that is, allowing some users to edit certain pages, but not others) but I think that violates the philosophy of a wiki to some extent so unless somebody else wants it, security is an all or nothing approach.

Photo gallery

Right before I started driving across the country, I bought a new digital camera and started investigating options for putting a large number of pictures online. I knew that there were quite a few sites like Flickr, SmugMug, Picasa, etc that offered free photo hosting, but they all impose limits upon the number of photos you can upload or bandwidth usage or some other metric due to the realities of disk space and bandwidth (their tubes are clogged). In an attempt to avoid the inevitable situation of having too many pictures for the service I subscribe to, I decided to pursue the idea of hosting it on my own servers. I've built my environment to scale reasonably, so in theory I should be able to just add more disk/servers if I run out of space. This may not be cheaper, but in my mind, it's more fun.

Unfortunately, the open source photo galleries I was able to find were somewhat disappointing. They were either too limited in the features they offered, too reliant on things I don't want to deal with (read: PHP) or too difficult to integrate into an existing site without rewriting a large number of templates. This got me thinking that I could write a custom photo gallery similar to the way I wrote nikiwiki. Then I decided to take the idea a step further and just build it on top of the same codebase, essentially treating each directory full of pictures as a singular wiki page (with a predefined template for rendering the content of the page).

I still haven't worked out all the details, but I'm pretty sure that directories will be separate "albums" and each image under that will be accessible as a separate page containing a scaled version of the image along with any metadata gleaned from EXIF tags. The URL layout will reflect the on-disk representation closely, so where a normal wiki page would be /My_favorite_page, a photo gallery would be /My_favorite_album, and each image under that would simply be another page /My_favorite_album/First_photo with various GET variables to determine the presentation of the page.

I'm not entirely sure what effect the POST and PUT methods will have on these pages, if I implement them at all. Uploading new images will probably need to be done manually to start, in order to make sure the file layout is correct. I'm not really a fan of web-based uploading tools as they tend to cause more problems than they solve. Until I change my mind, you'll probably just have to use SFTP or some similar method to get the files to the server.

Blog

Coming full circle back to my original intent for nikiwiki, I'd like to find a way to build a blog on top of it, eliminating the need for me to maintain a separate (yet extremely similar) codebase for my blog. The biggest requirement for blogs is that posts (pages) be displayed chronologically. Currently, there is no metadata stored with wiki pages aside from what's on the filesystem. The mtime attribute might suffice, but that may cause problems down the road if I need to recover the pages from a backup or something. In my blog's current codebase, I've solved this problem by maintaining a separate file containing a list of two item tuples in the form (page name, timestamp) where timestamp is elapsed seconds since the epoch. This works well, but I don't really like that the index needs to be updated every time a page is added or changes because it provides a case where the index could get lost somewhere. Maybe I could pull the modification times from the git log of the underlying pages...

The remaining problem with storing blog posts in the wiki is that they will need to be differentiated from normal pages in some way, so that normal pages aren't indexed and displayed on the front page. I haven't given this one much thought, but I think something like a simple naming convention may work.

Site index and tagging

Finally, I think it would be nice to have some automatically generated site map pages similar to Special:NewPages in MediaWiki. Some form of taxonomy, perhaps in the form of tags might also help people to find relevant content, but this brings me back to the problem of maintaining a separate index of all pages, rather than something generated by the content.

It appears that I am now too tired to come up with coherent, or even useful ideas for nikiwiki, so I should probably call it a night. Feel free to send me questions and comments about nikiwiki by sending email to synack at neohippie.net. Hmm... Maybe nikiwiki also needs some form of comment system...


Where am I?

I got a new job with a company out in San Francisco and am going to be driving out there next week. I thought it might be fun for people to watch my progress but I haven't really had the time to setup a good GPS logger... So I bring you location updates via twitter.

Note: Date/time stamps are off. Meh.


Dead simple code deployment

I run neohippie.net on a number of servers at geographically separate locations... For fun.

For a while now, I've been looking for a good way to do code deployment from my development server to the two production servers. Not surprisingly, a lot of people pointed me at Capistrano, the flavor of the day for doing Rails deployment. Seeing as how I don't use Rails (this site runs on Django), Capistrano's builtin deployment recipies were of little use to me... Sure, I could modify the recipies or write my own, or even adapt somebody else's, but at the end of the day I'm left with a large chunk of unmaintained Ruby code that can break anytime.

I also found that Capistrano's documentation is weak at best... http://www.capify.org/ only gives a very light introduction to Capistrano and how to get it installed, beyond that, there is little useful information. The Rails website has a fairly large section on Capistrano, but little of it is relevant to the current version. The only remaining documentation I was able to find came in the form of blog posts from other users stranded by a lack of information. I managed to figure out how to get some basic stuff setup, and ended up just reading the source code to figure the rest out... Which I never should have needed to do.

For these and other reasons, I set off to find something better. I found a blog post that made mention of Fabric, a Capistrano-like deployment tool written in Python. Great! I thought, this should solve all my problems. While I agree with the author's decision to keep the tool as minimal as possible, he's taken it a little too far in that you end up with nothing more than an elaborate shell script, which is exactly what I'm using now. I think Fabric has potential, but the whole point of a deployment system is to give you the ability to do things that would be difficult or time consuming to do with shell scripts (eg. rollback a deployment).

So, if anybody knows of a deployment system written in Python that's well documented, extensible, and open source, let me know... Until then, I'm going to keep using the following shell scripts.

Usage: ./deploy sitename

#!/bin/bash
DEPLOY_ROOT="/web/sites"

source "$DEPLOY_ROOT/$1/.deploy"

function group_cmd {
    for server in $1; do
        echo -n "[$server] "
        ssh $deploy_user@$server "$2" &>/tmp/deploy.log
        if [ ! $? -eq 0 ]; then
            echo " error"
            cat /tmp/deploy.log
        else
            echo " ok"
        fi
    done
}

group_cmd "`cat /web/etc/deploy/web_servers`" \
    "mkdir -p $site_root && \
    cd $site_root && \
    $site_root/init stop && \
    git pull && \
    $site_root/init start"

/web/etc/deploy/web_servers

server1
server2
server3
...

/web/sites/sitename/.deploy

deploy_user="myname"
site_root="/web/sites/sitename"
site_repos="user@server:/path/to/repos"

Just because you can, doesn't mean you should.


Archives

openssh-key-visualization
more-nikiwiki-patches
nikiwiki-updates-and-ideas
where-am-i
dead-simple-code-deployment
rssnzb-masses
openwrt-and-vlans
sennheiser-hd515-vs-shure-es210
python-module-wesabecom
welcome-web-2023-r4
dns-examples
lights-out-without-lom
better-stronger-faster
server-problems-solved
beware-robots
mythtv-protocol-library-hd-on-a-ppc-mac-mini
synacks-basement-neko
apple-iphone
mythtv-protocol-library-updates
mythtv-protocol-library
big-drink-phone-client
finished-moving
upgrade
danger-high-voltage
contribution-to-the-arts
current-projects
python-seminar
pynewspaper-updates
its-a-longshot
summer-of-code-2006--final-report
mia
free-stuff-summer-of-code-updates-homeward-bound-o
sony-kills-puppies
summer-of-code-days-32-48
google-keynote-from-ces-2006
other-projects
summer-of-code-days-20-31
summer-of-code-days-16-19
summer-of-code-day-15
summer-of-code-days-12-14
summer-of-code-day-11
summer-of-code-days-09-10
backups-restored
more-rebuilding-summer-of-code-days-05-08
returning-from-the-dead
summer-of-code-day-04
summer-of-code-day-03
summer-of-code-day-02
more-random-code
summer-of-code-day-01
summer-of-code
internet-access-in-the-middle-of-nowhere
code-repository-open-for-business
rss-fixed
craziness
screamin-asian-man-screamin-asian-man
now-thats-what-i-call-a-rant
moving-to-csh-robot-invaders-a-new-addiction-and-v
neo-hippie
pong-gl
ihack-and-ishoot
csh-registered-for-classes
average-dorm-life
free-stuff-selling-stuff-newsvine
sunrises
ces-pictures
ces-updates
ces-day-1-south-hall-central-hall
day-2-south-hall-take-2-the-sands-race-for-larry-p
ces-day-0-flying-registration-bill-gates-keynote
ces-coverage
comments-fixed
the-second-christening
more-logs
coming-attractions
napster
its-that-time-of-month-again
corning-site-updates-ansa-magic-and-halloween
jon-stewart-colony-strikes-back-cheezy-bread-and-m
itso-tech-bookshelf-and-a-free-cd
serenity-and-free-headphones
iptv
website-updates
itso-bbq
care-package
pictures
classes-music-college-life
getting-cozy
moved-in
perl-irc-rss-and-other-assorted-letters
google-talk
new-site-new-blog
drupal
ipod-shuffle-second-opinion-webcam
ipod-shuffle-and-podcasts
more-rss
rss-fixed
hit-me
interrupted-existence-distributed-class-clustered-
itunes-49
a-week-in-recap-three-hour-cluster-graduation
amusing-quote
slashdots-last-breath-voices-kevin-rose-rit
xbox-360-ps3-and-ibms-salvation
microsoft-xna
comments
tasty-fruit-art-in-another-dimension-and-lots-of-c
a-picture-says-a-thousand-words
geneseo-buffalo-distcc-and-gentoo
new-cell-phone-and-online-games
a-poem-a-girl-a-twist-and-less-lofty-things
christmas-shopping-snow-crash-a-female-and-family
winter-wonderland-games-and-college-decisions
cameras-photos-and-linux-from-scratch
basement-blues
server-playgrounds
daily-grind
cameras-and-wireless-security-with-openbsd
archived-entry-03162004-0102-pm
archived-entry-03112004-0932-pm
archived-entry-03042004-1109-pm
archived-entry-02282004-1114-am
archived-entry-02282004-1112-am
archived-entry-02252004-1254-pm
archived-entry-02242004-0801-pm
archived-entry-02242004-1255-pm
archived-entry-02232004-1241-pm
archived-entry-02182004-0904-am
archived-entry-02112004-0122-pm
archived-entry-02072004-1115-pm
archived-entry-02072004-1054-pm
archived-entry-02052004-0648-pm
archived-entry-01282004-0313-pm
archived-entry-01282004-0102-am
archived-entry-01272004-0225-pm
archived-entry-01272004-1238-am
archived-entry-01262004-0548-pm
archived-entry-01112004-0452-pm
archived-entry-01112004-0451-pm
archived-entry-01102004-0102-pm
archived-entry-12282003-0328-pm
archived-entry-12272003-1230-am
archived-entry-12252003-1145-pm
archived-entry-12242003-1230-am
archived-entry-12222003-0536-pm
archived-entry-12182003-0915-pm
archived-entry-12052003-0303-am
archived-entry-12012003-0952-pm
archived-entry-11302003-1119-pm
archived-entry-11292003-0128-pm
archived-entry-11132003-0125-am
archived-entry-11112003-0705-am
archived-entry-11112003-0324-am
archived-entry-11112003-0112-am
archived-entry-11092003-0122-am
archived-entry-11012003-0308-am
archived-entry-10292003-0105-am
archived-entry-10282003-0439-pm
archived-entry-10272003-0830-pm
archived-entry-10262003-0606-pm
archived-entry-10262003-0128-pm
archived-entry-10262003-0127-am
archived-entry-10162003-0241-am
archived-entry-10112003-0524-pm
archived-entry-10072003-0240-pm
archived-entry-09242003-0649-pm
archived-entry-09222003-1250-pm
archived-entry-09222003-0802-am
archived-entry-09182003-0441-pm
archived-entry-09172003-0137-pm
archived-entry-09162003-0254-am
archived-entry-09132003-0847-pm
archived-entry-09102003-1258-am
archived-entry-09062003-0124-am
archived-entry-09032003-0255-am
archived-entry-08272003-0211-am
archived-entry-08252003-0505-pm
archived-entry-08242003-0735-pm
archived-entry-08212003-1057-pm
archived-entry-08212003-0153-pm
archived-entry-08182003-0428-am
archived-entry-08182003-0308-am
archived-entry-07282003-0118-am
archived-entry-06282003-0609-pm
archived-entry-06282003-0213-am
archived-entry-06272003-1217-am
archived-entry-06262003-0638-pm
archived-entry-06212003-0204-pm
archived-entry-06162003-1033-pm
archived-entry-06142003-0205-am
archived-entry-06142003-1245-am
archived-entry-06142003-1220-am
archived-entry-06142003-1202-am
archived-entry-06132003-1116-pm

or