NikiWiki updates and ideas

I've been playing around with a toy wiki implementation for a few months and apparently I haven't really announced it. NikiWiki is a simple file-based wiki written in Python using web.py and markdown syntax to avoid having to create yet another wiki syntax. It started out as a fork of the code that runs my blog in an attempt to clean up and genericize the code a bit. However, when I took a step back, I realized that I wasn't very far off from having a functional wiki, so I decided to pursue that course with the intent of eventually turning it into a wiki-based blog. I'm still not exactly sure how that's all going to fit together, but I have a few ideas.

In my mind, the definition of a wiki is a website that can be edited by anyone, including the ability to add, change, or delete arbitrary pages and/or content. At such an early stage, nikiwiki just about meets those criteria and little else. nikiwiki currently does not include any form of version control, making it prone to defacement or editing mistakes. In my own deployment of nikiwiki, I setup the data directory as a git repository, giving me some basic control over the pages, but this methodology has many obvious problems, particularly that I have to manually run git commit when I want to save the state of the site.

Todo

Version control

This brings me to my list of ideas for future features... I think version control is an important part of a wiki and will need to be addressed fairly soon. I like the idea of keeping everything within git or svn or some other existing versioning system, rather than attempting to re-invent the wheel. For that reason, I'll probably end up just building a REST interface for manipulating the git repository storing the pages and trigger certain actions automatically when a page is updated; git commit, for instance.

Authorization

I have already started implementing user authentication with PAM. I feel that PAM is the right way to go because it avoids having to create a separate user store and trying to figure out how to get it to integrate with the rest of your environment. Just configure PAM to supply authentication using whatever method you want eg. LDAP, NIS, RADIUS, Kerberos, etc. At some point, I may decide to implement some form of authorization as well (that is, allowing some users to edit certain pages, but not others) but I think that violates the philosophy of a wiki to some extent so unless somebody else wants it, security is an all or nothing approach.

Photo gallery

Right before I started driving across the country, I bought a new digital camera and started investigating options for putting a large number of pictures online. I knew that there were quite a few sites like Flickr, SmugMug, Picasa, etc that offered free photo hosting, but they all impose limits upon the number of photos you can upload or bandwidth usage or some other metric due to the realities of disk space and bandwidth (their tubes are clogged). In an attempt to avoid the inevitable situation of having too many pictures for the service I subscribe to, I decided to pursue the idea of hosting it on my own servers. I've built my environment to scale reasonably, so in theory I should be able to just add more disk/servers if I run out of space. This may not be cheaper, but in my mind, it's more fun.

Unfortunately, the open source photo galleries I was able to find were somewhat disappointing. They were either too limited in the features they offered, too reliant on things I don't want to deal with (read: PHP) or too difficult to integrate into an existing site without rewriting a large number of templates. This got me thinking that I could write a custom photo gallery similar to the way I wrote nikiwiki. Then I decided to take the idea a step further and just build it on top of the same codebase, essentially treating each directory full of pictures as a singular wiki page (with a predefined template for rendering the content of the page).

I still haven't worked out all the details, but I'm pretty sure that directories will be separate "albums" and each image under that will be accessible as a separate page containing a scaled version of the image along with any metadata gleaned from EXIF tags. The URL layout will reflect the on-disk representation closely, so where a normal wiki page would be /My_favorite_page, a photo gallery would be /My_favorite_album, and each image under that would simply be another page /My_favorite_album/First_photo with various GET variables to determine the presentation of the page.

I'm not entirely sure what effect the POST and PUT methods will have on these pages, if I implement them at all. Uploading new images will probably need to be done manually to start, in order to make sure the file layout is correct. I'm not really a fan of web-based uploading tools as they tend to cause more problems than they solve. Until I change my mind, you'll probably just have to use SFTP or some similar method to get the files to the server.

Blog

Coming full circle back to my original intent for nikiwiki, I'd like to find a way to build a blog on top of it, eliminating the need for me to maintain a separate (yet extremely similar) codebase for my blog. The biggest requirement for blogs is that posts (pages) be displayed chronologically. Currently, there is no metadata stored with wiki pages aside from what's on the filesystem. The mtime attribute might suffice, but that may cause problems down the road if I need to recover the pages from a backup or something. In my blog's current codebase, I've solved this problem by maintaining a separate file containing a list of two item tuples in the form (page name, timestamp) where timestamp is elapsed seconds since the epoch. This works well, but I don't really like that the index needs to be updated every time a page is added or changes because it provides a case where the index could get lost somewhere. Maybe I could pull the modification times from the git log of the underlying pages...

The remaining problem with storing blog posts in the wiki is that they will need to be differentiated from normal pages in some way, so that normal pages aren't indexed and displayed on the front page. I haven't given this one much thought, but I think something like a simple naming convention may work.

Site index and tagging

Finally, I think it would be nice to have some automatically generated site map pages similar to Special:NewPages in MediaWiki. Some form of taxonomy, perhaps in the form of tags might also help people to find relevant content, but this brings me back to the problem of maintaining a separate index of all pages, rather than something generated by the content.

It appears that I am now too tired to come up with coherent, or even useful ideas for nikiwiki, so I should probably call it a night. Feel free to send me questions and comments about nikiwiki by sending email to synack at neohippie.net. Hmm... Maybe nikiwiki also needs some form of comment system...

# NikiWiki updates and ideas I've been playing around with a toy wiki implementation for a few months and apparently I haven't really announced it. <a href="http://github.com/synack/nikiwiki/tree/master">NikiWiki</a> is a simple file-based wiki written in Python using web.py and markdown syntax to avoid having to create yet another wiki syntax. It started out as a fork of the code that runs my blog in an attempt to clean up and genericize the code a bit. However, when I took a step back, I realized that I wasn't very far off from having a functional wiki, so I decided to pursue that course with the intent of eventually turning it into a wiki-based blog. I'm still not exactly sure how that's all going to fit together, but I have a few ideas. In my mind, the definition of a wiki is a website that can be edited by anyone, including the ability to add, change, or delete arbitrary pages and/or content. At such an early stage, nikiwiki just about meets those criteria and little else. nikiwiki currently does not include any form of version control, making it prone to defacement or editing mistakes. In my own deployment of nikiwiki, I setup the data directory as a git repository, giving me some basic control over the pages, but this methodology has many obvious problems, particularly that I have to manually run git commit when I want to save the state of the site. ## Todo ### Version control This brings me to my list of ideas for future features... I think version control is an important part of a wiki and will need to be addressed fairly soon. I like the idea of keeping everything within git or svn or some other existing versioning system, rather than attempting to re-invent the wheel. For that reason, I'll probably end up just building a REST interface for manipulating the git repository storing the pages and trigger certain actions automatically when a page is updated; git commit, for instance. ### Authorization I have already started implementing user authentication with PAM. I feel that PAM is the right way to go because it avoids having to create a separate user store and trying to figure out how to get it to integrate with the rest of your environment. Just configure PAM to supply authentication using whatever method you want eg. LDAP, NIS, RADIUS, Kerberos, etc. At some point, I may decide to implement some form of authorization as well (that is, allowing some users to edit certain pages, but not others) but I think that violates the philosophy of a wiki to some extent so unless somebody else wants it, security is an all or nothing approach. ### Photo gallery Right before I started driving across the country, I bought a new digital camera and started investigating options for putting a large number of pictures online. I knew that there were quite a few sites like Flickr, SmugMug, Picasa, etc that offered free photo hosting, but they all impose limits upon the number of photos you can upload or bandwidth usage or some other metric due to the realities of disk space and bandwidth (their tubes are clogged). In an attempt to avoid the inevitable situation of having too many pictures for the service I subscribe to, I decided to pursue the idea of hosting it on my own servers. I've built my environment to scale reasonably, so in theory I should be able to just add more disk/servers if I run out of space. This may not be cheaper, but in my mind, it's more fun. Unfortunately, the open source photo galleries I was able to find were somewhat disappointing. They were either too limited in the features they offered, too reliant on things I don't want to deal with (read: PHP) or too difficult to integrate into an existing site without rewriting a large number of templates. This got me thinking that I could write a custom photo gallery similar to the way I wrote nikiwiki. Then I decided to take the idea a step further and just build it on top of the same codebase, essentially treating each directory full of pictures as a singular wiki page (with a predefined template for rendering the content of the page). I still haven't worked out all the details, but I'm pretty sure that directories will be separate "albums" and each image under that will be accessible as a separate page containing a scaled version of the image along with any metadata gleaned from EXIF tags. The URL layout will reflect the on-disk representation closely, so where a normal wiki page would be /My_favorite_page, a photo gallery would be /My_favorite_album, and each image under that would simply be another page /My_favorite_album/First_photo with various GET variables to determine the presentation of the page. I'm not entirely sure what effect the POST and PUT methods will have on these pages, if I implement them at all. Uploading new images will probably need to be done manually to start, in order to make sure the file layout is correct. I'm not really a fan of web-based uploading tools as they tend to cause more problems than they solve. Until I change my mind, you'll probably just have to use SFTP or some similar method to get the files to the server. ### Blog Coming full circle back to my original intent for nikiwiki, I'd like to find a way to build a blog on top of it, eliminating the need for me to maintain a separate (yet extremely similar) codebase for my blog. The biggest requirement for blogs is that posts (pages) be displayed chronologically. Currently, there is no metadata stored with wiki pages aside from what's on the filesystem. The mtime attribute might suffice, but that may cause problems down the road if I need to recover the pages from a backup or something. In my blog's current codebase, I've solved this problem by maintaining a separate file containing a list of two item tuples in the form (page name, timestamp) where timestamp is elapsed seconds since the epoch. This works well, but I don't really like that the index needs to be updated every time a page is added or changes because it provides a case where the index could get lost somewhere. Maybe I could pull the modification times from the git log of the underlying pages... The remaining problem with storing blog posts in the wiki is that they will need to be differentiated from normal pages in some way, so that normal pages aren't indexed and displayed on the front page. I haven't given this one much thought, but I think something like a simple naming convention may work. ### Site index and tagging Finally, I think it would be nice to have some automatically generated site map pages similar to Special:NewPages in MediaWiki. Some form of taxonomy, perhaps in the form of tags might also help people to find relevant content, but this brings me back to the problem of maintaining a separate index of all pages, rather than something generated by the content. It appears that I am now too tired to come up with coherent, or even useful ideas for nikiwiki, so I should probably call it a night. Feel free to send me questions and comments about nikiwiki by sending email to synack at neohippie.net. Hmm... Maybe nikiwiki also needs some form of comment system...