After my needs finally exceeded the limited capacity of GNOME's default bittorrent client, I tried a few alternatives but didn't have much luck. Azureus is almost definitely the single worst program I have ever used, and most of the interesting native-Linux alternatives are still too experimental for day-to-day use. A friend had repeatedly suggested that I consider uTorrent, so I eventually decided to try running it through Wine. It is now my bittorrent client of choice, and one that I highly recommend to others. Miraculously, uTorrent works almost perfectly on Linux. Even though I am running it through Wine, it is more stable, responsive, and robust than Azureus and signficantly less resource intensive.
The latest public beta release of uTorrent includes support for a remotely-accessible web interface. The web UI itself is relatively unremarkable and of little interest to me, but the underlying API is enormously useful. In order to facilitate the construction of a web interface, the uTorrent developers have exposed much of the application's underlying infrastructure through JSON. A little bit of simple reverse engineering has made it possible for me to construct my own interface, and adapt the program in unique and innovative ways.
My first experiment was a simple Python script that connects to uTorrent, logs in using proper authentication, extracts information about active torrents, and displays that information on stdout:
import httplib, base64
def parse_data(data):
data = dict((name, data[column]) for column, name in enumerate([
"hash", "state", "name", "total", "unknown", "received", "sent", "ratio",
"upload_speed", "speed", "eta", "label", "connected_peers", "peers",
"connected_seeds", "seeds", "available", "queue", "remaining"]))
data.update({
"hours_remaining": round(float(data["eta"] / 60 / 60), 2),
"percent": round(float(data["received"]) / data["total"] * 100,2),
"speed_kb": round(float(data["speed"]) / 1024, 2),
})
return data
h = {"Authorization": "Basic " + base64.encodestring("admin:").strip()}
con = httplib.HTTPConnection("localhost", 8080)
con.request("GET", "/gui/?list=1", headers = h)
response = con.getresponse()
if response.status == 200:
torrents = [parse_data(x) for x in eval(response.read())["torrents"]]
for t in torrents:
print "%-15.15s | %5.2f kb/s | %3.0f hours | %5.2f%%" % (
t["name"], t["speed_kb"], t["hours_remaining"], t["percent"])
print "\nTotal download speed: %s kb/s" % sum(t["speed_kb"] for t in torrents)
This is a relatively simple task, particularly since the syntax used to describe JSON data structures is virtually identical to Python's dict syntax. I cheat and use eval to eliminate the need for a third party JSON library. In general, using eval like that is something I would avoid and discourage since it could potentially create security vulnerabilities, but for a basic experiment it isn't really a big deal. In this example, the parse_data function is used to generate a python dict that associates key names with various torrent attribute values. Data from each torrent is parsed, and then displayed on the screen. At the very end, I use the sum function to compute the total download speed.
After completing my first experiment, I decided to take it to the next level. My Logitech G15 keyboard has a nifty programmable LCD that can be used to display various kinds of data. I haven't really done much with the LCD yet, so it seemed like the perfect opportunity. The G15 LCD is supported on the Linux platform by the open source G15 Tools collection, which includes a low-level hardware access library, a userspace daemon for managing the keyboard's macro buttons and LCD, a rendering library designed specifically for generating content suitable for the G15 screen, and a composer utility that enables users to paint the LCD with simple commands.
Unfortunately, there are no Python or Ruby bindings for the rendering library, so I had to either communicate directly with the daemon or use the composer as an intermediary. The Python bindings for the daemon only provide raw pixel access with no abstraction for text or anything like that. The composer provides such abstractions, but it has some weird limitations of its own. The composer binds to a named pipe on the file system, and then one communicates with it by writing commands into the named pipe. It's counterintuitive and messy, but it does work.
The composer accepts numerous commands, and allows the user to write text in fonts at several different sizes. It also supports basic drawing and graphics functionality. I used several of the composer's advanced features in my uTorrent display for the sake of experimentation. I added the following code to my initial script, and wrapped the entire thing in a while loop with a one second time.sleep invocation to make it repeat every second:
total_speed = sum(t["speed_kb"] for t in torrents)
text = "%(name)-20.20s %(speed_kb)3d %(hours_remaining)3d %(percent)3d"
headings = "%-20.20s %3s %3s %3s" % ("Name", "k/s", "eta", "%")
sys.stdout = f = open(sys.argv[1], "a")
print "PC 0\nTM \"%s\"" % headings
print "TO 0 10 1 0 \"%s\"" % '" "'.join(text % t for t in torrents)
print "TO 0 35 2 0 \"uTorrent %0.2f Kb/s\"" % total_speed
for x in [101, 121, 141]: print "DL %d 0 %d 32 1" % (x, x)
for x in [8, 32]: print "DL 0 %d 500 %d 1" % (x, x)
It's not exactly simple, but it works. My G15 LCD uTorrent display looks relatively nice. The grid layout in this particular examples uses up a lot of space, so only three active torrents will be visible. If you need to see more, you can reduce the font size of the total download speed value or remove the grid and labels. If you use the smallest font size and you don't include anything but the torrent data, you could probably fit a lot more on the screen. I usually only have two or three torrents actively downloading at any given time, so I prefer using the bigger fonts.
Since I plan on using this on a regular basis, I decided to clean up the code a bit. In the final version, I put the uTorrent stuff into its own file and threw together a few classes to make it easier to work with. You can see the complete version here. Those of you who want to use it, please remember that this implementation was designed specifically for the Linux platform and almost definitely wont work on Windows in its current state. I've already started to think about other ways to integrate uTorrent and the G15. For my next project, I want to see if I can find a way to control uTorrent's maximum download speed with the keyboard's volume dial.
Tags: programming, python, G15, bittorrent
Posted on 2007-01-18
I recently purchased a Logitech G15 gaming keyboard, which features a nifty integrated LCD and an assortment of macro keys. Unfortunately, Beryl only allows users to bind arbitrary commands to 11 keys. Since the G15 has 18 separate macro keys in addition to multimedia buttons, I had to find an alternate solution. I finally decided to implement my own binding manager so that I can associate keys with commands and Python expressions.
Obviously, the first step was to figure out how to set up global key bindings with Python. I started by poking around the GTK/GDK documentation in search of relevant functionality, but I didn't really find anything there. I asked in the #linux channel on the Ars Technica IRC server and got some useful suggestions, but nothing that seemed to do exactly what I wanted. It finally occurred to me to investigate the source code of a Python application that has global key bindings.
Deskbar, a really useful panel applet that I use on a daily basis, uses a global key binding for activation. The deskbar/Keybinder.py file in the Python site-packages directory reveals that Deskbar is actually using a file called _keybinder.so, which appears to have been created for the Tomboy note-taking program. The tomboy_keybinder_bind function in _keybinder.so makes it possible to associate a binding with a Python function. The binding is described as strings consisting of modifiers and single letters. The following is a simple example that shows how to associate a key binding with a function:
import gtk
from _keybinder import tomboy_keybinder_bind as bindkey
def onBindingPress(arg):
print "Binding initiated and parameter received: %s" % arg
bindkey("<Ctrl><Alt>b", onBindingPress, "Test")
bindkey("<Ctrl><Alt>q", gtk.main_quit)
gtk.main()
I find it a bit odd that this functionality had to be borrowed from Tomboy, and it makes me wonder how it is implemented in _keybinder.so. I'm not entirely sure why global key binding support isn't provided by the Python and Ruby GTK bindings, but I'm pleased that I found a way to do it with minimal hassle.
Tags: programming, python, gnome, G15
Posted on 2007-01-10
I recently discovered a Python IDE called PIDA that provides support for embedded Vim. I'm not a big fan of Python IDEs, but being able to use Vim as the editing component is definitely a killer feature. I haven't seen Vim embedded in a GNOME program since the unfortunate demise of the GVim Bonobo component and I had no idea it was even possible with current versions of Vim. A quick inspection of PIDA's source code revealed that it launches a new instance of GVim and then uses XEmbed to incorporate it into the user interface.
I had never heard of XEmbed before, but seeing it used in PIDA compelled me to pursue further investigation. Apparently, XEmbed makes it possible to embed virtually any X11 window inside of another application. GTK provides support for this with the Plug and Socket widgets, which are available in GTK's Python and Ruby bindings. XEmbed client support is also available in Qt with the QtXEmbedContainer object.
For the sake of experimentation, I wrote a simple Ruby script that uses my window management library and the Gtk::Socket object to facilitate window embedding. It provides two combo boxes which each contain a complete list of windows. The user selects a window in each one, and then clicks the Grab button to grab those windows and reparent them in Socket objects separated by a resizeable splitter. The source code for this experiment is relatively simple:
#!/usr/bin/env ruby
require "wmlib"
require "gtk2"
win = Gtk::Window.new("Embed Test")
win.signal_connect("destroy") {|w| Gtk.main_quit}
win.add vb = Gtk::VBox.new
vb.pack_start(hb = Gtk::HBox.new, false, false)
lst = Gtk::ListStore.new String, Integer
hb.pack_start(combo1 = Gtk::ComboBox.new)
hb.pack_start(combo2 = Gtk::ComboBox.new)
hb.pack_start(btnGrab = Gtk::Button.new("Grab"), false)
combo1.model = lst
combo2.model = lst
for w in WM::Window
i = lst.append
i[0], i[1] = w.title[0..30], w.id if w.title
end
vb.pack_start(pane = Gtk::HPaned.new)
pane.pack1 socket = Gtk::Socket.new, true, false
pane.pack2 socket2 = Gtk::Socket.new, true, false
win.show_all()
btnGrab.signal_connect("clicked") {|*a|
socket.add_id combo1.active_iter[1]
socket2.add_id combo2.active_iter[1]
}
Gtk.main
While experimenting with panes and XEmbed, one is reminded of tiling window managers like Ion. The ability to place arbitrary windows into tabs or panes is very compelling, and I'm certain I'll find some uses for the feature in the future.
Tags: programming, gnome, ruby
Posted on 2007-01-06
I've been trying to get my hands on a Nintendo Wii since the release last month. I haven't had much luck, and I'm not willing to pay a higher price to get one from e-bay or from the numerous retailers that offer overpriced bundles. To help me keep an eye on Wii availability, I made a simple notification utility that uses xpBargain's Wii Locator RSS feed. My utility checks the RSS feed every five minutes and displays the results in a compact notification window.
I was initially just going to keep a browser window open at the Wii Locator page and use the Firefox ReloadEvery extension to refresh it every five minutes, but that would waste a lot of screen space and be harder to read at a glance. I decided to use Ruby and GTK just to keep things simple. Coding simple data lists with a GTK treeview is more trouble than it is worth, so I decided to generate simple html and dump it into a GtkMozEmbed widget.
Making this sort of utility is relatively trivial in GTK, and I managed to write most of it without having to refer to the API documentation. I used REXML to parse the RSS feed and then I generate the HTML by hand. I could have converted the RSS to HTML with XSL, but that probably would have been overkill for such a simple throw-away app.
Everything seemed to work, but I ran into a small problem. Ruby's simple threading system doesn't work with GTK's main loop. To make a Ruby script perform an action every 300 seconds without stalling the rest of the program, I would typically do something like this:
Thread.new do
loop do
process_data
sleep 300
end
end
Unfortunately, that particular approach makes the program hang. After a little bit of research, I figured out how to perform timed events on a loop with GTK. The timeout_add method will evaluate a block at intervals when the program is otherwise idle. The block will continue to repeat at roughly the specified interval as long as the block returns the boolean true value when it is evaluated. In my simple Wii availability tracker, I used the following code:
Gtk.timeout_add(300 * 1000) do
process_data
true
end
It seems like the only places that have Wiis available right now are selling overpriced bundles. I probably wont be able to get one at a reasonable price until January, but at least my new utility will make it easier.
Tags: programming, ruby, gnome, nintendo
Posted on 2006-12-23
I have spent a lot of time lately working with Powershell, Microsoft's next generation command-line environment. Built on the .NET architecture, Powershell is extremely versatile and robust. It provides excellent support for dynamic manipulation of XML content, a feature that is particularly useful when used in conjunction with .NET's Internet functionality.
Earlier this week, I did a bit of experimentation and figured out how to use Powershell to interact with the Simpy social bookmark service. When I developed Ruby bindings for the Simpy REST API, the most frustrating impediment was Simpy's mildly awkward redirect system. Ruby's Net::HTTP class doesn't handle redirects automatically, so I had to manually implement support for handling a 302 response. I also had to manually create the authentication string for the header using Ruby's Base64 encoding library. Powershell handles both of those things automatically. The .NET Net.NetworkCredential object automatically generates the proper header string when provided with login information, and the Net.WebClient object's downloadString method does proper redirect handling on its own without necessitating further intervention.
Powershell handles XML very well, which also made the task much easier. Certain aspects of the XML API seem needlessly verbose or a little bit eccentric. For instance, when you use the select-object Cmdlet to access an element node, it returns the node itself rather than the value of the node in some cases. Based on my own experiments, I think that it fails to show the actual text value when the contents of the node are wrapped in a CDATA block. In order to extract the text value, one must use the get_innerText method.
The following example demonstrates how to use Simpy's REST API from the command-line to display the nicknames and tags of recent bookmarks:
> $connect = new-object Net.WebClient
> $connect.credentials = new-object Net.NetworkCredential("segphault", "XXXXXX")
> $links = [xml]$connect.downloadString("http://simpy.com/simpy/api/rest/GetLinks.do")
> $links.links.link | select {$_.nickname.get_innertext()}, {$_.tags.tag | % {$_.get_innnertext()}}
$_.nickname.get_innertext() $_.tags.tag | % {$_.get_innertext()}
--------------------------- ------------------------------------
GNOME Journal article about Tinymail {development, software, email, GNOME...
Comparison of various open source gr... {development, software, graphics, Li...
Agile programming considered (mostly... development
Metaprogramming with Ruby {Ruby, development, metaprogramming}
Linus talks like a pirate {LinuxArs, Linus, Linux Kernal, soft...
Mark Shuttleworth talks about Ubuntu... {LinuxArs, Ubuntu, Debian}
Optimal use of fonts on Linux {LinuxArs, X11, fonts}
IBM to provide commercial support fo... {LinuxArs, Eclipse, IBM}
McDonalds changes McFlurry cups to s... strange
GNOME orphaned thumbnail remover {LinuxArs, GNOME, Glade, Python...}
Note that the inability to consistently extract the text value of a node with select makes it necessary to use a loop in order to display the tags as a list. If CDATA blocks weren't so troublesome, the select invocation would be a lot more concise. Despite the minor deficiencies, Powershell is well suited for this sort of work.
I also experimented with listing tag information. The following demonstrates how to display a list of tags and the number of bookmarks associated with each tag:
> ([xml]$connect.downloadString("http://simpy.com/simpy/api/rest/GetTags.do")).tags.tag
name count
---- -----
LinuxArs 42
software 22
GNOME 8
strange 6
Mono 5
science 4
Qt 4
...
If I can find some time for further experimentation, I might throw together a few Powershell Cmdlets to automate link and tag manipulation.
Tags: shell, powershell, simpy, programming, dotnet
Posted on 2006-09-30
GNOME 2.16 has been released! With new features for many of my favorite programs, numerous bugfixes, and many aesthetic improvements, GNOME 2.16 is very polished and ready for widespread deployment. Although I tested it extensively in a virtualized VMware environment (the VMware image was kindly provided by Brent Smith), I think I'll probably wait until the official release of Ubuntu Edgy before using it on my main desktop computer.
I'm very pleased with the new features in the Evolution mail client, particularly the newly added support for a vertical message pane view. Evolution got a lot of really nice performance enhancements for this release, and the developers say that IMAP loading time is up to 48 percent faster! Maybe now Jorge will complain less often about Evolution's IMAP deficiencies. Gedit now has a sidebar file browser plugin as well as support for LaTeX authoring. With a built-in snippets system and support for scripting with Python, Gedit is quickly becoming a powerful tool.
For a more comprehensive overview of new features in GNOME 2.16, check out the review I wrote for Ars Technica. Published a day before the official release, my review attracted significant attention, and has been featured on most of the open source news sites that I read on a daily basis, including: Slasdhot, OS News, Footnotes, Linux Today, and Linux Weekly News. I also submitted it to Digg, where it rapidly climbed onto the top ten list for the technology section. Needless to say, my review has spawned inevitable GNOME vs. KDE arguments in discussion threads on several sites.
The 2.16 release is the first to include Mono as an official dependency. Support for C# applications and open source .NET technology became a controversial and highly divisive issue within the GNOME community in the weeks leading up to the official module decisions. Although I was initially a sceptic when Mono was still in its infancy (at the time, I felt that it would be more productive for the open source community to focus on Parrot), the steady stream of highly impressive C# applications developed by the GNOME community quickly changed my mind. In the past year, I have been a very vocal advocate of Mono because I see .NET as the future of desktop software development. I hope that the Mono developers can keep pace with Microsoft, and I hope to see compelling new .NET features like xlinq included in Mono in the future. Released yesterday, the latest version of MonoDevelop has a lot of exciting new features, including support for inline menu editing in the Stetic GUI builder, and integrated support for ASP.net development.
My Contributions
My contributions to 2.16 were minor but much-needed. I started rewriting the documentation for various applications in the gnome-games module. I have temporarily postponed additional work on gnome-games because my professional life is quite demanding at the moment, and because at least one of the games will be removed at some point in the next cycle and I don't really know which one yet.
I also helped with the release notes for 2.16. The release notes are typically done by Davyd and Murray, both of whom are relatively articulate. For whatever reason, they weren't available to it this time, and the responsibility fell upon other volunteers from the GNOME marketing list. The whole thing was done at the last minute and Claus Schwarm ended up writing most of the initial draft. I went through it and cleaned up a lot of minor problems. I primarily fixed spelling and grammatical errors, but I also rewrote several paragraphs for the sake of clarity. Others did the same, and the end result was more or less acceptable. Brent Smith converted it to DocBook practically overnight, and included screenshots augmented by my utility. The entire process felt very rushed, and I hope that it is planned better next time. If I hadn't noticed Quim's blog entry about the release notes on PGO and posted it to the documentation mailing list, fewer members of the GNOME documentation team would have been involved. Despite the minor problems, the release notes turned out fine. Claus and Brent deserve a lot of credit for their efforts. As a aside note, now that I have seen the effects in action on numerous screenshots, I think that drop shadows should only be used on screenshots with a single window, and edge fading should only be used on screenshots that include an arbitrary portion of the screen.
I want to take this opportunity to thank everybody that was involved in making GNOME 2.16 a reality. Everybody did a great job, and I'm looking forward to contributing as we march towards GNOME 2.18!
Tags: gnome, glade, gdp, mono, monodevelop, evolution, stetic, gedit, journalism
Posted on 2006-09-07
My blog recently got swamped with comment spam. I had hoped that a simple "click this to prove you are not a spammer" checkbox would provide adequate protection against automated comment bots, but that didn't accomplish very much. I managed to find a relatively good Ruby captcha library built with GD that will hopefully get the job done. Captcha systems are simplistic turing tests that automatically generate images with random garbled text. A user proves that they are not a bot by typing the text from the image into a text box. Integrating Captcha support was relatively easy:
captcha = CAPTCHA::Web.from_configuration($CAPTCHA_CONFIG)
captcha.clean; captcha.image
$cgi.div("id" => "commentForm") {
$cgi.img(captcha.image_uri + captcha.file_name) +
$cgi.hidden("digest", captcha.digest) +
$cgi.p {
"Text in image:" + $cgi.br + $cgi.text_field("key", "")
} +
$cgi.submit("Send Comment")
}
I also spent some time fixing up the recent activity links on my home page. I added support for recent Trac actions by parsing items from the Trac Timeline RSS feed. Right now, it only shows items that relate to Mage, but I may expand it in the future if I start working with other Cixar project repositories on a regular basis. I was having trouble with Trac's RSS feed at first, because it wasn't including an author tag in any of the items. I eventually discovered that the author tag only gets included if the associated user specifes an e-mail address on the Trac settings page. I still haven't found a way to use URL parameters to filter for a specific user, so I do it with an XPath expression in my Ruby script.
Tags: programming, ruby, blog, cgi
Posted on 2006-08-08
In the dynamic world of technology journalism, there are very few universal truisms. One of the most important things I have learned is that screenshots almost always improve the quality of an article. In written works about software, particularly in reviews or tutorials, screenshots fulfill a number of crucial functions and imbue content with elegance and authority that would otherwise be absent. Failure to include a screenshot in a news article about a new program, for instance, elicits criticism and rebuke from irritated readers. They don't just appreciate screenshots, they expect screenshots.
The screenshot rule isn't entirely applicable to documentation. Although screenshots are still important in documentation, it is a context in which they should be used sparingly. There are many reasons for this, but I think the most important one is that screenshots are more likely to be superfluous in task-oriented reference material that is meant to be used alongside the actual program. In many cases, screenshots in the GNOME documentation are used to help the user confirm that they are reading about the right program.
Although I routinely use screenshots to break up large blocks of text in articles in order to increase readability, screenshots in documentation actually have the opposite effect. As the GNOME documentation style guide points out, breaking up documentation text with screenshots inconveniences readers by making them do additional scrolling. There are also some pragmatic reasons. Each screenshot included in the documentation has to be replicated by translators in each of the 42 languages supported by GNOME (Update: the GDP is evaluating ways to automate this process). Screenshots also have to be updated constantly as continued development alters the appearance of applications.
One of the points mentioned in the style guide has been of particular concern to the GDP lately. Usability studies have shown that users confuse screenshots in documentation with real windows. Although this may seem far fetched, it is apparently a very real concern, and one that we are actively trying to address. Joachim suggests fading the edges of screenshots to make the distinction between windows and images more apparent to users. I think this is a great idea, but as Joachim points out in a comment on bug #348495, there is no way to ensure that the fading effect will be consistent in all screenshots. Additionally, opening up the GIMP and doing that sort of operation for every image could become very time consuming.
For the sake of simplicity and consistency, I think we need to automate the process. In order to facilitate discussion on the possibility of leveraging automated image effects, I have developed a simple proof-of-concept utility that automatically blurs the edges of PNG images and applies a drop shadow.
The New Utility
Written in Python, my new utility leverages Cairo, PIL, and GTK. I initially wanted to do all of the image processing with ImageMagick, but the ImageMagick bindings for Python don't appear to be maintained or widely distributed. When I discovered the binding deficiency, I thought about using Ruby instead of Python, but the utility really needs to be built with a language that is used and understood by other GNOME developers. I also want to avoid foisting Ruby/GNOME dependency headaches on other documentation writers. It would have been nice to be able to avoid using PIL, but Cairo doesn't seem to have a working guassian blur filter yet, and I need that in order to produce a drop shadow. There is actually a value in Cairo::Filter called FILTER_GAUSSIAN, but apparently it doesn't do anything yet.
Using PIL to render the drop shadow is problematic because there is no simple way to convert Cairo surfaces into PIL image instances with version 1.0.2 of Cairo. Right now, the only way to do it is to save the Cairo surface to disk and load it back into a PIL image instance. Since the Python Cairo bindings don't use Python's file interface conventions, I can't just save to a StringIO instance like I do when I want to get a PIL image into a GDK pixbuf. There are some nasty hacks that involve wrapping a Cairo surface around a binary array, but that particular trick causes some peculiar problems and distorts the images in some cases. Fortunately, a new surface method called to_rgba() is already in CVS, and possibly even in the next major version of Cairo. For now, I have the script automatically save the image and load it back in again, but hopefully I'll be able to eliminate that step in the future.
My working prototype isn't particularly elegant, but it is effective. I spent an extra hour adding a user interface that allows the user to customize the parameters of the operation. Users can drag sliders to change several values:
- border - specifies the size of the faded regions
- filled - specifies how far from the edges the fades should end
- offset - specifies the drop shadow offset
The UI is currently very kludgy, but it will make it easy to compare various configurations and determine exactly which values should be used by default for documentation screenshots. If there is sufficient interest, I plan to eventually produce a complete, streamlined screenshot utility specifically for use by documentation writers. It will include support for screen region selection and it will make it possible to capture multiple images and then choose which ones to save. (Update: It looks like other members of the GDP team are also interested in automation and have some very cool ideas of their own. I particularly like the idea of using DogTail for automatically grabbing the images.)
The current implementation of my utility can be run as a stand-alone script or loaded into other Python scripts as a module. To execute it in stand-alone mode, simply run it from the command line and provide the path to a png file as a parameter:
python edge_fader.py screenshot.png
The following example shows how to use the edge fader as a module:
import edge_fader
img = edge_fader.fade_edges("original.png",
border = 30, filled = 2, offset = 6, show_shadow = True)
edge_fader.save_to_disk(img, "output.png")
View the source code here, and download it here. In order to run it, you will need PIL, and Python bindings for Cairo and GTK. An example of the program's output can be found here, and this is a screenshot of the program itself. Comments and criticism are greatly appreciated.
Tags: programming, screenshots, python, gnome, cairo, gdp
Posted on 2006-08-02
I finally managed to set aside enough time to fix my blog. The frustrating limitations of PyBlosxom compelled me to search for better solutions, but I was unable to find blog software that meets all of my needs. Rather than attempting to integrate new features into an existing code base, I decided to write my own from scratch. Although I generally try to avoid reinventing the wheel, starting from scratch with Ruby ended up being less labor intensive than trying to fix PyBlosxom, and I will thankfully be able to maintain and extend my new system with much greater ease.
Although I still use nested categories under the hood, my new system emphasizes tags. Categories are poorly suited for blog entry classification, and tags provide a much higher level of flexibility. Consider my previous entry about the Vim Ruby bindings, for instance. Does that belong in the Software/Vim category or in the Programming/Ruby category? With tags, I can associate the entry with Programming, Ruby, Software, and Vim. Tags also facilitate some very intriguing and useful navigation paradigms. I'm particularly pleased with my new tag cloud feature, which displays tags in various sizes and colors based on frequency of tag use. For those of you that are not familiar with tag clouds, a brief description can be found alongside the clouds on my new Tags page.
I have also integrated Simpy functionality. The Recent Bookmarks section on the left is populated entirely from Simpy using my Ruby API. Despite my initial skepticism regarding social bookmarking, I have become an avid user, and I am now convinced that the concept is brilliant. My Tags page displays an additional cloud with tags from Simpy links. Unfortunately, the Simpy REST API does not provide temporal details for tags, so my script has to do some very inefficient processing that places excessive load on the Simpy servers. In order to limit the impact of that load, I have configured my script to cache the data and generate new tag clouds only when I add or update a blog entry. I plan to discuss this deficiency with Otis, one of the Simpy developers. Otis has been very receptive to my constructive criticism in the past, and I think that he will implement the features I need to make an efficient implementation of my Simpy tag cloud generator.
In addition to tags, I also implemented support for custom syndication exports. Now that I have a working blog, I would like to be able to add it to Planet ArsLinux and eventually Planet GNOME. The syndication export feature will enable me to control which entries get exported to various planets. I would also eventually like to create a Planet Cixar which will include the blog entries of all developers working on Cixar projects.
At present, my entire blog system (including the cloud feature) is about 300 lines of code. It is still too messy for public exhibition, but I plan to clean it up for official release at some point in the next few months. For those that are interested and impatient, private requests for access to the source code will be granted in the interim. The tag cloud code may be released sooner, because I think it would make a fun topic for an article. During the development process, I experimented with several different libraries and frameworks. I wanted to use Ruby's CGI library, but unfortunately, it doesn't support XHTML. I eventually found a nice patch that adds reliable support for XHTML 1.0, and I applied it to a copy of the Ruby 1.8 CGI library for deployment with my blog script. The CGI library has some particularly useful features for generating output and parsing parameters. I also started experimenting with WEBrick, a light-weight Ruby server that was immensely useful during the debugging process.
Tags: blog, programming, ruby, cgi, tagging, simpy
Posted on 2006-07-17
I have recently developed an interest in web services that enable users to store and share links on the Internet. Social bookmarking services are increasingly popular amongst bloggers, and could completely replace conventional bookmarks as ubiquitous broadband becomes reality. After comparing a number of available services, I finally settled on Simpy, which has several unique features and provides numerous advantages over competing services like del.icio.us. I'm particularly impressed with Simpy's support for third party development. Using the Simpy REST API, I can manipulate my bookmarks with virtually any programming language.
Although I initially had some problems with the authentication mechanism, I eventually managed to create a complete Ruby library for interacting with Simpy. Now officially released, the source code is available online through the Simpy Tools Sourceforge version control system. Documentation for the latest CVS version can be found here.
At the present time, simpyapi-ruby features support for adding, removing, modifying, and acquiring links as well as removing, renaming, and listing tags. Support for notes and watchlists will be implemented in a future version. Unsupported operations can still be performed with the generic Simpy::ClientBase.command method. Support is also provided for interfacing with Simpy RSS feeds.
The simpyapi-ruby library allows developers to leverage Ruby's syntactic sugar and interact with Simpy without having to parse, process, or manipulate XML data. Login and connection handling is all automatic, all you have to do is provide simpyapi-ruby with a Simpy username and password.
The simpyapi-ruby library is very easy to install. Simply run the setup.rb script included with the library. The library was designed with application development in mind (I plan to create a proof-of-concept GNOME utility for Simpy management), but it is also entirely suitable for simple scripting and web integration. For optimal exposure and licensing compatibility with other Simpy tools, my library is dual-licensed under the terms of the ASL and BSD licenses.
Tags: programming, ruby, tagging, simpy
Posted on 2006-06-21
I'm currently expanding the functionality of my AOL Instant Messenger interceptor for a project I'm working on with a researcher from IBM. For this particular project, the script will need to be able to intercept timestamps and screen names in addition to the data the program is already equipped to capture. Timestamps are easy, because they are handled entirely by the client application and aren't really part of the protocol. It's really as easy as adding a #{Time.now} to the relevant output string.
Screen names are a lot more challenging. The OSCAR protocol only embeds the destination screen name in outgoing messages and the sender screen name in incoming messages. As a result, it is very difficult to identify the screen name of the other party associated with each message. The packet sniffer will have to intercept the screen names during the sign-on process and associate those screen names with a local network IP address. I'm not sure yet, but the fact that the OSCAR protocol uses encryption for login strings might make this a lot more interesting.
The script has to be able to intercept data from two separate network interfaces simultaneously. In other programming languages this might be a problem, but thanks to Ruby's excellent support for simple threading, I can do it with relative ease. The following is a trivial example that demonstrates how to monitor packets from multiple interfaces at the same time using Ruby:
pc1 = Pcaplet.new("-s 1500 -i eth1")
pc2 = Pcaplet.new("-s 1500 -i eth3")
f1 = Pcap::Filter.new("tcp", pc1.capture)
f2 = Pcap::Filter.new("tcp", pc2.capture)
def display dev, pck
puts "#{dev}: #{pck.tcp_data}"
end
fork {
pc1.each_packet {|pk| display "eth1", pk}
}
pc2.each_packet {|pk| display "eth3", pk}
The power and elegance of Ruby never cease to impress me.
Tags: ruby, programming, pcap, oscar
Posted on 2006-06-14
GNOME's built-in screenshot utility really sucks. It must be invoked every time a single screenshot is taken, the interface doesn't facilitate capture of specific windows or screen regions, there is no easy way to save the same capture to more than one location, and the tiny preview makes it difficult to evaluate the captured image before I choose to save it. When I need to make screenshots for my articles, I typically use the import command provided by ImageMagick. I finally got tired of putting up with suboptimal screen capture tools and decided to write my own.
My new screenshot utility features a full-size preview, the ability to capture individual windows or specific screen regions, an integrated file selection component, support for saving the captured image directly to a remote location, and support for saving the same capture to multiple locations. The entire utility took me about an hour to build, including time spent researching the various APIs. The current version is approximately 25 lines code, not including comments or blank lines. I designed the interface with Glade and wrote the code with Ruby. The network transparency features are all implemented with GNOME's VFS library and the screen capture functionality is provided by Ruby ImageMagick bindings.
The effectiveness of my utility, the relative ease with which I developed it, and the brevity of the code are all a testement to the power of Ruby and GNOME. I'm convinced that Ruby and Glade provide the best solution available for rapid application development on Linux. The GNOME bindings for Ruby are excellent and relatively complete, but there are still problems in a few places. The GnomeVFS bindings are currently undocumented, and the API does not behave the way one would expect it to. GnomeVFS::File diverges from the traditional File class in several respects. The most frustrating difference relates to file creation. It should be possible to create a new remote file thusly:
GnomeVFS::File.new("sftp://cixar.com/home/segphault/test.txt", 2)
Unfortunately, the above yields a "File not found" error. I queried the folks in the relevant IRC channel, but didn't get much help. After a great deal of experimentation, I filed a bug report. I then proceeded to examine the relevant source code to see if I could find out why file creation was raising a "File not found" error. Careful examination of the file_initialize function revealed that file creation only transpires when the File.new method is passed three arguments. The missing argument indicates whether or not the program should refuse to overwrite existing files. The following successfully performs the creation operation:
GnomeVFS::File.new("sftp://cixar.com/home/segphault/test.txt", 2, false)
In any event, my new utility works quite well, and you can benefit from it too, because I am distributing it under the GPL. You can read the code or download the utility. Keep in mind that it has quite a few dependencies, so you might not be able to run it if you can't get the necessary libraries.
Tags: ruby, programming, imagemagick, gnome, glade
Posted on 2005-08-24
After reading an article about network monitoring with ngrep at Newsforge, I developed an interest in libpcap, a portable network packet sniffing library originally designed by LBL.
Packet sniffing is useful for a wide variety of tasks, particularly reverse engineering network protocols and spying on local network users. I found some Ruby bindings and started messing around. My first goal was to write a small network sniffer that would display http GET requests transmitted by any system on the network. I monitored packets transmitted by Firefox Get requests and managed to build a filter capable of extracting them. The filter also appears to work with Safari, and to a very limited extent, with IE, which appears to use a slightly different format for the requests.
Once I had basic GET filtering figured out, I decided to construct a system that would simplify filter extension. My PacketSniffer class allows me to add filters using pcap filtering strings and anonymous functions. The end result is a psuedo-language for network sniffing. Filter definitions look like this:
ps.filter 'tcp and dst port 80', proc {|pk|
if pk.tcp_data =~ /GET(.*)HTTP.*Host:([^\r\n]*)/xm
puts "(->) <#{pk.src}> http://#{$2.strip}#{$1.strip}"
end
}
The first argument of the filter method is the pcap filter string, 'tcp and dst port 80', which, in this case, tells libpcap to filter all tcp packets sent to port 80 of a remote server. The second argument of the filter method is an anonymous function that will be called every time libpcap receives a packet that meets the constraints of the filter string.
Next, I wanted to make an AIM message sniffer. Intercepting AIM packets is really not that difficult, it's just a matter of figuring out which port AIM uses (5190) and building pcap filter strings to grab packets sent to or recevied from that port. Parsing the packets OTOH is a pain in the ass. The OSCAR protocol is an insane mess and there are a lot of inconsistencies between various implementations. My aim_msg_parse function is really kludgy, but it is now capable of extracting the AIM SN of the user sending the message as well as the message itself.
The current version of my script works relatively well. In tests on my own network, it was capable of intercepting AIM and GET packets from OS X, Windows and Linux systems. If you want to try it out yourself, you will need my script, libpcap, and the ruby bindings. On Linux systems, you must run the script as root. I have not run it on a Windows system, but with a Windows port of libpcap, it might work.
Tags: ruby, programming, pcap, oscar
Posted on 2005-07-23
Ainalda and I did a software upgrade on the server today. This is our first upgrade since the Sarge roll-over. I am pleased to report that, to my knowledge, nothing is broken. If you notice any abnormalities, please let me know via e-mail. While I was tweaking the server, I took the opportunity to fix my Moot interface script so that my
I'm starting to get frustrated with the structural fallabilities of MoinMoin. Ainalda and I are both eager to develop a new Wiki system that will use an SVN back-end. Unfortunately, there are a number of issues that complicate CGI SVN access. We have discovered that local repository access from any user other than www-data will alter permissions and make the repository inaccessable.
We have looked at several systems that externally interface with a local SVN repository, and most of them are pretty convoluted. WebSVN is a PHP CGI system that provides a quality HTML interface for repository browsing. It uses a wrapper that sits on top of the svnlook command line utility, which is a really nasty hack. I started looking at other solutions, namely Trac, a Wiki, repository browser, and bug tracking system that integrates with SVN via Python. They use the official SVN Python bindings, which are the grotesque result of a failed Swig experiment.
Although no documentation is available for the SVN Python bindings, I did manage to find some semi-useful examples at Koders. Using those examples and some interactive Python pounding, I managed to produce a Python script that, when run as www-data, will display the requested file from a particular revision of the repository.
Tags: python, programming, subversion
Posted on 2005-07-11
Parrot 0.2.2 has been released! Parrot is a versatile virtual machine for dynamic languages. Though originally designed for Perl 6, it's power and flexibility have generated a lot of interest in the language development community. Parrot is distributed with a number of sample compiler implementations at varying stages of completion, including partial compilers for common languages like lisp, scheme, tcl and python.
The latest release features grammar and rule support in PGE, the Parrot Grammar Engine. Parrot also comes with utilties that convert PIR (Parrot Intermediate Representation) and PASM (Parrot Assembly) into PBC (Parrot bytecode). Those who are interested in learning how to implement languages for Parrot should start with the documentation. You may also be interested in my own (extremely weak) attempt at implementing PIR generators in Haskell and Ocaml.
I posted a blurb about it at LTU and got pretty decent response. A Pugs developer contacted me and invited me to contribute to his project. Pugs is a compiler that converts Perl 6 code into PIL and PIR. Apparently, the Pirate (Python-on-Parrot) people have a Google SOC participant working on a universal, generic AST layer, potentially with an S-Expression syntax. The Pirate and Pugs developers are now collaborating, with the intention of using a unified AST layer.
Tags: parrot, programming
Posted on 2005-07-09
I am currently working on designing an article management system to help me keep track of things. Unfortunately, progress has been slow. It took me a long time to pick a suitable back-end. At first, I wanted to use an XML serialization mechanism to store the data, but my experiments indicated that an XML solution would place severe limitations on scalability. After that, I began to look at databases.
SQLite was the obvious solution, but it has a lot of limitations that bother me. I also took a look at Metakit, an object oriented database library made by Equi4. Metakit is a good library, but I strongly dislike it's Python API, and it's lack of a true query system conflicts with the user interface paradigm that I want to implement. My final conclusion is that SQLite is best suited for the task.
I recently discovered that MozStorage, the Mozilla framework's next generation data storage system, will utilize SQLite. MozStorage will be used for Firefox's much needed new bookmark system. The current bookmark system has a vast number of deficiencies, and I ardently welcome improvements that will make it easier for me to manage and search my 900+ bookmarks.
Tags: python, programming
Posted on 2005-07-08
Every now and then I dig up a
bash tutorial
and learn a few new tricks. The fc built-in caught my eye this time. fc allows you to edit the last command in a traditional text editor. When you quit the editor, the contents of the buffer are evaluated by the shell.
fc is really useful for debugging long pipelines or long ruby operations performed from the command line. The default editor can be customized by changing the FCEDIT variable. I tried making vim the default, but with all my plugins, it loads too slowly. I ended up doing the following:
export FCEDIT="vim --noplugin"
I also figured out how to yank back in deleted strings. You can insert the most recently deleted string by hitting C-y. I learned a couple of neat history tricks as well. If you type !* in a command line sequence, it will replace it with all but the first word of the last command entered. !$ will be replaced with just the last word of the last command, making it the equivelent of the C-. shortcut.
While I was poking through the documentation, I took the time to learn the readline config file format. I configured M-h to perform the history-search-backward operation, which is like cycling back through the history, but only using lines that start the same way the current line does. Check out some of the other spiffy readline commands you can bind to shortcuts.
Tags: shell, bash, vim
Posted on 2005-04-04
Yesterday, I sent a message to the Vim development mailing list. I've been using GVim to manually edit the contents of a stream mid-pipeline:
cat testfile | sed '/[0-9]:\|=/d' | awk '{print $2 ", " $1}' |\
sort | cat -n | gvim -V0 - | grep -v "Vim:" | sed 's/1/*/' > test
Unfortunately, Vim emits a message when it reads from stdin, so I have to use grep -v to pull the notification out of the output stream after closing GVim. Bram responded with an explanation and some suggestions. I'll finally be able to fix the problem by making the message go to stderr. Piping through Vim will be less of a hassle now.
Tags: shell, vim
Posted on 2005-03-23
Once again, I have refined the English language. Prepare yourself for the spiffyness of my latest creation: 'soo'.
In the sentence "The mutant wombats are so evil." the word 'so' is used differently than it is in the sentence "the mutant wombats escaped, so we should probably run like crazed gophers and pray they dont eat our spleen."
In one case the word is used to exaggerate a property, and in the other case it's used to indicate a causal/conditional kind of relationship. In the case where it exaggerates a property, it should be spelled 'soo', which would be orthogonal with the distinction between 'to' and 'too'.
After I shared this discovery with Ainalda, he informed me that his ConLang group has been endevouring to come up with a word to describe the linguistically unimaginative. I suggested 'langlubber'.
Tags: language
Posted on 2005-03-03
I've been following the Vim development mailing list since early January in order to help me keep up with progress on Vim 7. On Friday, somebody mentioned the Vimshell patch in a message about integrated debugging support.
I'd never heard of Vimshell before, but I decided to investigate. One of Vim's major failings is that it can't run an ansi capable shell inside of a buffer. There are some relatively kludgy interactive shell buffer implementations that utilize scripting languages, but none of them are really very useful. I used to have one that uses Python. It doesnt do Ansi, it doesnt do vt100, it doesnt support tab completion, and it has a lot of other issues. Vimshell is different. Implemented in C, it provides complete support for almost all standard shell features. With Vimshell, it's possible to run an entire Vim session inside of a single Vim buffer. It's really a very impressive patch. Unfortunately, it too has some limitations. At the present time, it does not work with Gvim, and it doesnt allow you to use Vim normal mode key shortcuts inside of shell buffers.
My experiments with Vimshell were all quite succesful. I hade some compilation issues at first, because AAP doesn't like patches that add new files. When I dumped the AAP build and grabbed a normal tarball, I was able to patch and build without any errors. All of my tests were successful and impressive: I was able to run ssh, ftp, Vim, top, bash, and even emacs flawlessly in individual vim buffers.
Tags: vim, programming, shell
Posted on 2005-02-27
In my
article
on innovations in window management, I mention the
wmctrl
utility, which allows users to manage X11 windows from the command line. It is compatible with most netwm/gnome compliant window managers, and it supports the vast majority of the
EWMH specification.
I'm fascinated by the capabilities of this helpful little utility, but I'm frustrated with some of its limitations.
Using the source code for wmctrl as a guide, I have constructed a window management
library
for Ruby in C. This library provides users with an intuitive and concise syntax for window management and control. Here are some simple examples:
Window.each do |w|
w.close if
w.winclass == "gaim" and
w.title != "Buddy List"
end
Window.filter {|w| w != /Mozilla/}
Window.current.desktop = (Window//terminal/)[0].desktop
Window.each do |w|
puts "#{w.title} - #{w.winclass} - #{w.desktop}"
end
Eventually, I hope it will be comprehensive enough to provide any window manager with all the Sawfish features i've become accustomed to using. At that point I plan to construct a daemon that will allow me to associate keyboard shortcuts with individual Ruby functions. I'll be able to utilize my keyboard-fu with a more featureful window manager. Right now i'm thinking about switching to Flux, or something else with window tab support.
Tags: ruby, programming, libwm
Posted on 2005-02-26
There are a number of frustrating bugs in the Ruby interface for Vim that make it
difficult to utilize effectively. Deleted buffers are still accessible to the Ruby
interface, and are still counted by the Buffer count class method. Additionally,
changes to displayed but inactive buffers are not refreshed appropriately.
I have managed to fix the deleted buffer problem in the buffer access method and
in the buffer count method. I have also managed to fix the problems with screen
refresh for append and delete operations. There are probably other situations in
which the the refresh is not done correctly, but I havent found them yet.
In addition to my various bug fixes, I have written an extension to the interface
in native ruby that can be loaded at run-time in a vimrc file. Using my new Ruby
interface extensions, I've managed to port my PyRun tools to Ruby, and make some
nice improvements to them.
I plan on documenting my new Vim extensions at the moot eventually, but for now,
if you are interested you can get the code from the cixar web site. To add my
bugfixes to Ruby interface, get
my if_ruby.c file
and replace the one in your code base with it, and then recompile Vim. Then,
you can download
my native ruby interface extension
and put it somewhere in your .vim directory. Then you just have to add a line
in your vimrc file to include it.
Tags: ruby, programming, vim
Posted on 2005-01-11
