Canto | News

Hello again. It's been awhile, but this time hasn't been wasted. You can see in the Git log that progress is still being made. Of course, the age old open source trouble of having real work get in the way has reared its ugly head, but lately the changes have been mostly under the covers.

Among them are a massive drawing improvement that means that the most complex format strings with thousands of stories will still be rendered snappily, and as a few days ago the interface becomes informative and usable much faster. The plugin infrastructure has grown to be more robust, and I'm using it every day to include more interesting information in my Reddit feeds. JSON is being used for the entirety of the disk presence (config) and the protocol, so the daemon and client should be more interoperable with programs written across the spectrum of languages.

The visible improvements mostly include the ability to manipulate your feeds from inside the interface. Things like changing tag order, adding tags, using tags like folders. Also, adding, listing, and deleting feeds. Keybinds are now handled (queried/set) easily through the interface. The ability to change themes and colors on the fly is there too, if a bit rough at the moment. Theres also the ability to address ranges of items abstractly to say, "set all items in this tag, above my cursor read". OPML support and passworded feeds have been implemented for quite awhile.

I'm posting this because I'm currently slogging through the documentation on a Wordpress installation elsewhere on codezen. I'm taking the approach of adding polish as I document so it's less "document what I've done" and more "document how I'd like it to work and then make it work that way." This was the cause of the bind interface showing up, and now it's the cause of the color code being simplified.

Having improved the codebase and ironed out many of the bugs, we're still left with the following TODO:

• Documentation
• Polish
• Reader content support

As you can see these were resident on the previous list as welll, but I think that's understandable considering documentation and polish are ongoing as the interface continues.

Reader content support (i.e. more intelligent downloading) should be trivial, but I'm not sure whether it should be done as built in functionality or with a plugin.

Anyway, that's what's up on the Canto front. I'm off at the end of the week for the holidays, so I'd like to make better progress than I've been able to in quite awhile.

-Jack

Hello Canto-ers (Cantoites? Cantons?). Just a brief check-in about the status of 0.8.0. Clearly the end of 2010 didn't see a release so I must be operating on the same sort of schedule that game studios run on =). Honestly though, the chip I'm working on at my day job is getting closer and closer to golden, and the my other projects are starting to heat up with real hardware so I've been slammed.

Anyway things have come along nicely since the update last July (geez, it's really been that long). It's still not ready, but it's a lot closer to ready than ever. The server is maybe 90% done and the client maybe 75% done. Of course there are still a lot of things outside of code to be done. I'm personally dreading the documentation, but hey what's the point of writing software if nobody knows how to use it?

Things that are done:

• The whole system is configured with a canto-remote binary, no need to hand-edit the config and restart canto-daemon.

• The client responds to config changes dynamically to changes made with the remote. Includes things like interface tweaks, adding/removing feeds, themes etc.

• The client and daemon can communicate over a real network socket. This partially obviates the need to sync with a third party like Google Reader, although I'd still like to see such functionality.

• Filters and sorts are implemented server-side. However, unlike the previous post they are not done with persistent binaries. Even loading them into memory just once and feeding info through pipes it was dog slow compared to native Python objects. As a benefit though it allows for more expressive config options using Python's eval() in a restricted environment.

• Python plugin system that already works for powerful keybinds and the aforementioned filters and sorts.

• Practically everything else that's found in 0.7.x except the stuff in the next list.

A brief TODO:

• OPML support in canto-remote

• Password protected feed support.

• Binary hooks. I believe these are okay to do in binaries still as the hook events are much rarer and most of the canto internal state you'd want to mess with could be done via canto-remote from wherever.

• Automatic / script based tagging of items.

• Some form of protection for network socket ability so it's suitable to run over an open network (the internet) instead of just a LAN behind a firewall.

• Extend Python plugin system to include ability to fix/tweak important classes.

• Smarter reader download support.

• Polish, polish, polish, polish, polish.

• Documentation, man pages, etc.

I think that's a comprehensive list at this point. At least, fulfilling these items would be enough for an 0.8.0 release.

I want to refresh the site again with something that's a little more robust and better integrated with git (and that I don't have to maintain myself). Like Redmine or just Github or something like that. Haven't thought about it too much at this stage.

• Jack

You can grab the 0.7.10 release from the download page.

• Add canto-inspect.1 manpage
• Fix other manpages
• Fix some 2.6 incompatible abuse
• Add workaround for bad feed data caused by switching between the system feedparser and the builtin.

This is basically a maintenance release, mostly thanks to Canto having a new Debian maintainer (thanks Vincent!). Essentially some immaterial cleanups that won't make a difference to most. Some better Python practices (particularly string exceptions that apparently have always been buggy and now are not supported int 2.6+).

I would like this to be the last 0.7.x release, and thus consider it to be in maintenance mode. xulos and jaacoppi from IRC still have some nasty bugs, but try as I might I can't get them to reproduce. Resolving these bugs would be the only thing that would prompt an 0.7.11 release.

0.8.0 Plans

So, as I've mentioned before, Canto was basically the first piece of Python that I wrote. I've learned everything I know about Python from building it and every version from 0.4.1 to this latest 0.7.10 has been organically grown from the same codebase.

As a result 0.7.10 is about 30% a product of the beginner Pythonista I was, 50% the result of the intermediate Pythonista I came to be not recognizing the poor design, and 20% of the expert Pythonista I am now thrashing to salvage anything.

Now I reached the point where I stopped thrashing, evaluated the codebase on its merits versus the amount of effort to start again and found that, easily, it made more sense to start from scratch.

0.7.x evolved into a multi-process monster with canto-fetch flying around in a cronjob. Its insides were twisted around to communicate with a gimpy form of Python only picklable socket protocol. The dividing line between work that needed to be done in the interface and needed to be done in the backend was blurry and information that needed to cross the boundary ill-defined.

In short, it became a huge mess, but that's what happens when you start with the idea of threading and (through necessity, thanks to the Python GIL) end-up with processes. Among other bad choices.

0.8.0 rectifies the situation.

Instead of a hideously twisted gimp protocol of 0.7.x, a real, genuine socket protocol is being used to communicate with a (gasp!) daemon. Because the daemon exists, canto-fetch is history. As are slow exits, bouncing CPU usage, confusing macro systems, eye glazing Python based configs, pointless lock hammering on the filesystem, delayed updates, and lost information.

Most of the heavy lifting is done in the daemon, which is properly multi-threaded such that multiple clients could be running at once and cooperate. Not only that, but because it uses a simple text based protocol (instead of Python-only pickles), it would be trivial to write a GTK or a Qt based frontend to canto and still take advantage of its filters, sorts and other configuration all in any language that's smart enough to talk to a socket (read: all of them).

0.8.0 will be a great simplification.

Current Status

As of this writing, I have about half of the server functionality complete and about a third (although it's the hard third) of the client done. Neither are up yet for public consumption, but as soon as I start giving IRC users alphas, there will be another post and the project will appear in the usual gitweb.

I'm not sure how long it will take for that to happen, but I'm hoping to put out the initial 0.8.0 release before the end of the year (loose enough timeline for you?).

A short list of notable changes:

• Server / client (as mentioned above).

• Blocking event-based architecture = 0% CPU usage when not in use.

• INI style config, that can be machine manipulated. (Although Python extension will be worked in.)

• Built-in interface command line.

• Supports traditional text-cursor in addition to new commands making it trivial to do massive batch operations and use the interface without the text-cursor.

• Filters / sorts in arbitrary, persistently loaded binaries.

• All around more efficient methods for practically everything.

The basic gist you should take away from this post is that it's time for Canto to level up.

• Jack

You can grab the 0.7.9 release from the download page.

• Fix fresh install sans system feedparser
• Make canto-inspect use builtin feedparser

Some kinks had to be worked out with the import of feedparser and parts of the code were still falling back on the system. I am disappointed I haven't heard back from one of the original reporters so 0.7.10 might be coming sooner rather than later, but the part of the problem I was able to reproduce locally (and cropped up others' machines) was fixed in this commit.

Have fun! Submit bugs!

You can grab the 0.7.8 release from the download page.

• Fix feed exception encoding problem.
• Fix occasional zombies / extra pids floating around.
• Import feedparser into source tree.
• Add -s/--sysfp flag to canto-fetch to fall back on system feedparser
• Render improvements by honoring declared content types

Feedparser

Hot on the heels of the 0.7.7 release comes 0.7.8. It's been awhile since there's been a release that's only lasted a week, but in this case it had to happen. Why? Feedparser.

I love feedparser. It really makes my life much easier when it comes to getting feeds. I haven't touched XML with code since I started using it and it's been great. What I don't like about feedparser however is that the venerable Mark Pilgrim that authored it and maintains it has no time to fix or integrate fixes for about a million little bugs.

Right now, the Debian version of feedparser (the one I use everyday) has 14 patches against it. Understandable since the last release of feedparser was in April 2007. Things are bound to break. Arch has got 5. Gentoo, and I'm sure a lot of other distros have 0. So there are a number of versions floating around each with different behavior. I've known about this since Canto started using feedparser.

Over the last year or so a number of people have submitted bugs about this and that, and some of them were serious in that they rendered a feed entirely useless. Like misreporting titles. Or even worse, causing exceptions. Each time I just told them, "I'm sorry, it's feedparser, talk to your distro's maintainer about picking up this patch."

For the most part things were okay like this only because (at least, initially) I thought mpilgrim was going to step up and feedparser 4.2 would be coming out and maybe I wouldn't have to resort to importing the source. Then, in the last week, I got two bug reports from two separate people on two different distros that both came down to feedparser. The clock on 4.1 just passed three years and I've had enough. So I grabbed Debian's feedparser and tossed it into the source.

Now everyone will be on a level playing field and if (or when) someone else reports a bug that traces into feedparser. I can get in there and fix it myself, without having to wait on mpilgrim or the distros. Mission accomplished.

I don't foresee problems migrating beyond, perhaps, some items being marked as unread again because their IDs have changed.

Rendering

The other noticeable improvement on this release is the rendering of content that is explicitly reported as text/plain. Previous to this release, Canto ignored such things and attempted to render basically everything as HTML. It's a strategy that had served Canto well trying to cope with feeds published by moronic broken publishers and most of the time text/plain content wasn't changed in the process. However, this is sort of punishing good feed publishers for the others' stupidity by messing with the spacing of their content.

In particular, things like plaintext logs and the like would show up as a jumbled mess even though the feed declared "hey, don't format this."

So, to stop punishing the good guys, Canto honors the type. This may make bad feeds look like crap, but in that case the user can contact the publisher to complain.

0.7.7 Exception

Twice in the short lifespan of 0.7.7 it was reported that Canto excepted on line 289 of process.py. I didn't fix it directly, but I believe that it was fixed by restoring an accidentally dropped line from 0.7.6. If this isn't the case and you see such an exception, please report it.

Thanks

Lastly, thanks to jaacoppi (the pidwatcher), evg_krsk, and Caleb for reporting bugs for this release.

Have fun! Submit bugs!

You can grab the 0.7.7 release from the download page.

• Fix harmless widecurse.c warning
• Fix going to locale incompatible URLs
• Fix Python 2.6.5 weirdness
• Fix enclosure parsing/display exception
• Fix set_{filter,tag_filter,sort}(None)
• Improved config validation for gui options
• Work harder to maintain selections through updates
• Make cursor behavior more flexible
• Change default cursor behavior
• Documentation tweaks (thanks acoolon)

Another round of bugfixes. This one tackles some annoyances like occasionally getting an item with a link with characters that are both not URL-encoded and not compatible with your system locale. No symptoms other than the link just wouldn't open, which is crappy.

Some exception stuff has been cleaned up, and set_filter and friends now work properly when given None as the argument.

The most noticeable change, if you weren't suffering from the previous bugs, is that the cursor behavior is now configurable. For once I actually completed the official documentation before the release, so you can read about it here.

If you want to return to the <0.7.7 default scroll behavior, you can add this line to your config:

cursor_edge = 0

To cause the interface to scroll at the very edge of the screen, instead of the new default behavior with a scroll margin of 5 lines.

Have fun! Submit bugs!