Evergreen

Back in early May, I was in San Francisco for Google I/O. I had booked an extra day with the hopes of either doing some site-seeing or meeting up with the OpenLibrary team. After firing off an email to find out if anyone there was interested on working on some tighter integration between OpenLibrary and Evergreen, the answer from George Oates was an enthusiastic "Yes!". So, we spent a beautiful sunny day inside the Internet Archive headquarters discussing possible directions for this integration. Alcatraz, you can wait for my next trip...

As it turned out, the timing was great. I had spent a day hacking on the OpenLibrary "added content" module for Evergreen during the Evergreen hackfest (which I spent in an airport due to an eight-hour fog delay... different story), so I was quite familiar with the existing OpenLibrary Book API and their patterns of use were fresh in my brain. The biggest problem with the existing Book API, from my perspective, was that I had to make two calls for each work that I was interested in retrieving information about; one call returned the data (stable elements) and one call returned the details (unstable, but quite interesting elements like the table of contents, excerpts, etc).

The OpenLibrary team had this in their sights as well - but they wanted to tackle a bigger target. Rather than making one or more calls per work, they wanted to expose an API that would let users request info for multiple works in one shot: the Shotgun API (known amongst more polite company as the Read API). Loosely modelled on the Hathitrust API, it would also focus on exposing URLs for reading or borrowing (using the relatively recent OpenLibrary borrowing program) exact matches or similar editions. It sounded great, and we spent the afternoon fleshing out how we wanted it to look and work. My role was largely that of the third-party developer - the API customer - and we had great discussions.

Working code wins

Of course, discussions are one thing, and working code is another. OpenLibrary developer Mike McCabe was riding shotgun on the development of the Read API, and once he had enough working code in place, he contacted me to ask me to start developing against it. It was the usual development process: I started with a hard-coded sample JSON output, then as Mike pushed more functionality into a server environment I was able to test and expand my client-side code.

So where are we now? I can vouch that working with the all-in-one Read API, as a developer, is sweet. All of the data elements are readily visible in sweet, sweet JSON, in a single call, and it is utterly simple to pull the bits that you want to expose. I had been trying to pull together ebook links and the like from the Books API, and the use of the items list makes that absolutely painless for developers. Kudos!

Evergreen has a largely rewritten OpenLibrary added content module built against the Read API sitting in the Evergreen working repository user/dbs/openlibrary-read-api branch. As the Borrow and Read functions depend on IP address range matching, I have added the ability to proxy the Read API requests via the Evergreen server - so that if an Evergreen institution has special access rights to the OpenLibrary collection, their patrons will see the appropriate levels of access in the catalogue. Oh yes, the catalogue; as we were already using OpenLibrary by default for cover art, tables of content, and excerpts in Evergreen since the 2.0 release, the major difference that will be visible to Evergreen users will be in search results:

As you can see, if you have left the OpenLibraryLinks variable turned on in the result_common.js file, Evergreen will search for a matching record in OpenLibrary and tell you if an online version is available. It tells you whether the online version is an exact match, or similar, and will also expose items that you can borrow from OpenLibrary. Given the preponderance of print materials that still remains in our collections, and our users' general preference for anything electronic, I think this will be an extremely popular feature.

Moving forward

There are a number of areas that could use more polish and tender loving care. First and foremost, OpenLibrary supports matching based on ISBNs, LCCNs, OCLC numbers, and OpenLibrary IDs; right now, the Evergreen support is based strictly on ISBNs, which of course don't exist for many of the older materials in our collections. So a fruitful direction would be to take the regular dump of data that OpenLibrary thoughtfully provides (yay for open data) and use that to augment our records to include OpenLibrary ID numbers to use as a match point.

There is the small matter of merging these changes back into Evergreen proper. I developed against the Evergreen 2.0 branch because I wanted to be able to put this code into production as soon as possible, so there will be a tiny bit of merging pain to get this into master and backported properly. However, the changes are quite localized and should be agreeable, so hopefully this will not sit in a branch for too long.

At this early stage in the Read API's release, I have also found that it can be a bit slow to respond to requests containing a number of identifiers (or perhaps a large number of records and items). It is to be expected that functionality comes first and optimization comes later, so I have great hopes for improved performance once the Read API settles down.

Of course, once you have the Read API, you need an Write API - and I hope to be able to help pilot that as well, because the potential communal benefit of a Write API for library systems that have integrated with OpenLibrary is huge. Imagine a system where, when you ask for added content based on a given identifier, if the system says "Huh, I don't know anything about that identifier" it follows up with "Hey, can you POST what you know about it to this URL?". OpenLibrary could then run its algorithms and either add an edition to an existing work or generate a new work. We should also be able to expose OpenLibrary's metadata editing tools for our users, so they can flag bad cover art, or add a table of contents to works that they are passionate about, or post a favourite excerpt... Enabling a bi-directional give and take between systems has the potential to quickly make OpenLibrary a huge knowledgebase of open data. It would be a great boon for libraries, and I hope we can make it happen.

Update 2011-06-02 21:54 EDT: The omission of Mike McCabe's name has been corrected. Also, I forgot to thank my employer, Laurentian University, and the University of Windsor for allowing me to invest some of my time on strengthening Evergreen's ties to OpenLibrary. I believe this is the beginning of a solid, mutually beneficial partnership!

If you have been through an Evergreen upgrade, you know that the days after the upgrade can be painful. Users complain that the catalogue doesn't work right, there are mysterious glitches that happen on some machines and not others (even though the browser and operating systems are identical on each machine!), rebooting doesn't help... and then eventually the problem goes away.

The problem isn't all that mysterious, really, it's the result of the browser caching content. Normally, browser caching is a very positive experience: when a browser requests a file from a Web server, the Web server tells it to how long the browser should hold onto the file via a Cache-control directive. This means that if a page on your Web site is dozens of hundreds of images and CSS and JavaScript files, your browser doesn't have to download every one of those files on every page you visit; as long as the file hasn't expired, the browser can just serve it up from the local cache and only the fresh content needs to be fetched from the server. It's how the Web works, and it's really important for performance reasons.

However, if your Web server has told your browser to cache files for a month, and then during that month you upgrade your Web site so that there is new JavaScript and CSS files that your fresh content depends on, then you can run into trouble until those cached files expire. And that is exactly the case that we run into with Evergreen upgrades - only the problem is amplified by how heavily the Evergreen catalogue (which is just a Web site) relies on JavaScript for basic operations.

On the user side, you can handle the problem a few ways:

• Doing a hard refresh to force the browser to fetch fresh versions of all the files in its cache. You can force a hard refresh on most browsers by holding down the Shift key and clicking the Refresh or Reload button.
• Emptying the browser cache.

Neither of these user-side approaches is particularly convenient. Doing a hard refresh may work for one page, but as the user navigates to a different page that uses different CSS and JavaScript, they will have to do another hard refresh... and so on, which in the case of Evergreen means users will have to refresh around a half-dozen different pages (home page, search results, record details, account, advanced search). Hard refreshes are also not reliable, as resources fetched by XHR are not actually refreshed (this is a long-standing bug with Chrome and Firefox). If you don't know what XHR is, just know that Evergreen uses a lot of them. And emptying the browser cache is both painful (every browser has a different way of emptying browser cache) and overkill (you just want to discard the cache for one site, but most browsers will discard the cache for every site they have visited).

The "right" solution is to have the server tell the browser to fetch a new version of the resource. You could change the caching settings to be very short-lived - for example, change the cache time from one month down to one day for JavaScript and CSS - but unless you upgrade your site very frequently, that would mean that 99% of the time your users' browsers will be making unnecessary requests, and their experience of your catalogue will be that it is slower to load than other sites on the Web. Not so good.

The other approach is to change the pathname for the cached resources at upgrade time so that the browser doesn't find a match in its local cache and has to fetch the new version. There's some good news: some work has been going on in the Evergreen 2.1 release to tackle this problem, but it is not yet complete. And most sites are only looking at moving to 2.0 right now. As it happens, we made the jump from 1.6.1.8 to 2.0.6+ yesterday and boy howdy the browser cache was a problem after the upgrade, as one would expect. I took a quick stab at identifying the most likely paths that needed to be refreshed and threw together some shell commands to "munge" our catalogue skins so that browsers would be forced to pick up the new versions of the content.

Post-upgrade panic, I refactored those commands into a Perl script named cache-munger.pl (well, more precisely, a Perl script that generates shell commands). The Perl script has two hardcoded variables: a datestamp (which is really any uniquely identifying string that can appear in a directory name and URL) and a list of catalogue skins to munge. When you run the script, it generates a set of shell commands that you should be able to run on your Evergreen 2.0 instance to force browsers to cache the new version of your catalogue's JavaScript and CSS files.

Some limitations: I haven't written a script to convert your skins back to pristine mode (that's mostly a matter of updating the ack-grep commands and reversing the sed commands). And I haven't written a script to update a munged set of skins. And, I'm not 100% sure that I've hit every set of JavaScript and CSS that needs to be refreshed after an upgrade from 1.6 to 2.0. But it's a reasonable start, in my opinion, and hopefully it helps inform the Evergreen 2.1 effort so that we can have a standard, supported, painless means of telling browsers to fetch new resources as an automatic part of any upgrade in the future.

I'm at the Evergreen 2011 conference in balmy Decatur, Georgia... which wasn't a sure thing yesterday, given that the day started with an eight hour delay at the Sudbury airport due to fog - not to mention having to fly through the storm that spawn a tornado in Alabama. After all that, though, it's great to be back in the same physical space as the vibrant Evergreen community!

Yesterday afternoon I gave a presentation on Authorities in Evergreen 2.0, covering (as the title suggests) Evergreen's support for authority records in the 2.0 release (as well as a peek at the future of Evergreen 2.2).

The session appeared to be well-received - yay! - and I tried recording it on my colleague Rick Scott's Sansa Clip+. Hopefully that worked out and I'll be able to update this post with the audio, so you can have the full-on audio and slide experience.

The presentation is available under the Creative Commons Attribution Share Alike license, in the hopes that others will be able to use it for training purposes, to extend and improve it, and generally help out with the adoption of Evergreen.

tldr version: the Evergreen project now has a continuous integration server and build farm and needs testcases to make the best use of that infrastructure to help us provide higher-quality releases in the future.

Evergreen buildbot - past

Back in November 2009, Evergreen developer Shawn Boyette launched the Evergreen buildbot - a continuous integration server that ran basic tests with every commit to the OpenSRF and Evergreen repositories and created nightly tarballs of the code. It was a promising start towards a system that would provide us with instant feedback about the state of our code - at least as much as we had tests for it. Unfortunately, the server ran for only a few months before disappearing when Shawn parted ways with Equinox in early 2010.

I always thought it was a shame we had lost this piece of the development infrastructure, but Equinox had offered accounts on a server for anyone in the Evergreen community interested in taking on the task of setting up a new continuous integration test server - and through the rest of 2010, nobody stepped up to take on that responsibility. Most of us were busy developing and testing Evergreen 2.0, I suspect. So, in January of 2011, when I had a bit of breathing room, I scoped out the current state of continuous integration frameworks and discovered that the buildbot project (no relation to Shawn's code, other than a serendipitous name) was written in Python and therefore was much more approachable to me than the other leading alternative, Hudson... so I wrote up my findings and a quick proposal.

Evergreen buildbot - present

A few days later I had the buildbot running on the server provided by Equinox, providing reports on the status of the OpenSRF builds on Ubuntu Lucid. After putting out a call to the community for build servers to provide coverage for Evergreen on different operating systems, I had enough responses to focus my mind on improving the Evergreen build. Evergreen now has the same standard layout for Perl modules that we adopted a year ago for OpenSRF, along with some basic sanity tests in Perl (such as are there any syntax errors in this module?).

So, thanks to Equinox for providing the testing server that serves as the mothership for controlling all of the build tests. And many thanks to the University of Prince Edward Island Robertson Library and the Georgia Public Library Service for providing build servers for the build farm. We now have Evergreen test coverage on the Ubuntu Lucid and Debian Squeeze Linux distributions (huzzah) and OpenSRF test coverage on Ubuntu Lucid. If you have an interest in getting test coverage for a different distribution and have a server to spare, please feel free to contact me and we can get your server added to the build farm.

Checking build status

You can check the current state of the code for various OpenSRF and Evergreen branches at any time by visiting the Evergreen buildbot page and choosing one of the menu options. Recent builds provides a simple list of the success or failure of the 20 most recent builds. Waterfall, on the other hand, provides the detailed status of every tested combination of Linux distribution and code branch.

Evergreen buildbot - future

We still have work to do to deliver on the promise of the buildbot. Most important, I think, is that a continuous integration server can only run the tests that it has been given - and we have not given it many tests. It kills me that people discovered some fairly fundamental problems with the Evergreen 2.0 release (some recent examples include most identifier searches not working and limitations with Unicode in patron names). Now that we have a continuous integration server, we need a testing framework so that it becomes easy to add tests along the lines of "Import a set of sample bibliographic records, then run the following sets of searches (ISSN and ISBN with and without hyphens; EAN; UPC...) and ensure that the returned results match these expected results". It should be a human's job to set up that automated test once so that we're forever confident in the future that we're not screwing up those basic features, no matter what we change in our database schema or underlying code.

Now, there are very few people that can currently create that sort of a test. There might be none at the moment, in fact, because we need that previously mentioned testing framework to be sorted out and integrated into the buildbot, However, in the short term we can create these testing scenarios so that humans can reproduce them during testing blitzes, until such time as we have the testing framework sorted out and can begin automating these tests. Otherwise, I fear that we'll go into the Evergreen 2.1 alpha/beta/release candidate cycle and get reports from testing that indicate that all is well - but only because some of the more complex tasks haven't actually been attempted - and we'll find ourselves scrambling once again after the release to fix problems that become evident when sites actually start moving to the release.

Beyond tests, we need to teach it to create cleanly packaged tarballs on a regular basis - although that should arguably be nothing more than, or not much more than, the equivalent of running make package rather than pushing all kinds of specialized packaging logic into the buildbot itself. Autotools wizards, your assistance would be greatly appreciated.

Spreading Evergreen buildbot knowledge

To ensure that our project can survive the loss of the current master build server (or me, for that matter!), I've been committing a password-sanitized copy of the buildbot configuration to the examples directory of the OpenSRF repository. In addition to reducing the dependency on one person and one server, it also gives anyone else interested in contributing to the Evergreen buildbot the ability to easily define a build master and build slaves in a local environment.