Still spinning? My time at Canadiana.org prior to the CRKN merger

(See: Part 2, Part 3)
This series of articles is inspired by the journal article: “Spinning In”: the merger of Canadiana.org with the Canadian Research Knowledge Network

Leslie Weir giving me 5 year recognition

I started working at Canadiana in 2011, and was one of the Canadiana staff that transitioned to CRKN in 2018. Prior to Canadiana I spent most of my career involved in the Free/Libre and Open Source Software (FLOSS), which I’ve been involved in since 1992.

Canadiana had quite a bit of custom software, and that always made me uncomfortable. There wasn’t existing FLOSS software to fill the requirements when the custom ECO platform was launched in 1999 (decommissioned in 2012) or when the CAP platform was launched in 2012 (still being used). That changed over time.

AlouetteCanada, one of the organizations that merged to form Canadiana in 2008, had created an Open Source “Digital Collection Builder”. Artefactual was involved in that project, and they continued to build Open Source software for the community: the most visible and widely used being Access to Memory (AtoM) and Archivematica.

As well as custom Access software, Canadiana was managing a custom OAIS packaging system. (Disclosure: I was the primary developer/maintainer of that software and related infrastructure from 2014 until I left CRKN earlier this year. I had hoped all that custom software would have been decommissioned before I left).

While authoring custom OAIS packaging software might have felt necessary in 2012, Artefactual started working on Archivematica around the same time (See release notes). Archivematica almost immediately surpassed the functionality of the Canadiana OAIS packaging system. With a focus on digital preservation, being Open Source, and with archivists (rather than only librarians) involved, Artefactual received well deserved grants for, and community involvement with, their software.

I had a strong and constant urge to get rid of all the custom software I had authored and/or was maintaining as soon as possible. I started to advocate in 2015 within Canadiana to migrate to using Archivematica. Independently (mosty? I don't remember exactly.) Canadiana's Metadata Architect did an environmental scan in 2017 and also concluded we should migrate to Archivematica.

Canadiana had a custom REST API for accessing objects (images, etc) from storage. While the API was inspired by Amazon S3, I was aware OpenStack Swift had a module which provided an S3 compatibility layer. It seemed obvious to me that we should move away from any custom API to actually using a common API, enabling interoperability with other software without always having to customize.

Unfortunately, Canadiana generally had a “Do It Yourself” (DIY) and “Not Invented Here” (NIH) attitude, and thought of itself as a vendor. There was often push-back from colleagues against moving away from using custom software, custom data models, custom cataloging rules, etc. It took some time to convince colleagues to move from a private subversion repository to being more open on GitHub.

After years of internal advocacy, there was finally agreement on a few components:

• Move from custom Image server to a IIIF Image server. We picked Cantaloupe, and that functionality was launched in 2017.
• Move from custom object API to OpenStack Swift. In 2019 at CRKN there was a project to set up a temporary cluster using the SwiftStack management console and consulting services.
• Move from custom OAIS packaging system to Archivematica

There was still no agreement among staff on Access software, but at least if we could complete the above there would be a better understanding organization-wide of why Canadiana should move away from other custom software (and related custom data models, custom metadata application profiles, and custom cataloging rules).

I believed the best option for Access software to match the need was to use IIIF APIs for image/manifest/collection view/navigation and page search (possibly start a new FLOSS project given the scale of the image repository), and BlackLight-marc for document search. Blacklight closely matched the style of services that CAP offered, allowing the transition to not be as jarring to patrons as some of the other options.

By July 2016 the conversation about a merger with CRKN had been made public. At the time I saw this as an opportunity for positive policy change, including away from DIY/NIH towards coordinating enhancement of community software projects and coordinating cloud services among Library consortiums.

Around late 2016 or early 2017 I became aware of Scholars Portal's OLRC (Ontario Library and Research Cloud) that was launched in 2015.  This meant that the Archivematica and SWIFT object storage already existed in the larger community, and partnering with them rather than Canadiana (or later CRKN) doing any DIY/NIH was possible.