Data wrangling II

In my Data Wrangling for Aldus post last month, I talked about using OpenRefine to easily translate a big ugly spreadsheet full of bibliographic metadata into YAML metadata, effectively going from this kind of thing:

…into this sort of thing:

---
handle: 1500StCatherine
call-no: BX 4700 C4 E74 1500
author: Catherine, of Siena, Saint, 1347-1380 author.
year: 1500
title-marciana: Epistole devotissime
title-english: Letters
title-catalogue: "Epistole devotissime /"
status: Digitized
language: ita
pub-place: Italy
pub-date: [1500]
pub-city: "[Stampato in Venetia] :"
publisher: in casa de Aldo Manutio
---

The second step in this process was to take the resulting big batch of YAML metadata and split it into a set of individual files, each one a markdown/YAML file destined for this website.

Originally, I thought to use good old Unix split, a tool from the 1970s that will split a file into smaller files given some criteria. After trying that out, though, I realized that giving each of the resulting files a descriptive filename would be awkward, requiring me to wrap my split command in a shell script that was smart enough to assign filenames. And instead of going down that road, I thought I’d use python instead. After all, this kind of thing is why python exists!

We came up with a “handle” for each row in the spreadsheet – basically the pubdate combined with the author for each book: hence, “1500StCatherine” – not far from the way book historians and collectors refer to Aldine editions anyway. I added that as a column to the spreadsheet. Then, in my OpenRefine export template, I output the handle for each as if it were a markdown heading, just before the lines of YAML: “# 1500StCatherine”

Then I wrote a small python script (first one in years!) to split the file on those lines starting with # and assign a filename based on this ‘handle,’ so we end up with a file called 1500StCatherine.md that contains the YAML metadata block for the file. The python code looks like so:

import re
pattern = re.compile("^#")  # Look for lines starting with #

f = open('CatalogData-as-YAML.txt','r')
f_lines = f.readlines()

# 'book' is going to be a list containing the YAML lines for each record
book = ['dummy', '', '', '']    # A hack to catch the first item :-)

for line in f_lines:
    if pattern.match(line):
        handle = ("book[0].strip("# \n") + ".md")
        bookfile = open(handle, "w")   # Name the file
        # Start from 3rd line, write each into the file
        for l in range(2,len(book)): 
            bookfile.write(book[l])
        bookfile.close()
        book = [] # And clean up for the next iteration/book
    book.append(line) # Because outside the loop, we need the hack above

f.close()

The resulting .md file then goes straight into our Eleventy-generated website, and we get a webpage for it. Have a look at the 1500StCatherine.

Two other little enhancements: once we had a standard handle for each book, we could use it to refer to other things – such as an image of each book (just my own snapshots until such time as we get the proper book scans online): by naming the photos with the handle,^[1] we could automatically link them on to each page too.

Last piece was a bit of template code in Eleventy so that all this YAML metadata gets displayed in a reasonably pleasing manner. I chose to do this as a simple HTML dl list, CSS formatted as a table (you can check the CSS for this site if you’re curious). The “liquid” template code is simple enough, with a default fallback in case of null values. Like this:

<dl>
<dt>English title</dt> <dd>{{ title-english | default: "&nbsp;" }}</dd>
<dt>Marciana title</dt> <dd>{{ title-marciana | default: "&nbsp;" }}</dd>
<dt>SFU Catalogue title</dt> <dd>{{ title-catalogue | default: "&nbsp;" }}</dd>
</dl>

Now it only takes a few seconds to process the big ugly spreadsheet into 116 webpages… you can have a look at The Books on this site and browse through them. At this point (summer 2024) the data isn’t compete, so this is all provisional. But this little pipeline will get used again and again as the data matures.