Overview
HTRC Extracted Features are one of the ways in which users of HTRC's tools can perform non-consumptive analysis of text in the HathiTrust Digital Library's corpus. As with most HTRC functions, Extracted Features are available for HTRC Worksets. Worksets, which are user-create collections of volumes from the HTDL, can be small (one volume) to large (more than thousands of volumes), and at their core, consist simply of a list of HathiTrust Volume IDs.
Researchers have several options for creating their workset, including quering the /wiki/spaces/INT/pages/43417814. Researchers who do not yet have a workset and who only want to work with the public domain texts can create a workset in the /wiki/spaces/INT/pages/43418520.
Download Format
Files
The HTRC Extracted Features files are formatted in JSON. For more information about the fields, see the documentation for each release.
Sample file:
{ "id":"loc.ark:/13960/t1fj34w02",
"metadata":{
"schemaVersion":"1.2",
"dateCreated":"2015-02-12T13:30",
"title":"Shakespeare's Romeo and Juliet,",
"pubDate":"1920",
"language":"eng",
"htBibUrl":"http://catalog.hathitrust.org/api/volumes/full/htid/loc.ark:/13960/t1fj34w02.json",
"handleUrl":"http://hdl.handle.net/2027/loc.ark:/13960/t1fj34w02",
"oclc":"",
"imprint":"Scott Foresman and company, [c1920]"
},
"features":{
"schemaVersion":"2.0",
"dateCreated":"2015-02-20T11:31",
"pageCount":230,
"pages":[
{"seq":"00000015",
“tokenCount":212,
"lineCount":38,
"emptyLineCount":10,
"sentenceCount":7,
"languages":[{"en":"1.00"}],
"header":{
"tokenCount":7,
"lineCount":3,
"emptyLineCount":1,
"sentenceCount":1,
"tokenPosCount":{
"I.":{"NN":1},
"THE":{"DT":1},
"INTRODUCTION":{"NN":1},
"DRAMA":{"NNPS":1},
"SHAKESPEARE":{"NNP":1},
"ENGLISH":{"NNP":1},
"AND":{"CC":1}}},
"body":{
"tokenCount":205,
"lineCount":35,
"emptyLineCount":9,
"sentenceCount":6,
"tokenPosCount":{
"striking":{"JJ":1},
"his":{"PRP$":1},
"plays":{"NNS":1},
"London":{"NNP":1},
"four":{"CD":1},
".":{".":7},
"dramatic":{"JJ":2},
"1576":{"CD":1},
"stands":{"VBZ":1},
...
"growth":{"NN":1}
}
},
"footer":{
"tokenCount":0,
"lineCount":0,
"emptyLineCount":0,
"sentenceCount":0,
"tokenPosCount":{}}}]}}
Filepaths
HTRC Extracted Features files gathered through rsync will download in pairtree format. Pairtree format is a hierarchical filesystem developed by the University of California Curation Center that maps identifier strings (in this case the HathiTrust Volume ID) to directory paths two characters at a time. The filepaths keep the institutional short code (e.g. mpd, uc2) at the front of each HTID intact.
Preparing to Rsync
Converting HathiTrust Volume ID to Rsync URL (using HTRC Feature Reader)
If you already have a list of HT volume IDs, you can use a Python library developed by the HTRC called the HTRC Feature Reader library, to prepare to rsync your volumes of interest. (Note: The HTRC Feature Reader library can do many other things as well!) In the Feature Reader library, there is a convenience function in htrc_features.utils.id_to_rsync(htid) which aids in transforming a list of HT Volume IDs into URLs for rsync. Here is an example showing the conversion of one HT volume ID into an rsync url:
>> from htrc_features import utils
>> utils.id_to_rsync('miun.adx6300.0001.001')
'miun/pairtree_root/ad/x6/30/0,/00/01/,0/01/adx6300,0001,001/miun.adx6300,0001,001.json.bz2'
Converting HathiTrust Volume ID to RSync URL (using Python)
Researchers who have their list of HT volume IDs but prefer not to use the HTRC Feature Reader, can convert HT volume IDs into rsync URLs using a Python script. This example is a simplified part of a longer notebook, which further describes how to collect and download large lists of volumes: ID to EF Rsync Link.ipynb.
If you don't have it, you may have to install the pairtree library with: pip install pairtree (Python 2.x only).
import os
from pairtree import id2path, id_encode
def id_to_rsync(htid):
'''
Take an HTRC id and convert it to an Rsync location for syncing Extracted Features
'''
libid, volid = htid.split('.', 1)
volid_clean = id_encode(volid)
filename = '.'.join([libid, volid_clean, kind, 'json.bz2'])
path = '/'.join([kind, libid, 'pairtree_root', id2path(volid).replace('\\', '/'), volid_clean, filename])
return path
Example:
id_to_rsync('miun.adx6300.0001.001')
'miun/pairtree_root/ad/x6/30/0,/00/01/,0/01/adx6300,0001,001/miun.adx6300,0001,001.json.bz2'
The Extracted Features for this volume can be downloaded using RSync:
rsync -azv data.analytics.hathitrust.org::features/{{URL}} .
Using the HTRC Portal Algorithm
The HTRC Portal can aid some researchers who do not feel comfortable creating rsync URLs themselves can use the HTRC EF_Rsync_Script_Generator algorithm in the Portal to create a shell script they will download and run locally to rsync the files.
Go to the Algorithms page of the HTRC Portal
While logged into the HTRC Portal, click on the 'Algorithms' link near the top of the screen. From the list of algorithms that shows up, click on EF_Rsync_Script_Generator. This 'algorithm' generates a script for downloading the feature data files that correspond to your workset.
Execute the EF_Rsync_Script_Generator algorithm
Specify a job name of your choosing. You also have to select a workset that the EF_Rsync_Script_Generator algorithm will run against: Check the button saying “Select a workset from my worksets” and select your desired workset. Your screen should now look like the figure below. At this point, click the ‘Submit’ button.
Wait until the algorithm has finished
You can now see the status of the job. Refresh the screen to see its progress.
Open the completed job to download
Eventually, the job will have “completed”, and it will move to the "Completed Jobs" section of the page. Click on the link representing the job name to initiate downloading the EF_Rsync_Script.
At this point, the script will be downloaded to your computer’s hard disk wherever browser-initiated downloads are stored, likely in your Downloads folder.
Run the script returned by the EF_Rsync_Script_Generator algorithm
Windows users: Please follow the directions here before continuing to ensure your machine is equipped for rsync.
From the command line, navigate to the directory where the script file is located. This directory will typically be called Downloads, though the location may be different depending on your machine and if you have moved the file. Here is an example:
cd ~/Downloads
Then run the file you downloaded. It is a shell script. When you run it, a basic features data file and an advanced features data file for each volume in your workset will be transferred to your hard disk via the rsync utility.
sh EF_Rsync.sh
If your workset contained N volumes with HathiTrust volume IDs V1, V2, V3,... VN respectively, then executing the shell script as shown above will cause the following feature data files for the corresponding volumes to be transferred to your computer’s hard disk via rsync: V1.json.bz2, V2.json.bz2, V3.json.bz2, ..., VN.json.bz2. See Filepaths above to learn more about the pairtree structure the Extracted Features files follow.
(Optional) Uncompress the downloaded files
Because the feature data files are compressed, you may need to uncompress them into JSON-formatted text files, depending on your need. The compression used is bzip2. If you are using the files with the HTRC Feature Reader, the library will deal with the compression automatically.