| Table of Contents | ||
|---|---|---|
|
...
Sample Files
A sample of 100 extracted feature files is available for download through your browser: sample-EF202003.zip
Also, thematic collections are available to download: DocSouth (82 volumes), EEBO (234 volumes), ECCO (412 volumes).
Filepaths
The dataset is stored in a directory specification created by HTRC called stubbytree. (Note: This is a change from previous versions of the Extracted Features dataset that used the pairtree directory specification.) Stubbytree places files in a directory structure based on the file name, with the highest level directory being the HathiTrust source code (i.e. volume ID prefix), and then using every third character of the cleaned volume ID, starting with the first, to create a sub-directory. For example the Extracted Features file for the volume with HathiTrust ID nyp.33433070251792 would be located at:
...
| Code Block | ||
|---|---|---|
| ||
>>$ htid2rsync hvd.32044140344292 --oldstyle >> hvd/pairtree_root/32/04/41/40/34/42/92/32044140344292/hvd.32044140344292.json.bz2 |
Workset Builder 2.0
Extracted Features 1.5 files can also be downloaded for search results in HTRC's beta Workset Builder 2.0. After completing your search, you can download either the Extracted Features files for your results set or for single files from your results.
First, enter search terms for a desired set of volumes. Once results are returned, filter the results to remove any volumes for which you do not want Extracted Features files or to find the volume(s) most relevant to your work.
...
| title | Download for all results |
|---|
You can download the files for your entire results set. Once your results includes all of the volumes you'd like Extracted Features files for, click "JSON Extracted Features (ZIP)" under the "Export Search Results" heading at the top of the left sidebar. Clicking this link will start a download of the Extracted Features files for all volumes in your results. Since this process is fetching and compressing files, and result sets can be large, it may take a few moments to start and finish your download.
...
| title | Download single file |
|---|
If you'd like to download Extracted Features files for individual volumes, you can click the link "Download Extracted Features (complete volume)" under each volume in the search results. This will download uncompressed Extracted Features for the given volume, in JSON.
...
Use HTRC EF Download Helper Algorithm
To download the Extracted Features data for a specific workset in HTRC Analytics, there is an algorithm that generates the Rsync download script, the Extracted Features Download Helper. The tool can also be useful if you don’t want to go through the process of determining files paths. Select version 1.5 when you run the algorithm to get the Extracted Features 1.5 version of the files.
The algorithm creates a shell script that you can download and run from your local command line. The file lists the rsync commands for every volume in an HTRC workset. Once you have run the algorithm and downloaded the resulting file, here is the command you would run.
...
| title | Running the algorithm |
|---|
Go to HTRC Analytics
Navigate to https://analytics.hathitrust.org and log in.
Go to the Worksets page of HTRC Analytics
Click on the 'Worksets' link near the top of the screen. From the list of worksets that appear, choose the one you would like to get Extracted Features for and click on its name.
From the 'Analyze with Algorithm' drop down menu, choose the Extracted Features Download Helper algorithm.
This algorithm generates a script for downloading the feature data files that correspond to your workset.
Execute the Extracted Features Download Helper algorithm
Specify a job name of your choosing. Then, click the ‘Submit’ button.
Wait until the algorithm has finished and then open the completed job to download
Eventually, the job will complete, and it will move to the "Completed Jobs" section of the page. Click on the link representing the job name to see the results.
From the results page, click the blue button to download the shell script you will use to get the Extracted Features. The file will go wherever downloads typically end up on your machine, often the Downloads folder.
...
| title | Run the shell script to download |
|---|
For Windows users, make sure your computer is set-up to run rsync we recommend cwRsync - Rsync for Windows or Cygwin.
Navigate to the directory where the script file is located
From the command line, navigate to the directory where the script file is located. This directory will typically be called Downloads, though the location may be different depending on your machine and if you have moved the file. Here is an example:
| Code Block |
|---|
cd ~/Downloads |
Run the shell script
Then run the file you downloaded. It is a shell script.
| Code Block |
|---|
sh EF_Rsync.sh |
When you run it, the Extracted Features files for the volumes in your workset will be downloaded. Note that your workset could contain a volume for which an Extracted Features File has not been created due to the way that the versions of Extracted Features are snapshots of the HathiTrust corpus at specific points in time.
The files will download in pairtree directory structure.
(Optional) Uncompress the downloaded files
Because the feature data files are compressed, you may need to uncompress them into JSON-formatted text files, depending on your need. The compression used is bzip2. If you are using the files with the HTRC Feature Reader, the library will deal with the compression automatically.
...
Download Format
File Format
Sample Files
A sample of 100 extracted feature files is available for download through your browser: sample-beta.zip.
Also, thematic collections are available to download: DocSouth (87 volumes), EEBO (355 volumes), ECCO (505 volumes).
Filepaths
The data is stored in a pairtree directory structure, allowing you to infer the location of any file based on its HathiTrust volume identifier. Pairtree format is an efficient directory structure, which is important for HathiTrust-scale data, where the files are placed in directories based on “pairs” of characters in their file names. For example the Extracted Features file for the volume with HathiTrust ID mdp.39015073767769 would be located at:
mdp/pairtreeroot/39/01/50/73/76/69/39015073767769
When you download the files, they will sync in pairtree directory structure, as well.
Download Options
Rsync
Rsync will download each feature file individually, following a pairtree directory structure.
The Rsync module (or alias path) for Extracted Features .2 is data.analytics.hathitrust.org::features-2015.02
If you run the rsync command as written above, without specifying file paths, it will sync all files. Do not do this unless you are prepared to work with the full dataset, which is 1.2 TB. Make sure to include the final period (.) when running your command in order to sync the files to your current directory, or else provide the path to the local directory of your choosing where you would like the files to be synced to.
A full listing of all the files is available from:
| Code Block |
|---|
rsync -azv data.analytics.hathitrust.org::features-2015.02/listing/htrc-ef-all-files.txt |
Users hoping for a more flexible file listing can use rsync's --list-only flag.
To rsync only the files in a given text file:
| Code Block |
|---|
rsync -av --files-from FILE.TXT data.analytics.hathitrust.org::features-2015.02/. |
Converting HathiTrust Volume ID to RSync URL (using Python)
Researchers who have their list of HT volume IDs but prefer not to use the HTRC Feature Reader, can convert HT volume IDs into rsync URLs using a Python script. This example is a simplified part of a longer notebook, which further describes how to collect and download large lists of volumes: ID to EF Rsync Link.ipynb.
If you don't have it, you may have to install the pairtree library with: pip install pairtree (Python 2.x only).
| Code Block | ||
|---|---|---|
| ||
import os
from pairtree import id2path, id_encode
def id_to_rsync(htid):
'''
Take an HTRC id and convert it to an Rsync location for syncing Extracted Features
'''
libid, volid = htid.split('.', 1)
volid_clean = id_encode(volid)
filename = '.'.join([libid, volid_clean, kind, 'json.bz2'])
path = '/'.join([kind, libid, 'pairtree_root', id2path(volid).replace('\\', '/'), volid_clean, filename])
return path |
Example:
| Code Block | ||
|---|---|---|
| ||
id_to_rsync('miun.adx6300.0001.001')
'miun/pairtree_root/ad/x6/30/0,/00/01/,0/01/adx6300,0001,001/miun.adx6300,0001,001.json.bz2' |
The Extracted Features for this volume can be downloaded using RSync:
| Code Block | ||
|---|---|---|
| ||
rsync -azv |
| Expand | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||
Converting HathiTrust Volume ID to RSync URL (using Python)Researchers who have their list of HT volume IDs but prefer not to use the HTRC Feature Reader, can convert HT volume IDs into rsync URLs using a Python script. This example is a simplified part of a longer notebook, which further describes how to collect and download large lists of volumes: ID to EF Rsync Link.ipynb. If you don't have it, you may have to install the pairtree library with: pip install pairtree (Python 2.x only).
Example:
|
Workset Builder 2.0
Extracted Features 1.5 files can also be downloaded for search results in HTRC's beta Workset Builder 2.0. After completing your search, you can download either the Extracted Features files for your results set or for single files from your results.
First, enter search terms for a desired set of volumes. Once results are returned, filter the results to remove any volumes for which you do not want Extracted Features files or to find the volume(s) most relevant to your work.
| Expand | ||
|---|---|---|
| ||
You can download the files for your entire results set. Once your results includes all of the volumes you'd like Extracted Features files for, click "JSON Extracted Features (ZIP)" under the "Export Search Results" heading at the top of the left sidebar. Clicking this link will start a download of the Extracted Features files for all volumes in your results. Since this process is fetching and compressing files, and result sets can be large, it may take a few moments to start and finish your download.
|
| Expand | ||
|---|---|---|
| ||
If you'd like to download Extracted Features files for individual volumes, you can click the link "Download Extracted Features (complete volume)" under each volume in the search results. This will download uncompressed Extracted Features for the given volume, in JSON.  |
Use HTRC EF Download Helper Algorithm
To download the Extracted Features data for a specific workset in HTRC Analytics, there is an algorithm that generates the Rsync download script, the Extracted Features Download Helper. The tool can also be useful if you don’t want to go through the process of determining files paths. Select version 1.5 when you run the algorithm to get the Extracted Features 1.5 version of the files.
The algorithm creates a shell script that you can download and run from your local command line. The file lists the rsync commands for every volume in an HTRC workset. Once you have run the algorithm and downloaded the resulting file, here is the command you would run.
| Expand | ||
|---|---|---|
| ||
Go to HTRC Analytics Navigate to https://analytics.hathitrust.org and log in. Go to the Worksets page of HTRC Analytics Click on the 'Worksets' link near the top of the screen. From the list of worksets that appear, choose the one you would like to get Extracted Features for and click on its name.
From the 'Analyze with Algorithm' drop down menu, choose the Extracted Features Download Helper algorithm.
This algorithm generates a script for downloading the feature data files that correspond to your workset. Execute the Extracted Features Download Helper algorithm Specify a job name of your choosing. Then, click the ‘Submit’ button.
Wait until the algorithm has finished and then open the completed job to download Eventually, the job will complete, and it will move to the "Completed Jobs" section of the page. Click on the link representing the job name to see the results.
From the results page, click the blue button to download the shell script you will use to get the Extracted Features. The file will go wherever downloads typically end up on your machine, often the Downloads folder.
|
| Expand | ||||
|---|---|---|---|---|
| ||||
For Windows users, make sure your computer is set-up to run rsync we recommend cwRsync - Rsync for Windows or Cygwin. Navigate to the directory where the script file is located From the command line, navigate to the directory where the script file is located. This directory will typically be called Downloads, though the location may be different depending on your machine and if you have moved the file. Here is an example:
Run the shell script Then run the file you downloaded. It is a shell script.
When you run it, the Extracted Features files for the volumes in your workset will be downloaded. Note that your workset could contain a volume for which an Extracted Features File has not been created due to the way that the versions of Extracted Features are snapshots of the HathiTrust corpus at specific points in time. The files will download in pairtree directory structure. (Optional) Uncompress the downloaded files Because the feature data files are compressed, you may need to uncompress them into JSON-formatted text files, depending on your need. The compression used is bzip2. If you are using the files with the HTRC Feature Reader, the library will deal with the compression automatically. |
Anchor EF.2download EF.2download
Downloading Extracted Features version 0.2
| EF.2download | |
| EF.2download |
Download Format
File Format
Sample Files
A sample of 100 extracted feature files is available for download through your browser: sample-beta.zip.
Also, thematic collections are available to download: DocSouth (87 volumes), EEBO (355 volumes), ECCO (505 volumes).
Filepaths
The data is stored in a pairtree directory structure, allowing you to infer the location of any file based on its HathiTrust volume identifier. Pairtree format is an efficient directory structure, which is important for HathiTrust-scale data, where the files are placed in directories based on “pairs” of characters in their file names. For example the Extracted Features file for the volume with HathiTrust ID mdp.39015073767769 would be located at:
mdp/pairtreeroot/39/01/50/73/76/69/39015073767769
When you download the files, they will sync in pairtree directory structure, as well.
Download Options
Rsync
Rsync will download each feature file individually, following a pairtree directory structure.
The Rsync module (or alias path) for Extracted Features .2 is data.analytics.hathitrust.org::features-2015.02
If you run the rsync command as written above, without specifying file paths, it will sync all files. Do not do this unless you are prepared to work with the full dataset, which is 1.2 TB. Make sure to include the final period (.) when running your command in order to sync the files to your current directory, or else provide the path to the local directory of your choosing where you would like the files to be synced to.
A full listing of all the files is available from:
| Code Block |
|---|
rsync -azv data.analytics.hathitrust.org::features-2015.02/listing/htrc-ef-all-files.txt |
Users hoping for a more flexible file listing can use rsync's --list-only flag.
To rsync only the files in a given text file:
| Code Block |
|---|
rsync -av --files-from FILE.TXT data.analytics.hathitrust.org::features/{{URL}} .-2015.02/. |