Excerpt |
---|
Read about HTRC's most recent version of the Extracted Features derived dataset. |
Page-level features from 17.1 million volumes
Short URL for this page: https://go.illinois.edu/EF20_documentation
HTRC Extracted Features 2.0 is the most current version of a derived dataset consisting of metadata and data elements extracted from volumes in the HathiTrust Digital Library. The dataset is composed of 17+ million JSON files representing a snapshot of the HathiTrust corpus from February 2020.
This documentation describes the structure and data in the HTRC Extracted Features 2.0 files for users of those files. The specific features extracted are described in more detail below.
You can also refer to technical documentation of the Extracted Features 3.0 JSON-LD schema here. The schema was developed collaboratively with JSTOR-Portico, and it could be applied to data from non-HathiTrust sources to create compatible datasets. This version of the extracted features vocabulary is designed as a linked data standard (JSON-LD).
Downloading the files
See directions for Extracted Features v.2.0
Attribution
Jacob Jett, Boris Capitanu, Deren Kudeki, Timothy Cole, Yuerong Hu, Peter Organisciak, Ted Underwood, Eleanor Dickson Koehl, Ryan Dubnicek, J. Stephen Downie (2020). The HathiTrust Research Center Extracted Features Dataset (2.0). HathiTrust Research Center. https://doi.org/10.13012/R2TE-C227
This feature dataset is released under a Creative Commons Attribution 4.0 International License.
Data Stats
# of volumes represented | 17,123,746 |
# of pages represented | 6,221,631,336 |
# of tokens represented | 2,906,819,723,689 |
# files derived from in-copyright volumes | 10,550,952 |
# pages derived from in-copyright volumes | 3,743,561,467 |
# tokens derived from in-copyright volumes | 1,709,172,692,891 |
# files derived from public domain & Creative Commons volumes | 6,572,794 |
# pages derived from public domain & Creative Commons volumes | 2,478,069,869 |
# tokens derived from public domain & Creative Commons volumes | 1,197,647,030,798 |
Files and file format
1 Extracted Features file per volume
- There is one Extracted Features 2.0 file per volume from HathiTrust that has been processed into the dataset. (17.1 million volumes processed = 17.1 million Extracted Features files in the entire dataset.)
- The dataset is not continuously updated, and files are not created on request. There are volumes in HathiTrust for which an associated Extracted Features file does not exist.
Size of the entire dataset
- Each Extracted Features file is BZIP2 compressed.
- The entire compressed dataset is 4TB.
- For that reason, we advise against downloading the entire dataset unless you are prepared to store and work with that amount of data.
File sections
The main sections are the metadata block and the features block. For a basic overview, follow this introductory guide to the HTRC Extracted Features 2.0 format.
JSON-LD and the linked data format
Extracted Features 2.0 files are in JSON-LD format. The linked data integration is new to this version of HTRC Extracted Features, and incorporates semantics from the Schema.org web ontology with extensions of our devising.
Linked data is found primarily in the metadata section of the file, and it connects to ontologies and vocabularies like Schema.org, or to authorities databases such as Virtual International Authority File (VIAF) or Library of Congress linked data resources. When no entry was found in a 3rd-party database, a non-authoritative placeholder entity was created for the name in a named entity database maintained by HTRC.
Terminology
These terms found in the documentation may be unfamiliar.
Volume: A digitized item in the HathiTrust Digital Library, may be a book, an issue of a periodical, or several items bound together. Represents a physical item that would be pulled off a library shelf and digitized as one object.
Object: A key-value pair in the JSON file. Denoted by curly braces. E.g. {type:”Book”}
Node: A linked data concept, they are the individual elements that make up a JSON-LD document’s graph (the subject-object-predicate of a linked data triple). Read more. They can be:
- A number or string
- Arrays (a list)
- Node objects (0 to more node properties)
- Arrays that list node objects
Term: A label for a particular node
Feature File Documentation
Repeating terms
Some terms in the Extracted Features files repeat, and their values depend on their context within the file. The repeating terms are described below.
Background Color | ||||||||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ||||||||||||||||||
Anchor |
|
Anchor | ||||
---|---|---|---|---|
|
Anchorname name
name
name | |
name |
Description: Used wherever Person and Organization object types are found in the metadata block. It identifies a Person or Organization in a human readable manner.
File description section
Each Extracted Features file opens with metadata describing the file itself.
Background Color | ||
---|---|---|
| ||
@contextDescription: Links to a context document, created by HTRC, which maps terms in the Extracted Features 2.0 file. Value: The URL identifying the context document. Always “https://worksets.htrc.illinois.edu/context/ef_context.jsonld” schemaVersionDescription: The version of the extracted features schema. Value: “3.0” idDescription: See id Value: The URL identifying the Extracted Features file. E.g., "https://data.analytics.hathitrust.org/extracted-features/20200210/ufl2.uf00103078_00001" htidDescription: Every volume in HathiTrust is assigned a unique HathiTrust identifier (htid). Value: The HathiTrust volume identifier. E.g. “ufl2.uf00103078_0000” typeDescription: See type Value: “DataFeed” [Learn more] publisherDescription: Describes HTRC, the publisher of the Extracted Features files. Consists of three parts: id, type, and name. idDescription: See id Value: “https://analytics.hathitrust.org” TypeDescription: See type Value: “Organization” NameDescription: See name Value: “HathiTrust Research Center” datePublishedDescription: The date the Extracted Features file was created, in YYYYMMDD format [ISO 8601] Value: E.g. “20200221” |
AnchorMetadataSection MetadataSection
Metadata section
MetadataSection | |
MetadataSection |
Metadata primarily about the volume described by the Extracted Features file.
The metadata is received from HathiTrust in MARC (library catalog metadata) format. (Learn more about HathiTrust metadata.) It is then transformed to the Bibframe standard, which is a linked data standard. By shifting the metadata from MARC to Bibframe, we are able to improve the overall quality of the metadata by providing links to more contemporary authorities sources for metadata (such as VIAF, etc.). As a final step, the metadata is converted into the Extracted Features metadata section format.
If there would be no value for a field, then it is left out of the file. Therefore, you may not find all of the fields below in a given Extracted Features 2.0 file.
Background Color | ||
---|---|---|
| ||
metadataDescription: Denotes the start of the metadata part of the Extracted Features file. The metadata describes the volume from HathiTrust represented by the Extracted Features file. schemaVersionDescription: The schema version for the metadata block of the Extracted Features file. Value: A URL linking to the schema. "https://schemas.hathitrust.org/Extracted Features_Schema_MetadataSubSchema_v_3.0" Type: Single value idDescription: See id Value: The Handle URL for the volume, which will point to the volume in the HathiTrust Digital Library. E.g. "http://hdl.handle.net/2027/mdp.39015062779023" Type: Single Value typeDescription: See type Value: Either “Book”, “PublicationVolume”, or “CreativeWork” depending on the value found in the Issuance field of the corresponding Bibframe record. When the Bibframe Issuance value is ‘mono’, the Book type is assigned. When the value is ‘serl’, the PublicationVolume type is assigned. In all other cases the CreativeWork type is assigned. The Bibframe Issuance is derived from a variety of fields in the MARC record, including Control Fields and Physical Description Fields. Additionally, "DataFeedItem" will always be listed as a type to reflect the item's relationship with the Extracted Features file. Type: Array of values dateCreatedDescription: The date on which the metadata portion of the Extracted Features file is generated, in YYYYMMDD format [ISO 8601] Value: E.g., “20200209” Type: Single value titleDescription: The title of the volume when the type (above) is “Book” or “CreativeWork”. Value: Derived from the Bibframe title, which originates from the Title & Title-Related Fields in the MARC record. Type: Single value alternateTitleDescription: An alternate title for a bibliographic entity described by the Extracted Features file. Value: Derived from the Bibframe title where the rdf:type is “VariantTitle”. Originates from a variety of fields in the MARC record, including Main Entry Fields and Title & Title-Related Fields. Additionally, if there are multiple valid values for title, one value will be used and all additional values will be listed as alternateTitles. Type: Single value or array of values enumerationChronologyDescription: Information regarding which volume, issue, and/or year the HathiTrust volume was published. Value: Taken from the Bibframe enumerationAndChronology node, which is derived from a variety of fields in the MARC record, including Holdings Fields and Locally Added Fields. Type: Single value publisherDescription: Information about the publisher(s) of the volume described by the Extracted Features file. Includes type, name, and id. [Learn more] Type: Single object or array of objects typeDescription: See type Value: Either “Organization” or “Person”. Taken from the type node for the Bibframe provisionActivity’s agent. Type: Single value id Description: See id Value: A URL identifying the publisher, such as a Handle URL, ORCID, etc. Type: Single value NameDescription: See name Value: Taken from the label node for the Bibframe provisionActivity’s agent. Derived from the Imprint field in the MARC record. Type: Single value pubPlaceDescription: Information about where the volume was first published. Includes id and type.[Learn more] Type: Single object or array of objects idDescription: See id Value: Taken from the Bibframe Instance’s provisionActivity’s place rdf:about node, which are derived from the country codes in the MARC 008 field. Type: Single value type Description: See type Value: “Place” [Learn more] Type: Single value
Description: See name Value: Corresponds to information in the Imprint field in the MARC record. Type: Single value pubDateDescription: The year in which that edition of the volume was first published.[Learn more] Value: Taken from the Bibframe provisionActivity date, which is derived from the Imprint field in the MARC record. Type: Single value genreDescription: Information about the volume’s genre, as determined by the cataloger of the work. [Learn more] Value: Taken from the Bibframe Work’s genreForm node and corresponds to a controlled vocabulary, e.g., the MARC Genre Terms List. Values are derived from the Genre/Form Field in the MARC record. Type: Single value or array of values categoryDescription: The volume’s topic or topics. [Learn more] Value: Derived from the Bibframe Work’s ClassificationLcc node. Represents the natural language label for the Library of Congress Classification (LCC) value based upon the Library of Congress’s LCC standard documentation. Type: Singe value or array of values languageDescription: The cataloger-determined language or languages of the volume. Value: Taken from the Bibframe Work’s language’s identifiedBy’s value node, which is derived from the Language Code field in the MARC record. Type: Single value or array of values accessRightsDescription: The copyright status of the volume. Value: Corresponds to attributes in HathiTrust’s accessRights database. Derived from a HathiTrust-local MARC field (974r) that is added to bibliographic records as they are processed at HathiTrust. Type: Single value accessProfileDescription: Restrictions derived from source institution. Value: Mapping from source to document access used in HathiTrust's access determination logic. Derived from a HathiTrust-local MARC field (974a) that is added to bibliographic records as they are processed at HathiTrust Type: Single value lastRightsUpdateDateDescription: The most recent date the volume's copyright status was updated. Value: Derived from a HathiTrust-local MARC field (974d) that is added to bibliographic records as they are processed at HathiTrust. Type: Single value contributorDescription: Contains information regarding the author(s), editor(s), or other agents involved in creating the volume. Consists of id, type, and name. [Learn more] Type: Single object or array of objects id Description: See id Value: A URL taken from the Bibframe agent that links to an authorities database (e.g., VIAF). Type: Single value typeDescription: See type Value: Either the Person or Organization. Taken from the Bibframe agent type. Type: Single value nameDescription: The name of the person or organization who created the volume. See also name Value: Taken from the Bibframe agent’s label. Derived from a variety of fields in the MARC record, including Main Entry Fields, Title and Title-Related Fields, and Added Entry Fields. Type: Single value typeOfResourceDescription: The cataloger-determined resource type of the volume (e.g., text, image, etc.). Value: The data value of this node is taken from the Bibframe Work’s type node, which is derived from the MARC record’s 008 Fixed-Length Data Elements Field. Type: Single value sourceInstitutionDescription: Contains information about the institution that contributed the volume to HathiTrust. Always has a type node and a name node. [Learn more] Type: Single object idDescription: See id Value: A URL identifying the source institution Type: Single valuetypeDescription: See type Value: Always “Organization” Type: Single value nameDescription: The name of the source institution. See also name Value: Derived from a HathiTrust-local MARC field (974c) that is added to bibliographic records as they are processed at HathiTrust. Type: Single value mainEntityOfPageDescription: An array of URLs linking to various metadata records describing the volume represented by the Extracted Features file. [Learn more] Value: The array typically contains 3 URLs that point to the HathiTrust Bibliographic API: HathiTrust brief bibliographic record, HathiTrust full bibliographic record, and the HathiTrust catalog record. Type: Array of values oclcDescription: The OCLC number for the volume. An OCLC number is an identifier assigned to items as they are cataloged in a library. Value: Derived from the MARC record’s System Control Number field (035). Type: Single value or array of values lccDescription: The Library of Congress Classification number for the volume. An LCC number is a type of call number that would be used to locate an item on a library shelf. Value: Derived from the MARC record’s LC Classification Number field (053). E.g. PG7158.M76 E23 Type: Single value or array of values lccnDescription: The Library of Congress Control Number for the volume. An LCCN is a unique number that is assigned during cataloging. Value: Derived from the MARC record’s Library of Congress Control Number field (010). Type: Single value or array of values issnDescription: The ISSN of the volume (when a journal). [Learn more] Value: Taken from the Bibframe Instance’s identifiedBy’s Issn node, which is derived from the MARC record’s ISSN field. Type: Singe value or array of values isbnDescription: The ISBN of the volume (when a book). [Learn more] Value: Taken from the Bibframe Instance’s identifiedBy’s Isbn node, which is derived from the MARC record’s ISBN field. Type: Single value or array of values isPartOfDescription: Contains information on the larger work that the a serial volume is a part of. [Learn more] Type: Single object idDescription: See id Value: A WorldCat URL identifying the overarching work. Type: Single value typeDescription: See type Value: Always "CreativeWorkSeries" Type: Single value journalTitleDescription: The title of the parent work. See also Value: Take from the Bibframe Work's Title node, which is derived from a variety of fields in the MARC record, including Main Entry Fields and Title and Title-Related Fields. Type: Single value |
AnchorFeaturesSection FeaturesSection
Features section
FeaturesSection | |
FeaturesSection |
This portion of the Extracted Features file contains all of the unigram tokens (i.e. words), token counts, and other calculated or algorithmically-derived data from the HathiTrust volume.
The full text of the volume is tokenized using Stanford NLP, which includes part-of-speech detection. The tokens are tagged with part-of-speech tags using the tagsets from Stanford NLP.
After an initial section containing volume- and EF file-level metadata, this section is divided into JSON arrays for each page in the volume. Nested within those page-level arrays are arrays for the header, body, and footer of the page.
Fields that would be empty (i.e. if the algorithm does not detect content on a page) are given NULL values.
Background Color | ||
---|---|---|
| ||
featuresDescription: Denotes the start of the features part of the Extracted Features file. idDescription: See id Value: The Handle URL identifying the HathiTrust volume typeDescription: See type Value: “DataFeedItem” schemaVersionDescription: The version of the features part of the Extracted Features document model. Value: A URL linking to the schema. "https://schemas.hathitru…_FeaturesSubSchema_v_3.0" dateCreatedDescription: The date the data features were generated in in YYYYMMDD format. [ISO 8601] Note that this will be different from the dateCreated for both the metadata block and the overall Extracted Features file. Value: E.g. “20200209” pageCountDescription: The number of page scans in the volume, which is equivalent to the number of scanned images making up the digital object in the HathiTrust Digital Library. Value: A number corresponding to the number of pages. pagesDescription: An array of individual page objects (one per page of the volume). seqDescription: The sequence number of the page in the volume. Corresponds to the digital object, so that the first scan in the volume is “00000001”, which may be the cover, a title page, or something else. Value: E.g. “"00000001" or “00000055” versionDescription: A hash of the page content used to compute the features for the page. Volumes in HathiTrust may be updated to improve scan or OCR quality or correct an issue, which would cause the text data to change, and, if features are reprocessed, a new hash would result. Value: The alphanumeric hash. calculatedLanguageDescription: The most probable language of the text on the page. Determined algorithmically, and specified by language codes. Value: A code for the language. Will be null if no language detected, or if the language was not recognized by the algorithm. tokenCountDescription: The total number of tokens detected on the page. Value: A number representing the number of tokens. lineCountDescription: The total number of lines of text detected on the page. Value: A number representing the lines. emptyLineCountDescription: The total number of empty lines on the page. Value: A number representing the empty lines. sentenceCountDescription: The total number of sentences detected on the page. Value: A number representing the sentences. headerDescription: The header portion of the page. tokenCountDescription: The total number of tokens detected in the header of the page. Value: A number representing the tokens. lineCountDescription: The total number of lines detected in the header of the page. Value: A number representing the lines. emptyLineCountDescription: The total number of empty lines detected in the header of the page. Value: A number representing the empty lines. sentenceCountDescription: The total number of sentences detected in the header of the page. Value: A number representing the sentence count. beginCharCountDescription: An array of the first non-White Space characters detected on lines in the header and their occurrance counts. Value: The character and the number of times it occurs. E.g. “R:1” endCharCountDescription: An array of the last non-White Space characters on detected lines in the header and their occurrance counts. Value: The character (letter, number, or punctuation) and the number of times it occurs. E.g. “g:5” tokenPosCountDescription: An array of individual tokens with their corresponding part of speech and occurrence counts in the header of the page. If the language is one that is recognized by the Stanford NLP parser ("ar", "zh", "en", "fr", "de", "es") StanfordNLP models for the language are used to tokenize the text and do part-of-speech tagging. Otherwise, a whitespace tokenizer is used. Value: The token, its part of speech (represented by thePenn Tree Bank for English), and a number representing the number of times that token appeared in the header. bodyDescription: The body portion of the page. tokenCountDescription: The total number of tokens detected in the body of the page. Value: A number representing the tokens. lineCountDescription: The total number of lines detected in the body of the page. Value: A number representing the lines. emptyLineCountDescription: The total number of empty lines detected in the body of the page. Value: A number representing the empty lines. sentenceCountDescription: The total number of sentences detected in the body of the page. Value: A number representing the sentence count. beginCharCountDescription: An array of the first non-White Space characters detected on lines in the body and their occurrence counts. Value: The character and the number of times it occurs. E.g. “R:1” endCharCountDescription: An array of the last non-White Space characters detected on lines in the body and their occurrence counts. Value: The character (letter, number, or punctuation) and the number of times it occurs. E.g. “g:5” tokenPosCountDescription: An array of individual tokens with their corresponding part of speech and occurrence counts in the body of the page. If the language is one that is recognized by the Stanford NLP parser ("ar", "zh", "en", "fr", "de", "es") StanfordNLP models for the language are used to tokenize the text and do part-of-speech tagging. Otherwise, a whitespace tokenizer is used. Value: The token, its part of speech (represented by the Penn Tree Bank for English), and a number representing the number of times that token appeared in the header. capAlphaSeqDescription: The longest length of the alphabetical sequence of capital characters starting a line. Value: A number representing the length. footerDescription: The footer portion of the page. tokenCountDescription: The total number of tokens detected in the footer of the page. Value: A number representing the tokens. lineCountDescription: The total number of lines detected in the footer of the page. Value: A number representing the lines. emptyLineCountDescription: The total number of empty lines detected in the footer of the page. Value: A number representing the empty lines. sentenceCountDescription: The total number of sentences detected in the footer of the page. Value: A number representing the sentence count. beginCharCountDescription: An array of the first non-White Space characters detected on lines in the footer and their occurrence counts. Value: The character and the number of times it occurs. E.g. “R:1” endCharCountDescription: An array of the last non-White Space characters on detected lines in the footer and their occurrence counts. Value: The character (letter, number, or punctuation) and the number of times it occurs. E.g. “g:5” tokenPosCountDescription: An array of individual tokens with their corresponding part of speech and occurrence counts in the footer of the page. If the language is one that is recognized by the Stanford NLP parser ("ar", "zh", "en", "fr", "de", "es") StanfordNLP models for the language are used to tokenize the text and do part-of-speech tagging. Otherwise, a whitespace tokenizer is used. Value: The token, its part of speech (represented by the Penn Tree Bank for English), and a number representing the number of times that token appeared in the header. |