This repository contains the code to import and integrate the book and rating data that we work with.

Requirements

• PostgreSQL 10 or later with orafce
• Python 3.6 or later with the following packages:
  • psycopg2
  • invoke
  • numpy
  • tqdm
  • pandas
  • sqlalchemy
  • numba
  • colorama
  • chromalog
  • humanize
• A Rust compiler
• psql executable on the machine where the import scripts will run
• 500GB disk space for the database
• 30GB disk space for data files

The environment-linux-x64.yml file defines an Anaconda environment that contains all the required packages, with the exception of the PostgreSQL server and client executables.

All scripts read database connection info from the standard PostgreSQL client environment variables:

• PGDATGABASE
• PGHOST
• PGUSER
• PGPASSWORD

Downloading Data Files

This imports the following data sets:

• Library of Congress MDSConnect Open MARC Records — get the XML files from https://www.loc.gov/cds/products/MDSConnect-books_all.html and save them into the data/LOC directory.
• LoC MDSConnect Name Authorities - get the Combined XML file from https://www.loc.gov/cds/products/MDSConnect-name_authorities.html and save it in data/LOC.
• Virtual Internet Authority File - get the MARC 21 XML data file from http://viaf.org/viaf/data/ and save it into the data directory.
• OpenLibrary Dump - get the editions, works, and authors dumps from https://openlibrary.org/developers/dumps and save them in data.
• Amazon Ratings - get the 'ratings only' data for Books from http://jmcauley.ucsd.edu/data/amazon/ and save it in data.
• BookCrossing - get the BX-Book-Ratings CSV file from http://www2.informatik.uni-freiburg.de/~cziegler/BX/ and save it in data

Running Import Tasks

The import process is scripted with invoke. The first tasks to run are the import tasks:

invoke loc.import
invoke viaf.import
invoke openlib.import-authors openlib.import-works openlib.import-editions
invoke goodreads.import
invoke ratings.import-az
invoke ratings.import-bx

Once all the data is imported, you can begin to run the indexing and linking tasks:

invoke viaf.index
invoke loc.index
invoke openlib.index
invoke goodreads.index-books
invoke analyze.cluster --scope loc
invoke analyze.cluster --scope ol
invoke analyze.cluster --scope gr
invoke analyze.cluster
invoke ratings.index
invoke goodreads.index-ratings
invoke analyze.authors

The tasks keep track of the import status in an import_status table, and will keep you from running tasks in the wrong order.

Setting Up Schemas

The -schema files contain the base schemas for the data:

• common-schema.sql — common tables
• loc-schema.sql — Library of Congress catalog tables
• ol-schema.sql — OpenLibrary book data
• viaf-schema.sql — VIAF tables
• az-schema.sql — Amazon rating schema
• bx-schema.sql — BookCrossing rating data schema
• gr-schema.sql — GoodReads data schema