If you are having trouble with the sign in process, please <ContactLink>contact the SSH Open Marketplace team</ContactLink>.

If you are having trouble with the sign up process, please <ContactLink>contact the SSH Open Marketplace team</ContactLink>.

Subject

Social Sciences & Humanities Open Marketplace

Successfully submitted!

[Zotero](https://zotero.org) is a free and open source reference and bibliography management software. It [ingests structured publication metadata](https://www.zotero.org/support/adding_items_to_zotero) from files in various formats, and through a set of translators also by scraping from websites.

[Wikibase](https://wikiba.se) is a free and open source software solution for storing, editing, and querying Linked Data. It is the software underlying Wikidata.

[Wikidata](https://www.wikidata.org) is today the largest free and open Knowledge Graph. It contains millions of entities describing, for example, scholarly publications and their authors.

ZotWb, the tool presented in this workflow, helps to migrate records from a Zotero group library to a custom Wikibase instance, and performing entity reconciliation against Wikidata and/or the own Wikibase, using [OpenRefine](https://openrefine.org).

## Installation

This section explains what is necessary to start using the zotero-wikibase tool: 

* A Zotero account with read-and-write access to the Zotero group you want to export data from
  * You need a zotero user with read and write access to the Zotero group you want to use, a Zotero API key, and a Wikibase user with bot permissions and password.
  * A Zotere API key can be obtained [here](https://www.zotero.org/settings/keys/new). Be sure to grant read and write access through the key you create to the group library you want to export items from. The key consists of a single chain of characters.
* A Wikibase instance
* The Open Refine software, together with the module for connection to your Wikibase

Installation steps:
* This tool is a python3 app. You need python3 installed.
* Create a [virtual environment](https://realpython.com/python-virtual-environments-a-primer/), and run all subsequent steps, and the tool itself, always inside that.
* Clone this repository to a place of your choice, ideally the same project folder your virtual environment was created in.
* The tool requires the following packages; [install them in your virtual environment](https://realpython.com/python-virtual-environments-a-primer/#install-packages-into-it):
  * wikibaseintegrator
  * mwclient
  * pyzotero

# Configuration of the tool

The zotero-wikibase tool will guide you through the configuration process, which consists of the following steps:

* The tool will ask you to enter basic parameters
  * Zotero group to export data from (Zotero group ID, Zotero API key)
  * Wikibase to export data to (Wikibase name and URL, bot-user name and password)
    * Sensible information (usernames and passwords, API key) will be stored in a file called `config_private.json`, which is by default included in `.gitignore`, so that there is no risk that you share this information in case you keep your zotero-wikibase folder synced using git. This means, that if you use your zotero-wikibase tool instance on various machines, you have to manually copy this file to the other machine, or re-do this configuration step inside the tool.
* Definition of basic ontology entities
  * Some basic properties, and items describing ontology classes need to be defined. During the process, the tool will ask you whether you want to create new entities, or use entities already existing on your Wikibase.
    * Examples for basic entities: "Wikidata Entity", a property of datatype "ExternalId" which points to the equivalent entity on Wikidata, or "instance of" (equivalent to rdf:type and wd:P31).
    * Examples for ontology classes: "Language", "Creator Role", or "BibItem type". Instances of these classes will be linked to their class using the "instance of" property.
* The mapping of Zotero creator types and Zotero data fields (one set of fields per BibItem type) is done when exporting data from Zotero, i.e., in the next step.
  * When you export data from Zotero (see next step), the tool checks if all BibItem types and all fields containing data in the set of records you are exporting are already mapped to Wikibase properties.
    * You can specify existing properties to use, or create new properties along the process. You are able to map fields differently according to the BibItem type. For example, you may want to have titles of books mapped to a different property than titles of television broadcasts.



# Zotero export

Now you can already start to export Zotero records! The mapping of the Zotero fields to Wikibase properties (which still may have to be created or completed) is done when running the export tool.

* In the previous step, you have specified a tag you will use for marking Zotero items to be exported, such as `wikibase-export`. Mark some items with that tag (and sync with Zotero web, in case you work on a local Zotero instance.)
* Run the Zotero export function
  * The tool first checks what Zotero item types you want to export, and will create a Wikibase Item representing each item type, in case it does not exist. For this, it imports the corresponding item from Wikidata, as defined in `bots/mappings/zotero_bibtypes.json`.
  * The tool asks you then whether you want to check the mapping of the fields of these item types. As long as you are not sure all fields you want to export are mapped to a property, you should do this.
    * You can specify existing properties to use, or create new properties along the process. You are able to map fields differently according to the item type. For example, you may want to have titles of books mapped to a different property than titles of television broadcasts.
    * You can also choose to provisionally or permanently ignore certain Zotero fields, depending on the item type. It is recommended to ignore the `abstract` field for all BibItem types, since that regularly contains more than the amount of characters allowed for a Wikibase string property.
  * The script asks you then whether you want to do the same with the creator types present in the ingested dataset.
  * When that is done, the script transforms the Zotero API output, and uploads it to your Wikibase.
    * The Zotero API will deliver you chunks of 100 items. That is, if you tag more than 100 with your export tag, only one hundred of these will be processed in one run. Those that are not processed in one run will stay with the export tag.
    * Zotero items are patched the following way:
      * The Wikibase URI is written to the EXTRA field
      * The Wikibase URI is attached as URI link attachment
      * The tag used for marking Zotero items to be exported will be replaced with the tag you have specified in the previous step for successfully exported items, such as `on-wikibase`.

# Deduplicating and Reconciling person names in OpenRefine

This function will produce a CSV sheet to be imported into OpenRefine, containing all unreconciled (i.e., literal only) creator statements (including all creator role properties). The CSV file will be saved in your [`data/unreconciled_creators`](data/unreconciled_creators) folder.

## Deduplication (optional)

Deduplication using the [Clustering function](https://openrefine.org/docs/manual/cellediting#cluster-and-edit)

1. Starting point is the column containing the full name of persons.

2. To perform **clustering** without losing original column content, select the column menu:

	* `Edit column > Add column based on this column > value`.
    
	* Choose a column name, e.g. `fullName_clusters`.
    
3.  Clustering algorithms can be accessed with `Edit cells > Cluster and edit`.
    
4. Perform all clustering algorithms in the recommended order and manually validate the found name clusters. Different algorithms may detect different degrees and types of string similarity.
	* key collision - fingerprint
	* key collision - ngram fingerprint
	* key collision - metaphone3
	* key collision - cologne-phonetic
	* key collision - Daitch-Mokotoff
	* key collision - Beider-Morse
	* nearest neighbor - Levenshtein
	* nearest neighbor - ppm
    
	Validation of name clusters includes selecting one of the names as the normalized name variant. Within the column `fullName_clusters`, this preferred name variant will replace the various name forms previously found in the data.
    
5. To perform **merging of duplicates** for normalized person names in `fullName_clusters`, once again select the column menu: `edit column > Add column based on this column > value`. REQUIRED COLUMN NAME: `fullName_preferred`.
    
6. Before duplicates can be eliminated, sort alphabetically and re-order permanently:

	`Sort > a-z`
	`Sort > re-order rows permanently`
	`Edit cells > Blank down`

	These steps result in a list of preferred person names that appear only once in the column. Preferred full names are thus connected to preferred given names and last names.
    
## Reconciliation

**Reconciliation** will be based on the column `fullName_clusters`. If no clustering was performed, Reconciliation will be based on the raw person data in the column `fullName`.
    
1. Person names may already be included in your Wikibase. If not, they might be found in Wikidata. In both cases, the [Reconciliation function](https://openrefine.org/docs/manual/reconciling) searches for matches in the specified dataset.

	For each reconciliation process, create a separate column based on `fullName_clusters` (or `fullName`) column using `edit column > Add column based on this column > value`.

	REQUIRED COLUMN NAMES:
	fullName_recon_Wikidata
	fullName_recon_Wikibase
    
2. For each recon column, invoke the reconciliation process using `Reconcile > Start reconciling`. From the list of reconciliation services, pick the appropriate one, leading to your Wikibase or to Wikidata. If not already selected, select an entity type such as `human` (Q5) to restrict possible reconciliation targets. Be careful with the function `automatically match candidates with high confidence`; that is risky, since it is not at all unlikely that there exist several persons with the same or similar name on Wikidata, and only one (or none) of these is the one you want to link to.
    
3. Validate reconciliation candidates. Approved reconciled person names are now linked to their corresponding person item in Wikibase/Wikidata. 
4. Now create a new column containing the person item identifier from Wikibase/Wikidata using `Reconcile > Add entity identifiers column`
    
    Specify the new column's name as REQUIRED: 
     * `Wikidata_Qid`
     * `Wikibase_Qid`
5. Export as comma seperated value (csv) to your `data/reconciled_creators` folder using `Export > comma separated values`


# Reconciling other literal values

This function allows you to specify any Wikibase property with datatype `String` mapped to a Zotero field, the values of which you want to reconcile against entities on Wikidata and/or your own Wikibase.

Together with the `String` property, you define a property of datatype `WikibaseItem`, which will be later used for pointing to the reconciled entity. The tool allows you to do that in three ways:
* A property that already exists on your Wikibase
* A property that is newly created for this purpose
* A property that is imported from Wikidata for this purpose

Zotero literals may contain concatenated literal values that should be interpreted as multiple values. For example, the field `publication place` may contain a string like "Berlin; Amsterdam". You can specify what characters should be interpreted as separator of concatenated values in the source literals (in the example, a semicolon). The tool will than split values that contain such characters, and produce separate lines in the file to be sent to OpenRefine.

The tool produces a CSV file in your [`data/unreconciled_others`](data/unreconciled_others) folder, which you import into OpenRefine.

You run the reconciliation (see chapter about reconciling creator names) on the column named according to the property you chose, and when done, produce a new column with the REQUIRED name `Wikidata_Qid`, and/or `Wikibase_Qid`, respectively, which will contain the reconciled Q-identifiers, and then export the project sheet as comma separated value (CSV) to your [`data/reconciled_others`](data/reconciled_others) folder.
 

# Importing reconciliation results to your Wikibase

You have saved your reconciliation result CSV file in the `data/reconciled_creators` or `data/reconciled_others` folder.

* Run the function you need (for creators or others)

## Importing reconciled creators
* The tool will ingest the newest CSV file in the folder, and ask you to confirm that it is the file you want to process.
* For lines containing a Wikibase identifier, it will use the indicated item, and, in case the line also contains a Wikidata identifier, attach that to the item.
* For lines containing a Wikidata identifier, but no Wikibase identifier, it will check if an item aligned to that Wikidata entity already exists, and if not, create it on your Wikibase.
* The `unknown value` statement the ID of which is listed in the row will be replaced by a `custom value` statement pointing to the Wikibase item describing the creator.
* It will then process the name literals found in the CSV line:
  * The preferred full name variant (column `fullName_preferred`) will be written to `label`, in the languages you have specified in the configuration step.
  * Other name variants found in the row will be written to `alias`.
  * If the row `fullName_preferred` is not empty, creator first and last names are additionally written to the item in the following ways:
    * In the form [Lastname], [Firstname] to `alias`.
    * Creator first and last names will be written as string statements to the properties defined to that in the configuration step, replacing any other value possibly present for these properties.

## Importing other reconciled values
* The tool will ingest the newest CSV file in the folder, and ask you to confirm that it is the file you want to process.
* For lines containing a Wikibase identifier, it will use the indicated item, and, in case the line also contains a Wikidata identifier, attach that to the item.
* For lines containing a Wikidata identifier, but no Wikibase identifier, it will check if an item aligned to that Wikidata entity already exists, and if not, create it on your Wikibase.
* All values are attached to the correspondent Wikibase item (first column) using the property of datatype `WikibaseItem` you specified.

# Other functions of the zotero-wikibase tool

## Import entities from Wikidata to your Wikibase

This function enables you to import entities (items, properties) from Wikidata to your Wikibase in a few clicks.
* You can choose the languages you want labels, aliases, and descriptions to be imported
* You can import import statements describing the entity, as long the Wikidata property is aligned to your local equivalent property.
  * We encourage you not to massively import statements, and make use of [federated queries](https://www.mediawiki.org/wiki/Wikibase/Federation) instead.
  * Statements pointing to items (properties of datatype `WikibaseItem`) are excluded from this function in the current version.



David Lindemann

Christiane Klaes

This workflow explains how to migrate publication metadata from any Zotero group collection to a custom Wikibase instance using the ZotWb tool, how to do entity reconciliation using Open Refine, with Wikidata as authority file, and how to send reconciled data back to Wikibase.

LODification of bibliographical data: Zotero to Wikibase migration with ZotWb

Media

Related items(3)

Wikibase

Zotero

OpenRefine

Workflow steps(8)

1 Introduction

2 Requirements and installation

3 Configuration of the tool

4 Exporting Zotero records to your Wikibase

5 Reconciling person name literal strings against Wikibase and Wikidata entities

6 Reconciling other literals (publisher, place, etc.)

7 Writing OpenRefine reconciliation results to your Wikibase

8 Other functions of the zotero-wikibase tool

Media

Related items(3)

Tool or serviceWikibase

Tool or serviceZotero

Tool or serviceOpenRefine

Workflow steps(8)

Wikibase

Zotero

OpenRefine