Discovering data

The first thing users of the package will want to do is to identify datasets that might contribute to their research goals. Since some of these data packages are too big for CRAN (the main though not only R package repository), we expect that their developers will instead choose to make their packages available on Github. Github is huge though, either way it can be difficult to identify salient data. We have tried to make this easy by allowing the reporting of all datasets in the ecosystem that we know of that currently pass our tests.

In the future, this function will report more details about the datasets included in each package and their provenance.

Understanding data

Packages in the qData ecosystem have the advantage to facilitate comparison and analysis of multiple datasets in a specific domain of global governance. This is possible with a particular coding system which follows the same principles across the different packages.

In qStates for example, all datasets from the states database contain variables named Begand End which represent the beginning and ending date of an episode of state sovereignty.

In {qEnviron}, the agreements database also have the Beg and End variables but those are attributed to treaties (signature and term dates). For the memberships database, Beg and End represent when a relationship between states and an agreement starts (either signature, ratification or entry into force) and ends (either withdrawal or term).

This specific variable name allows the comparison across the datasets which have different sources but same informations. It enables to point out the recurrence, difference or absence of observations between the datasets and extract more robust data when researching on a particular governance domain.

Loading data

Let us say that we wish to download the qStates package, which offers a set of datasets related to state actors in global governance. We can download and install the latest release version of the qStates package using the same function as before, only specifying which package we want to ‘get’, ‘get_packages(“globalgov/qStates”)’.

For now, let’s work with the Roman Emperors database included in qData. We can get a quick summary of the datasets included in this package with the following command:

data(package = "qData")
data(emperors, package = "qData")
emperors

We can see that there are three named datasets relating to emperors here: wikipedia (dataset assembled from Wikipedia pages), UNVR (United Nations of Roman Vitrix), and britannica (Britannica Encyclopedia List of Roman Emperors). Each of these datasets has their advantages and so we may wish to understand their differences, summarise variables across them, and perhaps also rerun models across them.

To retrieve an individual dataset from this database, we can use the pluck() function from the purrr package.

wikipedia <- purrr::pluck(emperors, "wikipedia")

However, the real value of qData packages is that multiple datasets relating to the same phenomenon are presented together. This facilitates comparison, the subject of the next section.

Comparing data

First of all, we want to understand what the differences are between these datasets. One important way to understand the relationship between these datasets is to understand what their relative advantages and disadvantages are. For example, one dataset may be long (has many observations) while another is shorter but wider (has more variables). One might include details further back in history while the other is more recent, but include more missing data or less precise data (i.e. coded at a less granular level) than another with a more restrictive. Or one might appear complete yet offer less information on where the original data points were sourced or how certain variables were coded, while another provides an extensive and transparent codebook that facilitates replication.

Differences in documentation

We can bring up the database level documentation using: ?emperors. This informs users on the datasets present in the database as well as the variables in the various datasets. However, if we want a more detailed summary of the various levels of data and sources, we can use data_source() and data_contrast().

Using data_source() and data_contrast()

The data_source() function displays bibliographic references for the datasets in a qPackage or within a database.

data_source(pkg = "qData", database = NULL, dataset = NULL)

The data_contrast() function returns a data frame with the key metadata of each level of data objects (qPackage, database, and dataset). This metadata includes the following elements:

  • Number of unique observations
  • Number of rows
  • Number of columns
  • Earliest beginning date
  • Latest end date
  • Source URL
data_contrast(pkg = "qData", database = NULL, dataset = NULL)

If we were specifically interested in the one dataset of the database, we can further specify the arguments.

data_contrast(pkg = "qData", database = "emperors", dataset = "wikipedia")

An example of inference sensitivity to data sources:

Next we may be interested in whether any relationships we are interested in or inferences we want to draw are sensitive to which data we use. That is, we are interested in the robustness of any results to different data specifications.

We can start by exploring whether our conclusion about when emperors began their reign would differ depending on which dataset we use. We can use the purrr::map() function used above, but this time pass it the mean() function and tell it to operate on just the “Beg” variable, wich represents when emperors began their reign (removing any NAs). Since qData datasets are always ordered by “Beg” (and then “ID”), we can remove any subsequent (duplicated) entries by ID to concentrate on first appearances.

emperors %>% 
  purrr::map(function(x){
    x %>% dplyr::filter(!duplicated(ID)) %>%
      dplyr::summarise(mean(Beg, na.rm = TRUE))
  })

It seems that there is some variation there.

Consolidate data

Now that we have visualized the data and looked at some of the different inferences drawn using different datasets, let us examine how to select and collapse databases. qData contains the consolidate() which facilitates collapsing multiple datasets into a dataframe for analysis.

consolidate() collapses a set of (q)datasets, or a database, into a single dataset with some combination of the rows and columns. The function includes separate arguments for the rows and columns, as well as for how to resolve conflicts in observations across datasets. The key argument indicates the column to collapse datasets by. This provides users with considerable flexibility in how they combine qData. For example, users may wish to stick to units that appear in every dataset but include variables coded in any dataset or units that appear in any dataset, but only those variables that appear in every dataset.

consolidate(emperors, rows = "any", cols = "any", resolve = "coalesce", key = "ID")
consolidate(emperors, rows = "every", cols = "every", resolve = "coalesce", key = "ID")