Skip to contents

Outlining the problem

One of the central problems in ensembled data packages is setting equivalences between observations from different datasets. This is a central problem because it is what enables us to relate datasets to one another.

What is required is some classification scheme so that same is recognised as same and different observations are recognised as different. In many cases, it is useful to have some kind of registry of unique units that existing and, importantly, new data can be read against.

There are several important features of such a registry that, while not required are certainly very helpful:

  1. Authoritative: the registry should be a singular source
  2. Comprehensive: the registry should ideally cover all possible units
  3. Succinct: the registry codes should be as short as possible
  4. Intelligible: the registry codes should be as meaningful for humans as possible

For some datacubes, unique identifiers are not too much of a problem because there exists a relatively comprehensive, authoritative, succinct, and intelligible registry. An example of this is the central states datacube in the manystates package. Here we can rely on ISO-3166 alpha-3 codes for most modern states, complementing them with COW or other codes for more historical states.

But we are not aware of comprehensive, authoritative, succinct, and intelligible registry of treaties. To solve this problem, we need to think both about how observations might be coded and what this codes might represent.

Setting up the problem

If we take a sample of environmental treaty titles from all the different datasets in the manyenviron::agreements datacube, we see that there are several treaties present in the different datasets. However, titles referring to the same agreement are not necessarily the same (e.g. a preposition might be missing from the title).

library(manydata)
manydata::get_packages("manyenviron")
samples <- lapply(manyenviron::agreements, function(x) x[x$Beg > "1990-01-01" & x$Beg < "2000-01-01", ])
titles <- unlist(purrr::map(samples, "Title"))
dates <- unlist(purrr::map(samples, "Beg"))
sample <- cbind.data.frame(titles, dates)

To compare possible identification strategies, we take the first 1000 treaties (ordered chronologically from the sample above) and hand code treaties that are the same across datasets. Notice that we remove all observations from HUGGO and AIGGO datasets, datasets coded by us, and use the original titles from other datasets.

library(readr)
agreements <- read_csv("hand-coded-sample.csv")

Options for solutions

A perfect match

If we use the hand coded duplicates to correctly identify all duplicates in the sample, we get a perfect match! That is, we identify all true positives and no false positives. The rates of true positives and false positives are represented the receiver operating characteristics (ROC) graph below. The area under the curve (AUC score), in dark grey, in the graph is maximized (equals 1).

library(WVPlots)
WVPlots::ROCPlot(agreements, "duplicate", "duplicate", 1, title = "Duplicates Match")

# WVPlots::PRPlot(agreements, "duplicate", "duplicate", 1, title = "Duplicates Match")
# Precision recall (PR) plots are also available for comparing solutions.

In reality, we often do not get a perfect score as we rarely know all the true duplicates across several large datasets.

Everything unique

First we could simply offer a unique code for each title, no duplicates. Obviously this would generate errors on the side of false negatives for many titles that are quite obviously (for humans) equivalent. That is, we would find zero false positives as well as zero true positives, as the ROC plot illustrates.

agreements$unique <- 0
WVPlots::ROCPlot(agreements, "unique", "duplicate", 1, title = "Unique")

# WVPlots::PRPlot(agreements, "unique", "duplicate", 1, title = "Unique")

Strict matching

We an also only consider equivalent those titles that are exactly the same as one another.

agreements$strict <- ifelse(duplicated(agreements$titles) |
                                  duplicated(agreements$titles, fromLast = TRUE), 1, 0)
WVPlots::ROCPlot(agreements, "strict", "duplicate", 1, title = "Strict")

# WVPlots::PRPlot(agreements, "strict", "duplicate", 1, title = "Strict")
# PR plot is consistent with ROC plot and shows little improvement from a random match.

Note that the this approach probably misses a number of more problematic cases. For example a “Treaty Of…” and “Treaty For…” would be considered distinct, as would “Antarctic Treaty” and “Antarctic Treaty 1959”, even though they ought to be rendered equivalent. Strict match is very specify, but not very sensible, as it generates no false positives but it only captures a so many of the true positives.

Fuzzy matching

We could also use fuzzy matching to match observations that are mostly similar but not identical.

library(stringdist)
fuzzy <- stringdist::stringsimmatrix(agreements$titles, agreements$titles)
diag(fuzzy) <- 0
agreements$fuzzy <- apply(fuzzy, 1, max)
agreements$fuzzy <- ifelse(agreements$fuzzy > 0.9, 1, 0)
WVPlots::ROCPlot(agreements, "fuzzy", "duplicate", 1, title = "Fuzzy")

# WVPlots::PRPlot(agreements, "fuzzy", "duplicate", 1, title = "Fuzzy")
# Improvements are also visible in the PR plot.

The risk of finding incorrect matches using fuzzy matching increases in relation to strict matches. That is, although we might do a better job at finding true matches with fuzzy, we also might get more false positives.

Extracting key identifying information

Another option is to extract key identifying information from the titles and dates, rendering as equivalent any titles that have the same key identifying information.

The question here is which information to extract from the titles. It needs to be enough information to render as distinct those titles that are indeed distinct, but that ignores both less salient information (e.g. “Of” versus “For”) and addenda (e.g. the year at the end of the treaty title). The less information we require for a match, the higher the false positive rate, and the more information we require, the higher the false negative rate.

This is what we currently do with the manypkgs::code_agreements() function. We programatically generate “treatyIDs” from treaty titles and dates by extract meaningful information. For more details on how the function works and how to understand the IDs generate, please see the sections below on “How to read a treatyID” and on “How code_agreements() work”.

agreements$treatyID <- manypkgs::code_agreements(title = agreements$titles, 
                                                 date = agreements$dates)
#> 0 entries were not matched at all.
#> There were 340 duplicated IDs.
agreements$treatyIDs <- ifelse(duplicated(agreements$treatyID) |
                                  duplicated(agreements$treatyID,
                                             fromLast = TRUE), 1, 0)
WVPlots::ROCPlot(agreements, "treatyIDs", "duplicate", 1,
                 title = "treatyID Strict")

# WVPlots::PRPlot(agreements, "idinfo", "duplicate", 1, title = "treatyID Strict")
# PR plot illustrate similarities between a perfect match and treatyID match.

The AUC score with code_agreements() is higher than what we get with fuzzy and with strict matches. In relations to other options, code_agreements() improves the number of true observations matched while not generating many false positives.

We can also compare the specificity versus sensitivity for each of the possible solutions discussed above in the same ROC plot.

WVPlots::ROCPlotList(frame = agreements,
                     xvar_names = c("strict", "fuzzy", "treatyIDs"),
                     truthVar = "duplicate", truthTarget = TRUE,
                     title = "Comparing the different solutions")

Still, this is not perfect and some true matches are still missing. This happens when the titles differ a great deal, even if they are referring to the same agreement. In those cases, we prefer to stay on the side of caution and refrain from trying to code these titles as duplicates, especially as such changes would likely lead to higher rates of false positives.

Once we have identified duplicates, we can also make sure the treaty information is standardized regarding how treaties relate to other treaties in the datacube. For this we can use the condense_agreements() function.

qsim <- manypkgs::condense_agreements(idvar = agreements$treatyID)
library(dplyr)
agreements <- dplyr::left_join(agreements, qsim)
agreements %>% 
  dplyr::filter(manyID != treatyID) %>%
  dplyr::select(treatyID, manyID)
#> # A tibble: 10 × 2
#>    treatyID                  manyID                   
#>    <chr>                     <chr>                    
#>  1 AUS-RUS[PEE]_1990A        AUS-RUS[PNE]_1990A       
#>  2 AUS-RUS[PEE]_1990A        AUS-RUS[PNE]_1990A       
#>  3 AUS-RUS[PNE]_1990A        AUS-RUS[PEE]_1990A       
#>  4 BRA-URY[DRH]_1991A        BRA-URY[DRC]_1991A       
#>  5 BRA-URY[DRC]_1991A        BRA-URY[DRH]_1991A       
#>  6 CTPOTA_1991A              CNTPOA_1991A             
#>  7 ANIPPS_1992E:MARPOL_1973A AIIPPS_1992E:MARPOL_1973A
#>  8 BLR-UKR[CCD]_1992A        BLR-UKR[CCC]_1992A       
#>  9 COL-JAM[LRD]_1993A        COL-JAM[MRD]_1993A       
#> 10 ONTRIN_1994A              INTRTT_1994A

How code_agreements() work

The function extracts important information from treaty titles and dates at each step of the way to make sure duplicates are actual duplicates.

Standardises titles

The first step of the function is to standardise treaty titles with the standardise_titles() function. The functions standardises character title variable to improve readability, facilitate string matching and enable more accurate comparisons for variables in different datasets.

Parties

The first element extracted from the title are the parties to a treaty. The function code_states()returns the 3 digit ISO codes for states when there are two parties identified (i.e. bilateral treaty). The three letters abbreviation in brackets pasted next to the parties represent the activity of the title. The activity is generated with code_activity(), which uses core words at the end of treaty titles to generate 3 letter acronyms for activity. Parties and activity are only returned for bilateral treaties.

manypkgs::code_states(agreements$titles)

Acronyms

Multilateral treaties start with the acronyms generated by the manytreaties::code_acronyms() function. code_acronyms() extracts letters from the title once specific words, numbers and special characters are deleted from the title. If the title exceed a certain length (6 letters), some of the acronyms contain the numbers of the letters ommited when generating the acronym.

manytreaties::code_acronyms(agreements$titles)

Date

The year of the treaty’s signature is extracted with messydates::year().

messydates::year(agreements$dates)

Type

code_type() detects the type of the treaty and assigns a letter for each agreement type.

manypkgs::code_type(agreements$titles)
  • Agreements = A
  • Amendments = E
  • Protocol = P
  • Notes = N
  • Resolutions = R
  • Strategy = S

To see which key words are used to identify the treaty type, please run code_type().

For amendments or protocol, their ordering numbers, if present, are also extracted in code_type()

Known Agreements

Are these a famous agreements for which abbreviations are known? Some treaties already have well-known abbreviations.

manypkgs::code_known_agreements(agreements$titles)
Know Agreements Abbreviation
“United Nations Convention On The Law Of The Sea” UNCLOS1982
“Convention On Biological Diversity” CBD1992
“Convention On The Conservation Of Antarctic Marine Living Resources” CCAMLR1980
“Convention On International Trade In Endangered Species Of Wild Fauna And Flora” CITES1973
“International Convention On Civil Liability For Oil Pollution Damage” CLC1969
“International Convention For The Prevention Of Pollution From Ships” MARPOL 1973
“Constitutional Agreement Of The Latin American Organization For Fisheries Development” OLDEPESCA1982
“Paris Agreement Under The United Nations Framework Convention On Climate Change” PARIS2015
“Convention On Wetlands Of International Importance Especially As Waterfowl Habitat” RAMSA1971
“United Nations Framework Convention On Climate Change” UNFCCC1992

For the complete list of known agreements coded, please run code_known_agreements().

Linkage

Detects the family a treaty might belong to. Treaties from the same family can be detected by removing predictable words that are added to treaty titles (e.g. amendment, protocol, meeting) and identifying duplicates based on the core words used to refer to a main agreement.

manypkgs::code_linkage(agreements$titles, agreements$dates)

Once added together, the information extracted from treaty title and date forms a treatyID which is more authoritative, comprehensible, succinct and intelligible than other conventions.

How to read a treatyID

A treatyID is a meaningful shorthand ID created from a combination of elements extracted from the agreement title and date. The treatyID allows users to identify date, type and linkage.

Type Pasting Pasting
Bilateral FRA-TON[DEZ]_1980A parties[activity]_year(type)
Bilateral + Protocol RUS-USA[FKC]_1967E:RUS-USA[FKC]_1967A parties[activity]_year(type):linkage

A bilateral treaty that is an agreement will have the following treatyID: “FRA-TON[DEZ]_1980A”. This is a combination of the parties to the agreement (FRA-TON), the activity described in the treaty, which is here “Delimitation of Economic Zone (DEZ)”, with the year of signature (1980), followed by the type (A = Agreement).

A bilateral treaty that is any other type than an agreement (e.g. protocol, amendments) will have the treatyID under this format: “RUS-USA[FKC]_1967E:FKC_1965A”. It is composed of the parties (RUS-USA), the activity abbreviation from treaty title “Fishing for King Crab (FKC)”, the year of signature of the amendment (1967), and the type (E) which refers to Amendment. The linkage portion links the agreement to the treatyID its “mother” treaty (RUS-USA[FKC]_1967A).

Type Pasting Pasting
Multilateral HSPDF_2005A acronym_year(type)
Multilateral + Protocol SFDP_2007E2:H08F_1992A acronym_year(type and number):linkage

A multilateral treaty that is an agreement will have the following treatyID: “HSPDF_2005A”. The treatyID indicates the acronym (HSPDF), the signature year of the agreement (2005), and the type (A).

A multilateral treaty that is not an agreement will have this treatyID format: “SFDP_1992E2:H08F_2007”. This represents the acronym (SFDP), the signature year of the amendment (2007), the type (E = Amendment), the number of amendment (2), and the linkage number (H08F_2007) of the “mother” treaty.

Type Pasting Pasting
Known treaties UNCLOS1982A abbreviation, uID, type
Amendment of known treaties J09H_1990E2:MARPOL1973A acronym_uID, type(number):linkage

Famous multilateral treaties have a simplified treatyID with a known abbreviation. For example, the United Nations Convention On The Law Of The Sea will have the following treatyID: “UNCLOC1982A”. This is the known abbreviation (UNCLOS), the signature year (1982) and the treaty type (A).

Protocols or amendments of the known treaties will have this treatyID format: “J09H_1990E2:MARPOL1973A”. It indicates the acronym (J09H), the signature year of this specific amendment (1990), the type of treaty (E), its number (2), and the treatyID of the “mother” treaty (MARPOL197A).