RIPE Historic RPKI Data Archive

Data description

Below is a concise description of the archived RPKI data available from RIPE. This data can be accessed using the following URL:

https://ftp.ripe.net/rpki/

Overall archive structure

The root of the repositories contains a number of directories, each of which have data for (part of) the RPKI data for a specific Regional Internet Registry (RIR). The table below lists these directories and their specifics:

¹ The APNIC repository was split up into a main repository (-iana) and subrepositories for the other RIRs, where these specific subrepositories contain those prefixes the covering /8 of which is allocated to another RIR in the IANA registry. This is an artefact resulting from the one-time desire to have the IPv4 “/8 majority” prefixes in the IANA registry on each RIR’s certificate. The RIRs would then sign certificates to each other for the small bits of early registration and transferred space out of such a /8 that is registered at the other RIR. APNIC discontinued this split in 2018, and since then uses a single RPKI repository like the other RIRs do. To get the unified view for the APNIC repository over the period of the split, the various subrepositories should be merged into a single view. The Ziggy tool (more information below) does this automatically.

² This is the unified repository for APNIC as it was maintained before October 2012, and is it is currently maintained.

³ This repository is produced by a RIPE NCC test environment and can be ignored.

Each of the repository directories is further structured into year, month and day directories. The day directories contain a file called repo.tar.gz that contains the archived RPKI data for that repository on that day. The contents of this file are discussed in more detail below.

Repository files

The .tar.gz files that can be downloaded for each RIR all have the same basic structure. They contain a directory with the name of the respective repository (e.g. apnic.tal), and inside that directory are deeper subdirectories for the year, month and day to which the archived file pertains (e.g. apnic.tal/2019/08/27).

In this directory, you will typically find two files:

• The Trust Anchor certificate for the repository; this is the root of trust from which the content of the repository should be validated (typically ending in .tal.cer, e.g. apnic.tal.cer).

• A directory called rta which has two subdirectories called validated and unvalidated. As the names imply, validated only contains valid RPKI objects as validated by the RIPE NCC RPKI Validator application, the unvalidated directory contains all RPKI objects regardless of validation status.

The reason for this layout is that it is the directory structure used by the RIPE NCC RPKI Validator application. Also note that this application has also fetched all data from subrepositories under the trust anchor to which the archived file belongs, so for example for the APNIC region it will also have fetched the NIR RPKI repositories for Japan and Taiwan.

Notes on data completeness

Apart from a the odd missing day, the data is mostly complete for the entire period covering January 2011 until the present day. We note, however, that the archived repo.tar.gz files before March 10, 2015 miss the trust anchor certificate for the respective RIR repositories, making it harder to validate the data in these files. In addition to this, prior to early 2014, most of the data cannot be validated by modern RPKI validators as it does not comply with the current RPKI standards (e.g. because it uses smaller keys in manifest end-entity certificates than the present day standard requires).

Validating the data using Routinator and Ziggy

If you are interested in independently validating the data, and producing a set of VRPs for a given day, you can use the Ziggy script released by NLnet Labs. This script uses the Routinator RPKI Relying Party software to validate data it downloads directly from the RIPE archive.

Below, we provide detailed instructions on how to install Routinator and how to run Ziggy.

Installing Routinator (and Rust)

The first thing you’ll need to do if you want to use Ziggy is to install Routinator. For detailed information on how to do this, we refer to the README in the Routinator repository, but we reproduce the simple approach below.

Step 1: install Rust

If you have not install the Rust programming language, you’ll need to install that first. Assuming you are on a UNIX-alike system (read: Linux, *BSD, macOS, …), the simplest way to install Rust is to run the following command from your shell prompt:

curl https://sh.rustup.rs -sSf | sh

Follow the instructions on your screen. The default settings are typically fine, as this will install Rust for your own user account only (also making it easy to clean stuff up if you decide you want to get rid of it). It’ll tell you when it’s done by saying: “Rust is installed now. Great!”. You’ll need to reload your shell profile, so log out and back in or re-execute the profile like so (example is for the Bash shell):

. ~/.bash_profile

Step 2: build and install Routinator

Rust comes with its own package management in the form of so-called “crates” that you can install using the “cargo” tool. Using cargo, installing Routinator couldn’t be simpler. Simply execute the following command on your shell prompt and be a little patient while Routinator is compiled:

cargo install routinator

Step 3: initialise Routinator

Because Rust has added itself to your path, you will now immediately be able to run Routinator from the command line. Before Routinator can do its work, you’ll need to initialise it. Do this by invoking it from the command line as shown below:

routinator init

It’ll tell you you’ll have to agree to the ARIN Relying Party Agreement, which you can do by specifying the appropriate flag on the command line.

Getting and running Ziggy

Ziggy requires Python 3 to be installed, and you may optionally need to install the dateutil Python package (which, for example, is available for Ubuntu as the python3-dateutil package).

Step 1: get faketime (optional)

In order to be able to run Routinator on old date, we’ll need to be able to fake time. Fortunately, there is a good tool for that, called faketime. If it isn’t installed yet, you’ll need to install it using your package manager (e.g. yum, apt or brew).

Step 2: get Ziggy

The next thing you’ll need to do is grab Ziggy from Github:

git clone https://github.com/NLnetLabs/ziggy

Step 3: configuring and running Ziggy

Ziggy comes with a sample configuration file called sample-ziggy.conf, which you can actually use straightaway. So let’s go ahead and do that, and let’s run Ziggy for July 1st, 2019:

./ziggy.py -c ./sample-ziggy.conf -d 2019-07-01

This should produce output that looks like this:

Ziggy is processing data for 2019-07-01

Got https://ftp.ripe.net/rpki/afrinic.tal/2019/07/01/repo.tar.gz

No data found at https://ftp.ripe.net/rpki/apnic-afrinic.tal/2019/07/01/repo.tar.gz

No data found at https://ftp.ripe.net/rpki/apnic-arin.tal/2019/07/01/repo.tar.gz

No data found at https://ftp.ripe.net/rpki/apnic-iana.tal/2019/07/01/repo.tar.gz

No data found at https://ftp.ripe.net/rpki/apnic-lacnic.tal/2019/07/01/repo.tar.gz

No data found at https://ftp.ripe.net/rpki/apnic-ripe.tal/2019/07/01/repo.tar.gz

Got https://ftp.ripe.net/rpki/apnic.tal/2019/07/01/repo.tar.gz

Got https://ftp.ripe.net/rpki/arin.tal/2019/07/01/repo.tar.gz

Got https://ftp.ripe.net/rpki/lacnic.tal/2019/07/01/repo.tar.gz

Got https://ftp.ripe.net/rpki/ripencc.tal/2019/07/01/repo.tar.gz

Cleaning out /Users/rijswijk/.rpki-cache/repository ... OK

Cleaning out /Users/rijswijk/.rpki-cache/tals ... OK

Ziggy is processing /tmp/afrinic.tal.tar.gz ... OK (914 objects)

Moving TA to /Users/rijswijk/.rpki-cache/repository/rpki.afrinic.net/ta/ta.cer ...OK

Creating a TAL for this TA ... OK

Ziggy is processing /tmp/apnic.tal.tar.gz ... OK (10705 objects)

Moving TA to /Users/rijswijk/.rpki-cache/repository/rpki.apnic.net/ta/ta-apnic.cer ...OK

Creating a TAL for this TA ... OK

Ziggy is processing /tmp/arin.tal.tar.gz ... OK (6802 objects)

Moving TA to /Users/rijswijk/.rpki-cache/repository/rpki.arin.net/ta/ta.cer ...OK

Creating a TAL for this TA ... OK

Ziggy is processing /tmp/lacnic.tal.tar.gz ... OK (6215 objects)

Moving TA to /Users/rijswijk/.rpki-cache/repository/repository.lacnic.net/ta/ta.cer ...OK

Creating a TAL for this TA ... OK

Ziggy is processing /tmp/ripencc.tal.tar.gz ... OK (35375 objects)

Moving TA to /Users/rijswijk/.rpki-cache/repository/rpki.ripe.net/ta/ta.cer ...OK

Creating a TAL for this TA ... OK

Ziggy thinks the Routinator should travel back to: 2019-07-01 07:20:52

Invoking the Routinator as:

faketime '2019-07-01 07:20:52' ~/.cargo/bin/routinator -vv --logfile ./routinator-2019-07-01.log vrps -n -o ./vrps-2019-07-01.csv -f csvext

Routinator indicated success!

Now that’s quite a bit of output, so let’s go over what Ziggy has told you, starting from the top:

• First, Ziggy will tell you which date it its running for:

Ziggy is processing data for 2019-07-01

• Then, you’ll see it trying to fetch data from RIPE’s archives. It will try to fetch data for each of the five RIRs. Note that between October 2012 and April 2018, the APNIC repository had a somewhat different structure. Ziggy will automatically attempt to fetch both the ‘regular’ APNIC repository as well as the structure used between those dates, so you can safely ignore messages such as the one shown below; these simply indicate that that particular layout for the APNIC data did not exist on the specified date.

No data found at https://ftp.ripe.net/rpki/apnic-afrinic.tal/2019/07/01/repo.tar.gz

• Once Ziggy has all the data it needs, it will clean out Routinator’s cache:

Cleaning out /Users/rijswijk/.rpki-cache/repository ... OK

Cleaning out /Users/rijswijk/.rpki-cache/tals ... OK

• Then, it will unpack the data for each RIR repository and attempt to reconstruct the trust root (the “TAL” or “Trust Anchor Locator”). Routinator needs this in order to know what key to use to start verifying the objects in the RIR’s repository. The example below shows the output for the RIPE repository. First, Ziggy will show how many objects the repository contains:

Ziggy is processing /tmp/ripencc.tal.tar.gz ... OK (35375 objects)

Then, Ziggy will move the TA certificate (if it found one) to the correct location (note that this is inside the Routinator cache):

Moving TA to /Users/rijswijk/.rpki-cache/repository/rpki.ripe.net/ta/ta.cer ... OK

Finally, Ziggy will recreate the TAL for the repository:

Creating a TAL for this TA ... OK

• Ziggy uses the timestamps in the RIR tarballs to work out the exact time of day when the dataset was collected. This time of day is then used when running Routinator. This is important because ROAs have a limited validity, and running Routinator with the wrong time of day may result in some objects not being accepted as valid yet, or as expired. For the example above, Ziggy told us the following:

Ziggy thinks the Routinator should travel back to: 2019-07-21 07:20:52

• Finally, Ziggy will invoke Routinator and shows you exactly which command it used to do so (so you can re-run this command later on, if you want to).

The output will have been written to the directory from which you executed Ziggy into a file called vrps-2019-07-01.csv. If you used the default configuration, the output is the extended CSV output, which lists the URI for the validated object, the associated AS number, the IP prefix, the maximum length as set in the ROA and the validity (not before, not after) of the object.

And that’s it, we hope these instructions have proven useful. Please reach out to one of the paper authors if you have any questions.