fetch-chart-data

Once you have identified a chart url like https://ourworldindata.org/grapher/literate-and-illiterate-world-population, there are two key artifacts you can retrieve: the metadata json file, and the data csv file. The urls for both are $CHARTURL.metadata.json and $CHARTURL.csv respectively.

Always fetch the metadata first before fetching the data as it gives important context. Pay special attention to the descriptionKey field if it is given.

Query Parameters

Both endpoints support these query parameters to filter data and reduce response size:

Recommended base parameters: ?useColumnShortNames=true&csvType=filtered

Example with filters:

https://ourworldindata.org/grapher/life-expectancy.csv?useColumnShortNames=true&csvType=filtered&country=USA~GBR&time=2000..2020

Token Optimization

To reduce token usage when working with the data:

1. Remove the Entity column - The CSV has both "Entity" (country name) and "Code" (ISO3). The Code column is sufficient for analysis and joining. Remove Entity to save tokens:

``bash curl -s "$URL" | cut -d',' -f2- ``

1. Filter metadata - The full metadata is verbose. Key fields to extract:

- chart.title, chart.subtitle, chart.note, chart.citation - columns..titleShort, columns..descriptionShort, columns..unit, columns..shortUnit

Skip these verbose fields: titleLong, citationLong, fullMetadata, owidVariableId, lastUpdated, nextUpdate

Metadata

The metadata outlines key information about the chart, as well as information on each time series used in the chart, each of which will be one column in the data file.

The typescript type of the metadata json file is as follows:

export type MetadataColumn = {
    titleShort: string
    titleLong: string
    descriptionShort?: string
    descriptionKey?: string // a markdown bulleted list (one `- ` item per line), curated by experts at Our World In Data to collect important information or caveats about this data. If given, it might make sense to surface this information to the user.
    descriptionProcessing?: string // notes about how this data was processed by Our World In Data in case it is not a straightforward republishing from the original source
    shortUnit?: string
    unit?: string
    timespan?: string // timespan of the data covered in the form `YYYY-YYYY`
    tolerance?: number // default tolerance for this time series. Data from Our World In Data is sometimes sparse for some countries. In these cases, OWID charts use this tolerance threshold to display values on charts from neighboring years up to this limit
    type?: string // Numeric, Categorical, Ordinal, Integer
    conversionFactor?: number // Conversion factor that was used internally. This is already applied for the data in the csv, don't apply it again
    shortName?: string // short column name that matches the csv column names when the useColumnShortNames query param is set to true
    lastUpdated?: string
    nextUpdate?: string
    citationShort: string
    citationLong: string
    fullMetadata: string // URL of the expanded metadata for this time series
}

export type ChartMetadata = {
    title?: string
    subtitle?: string
    note?: string
    xAxisLabel?: string
    yAxisLabel?: string
    citation: string
    originalChartUrl?: string
    selection: string[]
}

export type GrapherMetadataResponse = {
    chart: ChartMetadata
    columns: Record<string, MetadataColumn>
    /** Date in YYYY-MM-DD format */
    dateDownloaded: string
    /** Filtered grapher query params, or undefined if no params */
    activeFilters?: Record<string, string>
}

CSV Structure

The high level structure of the CSV file is that each row is an observation for an entity (usually a country or region) and a timepoint (usually a year).

The first two columns in the CSV file are "Entity" and "Code". "Entity" is the name of the entity (e.g. "United States"). "Code" is the OWID internal entity code that we use if the entity is a country or region. For normal countries, this is the same as the iso alpha-3 code of the entity (e.g. "USA") - for non-standard countries like historical countries these are custom codes.

The third column is either "Year" or "Day". If the data is annual, this is "Year" and contains only the year as an integer. If the column is "Day", the column contains a date string in the form "YYYY-MM-DD".

Note that useColumnShortNames=true lowercases these first three headers, so with the recommended base parameters they arrive as entity,code,year (or entity,code,day). Match column names case-insensitively rather than hardcoding either form.

The other columns are the data columns, which are the time series that power the chart.

Prefer to download the CSV and metadata into a file and process it from there.