dewey

Contents

• [What Dewey Is](#what-dewey-is)
• [Credential Enforcement](#credential-enforcement)
• [Download Enforcement](#download-enforcement)
• [Access Method Decision Table](#access-method-decision-table)
• [Authentication](#authentication)
• [Quick Reference: Featured Datasets](#quick-reference-featured-datasets)
• [SafeGraph Global Places Quick Reference](#safegraph-global-places-quick-reference)
• [Additional Resources](#additional-resources)

What Dewey Is

Dewey Data is an academic data marketplace — one institutional Platform Subscription unlocks a catalog of ~300 datasets from ~40 providers (foot traffic, POI, mobility, consumer transactions, real estate, labor). UVA Library and NYU both hold the institutional subscription; SafeGraph and most providers are free under it.

Dewey is not a SQL warehouse like WRDS. Data is delivered as partitioned Parquet/CSV.gz files downloaded via an API key. You discover datasets, read metadata, sample, filter (by date partition + columns), then download. Think "S3 of presigned Parquet links," not "PostgreSQL."

Credential Enforcement

IRON LAW: NEVER GUESS, INVENT, OR HARDCODE THE API KEY

<EXTREMELY-IMPORTANT> The Dewey API key belongs to the user's account (app.deweydata.io → Connections → Add Connection → API Key). It is shown once. You do not have it and cannot derive it.

• ALWAYS ask the user for the key before any real data pull. No exceptions.
• NEVER write a placeholder like apikey = "yourapikey" and run it — it will 401 and waste a round trip. Read from DEWEYAPIKEY env var or a gitignored file (~/.config/dewey/apikey).
• NEVER commit the key, echo it back, or paste it into a script that gets committed.

Guessing or hardcoding the key is NOT HELPFUL — every call 401s, and a committed key is a security incident the user must rotate. </EXTREMELY-IMPORTANT>

Each product (dataset) has its own product path / project ID (prj…), obtained from the dataset page: Get Data → (Skip filtering) → Connect to API / Bulk API → API URL. One API key, many product paths. If you don't have the product path, discover it via the [MCP server](references/mcp.md) (searchdatasets) rather than guessing.

Download Enforcement

IRON LAW: NO BULK DOWNLOAD WITHOUT METADATA + SAMPLE + FILTER FIRST

Before downloading ANY Dewey dataset, you MUST:

1. IDENTIFY the product path and what partitions/columns you actually need
2. META — call getmeta (deweydatapy) / getdownload_info (MCP) to learn partition columns, date range, file count, total size
3. SAMPLE — pull 100 rows (readsample / MCP sampledataset) and INSPECT the schema before committing to a full pull
4. FILTER — restrict by date partition (partitionkeyafter/before) AND columns; for selective pulls use DuckDB COPY TO over the presigned URLs, never download the whole catalog
5. DOWNLOAD the filtered subset, then verify row counts / NULLs / date range on disk

This is not negotiable. Skipping the sample-and-filter step is NOT HELPFUL — Dewey datasets are routinely hundreds of GB to multiple TB; an unfiltered pull burns hours of bandwidth and disk for data you'll immediately throw away.

Dewey Facts

• SafeGraph Patterns is multi-TB; "download everything and filter in pandas" fills the disk before the filter ever runs — counterproductive on its own terms. Use DuckDB COPY TO with a WHERE clause on the remote parquet to pull only the rows/columns you need.
• Column names differ by provider and release (naicscode vs NAICSCODE; openedon may not exist at all). A full pull against guessed columns is the exact incompetence the sample step exists to prevent — readsample(nrows=100) BEFORE the full pull.
• Most datasets are date-partitioned weekly; "all of it" means every weekly file ever shipped. Set partitionkeyafter/before to the study window.
• Presigned links expire in 24h (downloadfiles0). For large multi-day pulls use downloadfiles1 (page-by-page, refreshes links) — a long job on download_files0 dies mid-pull.
• A wrong prj product path 404s or returns someone else's data. Get the path from Connect to API or MCP searchdatasets; hardcoding a guessed path is an unverified claim presented as fact.
• Use deweypy.getdatasetfiles, not deweydatapy.getmeta/getfile_list — the latter's external-api/v3 endpoint is dead (returns non-JSON / 500 → JSONDecodeError), confirmed 2026-06-10. See references/deweypy-client.md.
• The download service throws transient HTTP 500s on individual presigned URLs, and one bad file aborts a whole-batch DuckDB COPY readcsv([...]). For filtered pulls: chunk (~20 files), retry per chunk re-minting fresh URLs, fall back to per-file skip; restartable via per-chunk parquet. Set SET httptimeout=120000; SET http_retries=3;. Worked example in references/deweypy-client.md.
• Some providers gate access behind extra terms (e.g. ConsumerEdge): the web "Get Data" flow shows an "I acknowledge…additional terms" modal you must accept once before the dataset is usable / its prj_ path mints. Don't auto-accept a provider license without the user's OK.
• MCP tools load only at session start. After claude mcp add … dewey-prod, the searchdatasets/sampledataset/etc. tools are NOT available in the current session — start a new session to use them.

Red Flags — STOP Immediately If You're About To:

• **Call downloadfiles* without first calling getmeta + read_sample** → STOP. Meta + sample first.
• Download a dataset with no startdate/enddate / partition filter → STOP. Scope the date range.
• Load a whole remote dataset into a DataFrame → STOP. Use DuckDB COPY TO … (FORMAT PARQUET, PARTITION_BY …) to persist a filtered subset to disk.
• Run a pull with apikey="yourapikey" or any guessed key → STOP. Ask the user; read from env/file.
• Write the API key into a script you'll commit → STOP. Env var or gitignored file only.

Access Method Decision Table

Authentication

Get the key once from app.deweydata.io → Connections → Add Connection → API Key. Store it out of source control:

mkdir -p ~/.config/dewey && echo 'YOUR_KEY' > ~/.config/dewey/apikey && chmod 600 ~/.config/dewey/apikey
# or: export DEWEY_API_KEY=...   (add to .envrc, which should be gitignored)

import os, pathlib
apikey = os.environ.get("DEWEY_API_KEY") or pathlib.Path("~/.config/dewey/apikey").expanduser().read_text().strip()

Institutional login (to browse the catalog / create the key) is via UVA NetBadge (use your UVA email) or NYU SSO. The Platform Subscription is what makes SafeGraph etc. free — see references/datasets.md.

Quick Reference: Featured Datasets

Full catalog (all ~250 datasets): references/catalog.md — every dataset grouped by category with time coverage, row count, size, and download access (machine-readable: references/catalog.csv). Featured-dataset detail + discovery workflow: references/datasets.md.

SafeGraph Global Places Quick Reference

Core POI schema — columns are UPPERCASE, NAICSCODE is a string, BRANDS is a JSON-array string (extract with jsonextractstring(BRANDS,'$[0].safegraphbrand_name')). Always sample before filtering.

Resolved empirically: crypto/Bitcoin ATMs do exist as standalone POIs under NAICSCODE='522320'; all major operators are present. But OPENEDON/CLOSEDON are NULL for BTMs in the current release → it's a cross-section, not a time series. Full details, the 7 BTM operators, and the worked example: references/safegraph-places.md and examples/btmsafegraph_pull.py.

Additional Resources

Reference Files

• references/access-options.md — all download methods (UI, deweypy, deweydatapy, DuckDB, MCP, R), 24h link expiry, partitioning, reading data on disk
• references/deweypy-client.md — deweypy (modern CLI + auth/download) and deweydatapy (getmeta, getfilelist, readsample, download_files0/1) function reference; deweyr for R
• references/duckdb.md — selective remote-Parquet pulls, COPY TO … PARTITION_BY pattern, querying downloaded files
• references/govfiles-business-entity.md — GovFiles US business-entity registry: all 7 table schemas w/ fill rates, the SEC CIK / LEI / FEIN bridge in Identifiers, the missing officers/parties table, the ungraphable Relationships table (1% counterparty key), FILED_ON sentinel dates, launched Jun 2026 / empty changelog — which sparse fields may backfill and which won't, worked DuckDB join
• references/linkup-job-postings.md — LinkUp: 12 tables across 2 products (incl. Extracted Salary, Job Descriptions, Structured Fields, Remote Tag — added Jun 2026, NOT in the stale catalog.csv), join keys (COMPANYID/JOBHASH/BASE_HASH/REQID), salary top-coding at 12k/1M, point-in-time ticker joins, the scrape-log structural-break trap, worked firm-quarter vacancy panel
• references/edi-corporate-actions.md — EDI WCA/RCAN corporate actions: event grain (options=ORs, serials=ANDs), identifier hierarchy + outturn ids, generic label/value slots, future-dated rows, Notes-table coverage gap, worked dividend/CUSIP query + EVENTCD enumeration
• references/mcp.md — Dewey MCP server URL, JSON config, the 9 tools, discovery → schema → sample workflow
• references/datasets.md — featured-dataset catalog, UVA NetBadge / NYU institutional access, discovery workflow
• references/catalog.md + catalog.csv — full enumerated catalog (~250 datasets / 39 partners) by category, with coverage / rows / column count / size / access
• references/schemas.json — full column schemas for all ~250 datasets (keyed by slug → columns[] with name/type/description; 11,264 columns). Look up a dataset's columns here before pulling, instead of a live getdatasetschema call
• references/linkage.md — cross-dataset join-key map (placekey, ticker, cusip/cik, domain, person id, lat/long, fips, zip…) — which datasets combine and on what spine
• references/safegraph-places.md — Global Places schema, NAICS 522320, BTM operator brands, openedon/closedon, the Bitcoin-ATM worked example

Example Files

• examples/btmsafegraphpull.py — acceptance test: filter SafeGraph Global Places to the 7 BTM operator brands + NAICS 522320, verify standalone-POI / open-close coverage, export the US subset to ~/projects/batm/