Run an indexer

The indexer turns the blockchain into something you can query. It reads the Cabdell application’s events, keeps them in a SQLite database, derives the tables a reader needs (articles, authors, reviews, comments, reputation…), fetches and checks the content from IPFS, and serves everything as the Indexer API. The chain is the truth; the database is a cache that can always be rebuilt.

The source of Cabdell’s indexer is not public yet (it lives in projects/indexer of Cabdell’s repository, whose README.md is the detailed reference); we intend to publish it under an open licence. This page describes it for those who run it with us, and so that anyone can see what it does and build an equivalent from the published rules, for instance to check cabdell.press (see Verify without trusting cabdell.press). Terms such as round, event and pin are explained in the Glossary.

Algorand application calls (ARC-28 logs)
  -> algokit-subscriber (watermark, catch-up)
  -> ARC-28 decoder built from the ARC-56 file        src/events.ts
  -> chain_events (append-only)                       src/db.ts
  -> projections, one transaction per round           src/projections.ts
  -> content checks (re-hash, header against chain)   src/content.ts
  -> REST API (read-only JSON)                        src/api.ts, src/queries.ts

One process serves one network and one application, with its own database.

Requirements

• Node 24 or later (engines in package.json; developed and accepted on Node 26.5.1, which the Docker image uses).
• No build step: Node runs the TypeScript sources directly. No native modules: the database is Node’s built-in node:sqlite.
• Network access to an Algorand node, and preferably an Algorand indexer for fast catch-up (both named in spec/networks.json).
• The repository: the indexer reads spec/networks.json and the contract’s ARC-56 file from it.

cd projects/indexer
npm ci
npm run check        # type check (tsc --noEmit)
npm test             # offline tests
npm run fmt:check    # formatting

Quick start on TestNet

cd projects/indexer
ARIADNE_NETWORK=testnet node src/main.ts serve

In PowerShell, set the variable first: $env:ARIADNE_NETWORK = "testnet"; node src/main.ts serve.

The indexer checks the node’s genesis hash, starts at the application’s creation round (startRound in the manifest), catches up through the Algorand indexer, and serves the API on http://127.0.0.1:3000:

curl http://127.0.0.1:3000/status

Without a Kubo node it reads content from the manifest’s IPFS gateway, verifying every block, and keeps no IPFS copy of its own. The database is data/testnet.sqlite inside projects/indexer unless you set ARIADNE_DB.

Configuration

The network entry of spec/networks.json gives the application id, the Algorand node and indexer, the IPFS gateway and the outside services (see Networks and addresses). Environment variables select the network and override or complete it. Numbers must be non-negative integers, or the indexer refuses to start.

Variable	Default	Meaning
ARIADNE_NETWORK	localnet	the key of spec/networks.json to serve
ARIADNE_NETWORKS_JSON	the repository’s spec/networks.json	another manifest file
ARIADNE_DB	data/<network>.sqlite	the database file; {appId} in it is replaced by the application id, so a new deployment starts with a new database
ARIADNE_HOST	127.0.0.1	the address the API listens on (the Docker image sets 0.0.0.0)
ARIADNE_PORT	3000	the API port
ARIADNE_POLL_SECONDS	20 when reading through the Algorand indexer, else 2	pause between polls once caught up
ARIADNE_SYNC	indexer for a public network with algoIndexer, else algod	how the chain is read (see Ingestion)
ARIADNE_MAX_ROUNDS	500	rounds read from the node per poll while catching up
ARIADNE_ALGOD	the manifest’s algod	Algorand node
ARIADNE_ALGOD_TOKEN	the manifest’s algodToken	its API token
ARIADNE_APP_ID	the manifest’s appId	the application
ARIADNE_START_ROUND	the manifest’s startRound	where an empty database starts
ARIADNE_ALGO_INDEXER	the manifest’s algoIndexer	Algorand indexer for catch-up; empty: every round is read from the node
ARIADNE_IPFS_GATEWAY	the manifest’s ipfsGateway	trustless gateway used as content source; empty: none
ARIADNE_CONTENT_DIR	unset	a local content source: <dir>/<cid>/ holds an article folder, <dir>/<cid> a review or comment (tests, LocalNet)
ARIADNE_KUBO_API	unset	Kubo RPC address (for example http://kubo:5001): content source and second copy. Never expose this RPC publicly
ARIADNE_CONTENT_RECHECK_SECONDS	21600 (6 hours)	how often a content record is checked again
ARIADNE_CONTENT_PER_CYCLE	20	article CIDs checked per content cycle (review and comment texts: five times as many)
ARIADNE_CONTENT_CONCURRENCY	4	fetches at the same time (also pins at the same time)
ARIADNE_CONTENT_TIMEOUT_SECONDS	30	seconds before a fetch through Kubo gives up
ARIADNE_PINS_PER_CYCLE	20	pins attempted per pin cycle
ARIADNE_PIN_QUOTA_MB, ARIADNE_PIN_QUOTA_DAYS	200, 30	at most this much of one address’s content is pinned by our Kubo per window (see Background loops); 0 MB: no quota
ARIADNE_PROGRAM_CHECK	on	0 skips the check of the application’s program at start (see below)
ARIADNE_STALL_SECONDS	300	/health answers 503 after this long without a successful ingestion poll
ARIADNE_APP_URL	the manifest’s appUrl	public origin of the web app; ORCID and DOI records must link back under it
ARIADNE_ORCID_API	the manifest’s orcidApi	ORCID public API; empty: ORCID iDs are not checked
ARIADNE_ZENODO_API	the manifest’s zenodoApi	this network’s Zenodo
ARIADNE_DATACITE_API	the manifest’s dataciteApi	DataCite API; empty: DOIs other than Zenodo’s stay “declared”
ARIADNE_OPENALEX_API	https://api.openalex.org	OpenAlex, for verified ORCID iDs; empty turns it off
ARIADNE_ORCID_CLIENT_ID, ARIADNE_ORCID_CLIENT_SECRET	unset	ORCID /read-public credentials, for higher rate limits; without them the public API is used anonymously
ARIADNE_ORCID_TOKEN_URL	https://orcid.org/oauth/token	where the ORCID token is requested
ARIADNE_VERIFY_PER_CYCLE	10	ORCID iDs and DOIs checked per verification cycle, each
ARIADNE_VERIFY_RECHECK_SECONDS	604800 (7 days)	how often a settled ORCID or DOI result is checked again
ARIADNE_VERIFY_TIMEOUT_SECONDS	20	timeout of one request to an outside service
ARIADNE_OPENALEX_API_KEY	unset	the operator’s OpenAlex key, sent to OpenAlex only, as a bearer token (never in a URL)
ARIADNE_CONTACT_EMAIL	unset	the operator’s contact: named in the User-Agent of every outside request and in every statement of reasons
ARIADNE_CONTACT_URL	<appUrl>/<network>/legal (<appUrl>/legal on MainNet, TestNet’s page for the sandbox)	the page where a decision to withhold is contested
ARIADNE_WITHHELD	the repository’s spec/withheld.json	the list of what this service withholds, read again whenever it changes (see below)
ARIADNE_DENYLIST_DIR	unset	where the gateway’s deny lists are written (a directory Kubo reads)
ARIADNE_BADBITS_URL	unset	the IPFS Bad Bits deny list, kept beside ours and refreshed daily (one indexer per node)
ARIADNE_PRESERVE_KEY, ARIADNE_PRESERVE_DIR	unset	the key (32 bytes, hex or base64) and the directory of terrorist content preserved for six months (see below)
ARIADNE_GC_HOURS	24	hours between garbage collections of the Kubo node; 0: none on schedule

Commands

Run node src/main.ts <command> from projects/indexer (ingest, rebuild, serve and content also exist as npm scripts).

Command	What it does
serve	The normal way to run it: the API, the ingestion loop and every background loop.
ingest	Catches up with the chain once, without the API, and stops. ingest --follow keeps polling.
rebuild	Drops every projection table and rebuilds it from the stored chain_events (after a schema change or a fix). The result is identical to ingesting round by round.
content	Fetches and checks the article content that is due, once.
pins	Pins on the Kubo node what is due, once (needs ARIADNE_KUBO_API).
verify	Checks the declared ORCID iDs and DOIs that are due against ORCID, Zenodo, DataCite and OpenAlex, once.
reindex	Drops and rebuilds the search indexes from the database (serve keeps them current).
status	Prints the network, application, genesis, watermark, schema version and row counts as JSON.
unseal <file> <out.car>	Decrypts content preserved under the TCO Regulation, for an authority (needs ARIADNE_PRESERVE_KEY).

What the indexer refuses

It stops with an error, rather than serving wrong data, when:

• the manifest’s appId is 0 (“deploy the contract first, or set ARIADNE_APP_ID”);
• the Algorand node reports a genesis hash different from the manifest’s;
• the database was built for another genesis or another application;
• the application runs another program than the one this indexer was built for: at start it asks the node for the application’s approval program (one request) and compares its SHA-256 with the compiled program in the contract’s ARC-56 file (“this indexer was built for another version of the contract”). A new version of the contract is a new application with its own indexer: this version reads contract v4 (D-163) only. The check is skipped when the ARC-56 file holds no compiled program, or with ARIADNE_PROGRAM_CHECK=0 for a node that cannot answer it;
• the database schema is newer than the code.

Ingestion

Each poll asks for the application’s calls (and for the payment notes of public lists) after the last round stored, the watermark. With an Algorand indexer configured, a poll catches up by up to 100 000 rounds through it; otherwise every block is read from the node, ARIADNE_MAX_ROUNDS at a time, which needs an archival node. A new database starts just before startRound, so it never scans the years before Cabdell existed.

On a public network (TestNet, MainNet) with an Algorand indexer, every poll goes through that indexer alone: one request for the round it has reached and one search per filter up to it, never whole blocks and never the node’s status, every 20 seconds. Public endpoints such as Nodely’s free tier count every request: this is about 9 a minute, where reading blocks every 2 seconds took 48. Development chains, which have no Algorand indexer, keep reading blocks from the node; ARIADNE_SYNC=algod or indexer chooses explicitly. A failed poll is retried after the interval doubled each time (at most 10 minutes, and at least a minute after a 429 or 403 answer), and every request carries the User-Agent Cabdell-indexer/1.0 (+https://cabdell.press; mailto:<ARIADNE_CONTACT_EMAIL>) (without the e-mail when it is not set).

Every round is one SQLite transaction: the new events, the projections and the watermark together. Storing an event twice changes nothing, so a crash or a restart simply resumes from the watermark. The projections apply the same rules as the contract’s reference implementation; an event stream that breaks them (which must never happen) aborts the round, the indexer logs it and keeps retrying, and /health turns 503 after ARIADNE_STALL_SECONDS.

No event is skipped. Contract v4 numbers its events from 1 without gaps (event_seq), and the indexer stores the number with each event. When the next event read does not follow the last one stored, some were missed (a public endpoint may answer from an Algorand indexer that is behind): the poll fails with “missing events”, the watermark does not move, and the next poll reads the same rounds again. rebuild checks the same sequence in the stored events.

Background loops

serve runs the ingestion loop and six background loops side by side, so a slow outside service never delays a new article. Each background loop pauses 10 seconds between cycles.

Loop	What it does
ingestion	Polls the chain (see above), every ARIADNE_POLL_SECONDS once caught up.
withheld	Applies the list of what this service withholds (see below) within seconds of an edit of its file.
jury	Opens, draws and settles the juries of disputes.
content	Checks the article CIDs that are due: never-checked ones first, newest article first, then re-checks, least recently checked first; up to ARIADNE_CONTENT_PER_CYCLE per cycle, ARIADNE_CONTENT_CONCURRENCY at a time. Also reads the text of new reviews and comments for search, keeping a text only when its bytes hash to its CID.
pins	With ARIADNE_KUBO_API, pins the second copy: every article CID checked as valid and every review and comment CID. Each pin may take 60 seconds; failures are retried after 1, 2, 4… minutes, at most a day. At most ARIADNE_PIN_QUOTA_MB (200 MB) of one address’s content (the article’s submitter, the reviewer, the commenter) is pinned in any ARIADNE_PIN_QUOTA_DAYS (30 days); beyond it the content is still checked and served but not pinned by us, and is tried again a day later. The node also collects its garbage every ARIADNE_GC_HOURS, and the Bad Bits list is refreshed once a day.
verify	Checks declared ORCID iDs and DOIs, and asks OpenAlex for what the seal needs (see below).
search	Recomputes the citations between articles and the Thread Score, then the Peer-reviewed seal, then the search indexes, each only when something it depends on changed. Thread percentiles are also recomputed every day.

Content sources are tried in this order: ARIADNE_CONTENT_DIR, then Kubo (ARIADNE_KUBO_API), then the manifest’s IPFS gateway as a trustless gateway (/ipfs/<cid>?format=car, every block verified, the folder re-hashed with the canonical parameters). With no source at all, every article is unavailable: served with a warning, never hidden. The validation statuses are described in The article header.

Database

One SQLite file per network and application, in WAL mode (so -wal and -shm files sit beside it). The schema version is 19 (D-164: the table of withheld CIDs and the objections to profiling; the migration deletes the e-mail addresses and the data of unverified or undeclared ORCID iDs kept before). Opening an older database migrates it in place; a newer one is refused.

Kind	Tables	Rebuilt by
Base tables, kept across rebuilds	meta, chain_events, watermark, content, pins, orcid_checks, doi_checks, post_texts, coauthor_checks, the jury’s jury_cases, jury_seats, jury_answers, notification_ids, and since schema 19 withheld_cids	never: they are the inputs (events) or cached results of IPFS and outside services
Projections, from the events	fields, governance, articles, versions, coauthors, article_votes, reviews, review_votes, comments, comment_votes, mentions, flags, disputes, reputation, reputation_changes, claims, notifications, identities, article_dois, doi_declarations, follows, lists, list_items, journal_editors, journal_volumes, watches, flag_notes; since schema 18, dispute_outcomes, flag_settlements, damping and governance_pending; since schema 19, objections	rebuild
Derived views	citations, article_duplicates and article_impact (Thread Score); article_seals, review_qualifications, seal_state (the seal); search_articles, search_posts, search_state, search_posts_done (search)	the serve loop; reindex for search

Since schema 17 (D-162) every stored event keeps its raw bytes as the blockchain holds them, and rebuild decodes them again, so a fix of the decoder fixes every event; a notification keeps its number across rebuilds (notification_ids), so browsers and the push sender never see one twice.

chain_events holds every decoded event, keyed by transaction id and log position, with its round, name, fields and, since schema 18, its event_seq. Every projection row points back to the event that produced it. Public lists are stored there too, as ListNote rows, so that rebuild replays them in chain order.

Nothing in the database needs a backup: delete it and the indexer rebuilds it from the chain (from startRound), then asks IPFS and the outside services again.

Outside services

Besides Algorand and IPFS, the indexer asks outside services about the declarations people make on chain. Their answers are cached in the base tables; no reputation is ever derived from them. Every request names Cabdell in its User-Agent.

Service	Used for	Configured by
ORCID public API	Is a declared ORCID iD verified (its public record lists the person’s profile page)? The record’s names and public contact data	orcidApi / ARIADNE_ORCID_API; optional credentials
Zenodo records API	Is a declared Zenodo DOI published, and identical to the article page (and, for one version, its CID)?	the instance of the DOI’s prefix: 10.5281 on zenodo.org, 10.5072 on sandbox.zenodo.org (the network’s own zenodoApi when the prefix is its own)
DataCite	The same check for other DOIs	dataciteApi / ARIADNE_DATACITE_API
OpenAlex	Citation counts and works per field of verified ORCID iDs; works shared by a reviewer and an author (a conflict of interest for the seal)	ARIADNE_OPENALEX_API

Schedule: a settled result is checked again weekly (ARIADNE_VERIFY_RECHECK_SECONDS); during the two days after a declaration that is not verified yet, every ten minutes; a failed request is retried after 1, 2, 4… minutes, at most a day, and the last result is kept meanwhile. Reviewer-author pairs are checked on OpenAlex every 30 days. A service that answers 429 (or 503 with Retry-After), or says its allowance is spent (X-RateLimit-Remaining: 0), is not asked again before the time its Retry-After or X-RateLimit-Reset gives (a minute by default), and the iDs waiting meanwhile count no failure.

What is kept of an ORCID record (D-164): never its e-mail addresses; its names, websites, countries, current employments and OpenAlex’s counts only while the iD is verified (they are deleted when it stops being verified); and nothing at all once the iD is no longer declared, with the works in common counted for it. The API never returns an iD that is not verified.

Turning them off

To stop	Set	Effect
ORCID	ARIADNE_ORCID_API= (empty)	declared iDs stay pending
DataCite	ARIADNE_DATACITE_API= (empty)	DOIs that are not Zenodo’s stay declared
OpenAlex	ARIADNE_OPENALEX_API= (empty)	no citation counts; the seal’s OpenAlex path to expertise and its shared-works conflict check are not available
Every ORCID and DOI lookup	ARIADNE_APP_URL= (empty)	ORCID iDs are not checked, DOIs are declared (malformed ones invalid), and links to article pages are not recognised as citations

Zenodo DOIs are checked at the Zenodo instance of their prefix even when ARIADNE_ZENODO_API is empty; only an empty ARIADNE_APP_URL stops those lookups. An indexer with outside services turned off computes the seal and the citations from less information, so its results can differ from those of cabdell.press.

Docker

deploy/indexer.Dockerfile builds an image from the repository root: Node 26.5.1 (pinned by digest), production dependencies only, the sources, the ARC-56 file and spec/. It runs node src/main.ts serve as the node user, with ARIADNE_HOST=0.0.0.0, ARIADNE_PORT=3000 and a volume at /data.

docker build -f deploy/indexer.Dockerfile -t ariadne-indexer .
docker run -d --name ariadne-indexer-testnet \
  -e ARIADNE_NETWORK=testnet \
  -e "ARIADNE_DB=/data/testnet-{appId}.sqlite" \
  -v ariadne-indexer-testnet:/data \
  -v "$PWD/spec:/app/spec:ro" \
  -p 127.0.0.1:3000:3000 \
  ariadne-indexer

Mounting spec/ read-only, as the hosting stack does, lets a change to the manifest take effect with a restart instead of a rebuild. Other commands run in the same image:

docker exec ariadne-indexer-testnet node src/main.ts status

The hosting stack adds a Kubo node (ARIADNE_KUBO_API=http://kubo:5001) and a Docker health check that reads /health every 60 seconds; see Host the full stack.

What this service withholds

The indexer reads a list of what the service it belongs to no longer shows (ARIADNE_WITHHELD, a JSON file; its format is described in spec/withheld.json), and reads it again within seconds whenever the file changes, without a restart. An entry names a record (an article, a review, a comment, a list or journal, a journal volume, a flag’s note, a juror’s reasons, a person, or a raw CID), the application it belongs to and its ground (order, notice, terms, data_protection, copyright, csam, terrorist, other). Then:

• the API leaves out its texts, names and CIDs and puts a statement of reasons in their place (see the Indexer API); GET /withheld is the transparency report;
• the indexer purges the texts it kept of it (content, review and comment texts, search rows, list texts, notes, reasons, a person’s ORCID data) and never fetches, checks or pins it again;
• with Kubo, it drops the pins and has the node collect its garbage, and appends the CIDs to the gateway’s deny list (<ARIADNE_DENYLIST_DIR>/ariadne-<network>.deny; Kubo then answers 410 Gone and neither serves nor fetches them; Kubo reads a new list file after its next restart, and follows lines appended later);
• the people concerned get a notification of kind withheld (unless the entry says notify: false);
• terrorist content is first exported from Kubo and kept encrypted (AES-256-GCM, ARIADNE_PRESERVE_KEY) in ARIADNE_PRESERVE_DIR for six months, as the TCO Regulation requires, then deleted; without the key it stays pinned, blocked, until the key is set. Child sexual abuse material is kept nowhere.

How the operator of cabdell.press uses it in an emergency is in deploy/README.md (section D5) of the repository.

Logs

The indexer writes one line per event worth knowing: polls that found calls or caught up, content checked, pins, ORCID and DOI results, what it withheld, errors, and API requests that failed with a 5xx status, by their path only (never the query, the client’s address or its User-Agent). No access log is kept, by the indexer or by the reverse proxy of the hosting stack (see Host the full stack).

LocalNet

For development, start the repository’s LocalNet and deploy the contract (see Networks and addresses), then:

cd projects/indexer
ARIADNE_CONTENT_DIR=/path/to/content node src/main.ts serve

ARIADNE_NETWORK defaults to localnet. Point ARIADNE_CONTENT_DIR at the same folder the web app uses for LocalNet (its own ARIADNE_CONTENT_DIR): the web app’s development pin target unpacks each published folder there, and the indexer validates it from there. The LocalNet acceptance test catches up from round 0 and compares the projections with every Article and CoAuthor box: populate the chain with the contract suite (poetry run pytest tests/localnet in projects/contracts), then run npm run test:localnet.