> For the complete documentation index, see [llms.txt](https://diffio.ai/llms.txt). This page is also available as HTML at https://diffio.ai/docs/developers/guides/clean-up-a-back-catalogue/.

# Clean up a back catalogue

Restore every recording in a folder, a few at a time, pick up where a stopped run left off, and total what it used.

## How it works

1. State file → Your script: `done files`, skipped if digest and output match
2. Your script → Diffio API: `POST /projects`, Idempotency-Key from path and content, 3 at a time
3. Diffio API → Your script: `upload or upload: null`, null: it already arrived
4. Your script → Diffio API: `GET …/generations/{id}`, until complete
5. Your script → Diffio API: `GET …/media`, download the restored MP3
6. Your script → State file: `done`, after each file
7. Your script → Diffio API: `GET /usage?start=…`, the run's total

**Stop the script at any point; the next run picks up from the state file and the same keys.**

## Build it

### 1. One Idempotency-Key per file

The key hashes the file's relative path with a SHA-256 of its content, under a batch name. Running again finds the same project instead of creating and billing a second one; a changed file gets a new key. Keys belong to your account (and end user, if you send one).

### 2. Skip what already arrived

A repeated `POST /projects` answers `upload: null` once the file has arrived: skip the upload and wait for the project's first generation (`latest_generation`).

### 3. A few at a time

Three workers keep uploads and downloads moving without flooding your network; restores queue on Diffio's side (see `queue` on a generation). Each worker waits at most 2 hours for one file.

### 4. Resume from the state file

The state file is named after `BATCH` and the source and target folders, so another batch or folder pair never reuses it (give each source its own target). A file counts as done only when its stored digest matches its content and its output exists; outputs are written to a `.part` file and renamed. A failed generation fails again with the same key; a new `BATCH` name starts over with new keys and restores every file again.

### 5. Total the usage

[GET /usage](https://diffio.ai/docs/api-reference/usage-summary/) from the first run's start time. It counts the whole account's generations in that period; send `Diffio-End-User` on the run to count only its own.

### The complete script

Pass the folder to read and the folder to write; `ep01.wav` becomes `ep01.wav.restored.mp3` at the same relative path.

**catalogue.py**

```python
"""Restore every recording in a folder, three at a time, and resume where a stopped run left off.

python catalogue.py ./archive ./restored
"""
import hashlib
import os
import sqlite3
import sys
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone
from pathlib import Path

from diffio import DiffioClient, DiffioError

SOURCE, TARGET = Path(sys.argv[1]).resolve(), Path(sys.argv[2]).resolve()
EXTENSIONS = {".mp3", ".wav", ".m4a", ".aac", ".flac", ".ogg", ".mp4", ".mov", ".webm"}
WORKERS = 3
BATCH = "catalogue-v1"  # a new name starts a new state file and new projects: everything is restored again
client = DiffioClient()  # reads DIFFIO_API_KEY

# One state file per BATCH, SOURCE and TARGET: another batch or folder pair never reuses this progress.
pair = hashlib.sha256(f"{BATCH}\n{SOURCE}\n{TARGET}".encode()).hexdigest()[:16]
state = sqlite3.connect(f"catalogue-{pair}.db")
state.executescript("""
    create table if not exists files (path text primary key, digest text, status text, project_id text, error text);
    create table if not exists run (started_at text);
""")
with state:
    if not state.execute("select 1 from run").fetchone():
        state.execute("insert into run values (?)", (datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),))


def sha256_of(path):
    digest = hashlib.sha256()
    with open(path, "rb") as source:
        for block in iter(lambda: source.read(1 << 20), b""):
            digest.update(block)
    return digest.hexdigest()


def restore(path, done_digest):
    relative = path.relative_to(SOURCE).as_posix()
    output = TARGET / f"{relative}.restored.mp3"  # ep01.wav -> ep01.wav.restored.mp3
    digest = sha256_of(path)  # identity is the content, not the size
    if digest == done_digest and output.exists():
        return digest, None  # finished on an earlier run, unchanged since
    key = f"{BATCH}-" + hashlib.sha256(f"{relative}\n{digest}".encode()).hexdigest()
    project = client.projects.create(file_name=path.name, size_bytes=path.stat().st_size,
                                     generation={"model": "diffio-flash"}, idempotency_key=key)
    if project.upload is not None:  # None: an earlier run already uploaded this file
        client.projects.upload(project, path)
    generation = client.generations.wait(project.id, project.latest_generation.id, timeout=7200)
    output.parent.mkdir(parents=True, exist_ok=True)
    partial = output.with_name(output.name + ".part")
    client.generations.download(project.id, generation.id, path=partial)
    os.replace(partial, output)  # a stop never leaves a half-written output
    return digest, project.id


done = dict(state.execute("select path, digest from files where status = 'done'"))
todo = [path for path in sorted(SOURCE.rglob("*"))
        if path.is_file() and path.suffix.lower() in EXTENSIONS and TARGET not in path.parents]

with ThreadPoolExecutor(max_workers=WORKERS) as pool:
    futures = {pool.submit(restore, path, done.get(path.relative_to(SOURCE).as_posix())):
               path.relative_to(SOURCE).as_posix() for path in todo}
    for future in as_completed(futures):  # state is written here, on one thread
        relative = futures[future]
        try:
            digest, project_id = future.result()
        except DiffioError as error:
            digest, project_id, outcome = None, None, ("failed", str(error))
        else:
            outcome = ("skipped", None) if project_id is None else ("done", None)
        if outcome[0] != "skipped":
            with state:
                state.execute("insert or replace into files values (?, ?, ?, ?, ?)",
                              (relative, digest, outcome[0], project_id, outcome[1]))
        print(outcome[0], relative)

started_at = state.execute("select started_at from run").fetchone()[0]
usage = client.usage.get(start=started_at)
print(f"since {started_at}: {usage.generations} generations, {usage.billable_seconds} s, "
      f"${usage.list_price_usd:.2f} at list price")
```

**catalogue.mjs**

```javascript
// Restore every recording in a folder, three at a time, and resume where a stopped run left off.
// node catalogue.mjs ./archive ./restored
import { createHash } from "node:crypto";
import { createReadStream, existsSync, mkdirSync, readFileSync, readdirSync, renameSync, statSync, writeFileSync } from "node:fs";
import { basename, dirname, extname, join, resolve, sep } from "node:path";
import { DiffioClient, DiffioError } from "diffio";

const [SOURCE, TARGET] = process.argv.slice(2).map((folder) => resolve(folder));
const EXTENSIONS = new Set([".mp3", ".wav", ".m4a", ".aac", ".flac", ".ogg", ".mp4", ".mov", ".webm"]);
const WORKERS = 3;
const BATCH = "catalogue-v1"; // a new name starts a new state file and new projects: everything is restored again
const client = new DiffioClient(); // reads DIFFIO_API_KEY
const sha256 = (text) => createHash("sha256").update(text).digest("hex");

// One state file per BATCH, SOURCE and TARGET: another batch or folder pair never reuses this progress.
const STATE_FILE = `catalogue-${sha256(`${BATCH}\n${SOURCE}\n${TARGET}`).slice(0, 16)}.json`;
const state = existsSync(STATE_FILE)
  ? JSON.parse(readFileSync(STATE_FILE, "utf8"))
  : { source: SOURCE, target: TARGET, started_at: new Date().toISOString().replace(/\.\d+Z$/, "Z"), files: {} };
const save = () => { // write then rename, so a stop never leaves half a file
  writeFileSync(`${STATE_FILE}.tmp`, JSON.stringify(state, null, 2));
  renameSync(`${STATE_FILE}.tmp`, STATE_FILE);
};
save();

const sha256Of = (path) => new Promise((done, fail) => {
  const hash = createHash("sha256");
  createReadStream(path).on("data", (block) => hash.update(block)).on("end", () => done(hash.digest("hex"))).on("error", fail);
});

async function restore(relative) {
  const path = join(SOURCE, relative);
  const output = join(TARGET, `${relative}.restored.mp3`); // ep01.wav -> ep01.wav.restored.mp3
  const digest = await sha256Of(path); // identity is the content, not the size
  const previous = state.files[relative];
  if (previous?.status === "done" && previous.digest === digest && existsSync(output)) return null; // unchanged
  const project = await client.projects.create(
    { file_name: basename(path), size_bytes: statSync(path).size, generation: { model: "diffio-flash" } },
    { idempotencyKey: `${BATCH}-${sha256(`${relative}\n${digest}`)}` }
  );
  if (project.upload) await client.projects.upload(project, path); // null: an earlier run already uploaded it
  const generation = await client.generations.wait(project.id, project.latest_generation.id, { timeout: 7200 });
  mkdirSync(dirname(output), { recursive: true });
  await client.generations.download(project.id, generation.id, { path: `${output}.part` });
  renameSync(`${output}.part`, output); // a stop never leaves a half-written output
  return { status: "done", digest, project_id: project.id };
}

const todo = readdirSync(SOURCE, { recursive: true })
  .filter((relative) => EXTENSIONS.has(extname(relative).toLowerCase()))
  .filter((relative) => statSync(join(SOURCE, relative)).isFile() && !join(SOURCE, relative).startsWith(TARGET + sep))
  .sort();

const queue = [...todo];
async function worker() {
  for (let relative = queue.shift(); relative; relative = queue.shift()) {
    let result;
    try {
      result = await restore(relative);
    } catch (error) {
      if (!(error instanceof DiffioError)) throw error;
      result = { status: "failed", error: error.message };
    }
    if (result) {
      state.files[relative] = result;
      save();
    }
    console.log(result ? result.status : "skipped", relative);
  }
}
await Promise.all(Array.from({ length: WORKERS }, worker));

const usage = await client.usage.get({ start: state.started_at });
console.log(
  `since ${state.started_at}: ${usage.generations} generations, ${usage.billable_seconds} s, ` +
    `$${usage.list_price_usd.toFixed(2)} at list price`
);
```

## What you get

| File | Run 1 | Run 2 |
| --- | --- | --- |
| `ep01.wav` | done | **skipped** |
| `ep02.wav` | done | **skipped** |
| `ep03.mp3` | uploaded, then stopped | **done (no second upload)** |
| `ep04.mp3` | not started | **done** |
| `ep01.mp3` | not started | **done (its own output)** |

- One `.restored.mp3` per recording, named after the whole original file name, so `ep.wav` and `ep.mp3` never collide.
- No file billed twice: a rerun reuses each unchanged file's project; a changed file is restored again.
- The run's generations, billable seconds and list price from `GET /usage`.

**Output of `python catalogue.py ./archive ./restored` (second run)**

**Terminal**

```bash
skipped ep01.wav
skipped ep02.wav
done ep03.mp3
done ep01.mp3
done ep04.mp3
since 2026-11-13T09:00:00Z: 5 generations, 14400 s, $20.00 at list price
```

## API reference

- POST [Create a project](https://diffio.ai/docs/api-reference/create-project/) (`/projects`): Idempotency-Key, generation and upload.
- GET [Get a generation](https://diffio.ai/docs/api-reference/get-generation/) (`/projects/{project_id}/generations/{generation_id}`): status, stage and queue.
- GET [Get a playback grant](https://diffio.ai/docs/api-reference/get-generation-media/) (`/projects/{project_id}/generations/{generation_id}/media`): The restored file.
- GET [Get usage](https://diffio.ai/docs/api-reference/usage-summary/) (`/usage`): Generations, billable seconds and list price.

## Related pages

- Previous: [Compare Flash and Pro](https://diffio.ai/docs/developers/guides/compare-flash-and-pro/): One project → two generations
- Next: [Bill your users from usage](https://diffio.ai/docs/developers/guides/bill-users-from-usage/): Usage per end user → invoice
