For AI agents: every API docs page in one plain-text file is at /docs/llms-full.txt, and the documentation index is at /llms.txt. This page is available as Markdown at https://diffio.ai/docs/developers/guides/clean-up-a-back-catalogue.md, and every docs URL returns Markdown when requested with Accept: text/markdown.

Clean up a back catalogue
Restore every recording in a folder, a few at a time, pick up where a stopped run left off, and total what it used.
How it works
- State file → Your script:
done files, skipped if digest and output match - Your script → Diffio API:
POST /projects, Idempotency-Key from path and content, 3 at a time - Diffio API → Your script:
upload or upload: null, null: it already arrived - Your script → Diffio API:
GET …/generations/{id}, until complete - Your script → Diffio API:
GET …/media, download the restored MP3 - Your script → State file:
done, after each file - Your script → Diffio API:
GET /usage?start=…, the run's total
Build it
- 1
One Idempotency-Key per file
The key hashes the file's relative path with a SHA-256 of its content, under a batch name. Running again finds the same project instead of creating and billing a second one; a changed file gets a new key. Keys belong to your account (and end user, if you send one).
- 2
Skip what already arrived
A repeated
POST /projectsanswersupload: nullonce the file has arrived: skip the upload and wait for the project's first generation (latest_generation). - 3
A few at a time
Three workers keep uploads and downloads moving without flooding your network; restores queue on Diffio's side (see
queueon a generation). Each worker waits at most 2 hours for one file. - 4
Resume from the state file
The state file is named after
BATCHand the source and target folders, so another batch or folder pair never reuses it (give each source its own target). A file counts as done only when its stored digest matches its content and its output exists; outputs are written to a.partfile and renamed. A failed generation fails again with the same key; a newBATCHname starts over with new keys and restores every file again. - 5
Total the usage
GET /usage from the first run's start time. It counts the whole account's generations in that period; send
Diffio-End-Useron the run to count only its own.
The complete script
Pass the folder to read and the folder to write; ep01.wav becomes ep01.wav.restored.mp3 at the same relative path.
"""Restore every recording in a folder, three at a time, and resume where a stopped run left off.
python catalogue.py ./archive ./restored
"""
import hashlib
import os
import sqlite3
import sys
from concurrent.futures import ThreadPoolExecutor, as_completed
from datetime import datetime, timezone
from pathlib import Path
from diffio import DiffioClient, DiffioError
SOURCE, TARGET = Path(sys.argv[1]).resolve(), Path(sys.argv[2]).resolve()
EXTENSIONS = {".mp3", ".wav", ".m4a", ".aac", ".flac", ".ogg", ".mp4", ".mov", ".webm"}
WORKERS = 3
BATCH = "catalogue-v1" # a new name starts a new state file and new projects: everything is restored again
client = DiffioClient() # reads DIFFIO_API_KEY
# One state file per BATCH, SOURCE and TARGET: another batch or folder pair never reuses this progress.
pair = hashlib.sha256(f"{BATCH}\n{SOURCE}\n{TARGET}".encode()).hexdigest()[:16]
state = sqlite3.connect(f"catalogue-{pair}.db")
state.executescript("""
create table if not exists files (path text primary key, digest text, status text, project_id text, error text);
create table if not exists run (started_at text);
""")
with state:
if not state.execute("select 1 from run").fetchone():
state.execute("insert into run values (?)", (datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ"),))
def sha256_of(path):
digest = hashlib.sha256()
with open(path, "rb") as source:
for block in iter(lambda: source.read(1 << 20), b""):
digest.update(block)
return digest.hexdigest()
def restore(path, done_digest):
relative = path.relative_to(SOURCE).as_posix()
output = TARGET / f"{relative}.restored.mp3" # ep01.wav -> ep01.wav.restored.mp3
digest = sha256_of(path) # identity is the content, not the size
if digest == done_digest and output.exists():
return digest, None # finished on an earlier run, unchanged since
key = f"{BATCH}-" + hashlib.sha256(f"{relative}\n{digest}".encode()).hexdigest()
project = client.projects.create(file_name=path.name, size_bytes=path.stat().st_size,
generation={"model": "diffio-flash"}, idempotency_key=key)
if project.upload is not None: # None: an earlier run already uploaded this file
client.projects.upload(project, path)
generation = client.generations.wait(project.id, project.latest_generation.id, timeout=7200)
output.parent.mkdir(parents=True, exist_ok=True)
partial = output.with_name(output.name + ".part")
client.generations.download(project.id, generation.id, path=partial)
os.replace(partial, output) # a stop never leaves a half-written output
return digest, project.id
done = dict(state.execute("select path, digest from files where status = 'done'"))
todo = [path for path in sorted(SOURCE.rglob("*"))
if path.is_file() and path.suffix.lower() in EXTENSIONS and TARGET not in path.parents]
with ThreadPoolExecutor(max_workers=WORKERS) as pool:
futures = {pool.submit(restore, path, done.get(path.relative_to(SOURCE).as_posix())):
path.relative_to(SOURCE).as_posix() for path in todo}
for future in as_completed(futures): # state is written here, on one thread
relative = futures[future]
try:
digest, project_id = future.result()
except DiffioError as error:
digest, project_id, outcome = None, None, ("failed", str(error))
else:
outcome = ("skipped", None) if project_id is None else ("done", None)
if outcome[0] != "skipped":
with state:
state.execute("insert or replace into files values (?, ?, ?, ?, ?)",
(relative, digest, outcome[0], project_id, outcome[1]))
print(outcome[0], relative)
started_at = state.execute("select started_at from run").fetchone()[0]
usage = client.usage.get(start=started_at)
print(f"since {started_at}: {usage.generations} generations, {usage.billable_seconds} s, "
f"${usage.list_price_usd:.2f} at list price")What you get
| File | Run 1 | Run 2 |
|---|---|---|
ep01.wav | done | skipped |
ep02.wav | done | skipped |
ep03.mp3 | uploaded, then stopped | done (no second upload) |
ep04.mp3 | not started | done |
ep01.mp3 | not started | done (its own output) |
- One
.restored.mp3per recording, named after the whole original file name, soep.wavandep.mp3never collide. - No file billed twice: a rerun reuses each unchanged file's project; a changed file is restored again.
- The run's generations, billable seconds and list price from
GET /usage.
Output of python catalogue.py ./archive ./restored (second run)
skipped ep01.wav
skipped ep02.wav
done ep03.mp3
done ep01.mp3
done ep04.mp3
since 2026-11-13T09:00:00Z: 5 generations, 14400 s, $20.00 at list priceAPI reference
- POSTCreate a project (
/projects): Idempotency-Key, generation and upload. - GETGet a generation (
/projects/{project_id}/generations/{generation_id}): status, stage and queue. - GETGet a playback grant (
/projects/{project_id}/generations/{generation_id}/media): The restored file. - GETGet usage (
/usage): Generations, billable seconds and list price.