Database management#

The polartools.manage_database module wraps databroker-pack and suitcase to import/export Bluesky catalogs. It’s how you

  • distribute a beamtime’s data as a self-contained directory,

  • ingest someone else’s exported catalog into your local databroker,

  • export to plain CSV/JSON for downstream analysis.

Scope: these helpers work with databroker catalogs only — tiled is not yet supported here (issue #60).

Export a catalog to portable msgpack#

Pack a subset of a catalog into a folder you can copy to another machine:

from polartools.manage_database import to_databroker

to_databroker(
    db,  # source databroker catalog
    folder="/exports/run3",  # destination
    query={"scan_id": range(100, 200)},
)

The destination will contain a documents/ directory with msgpack files, a catalog.yml, and a documents_manifest.txt.

Re-import a packed catalog#

On the receiving machine:

from databroker import catalog
from polartools.manage_database import from_databroker_inplace

from_databroker_inplace(
    folder="/data/run3",  # the directory created by to_databroker
    name="run3",  # the catalog name to register
    catalog=catalog,
)

# Now use it like any other catalog
db = catalog["run3"]
df = db[100].primary.read().to_dataframe()

from_databroker_inplace writes a config file under ~/.local/share/intake/ that points databroker at the packed data.

CSV / JSON export#

For analysis tools that don’t speak Bluesky:

from polartools.manage_database import to_csv_json

to_csv_json(
    db,
    folder="/exports/csv",
    query={"sample": "Fe2O3"},
    overwrite=True,
)

Each scan becomes a .csv file (per primary stream) plus a .json sidecar with the start/stop documents.

Known issue: to_csv_json doesn’t currently work under pytest because of a logging conflict (issue #56). It’s fine in interactive use.

Removing a registered catalog#

from polartools.manage_database import remove_catalog

remove_catalog("run3", catalog=catalog)

This removes the catalog registration but leaves the underlying data files on disk — you can re-add them later with from_databroker_inplace.

Catalog with image data#

If your scans reference external image files (Lambda HDF5, Eiger HDF5, SPE), pass external=True to also pack the image files alongside the documents:

to_databroker(
    db,
    folder="/exports/run3",
    query={"scan_id": range(100, 200)},
    external=True,
)

Without external=True, the packed catalog will reference the original image file paths — works if the destination machine can see them, breaks otherwise.

See also#