Adding data

How AIMBAT stores data

AIMBAT never modifies input files. When a data source is added, AIMBAT reads the metadata it needs (event location, station coordinates, initial pick time) and stores copies in the project database. After import, the original files are only accessed to read waveform samples.

Each file is validated before any record is written: required fields present, values in range, types correct. A file that fails validation is skipped and the database is left unchanged for it. Other files in the same batch still import.

Data types

AIMBAT reads three data types, selected with --type:

--type defaults to sac, so it only needs to be given for JSON. Both JSON variants use the .json extension, so --type is what tells them apart. The TUI file picker filters by extension (.sac, .bhz, .bhn, .bhe for SAC); the CLI reads whatever paths it is given as the requested type.

See aimbat data add for the full set of flags, including --use-event, --use-station, --dry-run, and --no-snapshot.

Adding SAC files

The common case is SAC files in one or more directories, each carrying station and event headers and an initial pick:

aimbat data add *.sac

data add file1.sac file2.sac

from pathlib import Path

from sqlmodel import Session
from aimbat.core import add_data_to_project
from aimbat.db import engine
from aimbat.io import DataType

with Session(engine) as session:
    add_data_to_project(session, sorted(Path().glob("*.sac")), DataType.SAC)

Import deduplicates, so the same command is safe to re-run:

aimbat data add *.sac          # first run: imports everything
aimbat data add *.sac          # second run: no-op, all files already known

Initial picks

SAC files carry named time markers (T0–T9). AIMBAT reads one of them as the initial phase pick (t0 in AIMBAT's terms). The default is T0; change it with AIMBAT_SAC_PICK_HEADER, per command or in .env:

AIMBAT_SAC_PICK_HEADER=t1

Selecting subsets

Standard shell glob patterns apply, at a normal command-line prompt:

aimbat data add event1/*.sac           # one subdirectory
aimbat data add data/**/*.sac          # recursive (bash 4+ with globstar; zsh)
aimbat data add data/*/BHZ.sac         # vertical component only
aimbat data add data/II.*.BHZ.sac      # network filter

How events are matched and de-duplicated

Event matching applies to SAC and --type json_event imports alike.

AIMBAT links a newly imported event to an existing one only if its origin time is exactly equal to that event's stored time, to the microsecond (the precision AIMBAT stores timestamps at). Otherwise it does not reuse that event.

A new origin time that nearly, but not exactly, matches an existing event's time is not silently turned into a second event. How close counts as "near" is controlled by event_duplicate_tolerance and event_duplicate_raise_tolerance (see Aimbat Defaults):

• Within event_duplicate_tolerance (default 0.1 s), the existing event is reused, exactly as an exact time match is, with a warning naming both origin times and the gap (a --dry-run reports the same in its preview). A gap this small is usually timestamp precision loss upstream, such as the classic v6 SAC header's 32-bit float o field not round-tripping identically across files, rather than a genuinely distinct event.
• Beyond that but within event_duplicate_raise_tolerance (default 2 s), import always raises, even during --dry-run. A gap this size usually signals a real timing problem in the source data, worth investigating before import.
• Beyond event_duplicate_raise_tolerance, the new origin time is treated as a genuinely independent event, with no warning.

The check runs against every event AIMBAT already knows about when a file is processed, including events created earlier in the same data add call. Reuse is always first-wins: the new import links to the existing event's stored time and location exactly as they are. It never merges, averages, or recomputes from the new data. In the ambiguous-gap band, re-run with --use-event <ID> to link explicitly to the existing event and get past the error.

Previewing before import

--dry-run shows what would be added without touching the database, for any --type:

aimbat data add --dry-run *.sac

data add --dry-run file1.sac file2.sac

(
    added_datasources,
    existing_station_ids,
    existing_event_ids,
    existing_seismogram_ids,
    duplicate_warnings,
) = add_data_to_project(session, paths, DataType.SAC, dry_run=True)

The TUI has no equivalent; its file picker imports as soon as a file is chosen.

The CLI and shell print which stations, events, and seismograms would be newly created versus already known (a near-duplicate within event_duplicate_tolerance shows as a reused event); the API returns the same information. Either way, a dry run surfaces any ambiguous-gap conflict a real run would raise. That makes it worth running as routine practice before a new batch: catching an ambiguous-gap conflict here means resolving it up front with --use-event <ID> rather than hitting it partway through a real import, which aborts the whole batch and rolls back everything already added in that call.

A gap in the ambiguous band raises during --dry-run just as it does on a real import.

Automatic snapshots

An import snapshots every event that receives at least one newly imported seismogram, at the end of the call. This happens regardless of --type, and gives a baseline to roll back to before any processing begins. Re-importing files already in the project creates no new snapshot, since nothing changed.

aimbat data add --no-snapshot *.sac  # skip it for this invocation

data add --no-snapshot file1.sac file2.sac

from aimbat.core import create_snapshot

create_snapshot(session, event, comment="import")

See Snapshots for working with snapshots generally.

Missing event or station metadata

SAC files from some sources omit event or station headers. Add the metadata from a JSON file first, then link the SAC files to the resulting record.

The TUI has no equivalent for either: its file picker can't link an import to a pre-existing event or station. Use the CLI, shell, or API instead.

Supplying a missing event

aimbat data add --type json_event event.json
aimbat data add --use-event <EVENT_ID> --type sac *.sac

data add --type json_event event.json
data add --use-event <EVENT_ID> --type sac file1.sac file2.sac

from sqlmodel import Session, select
from aimbat.core import add_data_to_project
from aimbat.db import engine
from aimbat.io import DataType
from aimbat.models import AimbatEvent

with Session(engine) as session:
    add_data_to_project(session, [event_json], DataType.JSON_EVENT)
    event = session.exec(select(AimbatEvent)).one()
    add_data_to_project(session, sac_files, DataType.SAC, event_id=event.id)

--use-event accepts a full UUID or any unique prefix from aimbat event list.

Supplying a missing station

aimbat data add --type json_station station.json
aimbat data add --use-station <STATION_ID> --type sac waveform.sac

data add --type json_station station.json
data add --use-station <STATION_ID> --type sac waveform.sac

from sqlmodel import Session, select
from aimbat.core import add_data_to_project
from aimbat.db import engine
from aimbat.io import DataType
from aimbat.models import AimbatStation

with Session(engine) as session:
    add_data_to_project(session, [station_json], DataType.JSON_STATION)
    station = session.exec(select(AimbatStation)).one()
    add_data_to_project(
        session, [waveform_sac], DataType.SAC, station_id=station.id
    )

JSON format

JSON event and station files must match the structure of AimbatEvent and AimbatStation. Use aimbat event dump or aimbat station dump to export existing records as templates.

Event:

{
    "time": "2024-03-15T14:22:11Z",
    "latitude": 37.5,
    "longitude": 143.0,
    "depth": 35.0
}

Station:

{
    "name": "ANMO",
    "network": "IU",
    "location": "00",
    "channel": "BHZ",
    "latitude": 34.946,
    "longitude": -106.457,
    "elevation": 1820.0
}

Records are identified by random UUIDs, so JSON exported from one AIMBAT project imports into another with no risk of ID collisions.

Ingesting seismograms directly

add_data_to_project reads from a file AIMBAT already understands (SAC, JSON_STATION, JSON_EVENT). add_seismograms_to_project is the counterpart for a caller that already has pysmo objects in memory, fetched from a web service, built in a notebook, or produced by a library such as pysmo.tools.project.PysmoProject, with no file round-trip required. It takes a list of (seismogram, station, event) triples, built from anything shaped like pysmo's IccsSeismogram, Station, and Event protocols:

"""
Ingest seismograms directly from a PysmoProject, with no SAC/JSON round-trip.

A PysmoProject fetches waveforms on demand and is just one possible producer
of (seismogram, station, event) triples. add_seismograms_to_project accepts
triples built from any pysmo Station/Event/IccsSeismogram-shaped objects,
however the caller likes (a notebook, an ObsPy Stream + inventory, ...).

This reuses one event and three stations from AIMBAT's own Alaska teleseismic
sample dataset (../aimbat-sampledata): the 2014-04-12 M7.6 Solomon Islands
earthquake. All three stations (AK/AV network) are in the Aleutians.
"""

import pandas as pd
from attrs import define
from sqlmodel import Session

from pysmo import Event, MiniEvent, MiniStation, Seismogram, Station
from pysmo.classes import StationXML
from pysmo.functions import clone_to_mini
from pysmo.tools.iccs import MiniIccsSeismogram
from pysmo.tools.project import FetchContext, ProjectEntry, PysmoProject
from pysmo.tools.signal import remove_response

from aimbat.core import add_seismograms_to_project
from aimbat.db import engine

station_atka = MiniStation(
    name="ATKA",
    network="AK",
    location="--",
    channel="BHZ",
    latitude=52.2016,
    longitude=-174.1975,
)
station_msw = MiniStation(
    name="MSW",
    network="AV",
    location="--",
    channel="BHZ",
    latitude=53.9148,
    longitude=-166.788,
)
station_akrb = MiniStation(
    name="AKRB",
    network="AV",
    location="--",
    channel="BHZ",
    latitude=54.1292,
    longitude=-166.0708,
)
event_solomon = MiniEvent(
    latitude=-11.3487,
    longitude=162.0025,
    depth=22600.0,
    time=pd.Timestamp("2014-04-12T20:14:39.300Z"),
)


@define(kw_only=True)
class ToMiniIccsSeismogramWithResponseRemoved:
    """seismogram_transform: instrument response removed, converted for ICCS."""

    pre_filt: tuple[float, float, float, float]

    def __call__(
        self, seismogram: Seismogram, context: FetchContext[Station, Event]
    ) -> MiniIccsSeismogram:
        response = StationXML.fetch(
            station=context.entry.station, time=context.starttime
        ).response
        corrected = remove_response(
            seismogram, response, pre_filt=self.pre_filt, replace=True
        )
        # t0 is set from the predicted arrival - AimbatSeismogram.t0 needs no
        # further fallback, since IccsSeismogram.t0 is a required field.
        return clone_to_mini(
            MiniIccsSeismogram, corrected, update={"t0": context.reference}
        )


project = PysmoProject(
    entries=[
        ProjectEntry(station=station_atka, event=event_solomon),
        ProjectEntry(station=station_msw, event=event_solomon),
        ProjectEntry(station=station_akrb, event=event_solomon),
    ],
    seismogram_transform=ToMiniIccsSeismogramWithResponseRemoved(
        # Teleseismic P on a broadband BHZ channel: comfortably within its
        # passband, well clear of microseismic noise below ~0.05 Hz.
        pre_filt=(0.03, 0.05, 0.5, 1.0),
    ),
)

# Build (seismogram, station, event) triples; project.seismogram() fetches
# and applies seismogram_transform. Nothing reaches disk until
# add_seismograms_to_project writes it to data_dir below.
items = [
    (project.seismogram(station, event), station, event)
    for event in project.events
    for station in project.stations_for(event)
]

with Session(engine) as session:
    add_seismograms_to_project(session, items, data_dir="waveforms")

PysmoProject is just one possible producer; any object with the right shape works identically, including one built by hand.

Every triple must carry a real Event: as with add_data_to_project, a seismogram cannot exist in AIMBAT without one. Reuse the same Event object across several triples, as above, to link them to a single AimbatEvent. The same near-duplicate detection described in How events are matched and de-duplicated applies.

Checking the import

aimbat data list              # data sources for the default event
aimbat data list all          # all data sources in the project
aimbat event list             # events extracted from imported files
aimbat station list           # stations extracted from imported files
aimbat seismogram list        # seismograms for the default event

data list
data list all
event list
station list
seismogram list

from sqlmodel import Session, select
from aimbat.db import engine
from aimbat.models import AimbatDataSource, AimbatEvent, AimbatSeismogram, AimbatStation

with Session(engine) as session:
    data_sources = session.exec(select(AimbatDataSource)).all()
    events = session.exec(select(AimbatEvent)).all()
    stations = session.exec(select(AimbatStation)).all()
    seismograms = session.exec(select(AimbatSeismogram)).all()

For the typical case of one waveform per station, the station and seismogram counts for an event should match. A mismatch usually means the source data are inconsistent: duplicate station entries with slightly different coordinates, missing headers, or files that failed to parse. Worth investigating before processing, since gaps or duplicates affect alignment quality.

Removing data

The unit of deletion is the seismogram: the record that ties a data source to a station and an event. Removing a seismogram severs that link and drops the data source entry. Events and stations are shared metadata containers, left in place even when no seismogram references them any more, since they may still be needed or were added deliberately via JSON.

aimbat seismogram delete <SEISMOGRAM_ID>   # one seismogram
aimbat event delete <EVENT_ID>             # an event and all its seismograms
aimbat station delete <STATION_ID>         # a station and all its seismograms, across events

seismogram delete <SEISMOGRAM_ID>
event delete <EVENT_ID>
station delete <STATION_ID>

from sqlmodel import Session
from aimbat.core import delete_event, delete_seismogram, delete_station
from aimbat.db import engine

with Session(engine) as session:
    delete_seismogram(session, seismogram_id)
    delete_event(session, event_id)
    delete_station(session, station_id)

Pruning vanished data sources

aimbat data prune reconciles a project after a data source disappears, such as a deleted SAC file or a moved directory. It probes every data source in scope. Any seismogram whose source can no longer be read is deleted. This is the batch, source-driven counterpart to the single-item delete commands.

aimbat data prune all --dry-run          # preview; delete nothing
aimbat data prune all                    # prune every event, with a prompt
aimbat data prune <EVENT_ID>             # prune one event only

Permanent removal has two parts. First the data source is removed, by deleting the file or regenerating the source without the seismogram. Then prune clears the leftover record.

prune only ever touches sources that have vanished. A source that is still readable is left alone, and a later data add would re-add anything it still provides. A seismogram that should stay in the project but sit out the analysis can be deselected with select = False instead.

From the API, prune_project returns a PruneReport that describes what was removed, or what a dry run would remove:

from sqlmodel import Session
from aimbat.core import prune_project
from aimbat.db import engine

with Session(engine) as session:
    report = prune_project(session, "all", dry_run=True)
    print(report.orphan_seismograms)