pmcprg.missing¶

Data layer for missing observations, free of any inference code: artificial missingness patterns (following the geometry of ImputeGAP's GenGap contamination module) and evaluation metrics restricted to the masked positions. pmcprg.pmc (imputation, forecasting, non-ignorable missingness) is the inference layer that consumes this one.

Patterns¶

mask_from_nan ¶

mask_from_nan(Y) -> np.ndarray

Boolean mask of the missing (NaN) entries of Y, same shape.

For a multichannel (N, d) array the mask is per entry; a row is missing for inference as soon as one of its components is (mask.any(axis=1)).

mcar ¶

mcar(Y, rate_series, *, block_size: int = 10, rate_dataset=1.0, offset=0.1, seed: Seed = None)

Missing completely at random, in blocks of block_size values.

GenGap's mcar: ceil(d * rate_dataset) series chosen at random (without replacement); in each, B = floor(W / block_size) block starts drawn without replacement in [P, N), and every block removes exactly block_size positions from its start onward. A block running past the end wraps around to P; a position already missing (an earlier block or a NaN of Y) is skipped forward. Blocks therefore never overlap — they may merge into longer gaps — and each contaminated series loses exactly B * block_size values, which can be less than W (GenGap's count too).

Differences from GenGap: when W < block_size (B = 0) GenGap prints a correction and shrinks block_size to W // 2; this function raises. When a series has fewer than B * block_size observed positions after the offset, GenGap skips it with a message; this function raises.

Parameters:

Name	Type	Description	Default
`Y`	`array(N) or (N, d)`		required
`rate_series`	`float in (0, 1] — fraction of each series' length N removed.`		required
`block_size`	`int >= 1`		`10.`
`rate_dataset`	`float in (0, 1]`	(ignored for a 1-D `Y`, see the module docstring).	`1.0 — share of series contaminated`
`offset`	`float`		`0.1 — protected leading fraction (< 1) or count.`
`seed`	int, Generator, SeedSequence or None — ``default_rng(seed)``.		`None`

Returns:

Type	Description
`(Y_masked, mask) — see the module docstring.`

aligned ¶

aligned(Y, rate_series, *, rate_dataset=1.0, offset=0.1)

One block [P, P + W) in each of the first ceil(d * rate_dataset) series.

GenGap's aligned: the gaps are synchronised across the contaminated series and start right after the protected offset. Deterministic (no seed, as in GenGap).

scattered ¶

scattered(Y, rate_series, *, rate_dataset=1.0, offset=0.1, seed: Seed = None)

One block of W values per series, each at its own random start.

GenGap's scattered: for each of the first ceil(d * rate_dataset) series, the start is uniform on [P, N - W] (inclusive), independently across series.

blackout ¶

blackout(Y, rate_series, *, offset=0.1)

Every series loses the same block [P, P + W) — all channels at once.

GenGap's blackout is aligned with rate_dataset = 1; so is this one. For a multichannel observation every row of the block is entirely missing. For a univariate series it coincides with :func:aligned.

disjoint ¶

disjoint(Y, rate_series, *, limit=1.0, offset=0.1)

Consecutive, non-overlapping blocks: series j loses [P + jW, P + (j+1)W).

GenGap's disjoint: the blocks tile the time axis from the offset on, one series after the other, until the series run out or the contamination reaches floor(N * limit); the block that reaches it is truncated there and the later series are left intact. rate_dataset is not a parameter — the number of contaminated series follows from the geometry (all d if they fit).

GenGap stops after writing index int(N * limit) - 1; the rule here (positions < floor(N * limit)) is the same. A limit too small for the first block is refused (GenGap would leave a one-value gap). A univariate series gets the single block [P, P + W).

overlap ¶

overlap(Y, rate_series, *, shift=0.05, limit=1.0, offset=0.1)

Consecutive blocks that overlap: each starts floor(N * shift) before the previous end.

GenGap's overlap: series j loses [P + j (W - sh), P + j (W - sh) + W) with sh = floor(N * shift), truncated at floor(N * limit) as in :func:disjoint; shift = 0 is :func:disjoint. Consecutive series therefore share sh missing timestamps.

Differences from GenGap: sh >= W is refused (blocks would stall or run backwards into the protected zone). GenGap's extra check int(N * shift) > int(N * offset) is not enforced: blocks move back relative to the previous block only, never below P, so it guards nothing and would forbid offset = 0.

gaussian ¶

gaussian(Y, rate_series, *, rate_dataset=1.0, std_dev=0.2, selected_mean: str = 'position', offset=0.1, seed: Seed = None)

Isolated missing values whose positions follow a Gaussian bump.

GenGap's gaussian: in each of the first ceil(d * rate_dataset) series, W distinct positions are drawn without replacement from [P, N) with probability proportional to norm.pdf(n, loc=center, scale=std_dev * (N - P)).

selected_mean="position" (GenGap's default) centres the bump in the middle of the unprotected range, center = (P + N) / 2. selected_mean="values" uses the series' mean value m, clipped to [-1, 1]: center = P + m (N - P). That rule assumes a normalised series (z-score or min-max, ImputeGAP's loaders); on raw data the centre is pinned to an end of the range. The mean ignores NaN here (GenGap's np.mean would propagate one).

distribution ¶

distribution(Y, probabilities, rate_series, *, rate_dataset=1.0, offset=0.1, seed: Seed = None)

Isolated missing values drawn from a user-supplied position distribution.

GenGap's distribution: in each of the first ceil(d * rate_dataset) series, W distinct positions of [P, N) are drawn without replacement with the given probabilities.

probabilities covers the unprotected positions P, …, N-1: shape (N - P,) — shared by every series — or (N - P, d), one column per series (GenGap takes (d, N - P), one row per series, because its matrix is transposed). Entries must be finite and non-negative; each column is renormalised to sum 1 (GenGap requires the caller to have done it). At least W observed positions need a positive probability.

state_dependent ¶

state_dependent(Y, X, rates, *, seed: Seed = None)

Rows missing independently given the hidden states: P(row n missing | x_n = i) = rates[i].

Simulates the "state" mechanism of :mod:pmcprg.pmc.missingness ([missingness] mechanism = "state"): given the state path X the rows are removed independently, row n with probability rates[X[n]] — missingness that depends on the hidden state, hence not at random given Y. One uniform u_n is drawn per row and row n is removed iff u_n < rates[X[n]], so a rate 0 never and a rate 1 always removes.

Differences from the GenGap patterns above: whole rows are removed (every column of a multichannel Y, as inference treats a row with a missing component), and there is no protected offset — every row follows the mechanism's law, which is what a simulation from the model needs. Values already missing in Y stay NaN and are not flagged; they do not change the draw (same seed, same removed rows).

Parameters:

Name	Type	Default
`Y`	`array(N) or (N, d)`	required
`X`	(N,) integer state path (e.g. from :func:`pmcprg.pmc.simulate`).	required
`rates`	`(K,) probabilities in [0, 1], one per state (K > max(X)).`	required
`seed`	int, Generator, SeedSequence or None — ``default_rng(seed)``.	`None`

Returns:

Type	Description
`(Y_masked, mask) — see the module docstring.`

state_markov ¶

state_markov(Y, X, onset, persistence, *, seed: Seed = None)

Bursts of missing rows whose onset and length depend on the hidden states.

Simulates the "state-markov" mechanism of :mod:pmcprg.pmc.missingness ([missingness] mechanism = "state-markov"): given the state path X the row mask m is a two-state Markov chain,

P(m_n = 1 | m_{n-1} = 0, x_n = i) = onset[i],
P(m_n = 1 | m_{n-1} = 1, x_n = i) = persistence[i],
P(m_0 = 1 | x_0 = i) = onset[i] / (1 − persistence[i] + onset[i]),

the last being the stationary missing probability of the mask chain under a constant state i. The mean length of a burst in state i is 1 / (1 − persistence[i]). One uniform u_n per row: m_n = 1 iff u_n is below the probability above. onset[i] = 0 with persistence[i] = 1 is refused (the initial probability is 0/0), as in the model.

Whole rows, no protected offset, values already missing: as in :func:state_dependent (the chain runs on the drawn mask only).

Parameters:

Name	Type	Default
`Y`	`array(N) or (N, d)`	required
`X`	`(N,) integer state path.`	required
`onset`	`(K,) probabilities in [0, 1], one per state.`	required
`persistence`	`(K,) probabilities in [0, 1], one per state.`	required
`seed`	int, Generator, SeedSequence or None — ``default_rng(seed)``.	`None`

Returns:

Type	Description
`(Y_masked, mask) — see the module docstring.`

Metrics¶

rmse ¶

rmse(Y_true, Y_hat, mask) -> float

Root mean squared error over the masked positions (no cap).

mae ¶

mae(Y_true, Y_hat, mask) -> float

Mean absolute error over the masked positions (no cap).

mutual_information ¶

mutual_information(Y_true, Y_hat, mask, *, bins: int = 10) -> float

Mutual information (nats) between truth and imputation on the masked positions.

Each array is discretised on its own bins equal-width bins (ImputeGAP's rule, see the module docstring), then I = sum p(a, b) log(p(a, b) / (p(a) p(b))) over the joint labels.

pearson ¶

pearson(Y_true, Y_hat, mask) -> float

Pearson correlation on the masked positions; NaN when either side is constant.

crps_from_samples ¶

crps_from_samples(Y_true, samples, *, mask=None) -> float

Mean CRPS estimated from predictive draws — unbiased energy form.

For each evaluated position with truth y and draws x_1 … x_M (M >= 2)::

CRPS = (1/M) sum_i |x_i - y|  -  1/(2 M (M-1)) sum_{i != j} |x_i - x_j|

the "fair" estimator of E|X - y| - E|X - X'| / 2, computed in O(M log M) by sorting. It is unbiased for the CRPS of the predictive law, so a single position can score slightly below zero.

samples has shape Y_true.shape + (M,) — the draws on the last axis. Returns the mean over evaluated positions.

crps_gaussian ¶

crps_gaussian(Y_true, mean, sd, *, mask=None) -> float

Mean CRPS of Gaussian predictive laws N(mean, sd**2), closed form.

CRPS = sd [z (2 Phi(z) - 1) + 2 phi(z) - 1/sqrt(pi)] with z = (y - mean) / sd (Gneiting & Raftery 2007, JASA 102:359–378, doi:10.1198/016214506000001437). mean and sd have the shape of Y_true or are scalars; sd must be > 0 at evaluated positions.

interval_coverage ¶

interval_coverage(Y_true, lo, hi, *, mask=None) -> float

Share of evaluated positions whose truth lies in the closed interval [lo, hi].

lo and hi have the shape of Y_true or are scalars; lo > hi at an evaluated position raises.

error_rate_split ¶

error_rate_split(X_true, X_hat, mask, *, align: bool = True) -> ErrorRates

Error rate on the missing positions, on the observed ones and overall.

Parameters:

Name	Type	Description	Default
`X_true`	`(N,) non-negative integer labels.`		required
`X_hat`	`(N,) non-negative integer labels.`		required
`mask`	`(N,) or (N, d) bool — True where the observation is missing. A`	multichannel row counts as missing as soon as one component is (the rule of the forward-backward with gaps).	required
`align`	`bool`	minimises the overall error (Hungarian on the confusion matrix, as :func:`pmcprg.pmc.inference.error_rate`), then split. `overall` then equals `error_rate(X_true, X_hat)`. One permutation serves both subsets: labels are a property of the model, not of the gaps. `align=False` compares labels as given.	True — relabel ``X_hat`` by the permutation that

Returns:

Type	Description
`ErrorRates(missing, observed, overall, n_missing, n_observed) —`
``observed`` is NaN when every position is missing; a mask with no
`missing position raises.`

ErrorRates ¶

Bases: NamedTuple

Classification error split by missingness (rates in [0, 1]).