Derived Variables — Transforms & Binning

Real measurements rarely arrive in the shape a control chart wants. A response can be heavily skewed, a rate lives on a bounded proportion scale, or the variable you’d most like to subgroup by is a continuous covariate with no natural categories. Derived variables let you reshape columns before you chart them:

Transforms (continuous → continuous): log, sqrt, arcsin(√x), inverse, square, power, zscore — tame skew, stabilize variance, or standardize.
Binning (continuous → ordered categorical): equal_freq, equal_width, breaks, sd — turn a continuous covariate into a rational-subgrouping factor, with labels you control.

Two ideas make this safe and reproducible:

Derivations are specifications, not code. Each is a small serializable object you can inspect, edit, and round-trip to JSON — never a stored function.
formulate() is the immutability boundary. Pending derivations ride on the ProcessBehavior; only when you formulate() do they materialize into the analytic dataset (freezing any data-dependent fit, like bin edges or a z-score’s μ/σ) and become referenceable by response= / factors= / time=.

We’ll work a fabricated craft-brewery fermentation log throughout.

The data¶

Twelve weeks of fermentation, four fermenters (A–D) each held near a different temperature setpoint, three replicate gravity readings per fermenter per week. Each reading records the fermentation temperature, the yeast cell count (cells/mL — a classically right-skewed measurement), and the attenuation (the fraction of sugar converted, a proportion in [0, 1]).

import numpy as np
import pandas as pd

from processbehavior import Derivation, ProcessBehavior, evaluate, validate

rng = np.random.default_rng(7)
weeks = np.repeat(np.arange(1, 13), 12)
fermenter = np.tile(np.repeat(['A', 'B', 'C', 'D'], 3), 12)
setpoint = {'A': 17.0, 'B': 20.0, 'C': 23.0, 'D': 26.0}

df = pd.DataFrame({
    'week': weeks,
    'fermenter': fermenter,
    'fermentation_temp': [round(setpoint[f] + rng.normal(0, 0.6), 1) for f in fermenter],
    'cell_count': rng.lognormal(mean=15.5, sigma=0.4, size=len(weeks)).round(0),
    'attenuation': rng.beta(8, 2, size=len(weeks)).round(3),
})
df.head()

df[['fermentation_temp', 'cell_count', 'attenuation']].describe().round(2)

Transforms — taming a skewed response¶

cell_count is right-skewed (mean well above median). A natural log pulls it back toward symmetry, which is what a chart of individual values wants to see.

evaluate(spec, column) previews a derivation against a column without attaching it to a dataset — it’s the same primitive the application’s live preview uses. (To actually use a derivation, you attach it with pb.transform(...), which returns a new ProcessBehavior — derivations never mutate in place.)

raw = df['cell_count']
logged = evaluate(Derivation.transform('cell_count', 'log'), raw).values

print(f'raw cell_count : mean={raw.mean():>12,.0f}  median={raw.median():>12,.0f}  skew={raw.skew():.2f}')
print(f'log(cell_count): mean={logged.mean():>12.2f}  median={logged.median():>12.2f}  skew={logged.skew():.2f}')

`arcsin` for proportions¶

attenuation is a proportion in [0, 1], so its variance is largest in the middle and shrinks at the extremes. The variance-stabilizing arcsine-root transform (arcsin here means arcsin(√x)) is the classic remedy. Its domain is the closed interval [0, 1]; floating-point noise a hair past a boundary (e.g. 1.0000000002) is clamped to it rather than flagged, while a value genuinely outside is reported as a domain violation. (ln is an accepted alias for log.)

att = df['attenuation']
res = evaluate(Derivation.transform('attenuation', 'arcsin'), att)
print(f'attenuation in [{att.min()}, {att.max()}]  domain violations: {res.n_invalid}')
print('arcsin(sqrt(x)) head:', res.values.head(3).round(3).tolist())

`zscore` standardization¶

zscore is data-dependent: it fits the mean and standard deviation (sample SD, ddof=1) of the column. Those fitted parameters are returned in fitted and are frozen when you formulate(), so the standardization stays reproducible afterward.

z = evaluate(Derivation.transform('cell_count', 'zscore'), df['cell_count'])
print('fitted (frozen at formulate):', {k: round(v, 1) for k, v in z.fitted.items()})
print(f'standardized -> mean {z.values.mean():.3f}, sd {z.values.std(ddof=1):.3f}')

Binning — a continuous covariate becomes an ordered factor¶

We measured fermentation_temp but never labeled temperature zones. Binning creates them. The default equal_freq method cuts the column into n quantile bins; the result is an ordered categorical. The fitted bin edges are recorded, and the default range labels are derived from those real edges.

temp = df['fermentation_temp']
binned = evaluate(Derivation.bin('fermentation_temp', method='equal_freq', n=4), temp)
print('fitted edges :', [round(e, 1) for e in binned.fitted['edges']])
print('range labels :', list(binned.values.cat.categories))

Changing the bin labels¶

The bins are fixed by the data, but what you call them is up to you. bin_labels accepts:

'range' (default) — the fitted interval edges, e.g. [18.65, 21.3);
'ordinal' — Low, Medium-Low, Medium-High, High;
'number' — Bin 1, Bin 2, …;
an explicit list of names — here the brewer’s own vocabulary.

The same four bins, relabeled four ways:

styles = ['range', 'ordinal', 'number', ['Cool', 'Optimal', 'Warm', 'Hot']]
for style in styles:
    cats = evaluate(
        Derivation.bin('fermentation_temp', method='equal_freq', n=4, bin_labels=style), temp
    ).values.cat.categories
    name = style if isinstance(style, str) else 'explicit list'
    print(f'{name:>13} -> {list(cats)}')

An explicit list must have exactly one name per fitted bin — validate() (below) catches a miscount before anything is committed.

Other binning methods¶

equal_width — bins of equal temperature span rather than equal count.
breaks — your own cut points. Values outside the range fall into explicit below/above categories, so a brewer’s known optimal band (say 18–22 °C) becomes three honest groups.
sd — zones at ±1σ / ±2σ around the mean (the SPC “zones” idea), fitting μ and σ.

(With equal_freq, ties can collapse duplicate edges so the fitted bin count is below the requested n; when that happens it’s reported in EvalResult.message rather than silently relabeled.)

ew = evaluate(Derivation.bin('fermentation_temp', method='equal_width', n=4), temp)
print('equal_width edges:', [round(e, 1) for e in ew.fitted['edges']])

br = evaluate(Derivation.bin('fermentation_temp', method='breaks', breaks=[18, 22]), temp)
print('breaks [18, 22] :', list(br.values.cat.categories))

sd = evaluate(Derivation.bin('fermentation_temp', method='sd'), temp)
print(f'sd zones        : {sd.fitted["n_bins"]} bins  (mu={sd.fitted["mu"]:.1f}, sigma={sd.fitted["sigma"]:.1f})')

Putting it together — formulate with derived columns¶

Now we chain the verbs and formulate(). We bin the temperature into the brewer’s named zones, log-transform the cell count, and analyze cell_count_log subgrouped by fermentation_temp_bin over week. The derived names are referenceable because they were attached before formulate() — a derivation added afterward would need a fresh formulate() to take effect.

study = (
    ProcessBehavior(df)
    .bin('fermentation_temp', method='equal_freq', n=4, bin_labels=['Cool', 'Optimal', 'Warm', 'Hot'])
    .transform('cell_count', 'log')
    .formulate(response='cell_count_log', factors=['fermentation_temp_bin'], time='week')
)
print(study)
print('zone order on the chart:', list(study.dataset['fermentation_temp_bin'].cat.categories))

The binned factor charts in zone order (Cool → Hot), not alphabetically — the ordered categorical is preserved end to end.

study.execute(chart='Xbar').plot()

Inspect, edit, and persist¶

The resolved derivations live on the study, fit-frozen. Each is a plain spec that round-trips to a dict (stable id and all), so saving a study preserves them exactly. And validate(spec, dataset) gives the application a structured pass/fail before committing — domain and label problems are returned as data, not raised as exceptions.

for d in study.derivations:
    print(f'{d.output_name:22} family={d.family:10} id={d.id}')

# Serializable: round-trips to/from a plain dict, including the frozen fit.
spec = next(d for d in study.derivations if d.family == 'bin')
assert Derivation.from_dict(spec.to_dict()) == spec
print('\nfrozen bin edges:', [round(e, 1) for e in spec.fitted['edges']])

# validate() catches a label-count mistake (5 names, 4 bins) without raising.
check = validate(Derivation.bin('fermentation_temp', n=4, bin_labels=['a', 'b', 'c', 'd', 'e']), df)
print('valid?', check.ok, '->', check.issues[0]['message'])

You can also revise the pending set on a ProcessBehavior before formulating — pb.remove_derived(id) and pb.replace_derived(id, new_spec) key off the stable id, never the (editable) label.

When to derive¶

Transform to stabilize. Log a skewed response, arcsine-root a proportion, z-score to put variables on a common scale — so the chart sees a well-behaved series.
Bin to subgroup. Turn a continuous covariate into an ordered factor when that is the rational way to group the process, and label the zones in the operator’s own language.
Trust the boundary. Data-dependent fits freeze at formulate(); the analytic dataset stays immutable; nothing is silently recomputed.

Box–Cox, derived-on-derived chaining, and free-form expressions are intentionally out of scope for this first version.