Bishop Reference Validation with PBTESTDATABASE_T100

This notebook replicates Tom Bishop’s slide-by-slide Minitab analyses of PM SDS 1, 2, and 3 from the PBTESTDATABASE_T100 -- a 4000-row dataset whose response columns each carry a structure matching one observed design state. For a quick tour of how all six design states classify, see the Design-State Tour.

The PBTESTDATABASE_T100¶

The dataset has:

FACTOR 1: 4 levels (1, 2, 3, 4)
FACTOR 2: 2 levels (1, 2)
PRODUCTION TIME: 100 time periods
PM SDS 1-6: Response columns, each with a sparsity pattern matching its SDS

`factors` vs `plan`¶

ProcessBehavior detects the design state using either factors or plan. SDS detection runs on raw data before NA rows are dropped, so cells where all response values are NA are counted as empty (N_kt=0) -- enabling SDS 4-6 detection even without a plan.

Approach	What ProcessBehavior Knows	Capabilities
`factors`	Observed structure (including all-NA cells)	All SDS detection (1-6), charts, basic design reports
`plan`	Expected + observed structure	All SDS detection (1-6), coverage analysis, rich design reports

from processbehavior import ProcessBehavior
# Load the PBTESTDATABASE_T100
pb = ProcessBehavior.read_csv('../../validation/PBTESTDATABASE_T100.csv')

print(f"Dataset: {pb.data.shape[0]} rows x {pb.data.shape[1]} columns")
print(f"Columns: {pb.data.columns.tolist()}")

Dataset: 4000 rows x 11 columns
Columns: ['PRODUCTION TIME', 'FACTOR 1', 'FACTOR 2', 'FACTOR 1xFACTOR 2', 'PM SDS 1', 'PM SDS 2', 'PM SDS 3', 'PM SDS 4', 'PM SDS 5', 'PM SDS 6', 'PM INERT']

/var/folders/cp/dvnklhkn2dd12f_pgg5mhydw0000gp/T/ipykernel_10922/489954322.py:3: ProcessBehaviorWarning: Found 11655 garbage/NA values across 5 column(s):
  • PM SDS 2: 3200 values
  • PM SDS 3: 958 values
  • PM SDS 4: 2134 values
  • PM SDS 5: 3359 values
  • PM SDS 6: 2004 values

These values were converted to NA and will be excluded from analysis.
  pb = ProcessBehavior.read_csv('../../validation/PBTESTDATABASE_T100.csv')

PM SDS 1: Full Validation Against Bishop’s Reference¶

Design state 1 -- full replication.

Every (factor x time) cell has 2+ observations. This is the ideal structure for Bishop’s VAS.

Capabilities: Exact within-cell variance, all VAS residuals (R2-R5), full interaction analysis, Xbar-S with exact limits.

Using `factors` (infer structure from data)¶

# SDS 1 with factors -- let ProcessBehavior infer the structure
study_sds1 = pb.formulate(
    response=pb.cols.PM_SDS_1,
    factors=[pb.cols.FACTOR_1, pb.cols.FACTOR_2],
    time=pb.cols.PRODUCTION_TIME,
    precision=5
)

print(f"Observed Design State: {study_sds1.observed_design_state}")
print(f"Valid charts: {study_sds1.valid_charts}")
print(f"Recommended: {study_sds1.recommended_chart}")

Observed Design State: SDSResult(sds=1, min_cell_size=5, reason='full_replication', n_empty_cells=0)
Valid charts: ['Histogram', 'Xbar', 'S', 'X', 'mR']
Recommended: Xbar

# Design report shows observed structure (no plan specified)
print(study_sds1.design())

Design Report (2 factors)
  Design-state lineage:
    PDS (Planned):    no plan supplied
    SDS (Sampling):   1 (Full Replication)
    ADS (Analytical): 1 (Full Replication)
  Min cell size: 5 | K: 8 | T: 100 | R: 800 | N: (min=5, median=5.0, max=5)

  Factors:
    FACTOR 1: observed=[1, 2, 3, 4]
    FACTOR 2: observed=[1, 2]

  Structure: Complete structure

  Available analyses (ADS 1):
    Primary: Histogram, Xbar *, S, X, mR
    R2: S, X
    R3: Xbar, S
    R4: Xbar, S
    R5: Xbar, S
    R6: Xbar, S
    Methods: Capability, Loss Function, Maximum Information

Original Performance Measurement - Sample Averages (matches)¶

study_sds1.execute(chart='xbar', by=[]).plot(theme='ggplot').show()

Original Performance Measurement - Sample Standard Deviations (matches)¶

study_sds1.execute(chart='s', by=[]).plot(theme='ggplot').show()

Analysis of Original Performance Measurement Behavior By Factor (Sample Averages and Standard Deviations)¶

study_sds1.execute(chart='xbar', by=[pb.cols.PRODUCTION_TIME], companion=True).plot(theme='ggplot').show()

Process Design Conditions Main Effects - Factor 1 and Factor 2 - Sample Averages (matches)¶

study_sds1.execute(chart='xbar', by=[pb.cols.FACTOR_1,pb.cols.FACTOR_2], value='R6',recentered=True).plot(theme='ggplot').show()

Process Design Conditions Main Effects - Factor 1 and Factor 2 - Sample Standard Deviations (matches)¶

study_sds1.execute(chart='s', by=[pb.cols.FACTOR_1,pb.cols.FACTOR_2], value='R6',recentered=True).plot(theme='ggplot').show()

Process Design Conditions Main Effects - Factor 1 - Sample Averages (matches)¶

study_sds1.execute(chart='xbar', by=[pb.cols.FACTOR_1], value='R6',recentered=True).plot(theme='ggplot').show()

Process Design Conditions Main Effects - Factor 1 - Sample Standard Deviations (matches)¶

study_sds1.execute(chart='s', by=[pb.cols.FACTOR_1], value='R6').plot(theme='ggplot').show()

Process Design Conditions Main Effects - Factor 2 - Sample Averages (matches)¶

study_sds1.execute(chart='xbar', by=[pb.cols.FACTOR_2], value='R6',recentered=True).plot(theme='ggplot').show()

Process Design Conditions Main Effects - Factor 2 - Sample Standard Deviations (matches)¶

study_sds1.execute(chart='s', by=[pb.cols.FACTOR_2], value='R6').plot(theme='ggplot').show()

Process Design Conditions by Production Time Effects (R3) - Sample Averages (matches)¶

study_sds1.execute(chart='xbar', by=[pb.cols.FACTOR_1,pb.cols.FACTOR_2,pb.cols.PRODUCTION_TIME], value='R3',recentered=True).plot(theme='ggplot').show()

Production Time Interaction Effects (R3) Standard Deviations (Match)¶

study_sds1.execute(chart='s', by=[pb.cols.FACTOR_1,pb.cols.FACTOR_2,pb.cols.PRODUCTION_TIME], value='R3',recentered=True).plot(theme='ggplot').show()

Potential Process Capability¶

sds1_capability = study_sds1.capability(lsl=232, usl=242, target=237)
print(f'Percent above USL:{round(sds1_capability.pct_above_usl,3)}| Percentage below the LSL: {round(sds1_capability.pct_below_lsl,2)}')
sds1_capability.plot(values=study_sds1.dataset['PM SDS 1'], nbins=10,show_potential=True).show()

Percent above USL:0.45| Percentage below the LSL: 0.53

Test of Maximum Information R2 (Match)¶

study_sds1.execute(chart='X', by=[], value='R2').plot(theme='ggplot').show()

Using `plan` (specify expected structure)¶

The same data formulated with a plan provides richer design reporting -- showing planned vs observed structure.

# SDS 1 with plan -- specify expected factor levels and time points
study_sds1_plan = pb.formulate(
    response='PM SDS 1',
    time='PRODUCTION TIME',
    plan={
        'factors': {
            'FACTOR 1': [1, 2, 3, 4],
            'FACTOR 2': [1, 2]
        },
        'T': 100,
        'N': 5
    },
    precision=3
)

print(f"Observed Design State: {study_sds1_plan.observed_design_state}")
print()
# Design report now compares plan to observed
print(study_sds1_plan.design())

Observed Design State: SDSResult(sds=1, min_cell_size=5, reason='full_replication', n_empty_cells=0)

Design Report (2 factors)
  Design-state lineage:
    PDS (Planned):    1 (Full Replication)
    SDS (Sampling):   1 (Full Replication)
    ADS (Analytical): 1 (Full Replication)
  Plan adherence: complete
  Min cell size: 5 | K: planned=8, observed=8 | T: planned=100, observed=100 | R: planned=800, observed=800 | N: planned=5, observed=(min=5, median=5.0, max=5)

  Factors:
    FACTOR 1: planned=[1, 2, 3, 4], observed=[1, 2, 3, 4]
    FACTOR 2: planned=[1, 2], observed=[1, 2]

  Structure: Complete structure

  Available analyses (ADS 1):
    Primary: Histogram, Xbar *, S, X, mR
    R2: S, X
    R3: Xbar, S
    R4: Xbar, S
    R5: Xbar, S
    R6: Xbar, S
    Methods: Capability, Loss Function, Maximum Information

PM SDS 2: Full Validation Against Bishop’s Reference¶

This section replicates Tom Bishop’s analysis of PM SDS 2 — an SDS 2 (No Replication) dataset where every (factor × time) cell has exactly 1 observation. We validate each chart against Bishop’s Minitab reference output.

study_sds2_val = pb.formulate(
    response=pb.cols.PM_SDS_2,
    factors=[pb.cols.FACTOR_1, pb.cols.FACTOR_2],
    time=pb.cols.PRODUCTION_TIME,
    precision=2
)

print(f"Observed Design State: {study_sds2_val.observed_design_state}")
print(f"Valid charts: {study_sds2_val.valid_charts}")
print(f"Recommended: {study_sds2_val.recommended_chart}")

Observed Design State: SDSResult(sds=2, min_cell_size=1, reason='no_replication', n_empty_cells=0)
Valid charts: ['Histogram', 'Xbar', 'S', 'X', 'mR']
Recommended: X

print(study_sds2_val.design())

Design Report (2 factors)
  Design-state lineage:
    PDS (Planned):    no plan supplied
    SDS (Sampling):   2 (No Replication)
    ADS (Analytical): 2 (No Replication)
  Min cell size: 1 | K: 8 | T: 100 | R: 800 | N: (min=1, median=1.0, max=1)

  Factors:
    FACTOR 1: observed=[1, 2, 3, 4]
    FACTOR 2: observed=[1, 2]

  Structure: Complete structure

  Available analyses (ADS 2):
    Primary: Histogram, Xbar, S, X *, mR
    R2: X
    R3: Xbar, S, X
    R4: Xbar, S
    R5: Xbar, S
    R6: Xbar, S
    Methods: Capability, Loss Function, Maximum Information

Analysis of Original Performance Measurement Behavior - PM Individuals Chart¶

Bishop’s reference values:

CL = 237.78
LBL = 235.47
UBL = 240.09

result_sds2_val = study_sds2_val.execute(chart='X', by=[], companion=True)
result_sds2_val.plot(theme='ggplot').show()

# Validate against Bishop's reference values
xmr_table = result_sds2_val.chart_table(chart='X')
print("ProcessBehavior XmR Statistics:")
print(xmr_table.head())

ProcessBehavior XmR Statistics:
   Obs subgroup  n   value  center     lpl     upl signal
1    1      1_1  1  236.93  237.78  235.47  240.09       
2    6      1_1  1  239.67  237.78  235.47  240.09       
3   11      1_1  1  239.67  237.78  235.47  240.09       
4   16      1_1  1  237.78  237.78  235.47  240.09       
5   21      1_1  1  237.46  237.78  235.47  240.09

Process Design Conditions Main Effects - Factor 1 and Factor 2 (R6 Recentered)¶

study_sds2_val.execute(
    chart='xbar',
    by=[pb.cols.FACTOR_1, pb.cols.FACTOR_2],
    value='R6',
    recentered=True
).plot(theme='ggplot').show()

Process Design Conditions Main Effects - Factor 1 (R6 Recentered)¶

study_sds2_val.execute(
    chart='xbar',
    by=[pb.cols.FACTOR_1],
    value='R6',
    recentered=True
).plot(theme='ggplot').show()

Process Design Conditions Main Effects - Factor 2 (R6 Recentered)¶

study_sds2_val.execute(
    chart='xbar',
    by=[pb.cols.FACTOR_2],
    value='R6',
    recentered=True
).plot(theme='ggplot').show()

Test of Maximum Information R2¶

study_sds2_val.execute(chart='X', by=[], value='R2').plot(theme='ggplot').show()

PM SDS 3: Full Validation Against Bishop’s Reference¶

This section replicates Tom Bishop’s VAS analysis of PM SDS 3 — an SDS 3 (Partial Replication) dataset with a mix of n=1 and n≥2 cells. We validate each chart against Bishop’s reference output.

Spec limits: LSL = 232, USL = 242, Target = 237

study_sds3_val = pb.formulate(
    response=pb.cols.PM_SDS_3,
    factors=[pb.cols.FACTOR_1, pb.cols.FACTOR_2],
    time=pb.cols.PRODUCTION_TIME,
    precision=2
)

print(f"Observed Design State: {study_sds3_val.observed_design_state}")
print(f"Valid charts: {study_sds3_val.valid_charts}")
print(f"Recommended: {study_sds3_val.recommended_chart}")
print()
print(study_sds3_val.design())

Observed Design State: SDSResult(sds=3, min_cell_size=1, reason='partial_replication', n_empty_cells=0)


Valid charts: ['Histogram', 'Xbar', 'S', 'X', 'mR']
Recommended: X

Design Report (2 factors)
  Design-state lineage:
    PDS (Planned):    no plan supplied
    SDS (Sampling):   3 (Partial Replication)
    ADS (Analytical): 3 (Partial Replication)
  Min cell size: 1 | K: 8 | T: 100 | R: 800 | N: (min=1, median=4.0, max=5)

  Factors:
    FACTOR 1: observed=[1, 2, 3, 4]
    FACTOR 2: observed=[1, 2]

  Structure: Complete structure

  Available analyses (ADS 3):
    Primary: Histogram, Xbar, S, X *, mR
    R2: X
    R3: Xbar, S, X
    R4: Xbar, S
    R5: Xbar, S
    R6: Xbar, S
    Methods: Capability, Loss Function, Maximum Information

Analysis of Original Performance Measurement Behavior — PM Individuals Chart (Slides 1-2)¶

Bishop’s reference values:

CL = 237.81
LBL = 234.9
UBL = 240.72

study_sds3_val.execute(chart='X', by=[], companion=True,).plot(theme='ggplot').show()

Analysis of Original PM Behavior — XmR by PDC RSG (Slides 3-18)¶

Individual XmR + Moving Range charts for each of the 8 factor combinations (PDC RSG: 1-1, 1-2, 2-1, 2-2, 3-1, 3-2, 4-1, 4-2). The by parameter with companion=True produces all 16 charts in one call.

study_sds3_val.execute(
    chart='X',
    by=[pb.cols.FACTOR_1, pb.cols.FACTOR_2],
    companion=True
).plot(theme='ggplot').show()

Current Process Capability (Slide 19)¶

Bishop’s reference values:

Process Mean = 237.81, Process StdEv = 1.74
PPL = 1.12, PP = 0.96, PPU = 0.8
Pct below LSL = 0.53%, Pct above USL = 0.46%

sds3_cap = study_sds3_val.capability(lsl=232, usl=242, target=237)
print(sds3_cap)
sds3_cap.plot(
    values=study_sds3_val.dataset['PM SDS 3'],
    view='current',
    histnorm='percent',
    nbins=10
).show()

CapabilityResult:
  Specs: LSL=232, USL=242, Target=237
  N=3042, Ybar=237.81, S=1.74, sigma_hat=1.74

  Current Capability (overall sigma):
    Pp=0.96  Ppk=0.8  (lower=1.12, upper=0.8)

  Potential Capability (R2 sigma):
    Cp=2.34  Cpk=1.96  (lower=2.72, upper=1.96)

  Z-scores:
    Z_lower=3.35, Z_upper=2.41

  Empirical (current):
    Below LSL: 16 (0.53%)  Above USL: 14 (0.46%)  Total outside: 30 (0.99%)
  Empirical (potential):
    Below LSL: 0 (0.0%)  Above USL: 0 (0.0%)

  Note: Stability not assessed; run study.execute() and review signals before interpreting indices.

Process Design Condition Main Effects — PDC RSG Sample Averages (Slide 20)¶

Bishop’s reference: CL = 237.83

study_sds3_val.execute(
    chart='Xbar',
    by=[pb.cols.FACTOR_1, pb.cols.FACTOR_2],
    value='R6',
    recentered=True
).plot(theme='ggplot').show()

Process Design Condition Main Effects — PDC RSG Sample Standard Deviations (Slide 21)¶

Bishop’s reference: CL = 0.71

study_sds3_val.execute(
    chart='S',
    by=[pb.cols.FACTOR_1, pb.cols.FACTOR_2],
    value='R6',
    recentered=True
).plot(theme='ggplot').show()

Production Time Main Effects — PT RSG Sample Averages (Slide 22)¶

Bishop’s reference: CL = 237.83

study_sds3_val.execute(
    chart='Xbar',
    by=[pb.cols.FACTOR_1, pb.cols.FACTOR_2, pb.cols.PRODUCTION_TIME],
    value='R3',
    recentered=True
).plot(theme='ggplot').show()

Production Time Main Effects — PT RSG Sample Standard Deviations (Slide 23)¶

Bishop’s reference: CL = 0.71

study_sds3_val.execute(
    chart='S',
    by=[pb.cols.FACTOR_1, pb.cols.FACTOR_2, pb.cols.PRODUCTION_TIME],
    value='R3',
    recentered=True
).plot(theme='ggplot').show()

Process Design Design Factor Main Effects — Factor 1 Sample Averages (Slide 24)¶

Bishop’s reference: CL = 237.83

study_sds3_val.execute(
    chart='Xbar',
    by=[pb.cols.FACTOR_1],
    value='R6',
    recentered=True
).plot(theme='ggplot').show()

Process Design Design Factor Main Effects — Factor 1 Sample Standard Deviations (Slide 25)¶

Bishop’s reference: CL = 0.71

study_sds3_val.execute(
    chart='S',
    by=[pb.cols.FACTOR_1],
    value='R6'
).plot(theme='ggplot').show()

Process Design Design Factor Main Effects — Factor 2 Sample Averages (Slide 26)¶

Bishop’s reference: CL = 237.82

study_sds3_val.execute(
    chart='Xbar',
    by=[pb.cols.FACTOR_2],
    value='R6',
    recentered=True
).plot(theme='ggplot').show()

Process Design Design Factor Main Effects — Factor 2 Sample Standard Deviations (Slide 27)¶

Bishop’s reference: CL = 0.71

study_sds3_val.execute(
    chart='S',
    by=[pb.cols.FACTOR_2],
    value='R6'
).plot(theme='ggplot').show()

Analysis of PDCxPT Interaction — R3 Residuals Individuals Chart (Slides 28-29)¶

Bishop’s reference: CL = 237.81, LBL = 234.87, UBL = 240.76

study_sds3_val.execute(
    chart='X',
    by=[],
    value='R3',
    recentered=True,
    companion=True
).plot(theme='ggplot').show()

Taguchi Loss Function Analysis (Slide 30)¶

Bishop’s reference (5-component Pareto):

PDCxPT INT = 42.4%
Unexplained = 25.0%
Mean = 16.4%
PDC = 13.7%
PT = 2.5%

sds3_loss = study_sds3_val.loss_function(target=237)
print(sds3_loss)
sds3_loss.plot().show()

LossResult:
  Target=237, Ybar=237.83, N=3042
  K=8 PDC levels, T=100 periods, SDS=3

  Expected Loss (EL) = 4.15

  Loss Decomposition:
         PDCxPT INT: 42.4%
        UNEXPLAINED: 25.0%
               MEAN: 16.44%
                PDC: 13.67%
                 PT: 2.49%

  PDC Decomposition:
           FACTOR 1: 8.27%
           FACTOR 2: 3.8%
            PDF INT: 1.6%

Taguchi Loss Function Analysis — Structured (Slide 31)¶

Bishop’s reference (per-factor decomposition):

PDCxPT INT = 42.4%, Unexplained = 25.0%, Mean = 16.4%
F1 = 8.3%, F2 = 3.8%, PT = 2.5%, PDF INT = 1.6%

sds3_loss.plot(structured=True).show()

Potential Process Capability (Slide 32)¶

Bishop’s reference values:

Process Mean = 237.830, Process StdEv = 0.710
PPL = 2.72, PP = 2.34, PPU = 1.95
Pct below LSL = 0%, Pct above USL = 0%

sds3_cap.plot(
    values=study_sds3_val.dataset['PM SDS 3'],
    view='potential',
    nbins=10
).show()

Test of Maximum Information — R2 Residuals XmR (Slide 33)¶

Bishop’s reference: CL = 0, LBL = -2.47, UBL = 2.47

study_sds3_val.execute(chart='X', by=[], value='R2').plot(theme='ggplot').show()

Maximum Information Analysis — R2 Histogram (Slide 34)¶

mi = study_sds3_val.maximum_information()
mi.plot().show()

Bishop Reference Validation with PBTESTDATABASE_T100

The PBTESTDATABASE_T100¶

factors vs plan¶

PM SDS 1: Full Validation Against Bishop’s Reference¶

Using factors (infer structure from data)¶

Original Performance Measurement - Sample Averages (matches)¶

Original Performance Measurement - Sample Standard Deviations (matches)¶

Analysis of Original Performance Measurement Behavior By Factor (Sample Averages and Standard Deviations)¶

Process Design Conditions Main Effects - Factor 1 and Factor 2 - Sample Averages (matches)¶

Process Design Conditions Main Effects - Factor 1 and Factor 2 - Sample Standard Deviations (matches)¶

Process Design Conditions Main Effects - Factor 1 - Sample Averages (matches)¶

Process Design Conditions Main Effects - Factor 1 - Sample Standard Deviations (matches)¶

Process Design Conditions Main Effects - Factor 2 - Sample Averages (matches)¶

Process Design Conditions Main Effects - Factor 2 - Sample Standard Deviations (matches)¶

Process Design Conditions by Production Time Effects (R3) - Sample Averages (matches)¶

Production Time Interaction Effects (R3) Standard Deviations (Match)¶

Potential Process Capability¶

Test of Maximum Information R2 (Match)¶

Using plan (specify expected structure)¶

PM SDS 2: Full Validation Against Bishop’s Reference¶

Analysis of Original Performance Measurement Behavior - PM Individuals Chart¶

Process Design Conditions Main Effects - Factor 1 and Factor 2 (R6 Recentered)¶

Process Design Conditions Main Effects - Factor 1 (R6 Recentered)¶

Process Design Conditions Main Effects - Factor 2 (R6 Recentered)¶

Test of Maximum Information R2¶

PM SDS 3: Full Validation Against Bishop’s Reference¶

Analysis of Original Performance Measurement Behavior — PM Individuals Chart (Slides 1-2)¶

Analysis of Original PM Behavior — XmR by PDC RSG (Slides 3-18)¶

Current Process Capability (Slide 19)¶

Process Design Condition Main Effects — PDC RSG Sample Averages (Slide 20)¶

Process Design Condition Main Effects — PDC RSG Sample Standard Deviations (Slide 21)¶

Production Time Main Effects — PT RSG Sample Averages (Slide 22)¶

Production Time Main Effects — PT RSG Sample Standard Deviations (Slide 23)¶

Process Design Design Factor Main Effects — Factor 1 Sample Averages (Slide 24)¶

Process Design Design Factor Main Effects — Factor 1 Sample Standard Deviations (Slide 25)¶

Process Design Design Factor Main Effects — Factor 2 Sample Averages (Slide 26)¶

Process Design Design Factor Main Effects — Factor 2 Sample Standard Deviations (Slide 27)¶

Analysis of PDCxPT Interaction — R3 Residuals Individuals Chart (Slides 28-29)¶

Taguchi Loss Function Analysis (Slide 30)¶

Taguchi Loss Function Analysis — Structured (Slide 31)¶

Potential Process Capability (Slide 32)¶

Test of Maximum Information — R2 Residuals XmR (Slide 33)¶

Maximum Information Analysis — R2 Histogram (Slide 34)¶

`factors` vs `plan`¶

Using `factors` (infer structure from data)¶

Using `plan` (specify expected structure)¶