Factorial Experiments: Why You Should Test Settings Together, Not One at a Time

The problem

Picture a machine that heat-seals packages. Its seals are too weak: they peel open too easily. Three settings could be to blame: the jaw temperature, how long the jaws stay closed (the dwell time) and how hard they press. The question is which settings matter, and what to change.

The obvious approach is to change one setting at a time and keep whatever helps. That works when each setting acts on its own. It fails when settings work together. In the lab’s process, a longer dwell makes seals weaker at 150 °C but stronger at 170 °C. Test dwell time only at the low temperature, and you’ll keep it short. You’ll never try the combination that works best.

A factorial experiment tests every combination instead. Three settings, each at a low and a high level, give 2 × 2 × 2 = eight combinations. From those eight results you can measure what each setting does on its own, called its main effect. You can also measure how much one setting changes what another one does. That’s called an interaction, like dwell time helping only when the temperature is high.

I built the lab so you can run that experiment and read the effects yourself. On the same process, you can also watch the one-at-a-time approach settle on weaker settings. The rest of this note explains what the lab calculates and how it tells real effects from noise.

How the lab works

The lab is a 2³ full factorial on a heat-seal process. That means three factors, or settings, each tested at two levels, in every combination. The factors are A, seal temperature (150/170 °C); B, dwell time (0.8/1.2 s); and C, jaw pressure (40/60 psi). The response, the result we measure, is peel strength in N/15 mm. That’s the force in newtons needed to pull apart a strip of seal 15 mm wide. Each factor is coded −1 at its low level and +1 at its high level. Then the eight combinations sit at the corners of a cube, called the design cube. Each replicate, or round, runs all eight in a new random order. The results come from a hidden model: y = 18 + 2.4A + 0.3B + 0.25C + 2.1AB − 0.2AC + ε, with ε ~ N(0, 0.6²). Here ε is random noise from a bell curve centered on 0, with a standard deviation (typical spread) of 0.6. You can run up to three rounds in total. Run orders and noise draws come from fixed seeds, so every run can be repeated exactly.

The math

Each effect is a simple difference of averages. Every effect has a contrast column, which gives each corner a + or a − sign. The effect is the average result at the + corners minus the average at the − corners. For a main effect, the column is just that factor’s own level. For AB, it’s the product A×B, and so on up to ABC. In a 2³ design the seven columns are orthogonal, meaning they don’t overlap: each one measures something separate. So every effect uses all eight corners, and it equals twice the matching coefficient in the coded model. The AB effect is also half the difference between B’s effect at high A (A+) and B’s effect at low A (A−). That’s how the lab’s verdict reports it. To decide if an effect beats the noise, the lab needs a noise estimate. With one round, it uses Lenth’s pseudo standard error, which estimates noise from the smallest effects. With two or three rounds, it pools the scatter between repeat runs at each corner into one number, s. The standard error SE = 2s/√n, where n is the total number of runs, is the typical size of noise in an effect. An effect counts as real when it’s bigger than SE times a two-sided 95% t threshold, a cutoff that pure noise rarely crosses.

An interaction means the effect of one factor depends on the level of another. It isn’t just two big main effects happening at the same time.

Using the lab

Run 8 trials runs one round, one trial at a time. Run 8 more repeats the design, up to three rounds in total, and tightens the noise threshold. Hover over or tap an effect to light up the corners where its contrast is +. That’s a face of the cube for a main effect, and a diagonal slice for a two-factor interaction. For ABC, it’s a tetrahedron, a pyramid with four triangle faces. The interaction plot shows any pair of factors, averaged over the third. If the lines aren’t parallel, that’s the interaction. Compare one at a time runs a one-factor-at-a-time (OFAT) study on the same process, starting from the corner where everything is low. Dwell time looks harmful at 150 °C, so OFAT leaves it low. It misses the best corner, which needs high temperature and long dwell together. The Run-to-run noise switch can set σ to zero, so you can check the estimates against the true coefficients. Show truth (under Answer key) draws the true corner averages and effects on top. Clicking a corner shows its value from each round.

What the lab assumes, and what it leaves out

The results are made up on purpose. They come from a known formula with a strong AB interaction and bell-curve noise. That way you can check the method against the true answer. Lenth’s method assumes most effects are really zero. The threshold for repeated rounds assumes the noise is the same size at every corner. The lab can’t spot curvature, where an effect bends instead of following a straight line. That would need center points, runs halfway between low and high. It also leaves out blocking (grouping runs, say by batch or day), drift over the run order and errors in the measuring system. Settings that are hard to change and residual checks (studying what the model missed) aren’t covered either. A real study would add center points, plus confirmation runs at the chosen settings, before changing a recipe. It would also ask whether an effect is big enough to matter in practice, not just whether it beats the noise.

Checks that it works right

With noise off, each estimated effect equals twice its true coefficient: A +4.80, AB +4.20, B +0.60, C +0.50, AC −0.40, and zero for BC and ABC. Changing one true coefficient moves exactly one estimate, because the contrast columns don’t overlap. AB equals half the difference between B’s effect at the two temperatures. With noise off, Lenth’s method still flags only A and AB. That’s a reminder that it treats the smallest effects as noise, even when they’re real. OFAT runs on its own seed, so comparing never changes the factorial results. With noise off, OFAT settles on A+ B− C+ at 18.05 N, while A+ B+ C+ gives 22.85 N.