FORTE turns feature removal into a staged production workflow

Feature removal is supposed to save computation, but a feature that looks weak alone may matter alongside others or for a subgroup.

It remains unclear how a production team can narrow many possible removals, test their combined effect, and still check real user-facing behavior before permanently cutting data and model dependencies.

FORTE combines temporal feature screening, joint perturbation checks, retraining, online tests, and lineage tracing to make model feature deprecation operational.

The authors compare a frozen production model on intact examples with the same model after a feature group is permuted or removed.

FORTE ranks candidates over time, checks proposed groups jointly, and sends only survivors to retraining and online testing, because weak features can substitute for one another.

In two production recommender systems, candidates stayed within reported offline bands and showed no statistically significant regression in the tested online metrics.

Online checks selected bounded removals, including 48 Search Relevance features and 8 Pin Ranker features, with measured efficiency gains after launch.

These results cover two Pinterest production models and the reported metrics and traffic segments. FORTE's permutation scores are not conditional importance, and coarse US/non-US monitoring does not establish fairness or rule out harm within a segment.