When Your “Better” Features Make Your Model Worse

When Your “Better” Features Make Your Model Worse

You know that feeling? You spend days engineering a beautiful new feature. It captures exactly the signal you wanted. You train, evaluate, and your Class 2 F1 jumps from 0.72 to 0.78. You smile. Then you submit to the leaderboard and your score dropped. What happened? You just met the feature engineering trade-off.

It’s a trap that almost every data scientist falls into at least once. I fell into it myself a few months ago, and it cost me a solid week of debugging. So let me walk you through what’s actually happening under the hood, why your clever manual features can backfire, and how to build a feature evaluation workflow that doesn’t lie to you.

The Classic Trap: One Metric’s Win Is Another Metric’s Loss

Let me paint the exact scenario. I was building a multi-class classification model for a client. Baseline was okay. Then I added two hand-crafted features: **Impact Severity** and **Energy Ratio**. Both were derived from raw sensor data, designed to catch a rare fault class. They made physical sense. On validation, Class 2 F1 improved from 0.72 to 0.78. Beautiful, right?

Not so fast. When I looked at the full picture, Class 0 and Class 1 F1 had both fallen. The leaderboard—which used a weighted average across all classes—was lower than my baseline. The model had effectively learned to sacrifice the majority classes to make the minority class look better.

This isn’t bad luck. It’s the mathematical reality of a shared probability budget. In most multi-class models, increasing one class’s confidence reduces another’s. If your new features are highly correlated with existing ones, you’re not adding information—you’re amplifying noise and skew.

Why Do Manual Features Backfire?

Multicollinearity Is a Silent Killer

When your new feature is correlated with two or three existing features, the model splits credit among them. In tree-based models, this can hide important splits. In linear models, it inflates coefficients. The result? Less stable predictions, especially on unseen data.

The worst part is that multicollinearity doesn’t always hurt training accuracy. It can even help you memorize noise. But the moment the test distribution shifts slightly, your fancy new feature becomes a liability.

The Precision/Recall Trade-Off

If you engineered a feature to find Class 2, you’re likely increasing recall for Class 2 while destroying precision for Class 1. Suppose your threshold shifts. More examples get classified as Class 2, but many of them are actually Class 1 or Class 0. Your macro score drops even though your “target class” looks better.

I see this all the time in fraud detection. Someone builds a feature that catches more fraud, but it also flags a ton of legitimate transactions. The model’s recall for fraud goes up, but the false positive rate doubles. In the real world, that means angry customers and a very stressed support team.

Leaderboard Metrics Are Usually Aggregated

Most leaderboards use a single scalar metric—accuracy, macro-F1, weighted kappa, or log loss. That scalar hides the per-class breakdown. If you optimize for one slice, you’re not optimizing for the whole. A five-point gain in Class 2 F1 can be wiped out by a three-point drop in two other classes, depending on class weights.

This is why the first thing I now do is read the competition or project metric extremely carefully. If it’s weighted F1, then improvements in a rare class are tiny. If it’s macro-F1, every class matters equally, even the ones you don’t care about.

A Real-World Walkthrough: The Sensor Data Case

Let me give you a concrete example. Picture a manufacturing plant with sensors measuring temperature, vibration, and pressure. Baseline model: 0.85 macro-F1 across three fault classes.

I created an **energy ratio** = vibration amplitude / pressure variance. It made physical sense. High-energy impacts should indicate harsh faults. On the training set, Class 2 F1 jumped to 0.81. But in cross-validation, the gain didn’t hold. Why? The energy ratio was highly correlated with raw vibration and pressure—multicollinearity again. In test data, sensor noise at low pressure made the ratio explode, sending the model into false Class 2 predictions.

The fix was counterintuitive: instead of adding the raw ratio, I used a residual feature—the part of vibration that was *not* explained by pressure. That orthogonalized signal actually added new information, and the leaderboard score improved across all classes. Lesson: don’t ask “does my feature make sense?” Ask “does my feature provide information that my current model doesn’t already have?”

Another Example: E-Commerce Churn Prediction

I worked with a subscription box company. The marketing team wanted a **customer engagement score** that combined login frequency, support tickets, and social media mentions. It sounded great. But the score was just a weighted sum of existing features. When I added it to the model, feature importance rankings shuffled, but AUC stayed flat. Worse, calibration became worse for low-engagement users because the score dominated the loss.

We ended up replacing the handcrafted score with a simple count of distinct feature categories contacted in the last seven days. That small, less correlated feature gave a real lift. The lesson repeated: sometimes the least flashy feature is the best one.

How to Evaluate Features Without Getting Tricked

1. Build a Multi-Metric Scorecard

Don’t just track F1 for Class 2. Build a scorecard that includes:

- Macro-F1 and weighted-F1
- Precision and recall for each class
- Log loss or Brier score (for calibration)
- A note on whether per-class metrics moved in the same direction

Here’s what I usually print out after every experiment:

| Metric | Baseline | New Feature | Verdict |
|---|---|---|---|
| Class 0 F1 | 0.88 | 0.84 | Bad |
| Class 1 F1 | 0.82 | 0.79 | Bad |
| Class 2 F1 | 0.72 | 0.78 | Good |
| Macro-F1 | 0.81 | 0.80 | Bad |

One glance and the story is clear. The shiny new feature helped one class but hurt the others. Without this scorecard, I would have submitted the model and gotten a nasty surprise.

2. Run a Correlation Audit

Before adding a manual feature, check its correlation with existing features. Use pandas `df.corr()` and look for absolute values above 0.7. If your new feature is too close to existing ones, consider:

- Dropping one of the redundant features
- Using residualization (regress the new feature on old features, take the residual)
- Using feature selection via scikit-learn’s `SelectKBest` or `RFECV`

Scikit-learn’s official feature selection guide is loaded with practical approaches: https://scikit-learn.org/stable/modules/feature_selection.html

3. Use SHAP to Understand Directional Effects

After training, compute SHAP values to see exactly how your new feature affects each class. Sometimes a feature is useful for Class 2 but actively harmful for Class 1. A SHAP summary plot will show you this immediately. The library is open-source and well maintained at https://github.com/shap/shap.

I’ve caught several “great” features that had strong positive SHAP values for one class but even stronger negative values for another class. That’s the trade-off you need to see before you submit.

4. Try Ablations

Ablation means removing a feature and seeing what happens. It’s surprisingly rare in manual feature engineering. I always run three experiments:

- Baseline with old features
- Baseline + new feature
- Baseline + new feature, minus one old correlated feature

If the third experiment beats the second, you’ve solved a multicollinearity problem. If the second is worse than the first, remove the feature. If the second is better but the leaderboard eval says otherwise, dig into your validation setup.

A Third Example: Medical Triage Classification

I once worked with a symptom-checker prototype. The goal was to triage patients into three urgency levels. I built a **comorbidity score** by summing up the number of pre-existing conditions. It improved detection of high-urgency patients, but it also made the model ignore the severity of the *current* symptom. A patient with one mild symptom and one comorbidity was classified as high urgency, even though their actual risk was low.

The problem was that my feature was too dominant. The model latched onto it because it was easy to split on, and it ignored the richer, more nuanced text features. This is another key trade-off: **feature dominance**. A strong handcrafted feature can crowd out weaker but more informative features. Regularization and feature interactions can help, but simpler is often better.

The Bigger Lesson: Features Are Not Free

Every feature you add increases model complexity, training time, and the chance of overfitting. Regularization can help, but it can’t fix a fundamentally redundant feature. You have to think of each feature as a bet: it must justify its place by adding new, stable signal across multiple metrics.

I’ve also learned that when a leaderboard drops after a feature addition, the root cause is almost never “the metric is wrong.” It’s that I was chasing a local optimum. The model isn’t stupid—it’s doing exactly what I asked. I just asked for the wrong thing.

FAQ

Why did my F1 score go up on one class but my total leaderboard score went down?

Because the leaderboard probably uses a weighted or macro average. Your model might be sacrificing precision or recall on other classes to gain on the one class you focused on. Always evaluate the full confusion matrix before getting excited.

What’s the best way to handle multicollinear features?

Check correlation first. If two features are highly correlated, drop one or use a residualized version of the new feature. Regularization, like L1 or L2, can help stabilize the model, but removing redundancy is usually more effective than trying to fix it with penalties.

Is manual feature engineering still useful in modern machine learning?

Yes, but only when it encodes domain knowledge that isn’t already in the model. If a feature is just a mathematical combination of existing features, it usually won’t help. If it captures a genuinely new signal, it can be powerful. The key is to verify that it’s actually adding independent information.

How do I know if a feature is actually helping?

Compare the baseline model and the model with the feature using multiple metrics and cross-validation. Do an ablation. Look at feature importance and SHAP values. If the improvement is inconsistent across folds or metrics, it’s probably overfitting noise.

Final Thoughts

I know the thrill of building the perfect feature. It feels like you’ve cracked the code. But the leaderboard is the ultimate referee. The sooner you build multi-metric evaluation into your workflow, the less time you’ll waste on features that only help one slice of the problem.

Next time you’re about to ship that clever “impact severity” feature, stop. Ask yourself: “Is this adding new information, or just amplifying a signal I already have?” Your future self—and your leaderboard ranking—will thank you.