Where would the dairy-free world even be without alt-milks and nut butters? These plant-based options make perfect add-ons to your coffee cup and your apple slices, but Stephen Pasterino, the trainer behind New York City’s P.Volve, has a super controversial opinion about what savoring both oh-so-delicious vegan staples in one day might mean for your fitness goals.

“I just have one message that I kind of keep saying over and over again. It’s lay off all those nut milks and nut butters. I mean, everyone’s going crazy with those right now,” says Pasterino on the most recent episode of the That’s So Retrograde podcast. But before you toss out your butter collection (I’m talkin’ almond, cashew, macadamia, etcetera), know that a nutritionist says Pasterino’s pretty aggressive and controversial advice comes with one major BUT: You have to be eating a lot of other fats in your diet for your love affair with these healthy varieties to become detrimental.

“If you’re using some almond milk in your coffee, even if you have several cups of coffee a day, that’s not really an issue,” explains Bonnie Taub-Dix, RDN, author of Read It Before You Eat It. “An ounce of almonds is about 170 calories, and it provides six grams of protein and healthy fat.” Unsweetened almond milk, for instance, contains only about 30 calories and 2.5 grams of fat per one cup serving, and chances are that your eight-ounce cup of morning joe doesn’t call for that much anyway. Almond or peanut butter makes a larger dent in your recommended daily intake at around 100 calories and 10 grams of fat per one tablespoon serving, but Taub-Dix points out it makes it a way better option than most other items you might grab out of the office snack drawer because it’s nutrient rich and well keep you satisfied longer.

“What I have found is that when people include these nuts and healthy fats as a part of a diet, especially when this kind of fat is replacing less healthy fat, they feel more satiated.” —Bonnie Taub-Dix, RDN

“What I have found is that when people include these nuts and healthy fats as a part of a diet, especially when this kind of fat is replacing less healthy fat, they feel more satiated. For those people who go fat-free or have a low-fat diet, that tends to backfire because it leaves you unsatisfied and the food you eat doesn’t act as long in your system,” the nutritionist says. But of course, any healthy food can be over-consumed, so eating your nut butter of choice by spoonful from the jar might knock the “healthy” right off the “healthy fat” title.

The takeaway? Be mindful of portion size, always, and ask yourself how the nut-derived noshing sesh will fit into the rest of your diet as a whole. For instance, Taub-Dix says, if you’re making an olive-oil-based pasta for dinner, think about leaving that jar of nutty goodness in the fridge for tomorrow’s fuel.

Here’s why that almond butter belongs in your smoothies, and how to know which all-star healthy fat is best for you.