We’ve all been there: You make sure to diligently wash your face (every. single. night.), towel off your skin, swipe on your nighttime serums and creams, and then go to bed with high hopes of a clear and copacetic complexion come morning. But overnight, when pimples sprout out from seemingly nowhere—despite the fact that you’ve followed our entire beauty regimen to a tee—it’s totally normal to wonder what gives.

Sure, acne can sprout up for a number of reasons—from hormones to stress—but there’s another culprit that could be right there in front of you without you even realizing it: your dirty towel. Gulp. “Oil, dirt, and makeup deposit onto your towel after you use it,” says New York City dermatologist Joshua Zeichner, MD. “This creates a breeding ground that allows bacteria to grow on it.” AKA: If you don’t wash it, the fibers become laced with bacteria and then that gets transferred to your face.

Worse still? If you leave your towel to dry in an always-humid environment like the bathroom. “Wet, used towels can carry bacteria and fungus, transmitting them to the skin can cause breakouts and rashes,” adds Robert Anolik, MD, a New York-based dermatologist. So despite your best intentions of washing away pollution and acne-causing oil and grime, by simply drying off your face with a used towel, you could be aggravating your freshly cleaned complexion, undoing all that good in one pat.

Towels are particularly problem-causing if you don’t have a normal skin type. “People who have extremely dry or sensitive skin, those who are acne prone, and people with skin conditions like eczema are the most at risk for a dirty towel causing breakouts since their skin barrier isn’t functioning as well as it should be,” says Dr. Zeichner. And this means that you could spread bacteria or fungus to your face, which blocks pores and makes flare-ups even worse.

I wish it stopped there, folks, but beyond creating an acne-friendly environment, towels can even make certain inflammatory skin conditions worse (sorry to bear the news). “Rough towels used to clean the skin can be harsh and cause irritation, dry skin, flaking, or even worsen eczema,” says Dr. Anolik. So pay close attention when towel shopping to stick with the soft, plush options that feel like they won’t provoke your skin.

As for preventing towel-caused breakouts, the key is all about keeping them clean. “It’s important to wash your towels regularly and not keep reusing a dirty towel,” says Dr. Zeichner. “I generally recommend that towels be washed once per week, or right away if there’s any visible staining.” No more living like ya did in a dorm room and using the same towel all semester (just me?). To maintain that glow, you’ve got to keep them fresh.

Speaking of cleaning, here’s how to use baking soda to keep your laundry smelling good. Or you can even DIY your own laundry detergent.