One problem is exposure to air. Anything that is open in the air is exposed to what is essentially a microbial rain. A single breath contains tens to hundreds of thousands of microbes, and your lungs aren't the ultimate source -- it's what you've breathed in from outside (though the lungs do have a microbiome, we've come to understand). A typical bacterial cell has a mass of about a picogram, which is 10^-12 grams... it doesn't take much air movement to keep it aloft. If you don't believe that, you can buy
mold test plates from the hardware store -- just pop em open and leave them on the counter where you typically work.
This impacts everything that you handle in open air. That means the carrier oil, the vials, the caps, and so forth. Anything that's been sterilized that's open in the air is now no longer sterile.
Filtration removes this because anything that goes into the vial passes the filter -- if your oil was opened in the air, who cares, cause it won't pass the filter. Another way to do it is what microbiologists do, which is to work under a flame. If you light a candle, the hot air rises and forms an effective curtain over the area you're working. Again, since these are insanely light particles that we're talking about (even 'heavier' stuff like dust) will just be diverted along the convection current. That won't remove any impurities already in your oil or raw compounds, but it will minimize the contamination associated with handling them.