Yes you can detect contamination in MilliQ by taste. I don’t know the threshold, but I do know that if it has a taste then the water is contaminated enough to ruin your experiments.

Are the contaminants dangerous? Or just normal microbial flora that you'd get on kitchen glassware?

No they are not dangerous. Pathogens are adapted to living in nutrient rich environments (i.e. our bodies) where their problem is avoiding our immune system. The bacteria that can grow in MilliQ water are those adapted to living in low nutrient environments and these bacteria are always harmless - except that is to your experiments :)

Wouldn't the very act of drinking it contaminate that source with bacteria from your lips?

I don't work in a lab, but I assume you'd pour it into a cup/beaker and drink out of that

Don't drink it directly out of the container...