If you're lactose intolerant or have an allergy to milk proteins - you're well aware of how milk wrecks your digestion and generally makes you feel like shit.
If you don't have either of these conditions, you could be forgiven for thinking that dairy is a reasonable thing to eat and might even be healthy. After all, the dairy industry has spent countless millions trying to convince you of this over the years (remember those "got milk" ads?).
I will make the case that dairy is simply not a healthy thing to eat for anybody, regardless of whether you have a specific intolerance or not. As I'll lay out below, there are simply too many troubling issues - if even one of these were present in a food it would be wise to avoid it - the combined effect of these factors makes it impossible to consider dairy even a remotely healthy food.
Mold Toxin Contamination
I've written previously about how dangerous mold toxins ("mycotoxins") can be. Not only are mycotoxins strongly oxidative and pro-inflammatory in the body, some mycotoxins can actually mimic our hormones and disrupt our endocrine system. Creepy stuff.
The primary way mycotoxins enter our food system is via large-scale grain production, specifically wheat and corn production. These grains sit in grain silos for months at a time and almost invariably become contaminated with mold in the process (some years, as high as 92% of corn crops test positive for mycotoxin contamination). So while minimizing consumption of these grains is a good step, it's not enough.
The digestive system of a cow is designed to process grass, and essentially grass alone. Yet, because it's cheaper and easier, 99% of dairy cows in the world are fed grains (generally corn). Beyond this, the corn fed to dairy cows is of a lower quality than what makes it to market for human consumption. This means cows are almost guaranteed to be eating feed that is significantly contaminated with mycotoxins.
The dairy cow effectively serves as a "mycotoxin accumulator", storing the mycotoxins in its fatty tissues, including in the milk fats. As several studies have shown, levels of mycotoxins in milk are often dangerously high. 
This "toxin accumulator" effect doesn't only apply to mycotoxins either. Other persistent organic pollutants like polychlorinated biphenyls (PCB's) and pesticides have also been shown to accumulate in dairy milk. 
Because organic grains are also highly likely to be contaminated with mycotoxins, buying organic milk does not reduce your risk of exposure.
Inflammatory Proteins (A1 vs A2 casein)
This is a somewhat new revelation, but perhaps the most compelling reason of all to minimize your milk consumption.
Casein, one of the primary proteins in milk, comes in two forms, referred to as A1 and A2 casein. Historically, the casein produced by dairy cows was almost entirely of the A2 variety. In the past few decades that has changed dramatically as dairy farmers discovered that cow breeds with genetics for A1 casein also happen to produce more milk.
These cow breeds naturally became preferred by commercial dairy producers and have increased dramatically in numbers over the past few decades. Far and away the most common breed of dairy cow today is the Fresian Holstein, which produces the highest levels of A1 casein. Almost all milk on supermarket shelves at this point contains significant levels of A1 casein.
Here's the problem:
Researchers have recently discovered that A1 casein triggers gastrointestinal inflammation in nearly everyone they test.  Studies on mice have confirmed that A1 casein, unlike A2 casein, triggers the pro-inflammatory Th2 pathway leading to gastrointestinal and systemic inflammation. 
I don't need to reiterate why inflammation is the enemy of a healthy body and brain. As I've written about at length, minimizing inflammation in the body is perhaps the single most important factor in preventing disease and improving mental and physical performance. Removing pro-inflammatory foods like dairy from your diet is the critical fist step.
Hormone and Antibiotic Residues
I don't know how else to say it - commercial dairy production is a dirty, dirty industry. Cows are confined to small spaces and fed low-quality feeds comprised of things they were not designed to eat in the first place. These are not healthy animals.
It's been well publicized that antibiotics are commonly used in commercial dairy production, but the reason why might surprise you. Decades ago, dairy and beef farmers started giving cows antibiotics to treat the diseases that were becoming a problem in new factory farm conditions. This did cut down on disease, but something unexpected also started happening - the cows started getting fatter and producing more milk. Very quickly, antibiotics started to be used as a tool for fattening animals and improving yields. Today, the vast majority of antibiotic use in dairy production is for this reason.
Unsurprisingly, antibiotic residues consistently show up in milk samples when tested.  As with mycotoxins, buying organic dairy products does not significantly reduce risk of exposure, as restrictions on use of antibiotics in organic milk production are lax and unenforced.
Consistently consuming any product with antibiotic residues will have a meaningful impact on your gut microbiome, which will have major adverse effects on your digestion, immunity and overall health. After all, the reason cows get fat when given antibiotics is because their gut bacteria is completely destroyed.
Non-organic dairy products add the risk of exposure to hormone residues. Bovine somatotropin (rBST) is widely used in conventional dairy production (also to increase yield). Studies have shown that residues of this cow hormone show up in conventional milk, along with increased levels of IGF-1. 
Histamines and Nitrosamines
This is a bonus for you cheese lovers, as I've learned that you are generally the most emotionally attached to your dairy and perhaps need a bit of extra encouragement to make some shifts in your diet.
You probably recognize the term histamine, these are one type of organic compound our bodies use to trigger an inflammatory response. When you have allergy symptoms, you take an anti-histamine, which will remedy your stuffy (i.e. inflamed) nose and sinuses.
Well, as it turns out, when you ferment a food with protein in it, histamines are one of the byproducts of this process. Most of the fermented foods we eat are low in protein (cabbage, grape juice, etc), but cheese is one of the notable exceptions. 
Generally, aged or ripened cheeses are allowed to ferment the longest, making them the most likely to have high histamine levels.
Also produced during the fermentation of some cheeses are nitrosamines, related organic compounds that have been connected to gastric and oesophageal cancer.  
The Only Time It May Be Safe(r) To Eat Dairy
As you can probably tell, it's my opinion that the negative effects of consuming dairy make it near-impossible to justify eating with any regularity. I know countless people who have seen their health and performance improve radically after dropping dairy from their diet. Most of these people likely did not have a lactose intolerance or specific milk allergy, but were just consistently eating a food that was pro-inflammatory and exposed them to numerous toxins. Pulling this out of the diet is inevitably going to produce big improvements.
Hypothetically, however, there may be a type of dairy that removes enough of these risks to make it a reasonable food to eat on occasion. If you can find a completely organic (eliminate risk of rBST), grass-fed (eliminate risk of mycotoxins) milk produced by exclusively Guernsey cows (lowest levels of A1 casein) - this milk would be free of a lot of the problems that plague 99.9% of dairy products on shelves today.
I haven't looked extensively, but I've never been able to find a milk that meets all of these criteria. Grass-fed dairy is rare enough, but compounding the problem is that most farmers do not know the genetics of their cows with enough certainty to guarantee they are full-breed Guernsey. Still, if you MUST have milk, this could be something worth inquiring about at your local farmers market.
Looking For More On Avoiding Toxic Foods?
Oxalates, Phytates and Saponins: The Secret Anti-Nutrients In "Healthy" Foods