the dairy thing

Several posts ago, in my little diet manifesto I mentioned that I try not to take in any dairy.  Here's the deal on that:  I don't (can't) have anything high-lactose, without immediate - uh - unsettling results.  So this rules out cow's milk, and lots of cheeses (which I love), and absolutely no ice cream, etc.  There are dairy products with lower lactose, like Greek Yogurt, which I eat all the time.  I also eat some cheese "like a spice", in salads, as long as there isn't too much of it.  I do think there have been some benefits to this apart from eliminating the "unsettling results", namely in terms of sinus allergies and getting colds.  But here's the problem with not having milk:  It is extremely hard for me to gain weight/maintain mass without it.
Gaining weight is not what a great many people spend time strategizing.  I  have a higher than average metabolism, so losing weight is super easy, while keeping healthy weight on is very difficult.  Truth be told, trying to gain enough weight to not be in the "skinny" category has been a life-long endeavor/struggle for me, but not only because of my metabolism -- I'm also a very picky eater who tended to pick the most unhealthy options earlier in life.  So these things combined, left me the way I was: pretty unhealthy and bony.  As I got older, my metabolism slowed down, and my poor eating/drinking habits caught up really quickly...my weight stayed low but my body fat % increased.  But, of course, after healthy diet and exercise changed me, things resolved, and at one point I was able to reach 183 lbs with a low body fat% (when I graduated highschool, I was 6'2", and around 130lbs!  not good).  So here's where milk figures in...in order to gain that healthy weight, I drank a lot of whole milk.  A LOT.  Without it, I find it really difficult to get back to that weight.  I currently weigh about 170, with a low (healthy) body fat %, but would like those extra muscle-pounds back.  I've been doing my best, but cardio is too effective... it just pulls everything off of me.  So.  I'm taking a break from the cardio-focused exercise (i.e. running, T25, etc.) that has been my last 8 months, and going back to moderate resistance training.  Nothing crazy -- I don't want another hernia.