"I guess legally he has a point" - I guess legally he absolutely doesn't. Certainly not under a criminal duty regarding 'health and safety'. Remember that H&S only applies where there is one party at least who takes the role of an employer. And it's the employer who hs to consider what THEY do that might be affecting you. Not the other way around. Sounds like a convenient excuse to me. If it comes to trading blows at all (not physically) be sure to ask for precise clarification on anything that he says is a valid reason. Don't argue it, just get the exact nature of what law, regulation, by law, duty of care, code of practice or whatever he's quoting. Being liable for something requires a test to say that you're liable in the first place. H&S is criminal law, not civil, for which there'd have to be a breach and that usually ends in a determination of guilt, not liability. Do remember that councils have to cover both criminal and civil requirements in some cases, so just because they come from the Environmental Health Dept, does not mean it's a criminal matter and enforceable. This sounds much more it's to do with occupation and landlord / tenant agreements.
Many people genuinely think that they're liable for things that they simply are not. Many old wives' tales out there.