Am I being unreasonable about my baby?

It is your job as a parent to make sure your child is well taken care of. This includes second hand smoke. If they want to watch your baby they should provide a non-smoking environment if that is what you request. There are tons of studies about the danger of second hand smoke you might want to bring some along to back you up. Good luck.