DH bothering neighbor with small children

DH has now become fixated with a neighbor family with 3 small children. Their house has very open picture windows and he’s walked over there to talk to them, apparently looking in the window and inviting the little ones to come over to our house. He likes people and likes kids and likes to have people come over. He’s harmless but I’m sure this is a bother to them. The dad is rightfully concerned. They know he has dementia as most all the neighbors do. I assured them that he’s never home alone. I’ve tried my best to keep track of DH’s whereabouts but sometimes he walks outside and crosses the street while I’m doing something else. If I see him over there I go over to collect him.

I’m sure you all understand how difficult it is to change his mind if he wants to walk over there. I can be more vigilant when he opens the front door and make sure to go with him. I have a special lock I use at night after he wandered out early in the morning but I hate to use it during the day and make him feel like a prisoner. Any suggestions?