He's squatting a big chunk of your square footage. I had a neighbor tried that and when I bought the house I made him move the fence. It was his responsibility to check property lines before he built it. In the county I live in you need a permit and an inspection for that fence.