A wealthy businessman is facing having to dig up his tennis court and tear down a home gym after building them without permission in his field.