They weren't trespassing
If you are on my property, and refuse to leave when asked, you are trespassing. Simple trespassing is a misdemeanor, enforceable under the criminal code.
Since the OP made no mention of asking them to leave, I agree with you, they were not trespassing, however, should you go back and review my post, I think you will find that was never my assertion.