You're not required to link back to someone who links to you. There are some that I link to that don't link back to me, and some that link to me that I don't link back to.
I separate mutual linking from comics that I just link to on my Links
page. I give the ones I'm trading with a short description so it isn't just a name on a page. In my own case, the best clickers-into-readers rates I've had have been from sites talking me up.
I know not everyone separates this, but I like to have the distinction.
If there was never an initial conversation about linking in the first place, there's no obligation. If someone emails you and asks but you haven't read theirs or just don't want to, let them down gently.