How to explain to a 4 year old why you can't visit him at his home.?

Yes my 4 year old asked me this today. I think hinting why I couldn't come to his family birthday birthday party.  How do you explain to him that if you go to his family birthday other people important to him would leave - and that is also not fair to a little boy. 

I am in the situation of a little boy who can't understand why I am not wanted there plus why would I want to spend anytime in a house full of abuse. My mother I law still claims she has no guilt to the way she treats me and operatively in front of the kids I hate it. I can't stand that abusive disrespectful atmosphere.  But I can't say that to my son. Because those guys are looking after my kids because I can't. - legally not and medically I am just barely keeping my head above water.  

I guess since I am religious I'll ask for a little assistance from up above on this one.