As a mother what should I do?

I met my husband when I was around 11 or 12, he always tried to Talk to me but i never accepted. When i was 21 i saw him on the corner 1nite and asked if he wanted to have sex.( i was having a real bad day) We had sex, I got pregnant, he wanted it i didn't. I ke pt the baby and while I was pregnant he disappears. My daughter was brn 10wks early and in the NICU for 6wks. The day she was born he called out of nowhere and I told him and he said he would be right down and I didn't hear from him for another 3yrs. He was in jail, but he didn't go to jail until over a month after our daughter was born. any way i get a new man we have two children and then things start going downhill, we breakup just so happen on the same day he gets out of jail. We end up together both with the understanding that before was just sex, now we are going to try. We fight, argue, he hits me, spits on me. he ends up getting caught in in crime life again and is on his way to jail. We get married and on our wedding day we get into a big fight and he hits me and I have him arrested. We get back together and he's awaiting sentencing and we're still fighting, and arguing, nothing physical (he got the message), but loud and constant. The last time he was locked up his fiance left and spent all his money and took all his stuff. so i constantly tell him i'm not her i'm not going anywhere, i'll wait. About three months before he is to turn himself in I find out that he cheated on me from my doctor. He says it was awhile ago but I don't believe that and it doesn't really make a difference. we work through it and i'm still playing boo boo the fool and saying i'll wait. I mean I really do love him. about three weeks before he is to surrender we are at the hotel and he gets a phone call and takes it in the bathroom. Well me and my nosy *** got my ear all the way to the door and I can hear him saying that " when you got out of the shower you went into my wallet and stole my mac card. I'm having it traced" and so forth. I confront him and once again i'm wrong it was some girl his cousin was seeing and they were all swimming and when she took a shower, even as I write this I feel so dumb!! N-Way I stay and he goes to jail and we're still arguing and fighting and the way he talks to me is like i'm beneath him. I haven't talked to him for three days now because he told me he lost respect for me for drinking. I know you read all of this and wander what the hell is her problem but here it is. I'm ready to leave, i'm ready to deal with all the pain that a divorce comes with but all that i said was about him and I, but we're not the only ones in the picture. We have three beautiful children and i say we because he adopted my other two. with all that I wrote he is the most amazing father. and i don't want to hurt their relationship in any way. I don't know what to do, do i stay because I know once I leave nothing will ever be the same for them or do I leave so I can be happy. am i a ***** to put my happiness ahead of theirss???