Lyric discussion by JuliaWiersum 

This is a song about an affair, in which both of those involved are ashamed. The female is the other woman, while the man is the one who has a significant other. The female begins, saying “leave me out with the waste, this is not what I do.” She means she knows she is doing something dirty and wrong, so she belongs where garbage belongs - where it isn’t to be seen. In a more literal sense, she hopes that she is left out of all of his conversation, because she can’t let anyone know what she is doing. She feels shame, and admits that this is something that she knows is morally wrong, and it’s not something that is within her character to do. It’s the “wrong kind of place” because the place, the setting, is that he is with someone else. Therefore, she shouldn’t even think of him. She’s either telling him that it’s the wrong time for “somebody new” meaning her; or she’s telling herself it’s the wrong time for somebody new on her part. Perhaps she has a significant other of her own on whom she is cheating as well. The line “it’s a small crime and I got no excuse” is very self-explanatory as within the world and all of its large crimes, being a mistress is not a big deal. ...but it’s still a crime. There is no way to prove it innocent or acceptable. So many times she has asked herself “is that alright?” because she feels the apprehension and the doubts. Deep down, she knows it’s wrong, but at the surface, she says it’s alright, because she is still engaged in the affair and the affair is something she wants. It’s a melancholy, thoughtless, almost sarcastic “yeah,” as it is said quickly, yet not with the assurance an truly affirmative reply would normally have.

As for the gun, there’s confusion between the two involved in the affair. There is one gun, and someone has to shoot it because someone will be hurt in the end. The gun is loaded and ready to shoot and kill, since the affair is happening. The shot happens when the affair is exposed. But who exposes it? Whose gun is it? Can it be given away so that someone else can do the shooting? She’s not used to hurting people the way she is, so how is she going to hold the gun if the man doesn’t shoot it for her?

I was going to write the exact same thing.

Just to add to this: I think the woman is singing to her lover, while the man is singing to his partner (therefore making it likely that your first assumption - the other woman and the adulterer - is right). The other woman is more involved in their relationship than her lover is, because he's in a relationship, hence why her verse/chorus sound like pleas, while his verse/chorus sound like he feels torn between the two, and therefore guilty about cheating on his actual partner.

An error occured.