Hard Time killing Floor Lyrics

Lyric discussion by joe2475 

Cover art for Hard Time killing Floor lyrics by Skip James

Starting to learn this on guitar (beware, VERY funky tuning). The one verse I would like to look at is this:

And the people are driftin' from door to door Can't find no heaven, I don't care where they go

This seems to me awfully dark. The protagonist(?) is looking at the other people and basically saying 'Listen, I'm so poor I got my own stuff to worry about. I don't care where you go.' That's harsh.

My Interpretation