Lyric discussion by AlfiePark 

Cover art for Dory lyrics by Grizzly Bear

This seems to be a more detailed version of the lines "And it was mamma, that got drowned in out, in that deep blue sea" from Deep Blue Sea.

Dory could be the mother's name, which is kind of a pun, as the definition of dory is " a flat-bottomed boat with high flaring sides"

So 'We’ll swim around like two dories" could be almost a macabre pun about Dory drowning...translated to "we won't swim well".

I don't know.

My Interpretation