I completely fail to understand the fast travel rules in Mementos.

Sometimes I can go up, sometimes I can go down, sometimes I can go anywhere.

It's inscrutable to me.