I suspect that you are feeling somewhat overloaded when you are not sure of where you're going and that this is causing a bit of a vicious cycle of thoughts that may undermine your abilities and then you make a mistake. You're right that you would benefit from giving yourself just a few minutes to collect your thoughts; it is your time after all
I think it's fair to say that many people take wrong turns and make mistakes, sat nav or not. It's how they respond to their mistakes which sets some people apart. I'm not suggesting that you try to memorise entire journeys, just that you look at them in useful ways before setting out on them. A few landmarks should be enough. Don't get wrapped up in the details of the journey, as if you've generally looked at a map, google earth, a sat nav etc etc, you'll remember the necessary information when you need it. I think you might be trying to memorise too much detail and it's causing you bother, so maybe break a journey down into little parts? Even if you get the general direction right but the incidental turns wrong, you can find your way back to the route you originally intended, should you get way laid.
There's a little trick which might help you. Just remember that this is about practice, not instant perfection. The more you practice this correctly, the better you will become at it.
No matter where you're going, before you even turn the ignition on, just sit in the seat for a couple of minutes in order to change your mental and physical state. Use this time to really calm your thoughts down, as sometimes they can fire at an alarming rate and cause a sort of background static which interferes with clear thinking. One way of doing this, I call '7-11' breathing. So you breathe in for a count of 7 and out for a count of 11. Do this for two minutes or until you feel certain that you have quietened your thoughts. You'll notice that at some point, the amount of interfering thoughts have reduced. If you find that you have become distracted while doing this task, that's fine, just notice that you have become distracted and give yourself permission to not think about whatever it is at that time, because you probably don't need to. Then calmly bring yourself back to the task at hand.
You know how to drive and you know where you want to go, so it's the getting there that counts. There will always be impatient people on the road, so don't get flustered if you meet one, as we all do. If you do meet someone who knows where they're going when you might not be sure where you are going, find a safe place to pull over and let them pass. Remember, this is a practice, so take a moment or two to let go of what just happened before moving off, as it's possible for one error to impact judgement such that it causes others.
Then when you're ready, take another moment to think about where it is you want to go next before you set off. It's fine. Nobody is going to be able to say they haven't been in the same situation and many Advanced Drivers would tell you that you haven't been on a drive unless you've been lost at least once.
Please let me know if the advice i'm giving you is helping or not, as i'll be better placed to help you. One last thing, have you considered taking a few observed drives with an IAM observer or a member of this forum? You could show them what happens and see how they might be able to help you.
Hope this helps.