What you described happened to me when I was deficient in one or more vitamins and minerals. For me it was Vitamin D, Calcium and Magnesium. I was never a big dairy eater and I have had to add a lot more to my diet, with magnesium, as my Vit D supplements were not getting absorbed. Since adding those 3 supplements, my dreams are much less fearful and I sleep better now.
Eating too much salt, having too much caffeine and even one can of diet soda can also result in fearful dreams for me.
If you are deficient in Vitamin B6 in particular (if I remember correctly) that can lead to poor dream recollection.
Since your problem has been so persistent, I am thinking you might have the same cause. You might want to look at your diet, visit with your doctor or see a naturopath.