I know the problem only too well - I have also been frustrated by the book giving a few definitions, a couple of worked examples then the exercises.
I can't understand it unless I can see what the numbers are actually doing to each other. I also find it easier if the topic is something physical, like mensuration or co-ordinates on graphs.
For those examples, I can relate the area of a rectangle or volume of a rectangular space, by considering the size of the room I am in. I found three-dimensional graphs (x,y,z) reasonably easy because by the time I was introduced to them I was used to the National Grid Reference and contour systems on Ordnance Survey maps (from navigating on long countryside walks).
I think though you and I had missed the point I took years to spot.
Books like those are not intended for self-teaching.
They are supposed to be companions to teachers who do the explaining... If the teachers are any good at helping people who struggle with the subject, so that was two of mine out of the running!
=
Mathematicians are not "scatter-brained" at all. I have known quite a few, and even more people who needed use a lot of maths in their work - including my Dad, a Chartered Electrical Engineer. They are though, very good at thinking of abstract concepts in logical steps, and that is a skill we don't all possess or have not cultivated. Well, I know I don't and haven't!
A former manager of mine was a scientist holding a PhD in Physics. I asked him about ability to learn advanced maths, and he said it is partly aptitude, but also lots of practice.