You declared about yourself "the types of math I enjoy only have a little bit practical usage outside of physics." However, I believe that you ought to give applied mathematics a try in some form, at least as a free elective course.
Many university students are like you and are at least somewhat undecided in their first year. If that is the case, make strong use of that first year to make strides in getting non-major requirements out of the way, perhaps taking a course in writing and a course in literature. Also, that first year can be good for taking courses to help get any foreign language requirement out of the way. Perhaps you will find out in that first year that university is not for you, or perhaps you will make up your mind about cosmology versus mathematics, or perhaps you decide to focus on a two-year degree in applied mathematics or computer science, and then get a couple of years of working experience before deciding whether to go further in university.
If you can, buy any textbooks a few days in advance of the start of the semester in order to be able to at least partially pre-read them, and perhaps get in some practice on problems posted at the end of the first chapter in the case of mathematics and physics courses. The extra time provided by getting your books at least a few days in advance will be a partial workaround your "horrible memory", and the increased familiarity of your environment from having those books for at least a few days before the first day of class will allow you to partially avoid the problems of ADHD. New memories are formed in various pathways taking up small regions of the brain that are each perhaps a cubic millimeter in volume. Where these pathways cross near each other, "retroactive interference" and "proactive interference" can result, somewhat akin to the construction phase of building a new overpass for a new superhighway that crosses over a pre-existing superhighway. We all know what construction can do to traffic patterns. Getting your books a few days early will give an efficiency benefit to your brain, the same type of efficiency benefit that a shopping trip will receive if one gives the construction crew a few extra days to get their job done before taking a rare trip to a favoured but distant shopping mall. This is also good advice even for someone with no memory problems and no ADHD. If you have occasionally watched a somewhat modern war movie, this method of operation is a version of what is referred to as "getting good intel".
As far a making good grades, the number one rule is "Do the homework, do the homework, do the homework." With many instructors, some test question are copied verbatim from homework assignments. A small fraction of instructors will use a sampling from their previously given homework assignments for all of the test questions.
I wish you well in your university studies. Part of university in addition to learning the course content is learning how to learn. Also, as a bonus, a few people also learn a little bit in university in terms of being able to apply knowledge in an interdisciplinary manner.
