Labels are often used as a convenience for quick identifications and sometimes to delineate boundaries. In this perspective, they are essentially tools (and not goals). With respect to labels, sometimes one finds a difficulty of explaining oneself of being a mathematical physicist. Are you in physics or are you in mathematics? My physics colleagues may ask the question why are you in an institute spearheading mathematical sciences, while my mathematician friends also may wonder why am I in the current position. My short answer is by accident.
In the past, I use to care about such labels but at this elderly age, I don't anymore and I have grown to respect all areas of sciences and art, each with their unique characteristics. In fact, being in the current position, it is necessary for me to care about all areas of mathematical sciences without favour. Of course, if pressed for an answer of what am I, I would say a theoretical physicist or even wider, a theoretical scientist because some of the systems that we study are not quite physical systems. I love both physics and mathematics because they are respectively grounded in 'reality' and a universal language to describe 'reality' both quantitative and qualitatively.
If pressed for my formal tertiary educational background, again you will find it ambiguous. My B.Sc. (Hons.) was from Department of Mathematical Physics, University of Adelaide. This department no longer exists and was fused with Department of Physics to become Department of Physics and Mathematical Physics (see history here) and now they are part of School of Physical Sciences. Note that one can also find mathematical physicists scattered in the School of Mathematical Sciences in University of Adelaide. Later I went to do Part III of Mathematical Tripos at the Department of Applied Mathematics and Theoretical Physics (DAMTP). This is now considered as a Master of Mathematics/Advanced Study. For this, I recall a colleague saying that theoretical physics is just applied mathematics. I told him that in some circles, applied mathematics are considered different from theoretical physics and I mention DAMTP. After Part III, I joined the Department of Mathematical Sciences, University of Durham for my PhD. Theoretical and mathematical physicists there, are divided into two departments, Physics and Mathematical Sciences; but there is an overarching centre, to which the theoretical and mathematical physicists from both departments belong to i.e. Centre for Particle Theory. Thus, with such background, personally I feel at home with both labels of physics and mathematics, if you ask me.
So what is really mathematical physics? And what is theoretical physics? Rather than try to define them here, I just would like to link to the wikipedia articles of mathematical physics and theoretical physics. For me, mathematical physics is a subset of theoretical physics, one which has more mathematical rigour. A paper in theoretical physics could dwell so much in conceptual matter, leaving little room for mathematics. The adjective of mathematical and theoretical is also used much in the same way as in mathematical and theoretical biology. The difference between theoretical and mathematical is not always clear cut and there may be a whole spectrum between these two labels. I would consider myself more of a theoretical physicist relying much more on intuition, while the late Prof. Twareque Ali for example is more of mathematical physicist.
Now is mathematical physics part of mathematics? Perhaps it is best to look at the AMS Mathematics Subject Classification. Mathematical physics used to be a subject on its own in the past (see here) but now it has gone under the various sections of "mechanics" in the 2010 classification (my own interest is in 81-XX, 82-XX, 83-XX). Note they also appear in the AIP Physics and Astronomy Classification Scheme (PACS). One can also check many mathematical topics are listed there. So the answer to the question at the beginning of this paragraph is in the affirmative.
Now the late Prof. Twareque once told me that it would be nice to have a centre of mathematical physics within the region. Indeed it would. However practically, I find that such idea is best to be part of the present institute. Theoretical physicists in the country are still a handful in number and we may not form a critical mass for such idea. Moreover, a centre that deals mainly with fundamental sciences may have difficulty in finding funding here. I also perceive proliferation of centres will not be a good idea (like proliferation of committees). Thus INSPEM would be best for us and it is a natural home for theoretical physicists particularly if we think of theoretical physics as an interdisciplinary subject, as I have told my younger colleagues.
Multidisciplinary and interdisciplinary research areas are now commonplace as scientists address more complex problems. Mathematical sciences are not excluded from this trend. Even in mathematics itself, the use of concepts and techniques from different subdisciplines in a particular subdiscipline has become more common. Once my younger colleague mentioned to me that he feared that he is drifting away from physics as he gets more involved in complex networks research. I pointed out that much of physics research in the department has become more chemistry-like and engineering-like and thus there is no need for such concern. Boundaries of traditional disciplines have very much disappeared, though there will always be room for pure disciplines such as in pure mathematics as much as a language can be studied for its own intrinsic beauty.
Finally I have no qualms in being labelled a mathematician or non-mathematician. What matters is what we do. In many ways, I feel sorry for those who feel the need of feeling superior by subscribing to a particular label.