Why do we travel? Spend enough time on the road and your fellow travelers start to look alike. Most are Caucasian, either from North America, Europe, or Australia. We are also from North America and our up bringing in the US gave us the ability to make choices for ourselves and to take chances. So, while in thinking, we're Americans our looks are Pakistani.
We found ourselves part of the tourist mix a few times as well but did stand out due to our brown color from being born and spending some time in Asia. The notion of minorities traveling is new, granted Chinese and other Asian nationalities are starting to get out but the numbers are still relatively small and we therefore raised eyebrows in several towns where the locals couldn't really place us; most thought we were Indian.
Everyone's travel experinces are unique, so for us, since we don't drink, we saw a lot of towns with bars and the usual travelers enjoying themselves, we would often opt for something like meditating at a mosque or spending time with the locals at a market or cooking a meal at our hotel. We never shut ourselves out from experiencing or being with people but wanted to break away from the tried and true of the tourist hordes and have a chance to engage and experience as much of the local life as possible. Did that mean, declining requests to spend time with fellow travelers and instead enjoy each other's company or explore the beauty of a small town, probably yes.