Well Granny, I'm not in a place with many foreigners so this may be a completely different point of view than in those areas.
Most Chinese people here have no expectation of foreigners to able to speak Chinese at all. Some of this is because many of them believe Chinese is far too difficult for foreigners to learn and if a foreigner is competent at all in Chinese he/she must be very, very clever and special indeed.
But really foreigners are mostly viewed as transient and therefore why would they learn? Look at the reaction on this forum when someone mentions that they might be interested in living in China forever... well, the Chinese are just as incredulous of this kind of idea.
As far as people speaking their own language to each other, meaning that others can't understand, that's just normal here. People do this in Chinese all the time. Two people from the same area might use their local dialect to talk to each other while others don't understand that dialect. In my area, Mandarin is the second language for everybody. They all have a local dialect that they learned first. I used to live in a city with four distinct dialects. Since no one could understand each other a fifth dialect was used in the city that was kind of a mash-up that everybody in that area could speak.
Now these are small areas. This city is about 45 minutes from Changsha but no Changsha natives could understand any of these dialects. Chinese people, in this province anyway, are used to having all kinds of conversations around them that they can't understand because of how many dialects there are, so some foreigners doing the same is no issue at all.
Last night I was having a few beers at a BBQ joint. My girlfriend and another girl where chatting in Changsha dialect, I and an American guy where speaking in English and another couple of people where using a different dialect from a nearby area. When anyone addressed the table it was in Mandarin. If there was some interesting point or joke to tell the table, someone would translate it into Mandarin for all to understand.
Even sign language was used quite often to order beer.
With the exception of wanting to practice Chinese, There's really no reason that I would use Chinese to speak to the American guy for a private conversation that doesn't include the others, it's just not as efficient for us.
I suspect the feeling for those people in Brisbane that get annoyed is more about how they feel the country is being taken over by a different culture. If you are hearing a couple people speaking Chinese to each other a couple of times a year, I doubt it would bother anyone, but the feeling (true or not) that English might getting pushed out in certain areas or feeling like you are foreigner in your own native country is unsettling. In the area of China I live in, no Chinese people would feel this way. There's not enough of us foreigners here for that.