Except for the part where there are almost always other Brits, both white and PoC saying the same things, yet everyone gets shot down as being Americans who don't know anything about how race works in the UK.

A lot of white people in the US get defensive about race for those same reasons. They think talking about race is disturbing the fragile harmony. It's called privilege.