The text will be off! Even though rectangles are perfectly centered.
But even if font can have its metrics unbalanced, it doesn’t mean it does. What happens in reality?
In reality, most of the popular fonts have metrics slightly off. Many have it significantly off: