An institutional value or quality is considered an important and typical feature of a particular society or group, usually because it has existed for a long time.
If someone accuses an organization of institutional racism or sexism, they mean that the organization is deeply racist or sexist and has been so for a long time.