Note that fontconfig matches color emoji fonts preferentially, so if you want to use a black and white font while having a color font installed (eg. Noto Color Emoji installed alongside Noto Emoji), fontconfig will still choose the color font even when it is later in the list.