Harold Hill tells the town they need to stop the pool hall because it will bring Black people with it.
Pardon my naivete, but where is that in the text?? (I assume you're referring to the lyric about ragtime, but if so, I wonder if you're overthinking it.)