Chief minister Yogi Adityanath on Wednesday said while some people attempt to divide society along caste lines for the sake of vote banks, they lose their nerve and fall silent when it comes to taking action against notorious mafia figures and criminals.