Why they remain

I thought I would post an interesting thought I found in God's word today. Sometimes I wonder why God leaves so many difficult people in our path; People who just make it a pain to go about our daily lives; People who hurt us, chastise us, imprison our spirits. Well, the nation Israel wondered this too, but there, in this case and in ours, is one reason:

20-22 And God's anger blazed against Israel. He said, "Because these people have thrown out my covenant that I commanded their parents and haven't listened to me, I'm not driving out one more person from the nations that Joshua left behind when he died. I'll use them to test Israel and see whether they stay on God's road and walk down it as their parents did."
23 That's why God let those nations remain. He didn't drive them out or let Joshua get rid of them. *Judges 2:20-23(MSG)

Hmmm... Could it be that sometimes in our life God leaves these people behind to sharpen and test us? Judgement will still fall upon all for their acts and motives, and the just shall be rewarded. However, God clearly states that one of the reasons that oppressors remain is to test us.