First off, the problem Paul was addressing in
1 Cor 5 was believers commiting fornication.
And, he ended by admonishing them to:
1 Cor 5
9 I wrote you in my letter not to associate with immoral people;
10 I did not at all mean with the immoral people of this world, or with the covetous and swindlers, or with idolaters, for then you would have to go out of the world.
11 But actually, I wrote to you not to associate with any so-called brother if he is an immoral person, or covetous, or an idolater, or a reviler, or a drunkard, or a swindler—not even to eat with such a one.
12 For what have I to do with judging outsiders? Do you not judge those who are within the church?
13 But those who are outside, God judges.
Remove the wicked man from among yourselves.
The "wicked" man that was to be removed from the assembly was a believer (at least a professed believer)
Secondly,
1 Cor 14
23
Therefore if the whole church assembles together and all speak in tongues, and ungifted men or unbelievers enter, will they not say that you are mad?
24
But if all prophesy, and an unbeliever or an ungifted man enters, he is convicted by all, he is called to account by all;
25
the secrets of his heart are disclosed; and so he will fall on his face and worship God, declaring that God is certainly among you.
Pretty clear!