You can change the registration policy of your community from "public" to "invitation". It means, the designated people will receive an invitation email from your community, with a link that gives them access to your community portal.
If the community access is public, anyone can join the community, with or without invitation. But, if you change the community access from "public" to "private" and the invitation registration is enabled, therefore, only your designated users with an invitation email are able to join and use your community.
Click on this link to learn more about the feature.