Residents of a small village fear their community will be lost forever as most of the homes are being used for holiday lets.