Residents of a picturesque Welsh hamlet fear their community will be lost as most houses are only used as holiday homes.