A former chairman of the National Federation of Fishermen's Organisations says the sector is worse off after Brexit.