Britain will have to abide by European Union fishing rules after Brexit but will not have a "voice at the table", a former environment minister has claimed.