I recently stripped half my boat too. Lots O fun
I had (still have) some of the patches like you mention. Most of it comes off with the heat gun and putty knife, but there are several areas that need sanding too.
If you are finishing bright I would think that you would have to remove it all or you might have color variations (but I do not know for sure).
I will be painting, but I am still sanding it all down so I know that I don't have to worry about the base.
I do not want any more issues