The barriers are those imposed by the EU, not Britain, and we cannot simply remove them.
The EU works as a single bloc with ease of movement across borders within itself, but not externally. It also has a policy called au communitaire, or similar, by which once a country becomes a "members" it is regarded as that permanently, with its own economic and legal systems subordinate to the EU.
This is one of the two main reasons it played so rough over the UK leaving, which it chose to do albeit by a very narrow majority.
Its other reason, and one reason Britons who voted to leave did so, is the loss of income. For the UK was one of four nett contributors to the EU - God knows where its money goes because it will not allow proper financial scrutiny and acts very harshly against auditors who try to investigate - along with Germany, France and Italy.
Now, it's all very well saying a nation can leave (if allowed by Brussels) but if it wishes to continue trading with the bloc it still has to follow many of the literally tens of thousands of regulations the Commission invents and the European "Parliament" nods through. A parliament in name only, although of elected politicians.
How many tens? No-one really knows but some estimates put it as high as ten, i.e,. 10 000 rules. The majority apply to specific trades and services which is why most of us know little or nothing of them. To be fair some are EU interpretations of genuinely world-wide agreements and laws by bodies like MARPOL, the UN and ISO. Others may though be at the behest of large commercial interests.
At the height of the debate some pro-EU politicians cried they wanted Britain to be at the "epicentre" of the EU. Oh the irony: they really meant "centre" but I fear their wish has unwittingly come about by the real definition of a geological term they just did not understand!