I removed the shortcut arrow by deleting the isshortcut entry in the registry.
To add or restore the entry again you can download this zip file
and unzip RestoreArrow.reg only (the others are not needed for restoring it). Click the unzipped .reg file and it is all good again.
Like the source suggests, removing the IsShorcut.reg entry can cause problems with features and programs that use links, which is what happened to me (shortcut folders disappeared in 'save as' windows).
Again, if you want to remove the arrow without my particular problem I found using TweakUI/explorer/shortcuts shortcut overlay options works. Thanks Giz for directing me to said software.