Electrical tape works wonders on things like that
You might as well do it right and solve this problem once and for all. It wouldn't take that much extra effort to slice the wire housing some more, expose more wire and simply redo the connections in the box. Then you'll have long term piece of mind.