GZ250 'throttle cap tightener bolt'

[TXtwowheeler]

The long thin bolt running into the handlebar through the throttle and through the cap on the end of the handlebar rattled loose. I was trying to tighten it back in, but way down 4 inches inside, the piece that's supposed to receive the threads got pushed too far in when I was trying to get the threads started. Now I can't reach it. The cap is off, and I can't get it back on until I get the piece back out where I can reach it.

First: Any tricks to getting it back out?

Second: Any tricks to avoiding having it rattle loose again?

I'm pretty handy, and have a moderate assortment of automotive tools, etc, but new to bikes.
Thanks in advance for your help.