I was in French immersion for 11 years, took classes in high school and university, and then took a long break from it -- about 10 years -- before I went back to it about 5 years ago. At first it was really hard! But things really got better over time. I practice 1-3 times a month with a French language meetup, and I also set my alarm to a French radio station. (If you can do one or both, I highly recommend!)

Yet even last year when I visited France for the first time, I was still somewhat petrified to speak to the locals. After 3 days I got over it, and it started to flow again.

Your angst is natural and common. Accept what you're feeling. You can push through it!