Greetings Earth invaders and network raiders, Alex here.

So… I just took my first steps into building my own ML / AI Model. Despite having been working on the Navi AI Interface for around a year now. I never actually built my own model. It has been a perpetual stream of using other models and frameworks. It is high time SSG and myself start working on our own. This post might be a little longer. Since its covering all the work from Mid-September till now.

Lets backtrack a little Rasa, Raices Con and CSI Linux.

Without going into details. SSG decided to move away from the Echo AI framework. Who’s developer is a fantastic person, however for what we (both SSG and CSI Linux) wanted out of Navi. We decided to move to Rasa in the interim. Which allowed us access to new features and avenues for deployment. A cleaner startup and the ability to serve the model we built with it using an inbuilt server system. Nice.

As of this point in time, I had been invited to speak about the Navi Project at Raices Con thanks to the pushing from Eric Belardo and some of the Raices community members. I decided to take the stage at my first cyber sec conference. It was a thrill and a half and honestly needs its own post.

One of the key takeaways from Raices Con was during my nearly day long discussion with Cindy Munoz of CISA. Lovely lady. I actually had the opportunity to go out to lunch with her, my significant other and her father during the conference. Oh right. Sorry. Memory lane snuck up on me. Don’t blame me. I got wrapped up the subtle nuance of realizing how crazy my network is these days. That and Its not every day you get to say you went to lunch with a fed. So her words stand out to me even today:

“In order to push the boundaries of what we can do and properly promote security within AI we need to build our own models. We have to stop relying on external platforms, frameworks and things like GPT.” – Cindy Munoz

Since then, I have had that target in my sights.

Sage advice from Eric

A while back I had the opportunity to talk to Eric Belardo one on one. He offered me some advice that I now call the “Learning in public” imperative. The basic gist of it is that every chance you get. Post what you are doing. Write a blog, do a video, do something to show your work. This is a great way to get noticed, build your contacts and even in my case. Get invited to speak at a conference.

Taking those first stumbling steps.

I am not going to lie to anyone here… I don’t know what I am doing. I am just doing. I am learning and there is absolutely zero wrong with that. So, I snagged a course on udemy after taking the time and talking to the team. We decided that when it came to our AI, we were going to go with PyTorch for the back end and build our own off of that. So the stage is set. So it was time to dig in. Dig I did.

The first ML / AI model you build with this course is a simple linear regression model. Used to train on a random set of numbers and predict future numbers. Nothing earth shattering. However, as simple as it is it is exhilarating. Let take a look at the code.

Okay, so its not that much of a stumble. On top of that it was a great learning experience for myself. Not to mention the team and I dug into the code a little later tinkering with things to see if we could get the training any closer to perfect. Spoiler alert, no. We couldn’t but we had a lot of fun with it. Despite it being what equates to a highly technical scatter graph.

So what did you learn?

Oh this is going to be a doozie. Lets talk Epochs real quick. An epoch is a term commonly used in AI / ML development to represent a number or passes or iterations the AI makes over the training data. The general number of passes an epoch holds is calculated as: number of epochs x 10.

So we discovered that the closest we can get to “perfect” using the default settings was trained at 183 epochs. Which would contain roughly 1830 iterations over the data per epoch. Even at that the data the model spit back was off by around 4% which is about the standard. Besides. Me thinks an AI will never be 100% accurate or it would be the same as hard coding responses at which point its not AI.

So this being said we started playing with the settings and default values and generally breaking things. We learned a good bit with that spare twenty minutes of hapless play that likely will stick with us through the rest of this journey.

A few takeaways were:

• Data density and required epochs to get responses within a 96% accuracy rating is proportional.
• No matter what, the 96% accuracy rating is the highest we could get it with the limited time we had to play with it.
• It was really easy to see how this would translate to what we are trying to do with the chat bot side of our big projects.

So, on that note about how this would translate into the chat bot. Let talk about that. Lets take a simple example here. You want to ask Navi about cryptography. So you feed it the prompt “What is cryptography.” On a single epoch here is what your accuracy looks like:

The line of red dots is representative of what the AI thinks you are asking, the green dots represent the proper answer. The blue is simply the data its trained on. Right now it might spit back something about cryptocurrency instead. Both have the term Crypto in them. They are still two very different things. Despite them also sharing an affinity for complex mathematics. If you are like me that is cringe worthy but also very fun. So lets give it some more training.

We found the sweet spot to be around 183 to get that 96% accuracy rating we mentioned.

So this graph looks much better with the red dots lining up painfully close to the green. In this case if you asked Navi “What is Cryptography” it would respond properly and actually tell you what you want to know.

I could also be completely off base here with this but it still gives a good visual representation of how AI training works and how it can effect things. So if you want to see where I get that 96% accuracy metric. If you zoom in on the green and red dots. You see this.

If you ditch the 0.8 and only look at the 56 – 60 you see we are around 4 off of the zero point or 100% accuracy. Thus we are around 96% accurate.

A few parting words

It has been a really exciting few months around SSG. Between the conference, getting our hands on new hardware and finally taking the steps toward getting the team ready to dive into building our own model we are setting up for an amazing 2024. So thank you so much for reading this.

Till next time, Happy hacking!

– Alex