The Future

I've been an IT Manager for 10 years. For most of that time I had a dream of finding a way to make all of the many systems I oversee easier to navigate. The dream is that I can just say "Hey J.A.R.V.I.S, do our configured password settings match our stated security policy?" or "Hey J.A.R.V.I.S, are our backups working?" It feels like an incredible waste of efficiency to have to manually remote into server or login to individual admin panels to find the answer to these things.

Over the years, I've conceived of a couple different ideas for surfacing the data. I became inspired when I experienced VR for the first time. I thought Wow! What if I could easily visualize our entire network in virtual space right in front of me? What if I can just point at a computer or group of computers and select "deploy software"!? Honestly this is still a good idea, I might revisit it sometime. But my second wave of inspiration came when I learned about using the OpenAI API and setting up Llama. As I began using ChatGPT and other LLMs for various workflows throughout my regular day, I started thinking "man, wouldn't it be easier if I could somehow squeeze our entire infrastructure into ChatGPT's memory so that I could ask any question and have an answer in seconds?" This was the birth of Sonny Insights.

Vision

The core vision of Sonny Insights is straight forward: a chat interface, like ChatGPT, that enables querying information about all systems that a system administrator might be responsible for. I began by building out some integrations for systems that are free or that I have experience with.

This all seemed fine, but I realized quickly that there would be more engineering than I originally anticipated...

Challenges

There were several immediate challenges I had to solve:

1. Very verbose data - When using LLM APIs like OpenAI, every token matters and many services and platforms provide data in very verbose packaging. As an example, retrieving Group Policy settings can easily be thousands of lines of XML.
2. Lack of standardization across environments - It's impossible to assume that because I do things one way in my environment, everyone else will do the same. This means any solution needs to be opinionated enough to be good, but generic enough to be portable.
3. Toolchains - What tooling do you build for the LLM to avoid ambiguity and allow the user to ask as many questions as possible?

Challenge 1 - Verbose Data

The first challenge I began tackling was the issue of verbose data. I won't claim to have a perfect solution, since I'm working on it still now, but I've got something that allows querying GPOs and getting usable results despite there being hundreds of thousands of lines of XML. I took this broad approach:

• Connectors are integrations to external systems. They provide a connection.
• Subconnectors pull specific pieces of data using an integration's connector. I was careful to build these to be generic and nestable. For example, imagine Active Directory is a connector, then the list of users is a subconnector, but what if you want details about a specific user? The layout would look like Active Directory (Connector) -> Users (Subconnector) -> User Details (Subconnector)
• Insight Engines are preconfigured instances of OpenAI compliant LLM API's. This allows you to have multiple different LLMs configured to handle varying degrees of complexity. As an example, you could use a local ollama instance for categorization since individual context size will be small and the UX doesn't depend on immediate results. Then you might choose to use OpenAI for the Sonny Insights chat interface since you want it to appear intelligent and use tools properly.
• Categorization allows placing broad labels on groups of data to rapidly filter out extra, unneeded context.

Through these systems, I was able to greatly pare down how much data would need to be passed to the LLM to get useful results.