Developing an AI Agent for DA: My journey, 100+ tools, and scalability challenges

[chuvadenovembro]

Hi guys,

NOTE: I apologize for my text (I use a translator), I'm from Brazil.

When I started working on this integration, I initially planned something simple. The idea was to have an AI assist users with site issues, using strict restriction rules to avoid problems.

However, during development, I kept the access open (without strict restrictions) to build out the features. One thing led to another, and I ended up creating almost 100 tools/integrations. The exponential capability of AI is both fascinating and scary at the same time!

Currently, the tools I’ve created (via AI) allow the agent to do almost everything clients usually ask for:

• Read, modify, and create files in public_html
• Manage emails
• Manage DNS zones
• Manage Cron jobs
• Manage Subdomains

Since I’m not an expert programmer, I opted to build this in modules. This way, if I need to fix or improve something, I don't break the rest. I also separated Reseller and User/Hosting access. The authentication is handled via DirectAdmin Login Keys, and access restrictions depend on the commands allowed in the API.

The Problem:I hit a wall I wasn't anticipating at the start: Scalability.Right now, I can't scale this easily because I need to replicate the setup for every access/user. I know I could solve this by exposing an API via HTTP, but that introduces major security concerns. I've already spent a lot of time ensuring the LLM doesn't have access to API info and that reports are sanitized.

I'm currently on vacation and working on this whenever I find a gap, but I wanted to highlight the potential here again. If I simply create a hook between my support system and this integration, the LLM (which has agentic behavior) could read a support ticket, interpret it, and if appropriate, actually execute the fix (for the simple tasks mentioned above). Obviously, this would require broader access.

I’ll continue studying this integration. Without trying to be a doomsayer here, if you don't realize the consequences of what I described above, check out the screenshots attached. I asked for simple things, and you can see the AI's performance using the tools.

 

[sysdev]

I'm not sure wat exactly your scalability wall is and why replicating a setup for a user can is a problem? It actually all depends on what 'this setup' means.

Assuming a (very minimized) workflow similar to ours:

x servers -> y users -> z tickets/mails/chats/questions -> AI Agents -> Results/actions

the scalability usually is raw AI power and security issues in everything the AI touches (hence the insane validations required).

From your post I understand that you have scalings issues related to the number of users/accounts and security issues relating the API?
Why is that?

 

[chuvadenovembro]

I'm not sure wat exactly your scalability wall is and why replicating a setup for a user can is a problem? It actually all depends on what 'this setup' means.

Assuming a (very minimized) workflow similar to ours:

x servers -> y users -> z tickets/mails/chats/questions -> AI Agents -> Results/actions

the scalability usually is raw AI power and security issues in everything the AI touches (hence the insane validations required).

From your post I understand that you have scalings issues related to the number of users/accounts and security issues relating the API?
Why is that?

If I offer this service to my clients today, I would have to manually replicate the solution folder for every single client. For each one, I’d also have to edit the configuration file manually.

Furthermore, regarding replication, if I update the solution, I'd need to come up with a mass update plan. It’s not a huge problem, but these details require time to strategize reliably.

I need to look into isolating each solution using Docker, for example, and automating the creation of each assistant (populating the necessary variables for the config file automatically).

I believe the effort required for these two stages (efficient/reliable replication and secure isolation on the same server) will consume precious time that I could otherwise spend refining the user experience.

For instance, I would like to improve the site editing capabilities. My goal is for the client to have an assistant as intelligent as Claude Opus to perform small website edits using natural language.