Agentic Orchestration Infrastructure

I designed and operate a Windows native multi agent system that turns requests into structured, reviewable work. It runs under Claude or Codex, and I use it to build and ship StreakUp. Its core is local, with cloud models and selected cloud services where useful.

In numbers

The system holds 17,133 active memory records, including curated facts and automatically extracted session memory, up from 6,715 five weeks earlier. Its transcript index contains 120,237 searchable chunks.

The operating surface includes 12 connected tools, more than 60 direct scripts, and 51 reusable skill templates. Cross provider review runs recorded costs of $0.62 and $0.17.

One product built through this system is live on Google Play and the App Store.

The journey

Stage 1: A Linux gateway in Docker

The system began as a persistent agent gateway running in Docker inside WSL Ubuntu. It established the basic pattern: messaging, tools, persistent state, and work spanning multiple chat sessions.

That gateway is now retired, but its lessons shaped the current system.

Stage 2: A Windows native core

I rebuilt the core around the environment I operate. The live system uses native Windows services, scheduled tasks, local storage, GPU access, Telegram, and command line agents, with no Docker or WSL in its active path.

Stage 3: Memory that compounds

I added structured procedural, semantic, and episodic memory, followed by a searchable transcript store. An hourly session watcher extracts useful memories with a daily backstop, while raw conversations remain available for explicit search.

The system can now recover relevant decisions, rules, and history across sessions.

Stage 4: One system, two model providers

I separated the operating rules from the model running them. The same rules, memory, skills, and tools now work under Claude Code or Codex, including model calls launched from within a session.

Stage 5: Adversarial review before shipping

I added structured review cycles in which Claude and Codex can critique the same artifact, challenge findings, merge overlapping issues, and stop when another round finds nothing material. A run ledger preserves state through interruptions.

What it does

A voice message arrives through Telegram as an audio file. The live path converts it to the required format and transcribes it locally with faster-whisper large-v3 on the GPU, avoiding a per message transcription API charge.

The transcript enters the active agent as an instruction. The system loads baseline rules, retrieves relevant memory when needed, and selects tools or specialist context for the task. Before parallel work starts, lifecycle checks look for duplicate work and apply the configured cost limit.

Agents write findings to shared state rather than making final decisions independently. I can follow progress through Telegram, while liveness monitoring alerts me when work appears stalled. The result is an inspectable task with human review where findings become decisions.

The differentiator

Provider agnostic core

The orchestration layer drives the vendors’ first party command line tools, Claude Code and Codex, authenticated through their subscription plans. Those sessions use plan based access rather than metered third party API keys, making their cost more predictable and reducing dependence on third party API availability and pricing.

The wider system remains hybrid, combining local compute and storage with selected cloud models and services. Its core rules, memory, tools, and skills are not locked to one model provider.

Provider diverse review

Because the same operating rules work under both providers, Claude and Codex can review each other’s work before it ships. Each reviewer produces structured findings; the system validates them, combines related issues, tracks agreement, and can run another bounded round when concerns remain unresolved.

Recorded review cycles cost $0.62 for a three round red team and $0.17 for a two round blackboard review. These runs prove the mechanism works. They do not prove that two providers always outperform one model given the same budget.

Architecture

Memory

Session context loads in tiers so simple work does not incur the full coordination stack. Persistent memory separates procedural knowledge, semantic facts, and episodic history. Raw transcript search sits alongside it and enters context only when retrieved.

Retrieval combines keyword search with vector similarity. Results are fused, diversified to reduce repetition, and adjusted with type aware time decay. An optional reranking stage exists but is currently dormant by choice.

Agent lifecycle

A duplicate spawn guard checks whether equivalent work is already running. A cost circuit breaker applies a configured financial limit before execution expands. Heartbeats and liveness monitors watch active processes and send escalating Telegram alerts when attention is needed.

These controls improve visibility and containment, but they do not make the system unattended. I remain responsible for reviewing alerts, resolving failures, and approving consequential decisions.

Tools and coordination

The connected surface includes 12 named tools, more than 60 direct Python scripts, and 51 skill templates. They cover memory search, context retrieval, transcription, browser capture, file delivery, monitoring, and maintenance.

Coordination follows a hub and spoke model. Specialists append structured results to shared state; the main agent reviews their output and relevant diffs before writing to the decision log. Oversight remains centralized when execution runs in parallel.

Deployment

Environment	Status	What it means
Windows native core	Running	The live agent system and its active operating path.
Original Linux gateway in Docker	Retired	The system’s working origin and the basis for parts of the current architecture.
Linux desktop in Docker	Running	An operator environment, not the agent core.
Portable core	Blueprinted and reviewed, not yet running	The container design exists, but the core has not completed a live container build and smoke test.

The core runs Windows native today. Its Linux lineage is real, while portable deployment remains designed rather than deployed.

What this demonstrates

I can design the architecture, build its operating parts, apply it to real product work, and keep it legible enough to audit and change. I can also distinguish what is running from what is planned, and what a successful mechanism proves from what it does not.