A local-first agent core separating LLM proposals from deterministic risk policy, approval state and tool execution, with optional Apple integrations.
Prototype
Python
FastAPI
MLX
Swift
Firebase
TypeScript
Original code-generated category illustration; not a product screenshot.
Overview
Jarvis Core places an explicit control layer between a model proposing a tool and the tool actually running. Its Python/FastAPI runtime handles risk classes, approval waits, expiry and repeated calls. Memory, native Apple bridges, Gmail and Firebase modules extend that core. It is independent of the J.A.R.V.I.S repository, which uses different packages, entry points and orchestration contracts.
Problem and objectives
A language model proposing a tool call does not establish that the action is authorized or trustworthy. Writes, external communication and destructive operations need different rules. Resuming a run after approval, controlling repeated call IDs and bounding execution also require explicit runtime behavior. This project treats model reasoning and application authority as separate layers rather than assuming a generated instruction should execute immediately.
My role
Development of the agent runtime, risk policy and integration contracts.
Solution approach
A chat request builds context and obtains a response or tool proposal from the model. A deterministic matrix evaluates proposed calls by operating mode and risk class. Allowed calls reach the executor, approval decisions enter a waiting state and denied calls return a result. Approval TTL, step/tool limits and repeated call IDs are runtime concerns. Mock adapters support local development; MLX, Apple IPC, Gmail and Firebase paths require separate configuration and permissions.
Implementation details
01
Binding approval to a specific action
ApprovalStore treats approval as more than a yes/no flag. Creation computes an action digest from the agent-run ID, tool-call ID, tool name and arguments; the request also records user identity and expiry. Approval and consumption verify that binding, and consumption is marked as one-time use. Changed parameters can therefore become an explicit integrity error. Because the store is in memory, this mechanism alone does not provide persistent approvals across process restarts.
02
A second policy check at execution
After the agent loop evaluates policy, ToolExecutor evaluates the call again; an is_approved flag cannot override a hard deny. Pydantic validates arguments before dispatch. Concurrent requests with the same tool-call ID are serialized by a per-call asyncio lock, and their result can be reused from a process-local cache. Layering checks reduces reliance on the model loop alone. An in-memory cache does not imply global exactly-once execution across services or restarts.
03
Resuming a waiting run with a tool result
When approval is required, the runtime enters WAITING_APPROVAL and retains the pending call on the run. Approval verifies action integrity, consumes the authorization and executes the tool. Its result becomes a TOOL message in the context before the multi-step loop resumes. Rejection has its own transition and explanation. Step and tool-call limits still bound continuation. The contribution is an inspectable lifecycle for proposal, authority and execution, not a claim that every integration has passed live-service acceptance.
Architecture flow
01
FastAPI chat request → context assembly
02
Mock / MLX adapter → response or tool proposal
03
Mode + risk matrix → allow / approval / deny
04
Approval state and executor → tool result
05
Result → next agent step / completion
Conceptual flow derived from project descriptions; not an application screenshot.
Core components
Observe / assist / autonomous policy modes
Deterministic decisions by risk level
Expiring approvals and resumable runs
Step and tool-call limits
Memory and native Apple IPC components
Optional Gmail / Firebase / iOS layers
Technical challenges
Resuming approval-paused runs with correct state and handling repeated calls.
Keeping tool risk labels aligned with execution policy.
Separating mock contracts from real-model, device and service acceptance.
Engineering decisions
Model proposals, policy decisions and execution occupy separate layers.
Default assist mode requires approval for writes, external and destructive operations; sensitive operations are denied.
External integrations are opt-in, with mock providers for local development.
A compact language-model study exposing Transformer internals through a custom BPE tokenizer, PyTorch components and local Apple Silicon training experiments.
