Harness Engineering for Agentic Network Operations

Why network agents require more than prompts, models, and tools

Language models can reason about network problems surprisingly well when they receive accurate operational context. Give a capable model the platform, software version, topology, configuration, symptoms, logs, and current command output, and it can often construct a useful investigation plan. It may identify a protocol interaction, compare several plausible causes, explain which observation is discriminative, and determine what evidence should be collected next. Network engineers have already learned how to improve these results. They specify the exact operating system, describe the topology, include recent state, and ask the model to rank hypotheses by probability. The answers usually become more accurate. Yet the engineer still owns a large part of the workflow. The engineer decides which devices matter, validates the requested commands, corrects invalid syntax, tracks what has already been checked, notices contradictions, distinguishes absence from unavailable evidence, and determines whether the conclusion is actually supported. The model reasons about the information it receives. The engineer manages the process that makes the information trustworthy. In this symbiotic human and model workflow, the engineer performs the function of the harness.

Harnesses

If you have used Claude Code or Codex, you have already interacted with a harness. The model is only one component. Both Anthropic and OpenAI have built a surrounding harness and an agentic loop that provides tools, working state, permissions, environmental feedback, and rules governing how the agent continues or stops.

These general-purpose harnesses extend this pattern beyond software development. Projects such as OpenClaw and Hermes Agent connect models to tools, messaging systems, memory, scheduled tasks, plugins, and local or hosted execution environments. They make it possible to assemble agents that operate through interfaces such as Discord, Telegram, Slack, iMessage, and other systems without rebuilding the entire runtime for each application.

These projects also make the harness independent of any single model. Depending on the application, a harness may use a frontier model through an API, an open model hosted by a provider, or a smaller model running on local hardware.

NetClaw applies a general-purpose harness concept to networking. It is built on OpenClaw and adds network-specific integrations and skills. It is an example of how an open-source harness can provide a foundation for network automation and operational workflows.

What a network-specific harness does

A network-specific harness is the system surrounding the model that controls how it interacts with the environment and is purpose-built for networking.

It determines which tools the model can use, which devices it can access, what actions are permitted, how evidence is stored, how failures are represented, and when a run is complete.

Prompt and context engineering is just one part of the system. The harness also includes inventory resolution, topology, device access, command validation, persistent state, evidence tracking, permission controls, retries, timeouts, and audit history.

Suppose the model decides that it needs the current BGP state from two routers. The harness must determine whether those routers are approved targets, identify the platform on each device, select or validate the appropriate commands, establish the connections, limit the output, preserve the results, and return the relevant information in a form the model can use.

When one router is unreachable, the harness must record that its state could not be verified. Missing evidence cannot be treated as proof that the router is healthy or unhealthy. This becomes an engineering challenge for long-horizon runtimes.

When the model requests an invalid command, the harness must reject it or provide a supported alternative. When the investigation begins to expand across the environment, the harness must apply limits to the number of devices, concurrent operations, retries, and total investigation scope.

These responsibilities place harness engineering close to runtime, security, and distributed systems engineering. Prompt quality matters, but the prompt operates inside a much larger execution system.

The manual troubleshooting loop

Consider an eBGP session that establishes and drops after several minutes.

A network engineer using a general-purpose model might provide the router platform, software version, peer address, autonomous system numbers, session history, interface state, and recent logs. The model proposes several possible causes and asks for additional output.

The engineer validates the requested commands, runs them, and returns the results. One hypothesis is eliminated and another becomes more likely. The model requests another command from the local router and perhaps a corresponding command from the remote side.

After several rounds, the investigation may converge on a likely cause.

Observe the problem
        |
        v
Form initial hypotheses
        |
        v
Choose the next evidence to collect
        |
        v
Run commands, query systems or data sources
        |
        v
Interpret the results
        |
        v
Eliminate, revise, or strengthen hypotheses
        |
        +------ More evidence needed ------+
        |                                  |
        +----------------------------------+
        |
        v
Produce a conclusion

The model participates in hypothesis formation and interpretation. The engineer manages scope, executes the checks, remembers previous findings, handles errors, and decides whether the available evidence is sufficient.

A network-agent harness implements these functions as a controlled workflow. It records which checks have already run, associates each observation with its source, enforces execution limits, and preserves the state required for the next reasoning step.

Some incidents require information that is unavailable through connected systems. An operator may need to confirm whether maintenance is in progress, whether a provider has acknowledged an outage, or whether an undocumented change occurred. The investigation should retain its state while waiting for that information.

Fixed workflows can handle narrow and predictable incidents. Broader diagnosis requires adaptive planning because relevant evidence may exist in device state, logs, telemetry, topology, configuration systems, ticketing platforms, cloud APIs, or provider services.

The model can decide which information would reduce uncertainty. The harness controls how that information is obtained.

Resolving the environment

A model may understand BGP, spanning tree, EVPN, MPLS, and common network operating systems. It does not know which devices exist in the current environment, how they connect, which software versions they run, or which services depend on them.

The harness needs a reliable way to resolve this operational context.

Inventory usually provides the first layer. A device record may contain an approved identifier, management address, network operating system, software version, site, role, access method, and authorization profile.

The model should reference the approved identifier instead of supplying an arbitrary address. The server can then resolve that identifier to the authorized device, connection method, and platform profile.

This separation keeps device access within the managed environment. It also allows the same logical operation to work across devices that require different credentials, jump hosts, transport protocols, or platform libraries.

Topology provides the relationships needed for fault isolation. A flat device list can identify infrastructure, but it does not describe physical links, Layer 2 adjacencies, routing peers, overlays, service paths, or shared upstream dependencies.

The required topology depends on the incident. An interface alert may involve the local device, its connected neighbor, and an upstream dependency. A routing problem may involve the affected peer, related route reflectors, and another router with the expected state. A service outage may cross switching, routing, firewall, cloud, and provider infrastructure.

A useful environment model can begin with a limited set of reliable relationships and expand when the evidence supports additional investigation. A complete digital twin may be useful, but many incidents can begin with a smaller and more focused representation.

Operational systems also describe different forms of state. A source-of-truth system describes intended state. A live device query describes observed state at collection time. Monitoring systems describe measurements and alert conditions. Logs describe historical events. The harness must preserve these distinctions when it constructs context for the model.

Evidence routing

Different operational questions require different evidence sources.

Current interface state usually comes from the device or recent telemetry. Historical transitions may require logs or time-series data. Flow records can describe traffic movement but do not establish device configuration. Inventory can identify the platform and role but cannot confirm current forwarding behavior. A configuration database may describe intended configuration while the device contains a different active configuration.

The harness needs a method for routing each question to an appropriate source.

Source selection becomes especially important when systems disagree. A source-of-truth platform may show that an interface should be enabled while the device reports that it is administratively down. Monitoring may report packet loss while a current device check appears healthy. Logs may record a failure that cleared before the investigation began.

These observations should remain separate records. Each record needs source identity, collection time, observation time when available, and a description of whether it represents intended, current, historical, or aggregated state.

The model can then reason about the disagreement without having to infer the authority and meaning of each unlabeled block of text.

Evidence routing also reduces unnecessary collection. A question about current neighbor state may require a live device query. A question about whether the session dropped repeatedly during the previous hour may be answered more efficiently through logs or telemetry.

The source selected for one question may be unsuitable for another. Flow records may confirm that traffic stopped crossing a link, but they do not explain why. A device query may show the current routing state, but it may not reveal a transient event that occurred thirty minutes earlier.

Tool boundaries

Tool design determines how much responsibility remains with the model and how much is enforced by the surrounding system.

A broad SSH tool that accepts an arbitrary target and arbitrary command places device selection, platform selection, syntax, and safety inside a probabilistic request.

A narrower interface allows the model to express the information it needs while the harness controls execution.

{
  "device": "ATL-EDGE-03",
  "operation": "get_bgp_neighbor_state",
  "arguments": {
    "neighbor": "192.0.2.10"
  }
}

The harness resolves the target platform and translates the operation into the appropriate command, API call, or parser.

Higher-level tools work well when the operation has a stable meaning across platforms. Direct command execution remains useful when the supported surface is broad, changes frequently, or differs substantially among vendors.

The correct balance is not obvious.

At one point during development, we expanded a small tool surface into approximately 120 generated tools based on a parsed vendor API catalogue. The larger tool set required a vector index so the model would not receive all tool definitions at once. It then required a separate model call to rewrite the user’s request into retrieval terms before the main model could receive the selected tools.

The retrieval machinery existed largely because the tool surface had become too large to manage directly.

That architecture was later replaced with a much smaller command-oriented interface. The model could request a command against a known device, while the harness handled platform resolution, validation, execution, output limits, and error classification.

The lesson was not that a small number of tools is always better. The larger lesson was that the shape of a vendor API should not automatically become the shape of the model’s action space. A large set of narrowly typed tools can move complexity from execution into retrieval, selection, and orchestration.

Tool validation must occur after the model generates the request and before the device receives it. Device-side read-only authorization provides another enforcement layer.

Tool responses should distinguish among authentication failure, connection timeout, unsupported syntax, parser failure, truncated output, successful empty output, successful output with data, and device rejection. These outcomes have different meanings inside an investigation.

Device access and command execution

Device access appears simple when described as “run a command over SSH.” In practice, the transport layer contains many of the most difficult reliability problems.

Network operating systems differ in prompt behavior, paging, privilege levels, error messages, line endings, command completion, and empty-output formats.

A generic transport may mistake a line inside command output for the device prompt and terminate collection early. The tool returns successfully with incomplete evidence. No exception is raised because the transport believes the command finished normally.

A device may reject a command by printing an error message as ordinary output. The SSH session succeeds, the command returns text, and the transport reports success. The investigation must still determine whether the device accepted the command.

A command may return no rows because the requested condition is absent. On one platform, this may appear as an empty body. On another, it may contain only a header. On another, it may contain a legend followed by no records. These outputs should be classified as successful empty results rather than execution failures.

Some commands page even when the command includes a bound. Others can hang a session under particular conditions. The harness may need to rewrite the request into a bounded form, reject it before execution, or apply a shorter command-specific timeout.

During evaluation, we observed a case where a device rejected a command, but the transport reported success. Repeated runs then produced different downstream behavior. One run treated the empty result as a factual absence. Other runs reported the input error. Another associated the requested object with a plausible value from unrelated output.

Changing only the command argument caused the control run to succeed. This isolated the failure to the distinction between transport success and device acceptance rather than to the model’s knowledge of the network.

The durable fix required several layers: platform-specific rejection classification, universal handling for general-purpose shell errors, structured error results that could return to the agent’s recovery path, and a bounded classifier for outputs that regular expressions could not reliably distinguish.

The classifier sees a narrow view of the command, transport status, output, and output length. Its effect is limited to classifying whether the device rejected the request. It does not diagnose the network condition.

This is an important division of responsibility. Deterministic code handles the cases that can be specified precisely. A model can help classify ambiguous output when its input and downstream authority are tightly bounded.

Raw evidence and structured findings

Network devices often produce large outputs. Passing every result directly into the model causes the active context to grow quickly. Important lines become harder to locate, and repeated collection consumes context that could otherwise be used for reasoning.

Immediate summarization creates another risk. A summary may remove a detail that becomes important later or convert an ambiguous observation into a stronger claim than the source supports.

A network harness needs to preserve the raw source while giving the model a manageable representation.

An early approach stored large outputs behind pointers. The complete output was compressed and saved, while the conversation received a small envelope containing a key and a preview. The model could retrieve the full artifact when it needed more detail.

The model used the retrieval tool, and the complete artifact returned to the same context the pointer system was intended to protect. Retrieval was also too coarse. The model selected whole artifacts rather than the relevant sections within them.

The more durable pattern separates storage from projection.

The complete, unmodified output is persisted before summarization or truncation. The model receives a bounded representation with a reference to the source. Mid-file evidence receives explicit handling because head-and-tail truncation often removes the part of a log that explains the incident.

{
  "finding": "Ethernet1/12 reported 814 CRC errors",
  "device": "DC1-LEAF-02",
  "command": "show interface Ethernet1/12 counters errors",
  "observed_at": "2026-08-02T16:14:22Z",
  "evidence_id": "ev-1042",
  "status": "confirmed"
}

The model can reason over the concise finding. The complete device output remains available through the evidence identifier.

Retrieval can then operate within bounds. The model refers to a source by a handle assigned by the system and supplies literal search terms. Matching is deterministic and the response is limited by lines and bytes. Search terms that find nothing are returned explicitly so the model can distinguish a failed search from a genuinely small artifact.

Large outputs can also be represented with verified source anchors. A small extraction step identifies short passages from the raw output, and deterministic validation confirms that each passage is a literal substring of the stored source.

This prevents an extracted anchor from containing words that are absent from the source. It does not prove that the model’s interpretation of the anchor is correct.

The evidence identifier must resolve to an actual stored artifact. Generated source labels without durable underlying data provide the appearance of provenance without the substance.

Current state and historical state

Network troubleshooting frequently mixes current and historical information.

A log entry may show that an interface went down thirty minutes ago. A current command may show that it is up. A monitoring system may still have an open alert because its recovery condition has not been met.

All of these observations may be accurate.

The harness should preserve the temporal meaning of each one.

The interface transitioned down at 15:42 UTC.
The interface returned to the up state at 15:43 UTC.
The interface was operational when checked at 16:14 UTC.
The monitoring alert remained open at 16:16 UTC.
The cause of the transition was not confirmed.

Combining these observations into one current-state claim would remove important information. Reporting only that the interface is up would omit the event that triggered the incident. Reporting that the interface is down would misrepresent the present condition.

Temporal correlation becomes more difficult when several systems are involved. Device clocks may differ. Monitoring intervals may aggregate short events. Logs may arrive late. External APIs may update more slowly than local telemetry.

The harness should preserve source timestamps and collection timestamps. When clock quality or event ordering is uncertain, the investigation should retain that uncertainty.

Transient incidents often require more historical evidence than current evidence. By the time the investigation begins, the original condition may have cleared. Counter changes, event logs, telemetry history, recent configuration changes, and correlated alerts may provide the remaining path to a useful explanation.

Hypotheses and findings

A network agent needs to distinguish direct observations from causal explanations.

A finding is supported by an identified source. A hypothesis proposes an explanation that remains under investigation. A contradicted hypothesis conflicts with available evidence. An unverified claim could not be checked because a required source was unavailable. An unresolved question remains open because the evidence does not distinguish among the remaining possibilities.

Consider an interface with increasing CRC errors. The remote side also reports input errors, and logs show two transitions during the incident window. Current optical power remains within the expected range.

The evidence supports a physical-layer problem, but it does not identify a specific cable, connector, or transceiver failure.

Hypothesis:
Physical degradation on the uplink

Supporting evidence:
The local interface reports increasing CRC errors.
The remote interface reports corresponding input errors.
Both interfaces transitioned during the incident window.

Conflicting evidence:
Current optical power is within the expected range.

Missing evidence:
Historical optical measurements are unavailable.
Cable and transceiver replacement history is unavailable.

Current assessment:
A physical-layer fault is likely but remains unconfirmed.

This representation supports investigation planning and final reporting. The model can identify which missing observation would reduce uncertainty, while the final response can distinguish direct evidence from inference.

Explicit hypothesis state also reduces anchoring. Competing explanations can remain active until enough evidence exists to reject them. Contradictory observations remain visible after the model develops a preferred explanation.

Negative results also have value. A command showing no interface transitions or no route changes can eliminate part of the search space. These observations need to persist with the same care as positive findings.

Provenance and verification

A plausible answer can still contain an unsupported relationship.

In one graded evaluation run, the system identified a real device and a real interface and described the association between them as verified. The command intended to locate the target had been rejected by the device. The interface came from unrelated output collected earlier in the same investigation.

Both objects were real. Their relationship existed only in model-authored text.

The failure was not a random invention of a device or port. It was an inferred association presented with more certainty than the evidence allowed. That distinction matters because the required controls differ.

A provenance verifier can check whether a cited excerpt exists in the persisted output and whether it contains both the subject and the value being attached to it. It can also downgrade a claim when the evidence source failed or when the required terms are absent.

The model remains responsible for making and citing the claim. Deterministic code verifies whether the citation supports the claimed relationship. The backend does not need to parse arbitrary device output and independently decide the diagnosis.

A not-found claim requires a successful source. A failed command cannot validate absence. If a device was queried and did not answer, the state is unavailable. If several devices were queried and only some returned negative results, the broader state remains unresolved.

This distinction is operationally important because absence and unavailability often lead to opposite next steps.

These failure patterns came from graded lab and evaluation runs rather than production incidents. Their value comes from exposing mechanisms that ordinary answer grading would have missed.

Persistent investigation state

A long investigation cannot live only inside a conversation transcript.

The system needs durable state that survives model calls, sub-agent activity, retries, worker changes, interruptions, and context reduction.

That state should include the original alert, incident scope, active hypotheses, eliminated hypotheses, collected evidence, failed checks, timestamps, source identifiers, open questions, and previous actions.

The need for external state becomes clear in asynchronous systems. During development, alerts handled by one background worker could lose conversation history that existed only in another worker’s memory. The solution required serializing the investigation state into shared storage and reconstructing it when a different worker continued the run.

This was an ordinary distributed systems failure. The presence of a language model did not change the underlying requirement that independent workers need a shared, durable representation of state.

Appending every tool result and model response to the next prompt works for short interactions. Longer investigations accumulate outdated hypotheses, repeated outputs, failed approaches, and low-value detail. Even when the content fits within the model’s context window, the additional material may make the current problem harder to follow.

Context construction therefore requires selection and compression. The harness retrieves the active hypotheses, relevant evidence, unresolved contradictions, recent failures, and available actions. The complete trajectory remains outside the prompt.

The state model should preserve why a hypothesis was rejected. Otherwise, a later model call may reintroduce the same explanation after earlier evidence has been compressed out of the active context.

Context as an engineering problem

As network investigations become longer, context tends to accumulate in separate surfaces.

One block contains prior-pass notes. Another contains execution summaries. Another contains coverage status. Another contains unresolved gaps. Another contains retrieval handles. Each addition solves a local problem, but the combined prompt begins to duplicate and contradict itself.

A more stable design projects the investigation state into one mechanically assembled facts representation.

That representation can contain the investigation goal, current request, scope, pass counters, target status, command rows, timestamps, raw evidence references, retrieval handles, authored notes, and explicit omissions. It should avoid backend-authored diagnostic judgments.

The facts representation should declare where information is lossy or incomplete. A partial projection should not appear complete simply because the missing data did not fit into the prompt.

Precise values deserve special handling.

In one controlled comparison, prior-pass findings were carried forward as prose in one arm and as structured fields in another. The prose arm lost exact operational values such as counters, timestamps, and event counts.

Two mechanisms produced the loss. One was a hard character limit that truncated a longer finding. The other was ordinary model choice. The value fit, but the model chose to write about something else.

The structured representation gave these values explicit fields. A follow-up evaluation recovered the values that prose had lost.

The practical conclusion is straightforward: when an exact value must survive several model calls, it needs an explicit field. Asking a model to repack precise operational values into prose creates a lossy compression step with no defined loss function.

Larger context windows reduce some pressure, but they do not remove this problem. The relevant question is not only whether the text fits. The question is whether the representation preserves the distinctions and values required for the next decision.

Scope control

An investigation should begin with the smallest reasonable fault domain.

If an alert reports a spanning-tree change on one access switch, the initial scope may include the affected switch, the relevant interface, the immediate neighbor, the current root bridge, and recent topology-change events.

Querying every switch in the environment would increase device load, investigation time, model cost, and the number of unrelated anomalies presented to the model.

The scope can expand when the evidence justifies it. If several access switches report changes at the same time, the investigation may move toward the distribution layer. If the local and remote interface state is healthy but several services share the same upstream path, the agent may investigate the common provider or cloud connection.

The harness should enforce limits on which devices can be reached, how many targets can be queried, how much parallelism is allowed, and when broader access requires human approval.

Scope decisions should remain visible in the investigation history. An operator reviewing the result should be able to see why each device or external system was queried.

A narrow initial scope also improves reasoning quality. The model receives fewer unrelated abnormalities and can concentrate on evidence connected to the reported symptom.

Sub-agents and parallel investigation

Network incidents often span several devices and data sources. Parallel collection can reduce investigation time when the work can be divided into separate tasks.

A supervisor may assign one worker to inspect the affected device, another to inspect the remote endpoint, another to review monitoring history, and another to compare routing state across peers.

Each worker should receive a narrow objective, bounded targets, a limited tool set, and the relevant portion of the investigation state. Its output should return as structured findings and evidence references.

Role separation should be enforced in the tool layer. A per-device worker should not be able to expand the investigation or operate on a different device simply because it generated different arguments. A supervisor should not bypass the worker layer and execute device commands directly when those responsibilities are intentionally separated.

Parallel execution introduces several engineering problems.

Progress messages from multiple workers can arrive out of order and interleave in the operator transcript. Device-access libraries may contend on shared initialization paths. Mutable connection objects may be unsafe to share across concurrent tasks. A refusal can become a loop when the model repeats the same malformed request without seeing enough information to correct it.

Some attempted optimizations make performance worse. Caching a mutable device object may reduce initialization overhead while creating unsafe concurrent use. Limiting connection setup may reduce contention while increasing total wall time. These changes need controlled comparison rather than assumption.

Loop bounds also need several layers. A pass budget can limit accepted fan-outs. An iteration cap can limit the routing loop. A framework recursion limit can remain as the final backstop. The earlier limits should still allow the system to produce a useful answer with explicit gaps.

Parallelism also creates a context problem at fan-in. Several workers may each return a manageable result, while their combined output is too large for efficient synthesis. The harness then needs an aggregate decision about which sources remain detailed, which are compacted, and which require retrieval.

Parallel agents improve latency and decomposition. They also require process management, ordering, isolation, bounded execution, and a plan for combined evidence.

Output limits, timeouts, and concurrency

Network tools require operational limits.

A command may return far more data than expected. A connection attempt may hang. An agent may request the same expensive operation across many devices. Excessive parallel collection may increase load on infrastructure during an incident.

The harness should enforce connection timeouts, command timeouts, output caps, concurrency limits, retry limits, target limits, and request rates.

These controls protect the network and the investigation process. Large outputs can overwhelm the active context. Long-running commands can block the workflow. Repeated retries can turn a minor access problem into a larger operational issue.

The investigation record should show when a limit affected the result. Truncated output should be marked as truncated. A timed-out command should remain a failed check. A rate-limited operation should remain visible in the history.

Retry behavior should depend on the failure. A temporary connection timeout may justify another attempt. Repeated authentication failures usually require operator intervention. Repeating a large command across many devices may increase load without improving the investigation.

Model roles

Different parts of the investigation require different model capabilities.

A strong reasoning model may be useful for planning the investigation, comparing hypotheses, resolving contradictions, and deciding whether the evidence is sufficient.

A smaller or more predictable model may be suitable for extracting source anchors, classifying command acceptance, converting parser results into a fixed schema, or checking whether required fields are present.

Command validation, target authorization, timeout enforcement, permission checks, and schema validation should use deterministic code where predictable behavior is required.

The orchestration model has a large influence on investigation quality because it decides which evidence to collect and how to interpret the combined result. A weak orchestrator may lose track of the original objective, expand the scope unnecessarily, ignore contradictions, or stop before the evidence supports a conclusion.

Collection workers operate on narrower tasks and can use stricter schemas. Their model requirements may differ from those of the orchestrator.

Model selection should be based on repeated evaluation within each role. Accuracy, latency, cost, tool-use behavior, schema reliability, and deployment constraints may lead to different choices across the system.

The surrounding architecture should make model replacement practical. Tool contracts, evidence records, permission controls, and investigation state should remain usable as models and providers change.

Failure handling

Tool failures are normal in production networks.

Devices may be unreachable. Credentials may fail. A parser may not support a software release. A command may time out. An API may return stale data. A monitoring platform may be unavailable.

The harness should represent each failure directly.

Requested check:
Retrieve remote interface state.

Execution result:
The check failed because SSH authentication was rejected.

Investigation impact:
Current state on the remote side could not be verified.

Affected hypothesis:
A physical fault remains possible, but the remote evidence is incomplete.

A failed check describes the investigation process and provides no direct evidence about the condition of the network.

An authentication failure does not show that the remote interface is down. A timeout does not show that no errors exist. A parser failure does not invalidate the underlying device output.

Fallback behavior should remain visible. If the system substitutes raw output after a parser failure, uses telemetry because live access failed, or relies on an older observation, the investigation should record that limitation.

Failures also affect confidence. A hypothesis supported only by local evidence should not receive the same assessment as one supported by both ends of a link.

Stopping decisions

An investigation needs a stopping condition because another check is almost always available.

Continued collection may add little information, repeat previous work, or expand into unrelated infrastructure. Early stopping creates a different problem when the system accepts its first plausible explanation before testing meaningful alternatives.

A stopping decision should consider whether the primary symptoms have been explained, whether the strongest competing hypotheses have been tested, whether important contradictions remain, whether the evidence is current enough, and whether additional collection is likely to change the conclusion.

Some investigations will end without a confirmed root cause. The available evidence may support a likely explanation, several unresolved possibilities, or no reliable conclusion.

The system should preserve that result and identify the missing information that prevents further progress. Human escalation may request provider confirmation, physical inspection, maintenance history, access to an unavailable device, or additional telemetry.

An unresolved result can still be operationally useful when it accurately defines the remaining uncertainty and the next action required.

Evaluation requires the full trajectory

A correct final diagnosis does not establish that the investigation process was reliable.

The agent may have queried the wrong device, used invalid commands, ignored contradictory evidence, fabricated a relationship, repeated unnecessary checks, or reached the correct conclusion through unsupported reasoning.

Evaluation should examine the complete trajectory.

Diagnostic accuracy measures whether the conclusion matches the underlying condition. Evidence precision measures whether cited observations support the claims. Command validity measures whether operations are appropriate for the target platform. Scope efficiency measures whether the agent gathered useful evidence without searching unrelated infrastructure.

Process status and answer quality need separate treatment. A run can complete successfully at the software level while producing the wrong investigation shape. It can also produce a useful answer despite inefficient retrieval, missing evidence, or an unnecessary fan-out.

During evaluation, we repeatedly observed correct or useful answers sitting on top of weak trajectories. In one case, the final answer was useful while the same run contained oversized context, retrieval calls that failed against hidden limits, and open evidence gaps that contradicted information already present. In another, the system finalized after a follow-up fan-out that collected no device evidence. Another completed quickly with a generic explanation while a reference run on the same prompt investigated two devices.

Useful evaluation methods include single-variable replay, recorded transport fixtures, known-ground-truth fault injection, explicit expected-absence cases, and active attempts to disconfirm the leading conclusion.

Single-variable replay freezes one model decision and changes one element at a time. This allows the effect of schema descriptions, field order, transient instructions, or context blocks to be measured without changing the entire run.

Recorded transport fixtures serve known device output through the real connection and parsing path. This helps detect line-ending changes, truncation, prompt-recognition errors, and platform-specific drift.

Expected-absence cases represent questions whose required data source does not exist. An honest statement that the information is unavailable should receive full credit, while a fabricated value should fail.

A mandatory disconfirmation step forces the evaluation to ask whether a success status concealed an empty result, rejected command, unrelated source, or incomplete collection.

Repeated runs are necessary because model behavior is probabilistic. Evaluation across models, platforms, incident classes, and failure conditions provides more information than a single successful demonstration.

Why network verification is difficult

Coding agents often operate in environments with strong feedback. Code can compile, tests can pass, and generated artifacts can be compared with expected results.

Network operations provide weaker verification.

A routing session can be established while application traffic still fails. A configuration can be syntactically valid and operationally harmful. An interface can remain up while dropping packets. A fault can clear before the investigation begins. Several visible symptoms may share one cause, while an unrelated anomaly may appear more severe than the actual fault.

Network state is distributed across devices and systems. Observations are collected at different times. Vendors use different commands, terminology, data models, and implementation behavior.

The investigation may never receive one definitive signal that proves the root cause. The harness must reason about source authority, time, uncertainty, missing evidence, and operational impact.

Evaluation should therefore account for calibrated conclusions and appropriate escalation rather than requiring a confirmed root cause in every case.

Read-only investigation and write control

Diagnosis and remediation have different risk profiles.

Read-only access still creates risk through excessive queries, sensitive output, and incorrect conclusions. The direct risk to forwarding behavior remains lower because the agent cannot modify configuration.

Early systems may rely on prompt instructions telling the model to ask for permission before applying a change. That is a behavioral instruction rather than an authorization boundary.

A stronger design removes write operations from the model-facing action surface and places them behind a separate, deterministic, human-authorized control path.

A proposed change should identify the target, exact operation, supporting evidence, expected result, affected scope, pre-checks, rollback procedure, and post-change validation.

The authorization system should verify the approved operation independently of the model. Post-change verification should collect the expected state after execution and compare it with the proposed outcome.

A successful command response confirms that the command executed. It does not confirm that the intended operational effect occurred.

Risk policy can account for device role, operation type, blast radius, maintenance status, and the availability of an out-of-band recovery path. Some operations may remain unsuitable for agent execution even when an operator requests them.

MCP and standardized tool interfaces

MCP provides a standard interface through which AI applications can discover and invoke tools exposed by external systems.

A network-related MCP server may expose inventory, observability, cloud connectivity, topology, ticketing, configuration, or device operations. Standardization can reduce integration work and allow vendors to maintain their own resource models and APIs.

The harness still performs source selection, investigation planning, target authorization, evidence management, failure handling, and stopping decisions.

An MCP tool may return the state of a cloud connection. The investigation still needs to determine whether that connection is relevant to the incident, whether another source reports conflicting information, and whether the evidence supports a conclusion.

Standardized tool interfaces primarily improve the connector layer. Cross-system investigation still requires a runtime that controls how those tools are selected and how their results enter the investigation.

Model capability and harness design

Stronger models will simplify some parts of the harness.

Improved instruction following may reduce prompt complexity. Better tool selection may require fewer routing rules. Larger context windows may reduce the frequency of context compression. Better reasoning may improve hypothesis ranking and contradiction handling.

The surrounding system will continue to manage authenticated access, approved targets, topology, current evidence, persistent state, permissions, operational limits, and audit history. These functions depend on the external environment.

Prompt complexity should be reviewed as models improve. Instructions that compensate for temporary model weaknesses can become unnecessary and may eventually reduce performance.

During development, the main prompt grew as individual failures produced new rules. It later shrank substantially after introducing a review policy that asked whether each rule belonged in the prompt, a tool contract, retrieved context, a platform profile, or deterministic code.

The useful mental model is that the system prompt initializes the agent, while the harness carries the operating state and enforcement mechanisms.

In the failures we could reconstruct from commits, tests, and evaluation artifacts, durable fixes usually required changes to tools, context, evidence handling, workflow, or controls. The historical record does not isolate model upgrades well enough to measure their independent contribution. Stronger models clearly made parts of the system easier, but the available record cannot quantify how much improvement came from the model itself.

When a model is used as a bounded component, such as an output classifier or source-anchor extractor, much of the engineering lies in controlling what it sees and limiting what its output can affect.

Agentic network operations

A practical network-agent system can receive an alert or operational question, resolve the relevant environment, determine which evidence is missing, and collect that evidence through bounded tools.

The system maintains investigation state across several rounds of reasoning and execution. It preserves raw evidence, tracks competing hypotheses, records failed checks, limits scope expansion, and reports conclusions according to the strength of the available evidence.

Its remediation workflow uses a separate authorization path with defined targets, operations, rollback procedures, and post-change verification.

The language model handles planning, interpretation, and synthesis where those tasks benefit from flexible reasoning. Deterministic code handles permissions, target resolution, command boundaries, evidence validation, execution limits, and other requirements that need predictable enforcement. Persistent storage preserves the source material and investigation state across model calls and worker processes.

Reliable agentic network operations depend on how these parts work together.