# Nightshift, full text

> The data and analytics layer for AI agents. Connect a source once and any agent gets exactly the data you allow, enforced down to the row, over MCP. The keys never leave your control.

Nightshift, in three verbs. Connect a data source and Nightshift builds a connector and a governed catalog; the data stays where it lives. An agent writes a request against that catalog and queries it live, with nothing to stand up. The compiler mints a policy-enforced endpoint, identity-aware and enforced the moment the request arrives: allow, deny, or mask by row, column, and value. Any consumer can call that endpoint (an MCP tool, a browser, a data app), with agents the expected primary consumer. When a policy needs a human, the request is held for an approval before anything flows. Credentials never reach the agent or its model, and observability falls out of routing every access through one compiler.

This is the full-text edition: page summaries followed by the complete text of every post. For the compact index, see https://nightshift.sh/llms.txt.

# Pages

## Product

- [Console](https://nightshift.sh/console): The governed data layer your team and your agents share: connect any source, query it live in notebooks or over MCP, and govern every read and write by identity.
- [Connect](https://nightshift.sh/connect): Point Nightshift at a source and it builds a connector and a governed catalog. The data stays where it lives; no copy into a warehouse, no key handed to an agent.
- [Govern](https://nightshift.sh/govern): Identity-aware policy, enforced the moment a request arrives: allow, deny, or mask by row, column, and value, down to a named identity.
- [Query](https://nightshift.sh/query): Any agent queries the governed catalog live: analytics that run at the edge of the source, with views, snapshots, and time travel, delivered over MCP.
- [Pricing](https://nightshift.sh/pricing): Usage-based pricing, shown on the page. A free workspace covers one source, with row-level policy and live queries.
- [Product tour](https://nightshift.sh/demo): A guided walkthrough of the platform: catalog, policies, identities, approvals, endpoints, and the audit log.

## Use cases

- [Financial services](https://nightshift.sh/use-cases/financial-services): Identity-aware, row-level access to portfolio, borrower, and fund data. PII and MNPI masked before they leave, sensitive actions gated.
- [Healthcare](https://nightshift.sh/use-cases/healthcare): Clearance-aware access to clinical and claims data. PHI masked in the compiler, restricted records denied, every access logged.
- [Industrial ops](https://nightshift.sh/use-cases/industrial-ops): Agents read the line through the catalog. The path that moves a machine is never exposed, so a wrong write cannot happen.
- [Logistics](https://nightshift.sh/use-cases/logistics): Each partner agent sees only its own lanes across the TMS, WMS, and carrier APIs. Consignee contacts masked, margins denied.
- [Public sector](https://nightshift.sh/use-cases/public-sector): Clearance-aware access to case, benefits, and records data. Sensitive fields masked, restricted records denied, every read on the record.
- [Retail](https://nightshift.sh/use-cases/retail): Governed access to commerce, CRM, and inventory data. Customer PII masked before it leaves, pricing kept behind a gate.

## Company

- [About](https://nightshift.sh/about): Who builds Nightshift and why.
- [Partners](https://nightshift.sh/partners): Working with Nightshift as a partner.
- [Careers](https://nightshift.sh/careers): Open roles.
- [Contact](https://nightshift.sh/contact): Talk to the team.

## Get started

- [Create a free workspace](https://nightshift.sh/signin?mode=signup): One source, with row-level policy and live queries.
- [Open-source spec](https://github.com/nightshiftco/nightshift): The Nightshift specification, developed in the open.

# Writing

## AI Should Meet You Where You Are

> May 29, 2026 · Ethan Henley · https://nightshift.sh/blog/ai-should-meet-you-where-you-are

*Why we built Nightshift to adapt to your infrastructure instead of asking your infrastructure to adapt to it.*

When a team evaluates a new piece of AI infrastructure, the first question is usually what it costs. The license, the compute, the seats. That number is almost never the real one.

The real cost is in the assumptions. An agent runtime is not a narrow tool. To do anything useful, an agent has to compute something, store state, authenticate as someone, hold a secret, reach the network, and call a model. That is six dependencies before it has done a single unit of work. Every platform has to make a decision at each of those six points. The only question that matters is whether it makes the decision for you, or lets you make it.

## The hidden migration

Most platforms make it for you. They run on their cloud. They manage identity their way. They keep your data in their tenancy. They route to the model they have partnered with. Each of those choices is reasonable on its own. Taken together, they describe a destination: a place you move your workload to. And moving a workload is a migration, whatever the sales deck calls it.

For a company starting from nothing, a destination is fine. There is nothing to migrate. But most companies that need agents are not starting from nothing. They have a cloud account with years of configuration buried in it. They have an identity provider that every internal system already trusts. They have a compliance boundary that took a long time and real money to stand up. They have hard-won opinions about where their data is allowed to live.

For them, every assumption the platform makes is a place where their environment has to bend. The sum of those bends is a migration wearing the costume of an integration. You do not notice it as a migration because it arrives one reasonable requirement at a time. Standardize on this cloud. Federate to this identity model. Send your data here. Use this model. None of them looks like a re-platforming on its own. All of them together are exactly that.

## Every dependency is an interface

We started from the opposite premise. A runtime should treat each of its dependencies as an interface, not a fixed choice. Compute is an interface: run it where you already run things. Identity is an interface: use the provider your systems already trust. Storage, secrets, network egress, and the model itself are interfaces too, each one pointed at a system you already operate.

This sounds like a small distinction. It is not. An interface is a promise that the thing behind it can be swapped. A fixed choice is a promise that it cannot. A runtime built on interfaces can meet your environment where it is. A runtime built on fixed choices can only ask your environment to come to it. That difference decides whether adoption is a configuration exercise or a project with its own budget line and a steering committee.

<Figure
  src="/blog/breadboard-inverted.png"
  alt="Each agent dependency wired to a system you already operate"
  caption="Each dependency is an interface pointed at a system you already operate."
  width={1063}
  height={516}
/>

## Adapting without giving up guarantees

Meeting you where you are is only worth anything if the runtime is trustworthy at its core. Adapting to someone else's environment is trivial if you are willing to ignore security. The hard version, the one worth building, is adapting while keeping strong guarantees no matter whose infrastructure you land on. A few ideas carry most of that weight.

**Isolation.** Every agent runs inside its own sandboxed execution environment with a hard boundary around it. A compromised or misbehaving agent cannot reach past that boundary. The isolation is the unit of trust, and it holds regardless of whose cloud it happens to be running in.

**Default-deny network policy.** Agent network access is governed by explicit, default-deny policy. The agent reaches what you have allowed and nothing more. You decide the egress, not the platform, and not the agent.

**System-level observability.** You can see what an agent actually does, not only what it reports back. Watching at the level of system calls is the difference between trusting an assertion and verifying a behavior. For anyone who is personally accountable for what runs inside their environment, that difference is the entire job.

**Secrets that never persist.** Credentials are injected into the isolated environment at the moment of use, scoped to the task, and never written down. The agent gets what it needs to do the work and holds nothing afterward.

None of these properties depend on a particular cloud, a particular identity provider, or a particular model. They belong to the runtime itself. That is precisely what lets the runtime travel to your environment without leaving its guarantees behind.

## Deployment becomes a decision you can reason about

The payoff of that design is that deployment is bounded. Because the runtime brings its own guarantees and points its interfaces at systems you already run, adopting it is additive rather than substitutive. You are not re-platforming. You are placing a runtime inside a boundary you already control and configuring it to use the compute, identity, and storage that are already there.

This keeps the blast radius of adoption small enough to actually reason about. You can deploy into a single boundary, hand it a real workload, watch what it does at the system level, and then decide whether to expand. The decision to try is not the decision to migrate. For most AI platforms those have quietly become the same decision. They should not be.

<RepoCallout />

## What stays yours

Because the runtime lives inside your boundary and treats the model as an interface, you keep the things that matter. The orchestration runs on your side. The secrets stay in your environment. The work your agents produce stays where you put it. If you call a hosted model, that request does go to the provider, and we would rather say so plainly than pretend otherwise. But everything around that call, the part that accumulates and compounds, remains yours.

Over time, that is what lets a team move from renting capability to owning it. That is a longer story for another post. It starts with a simpler decision than most people expect: whether the infrastructure you adopt fits the environment you already have.

## The right question

The industry has spent a few years asking how to get enterprises onto AI platforms. We think that is the wrong question. The better one is how AI fits into the infrastructure enterprises already have. The first treats your stack as a problem to be migrated away from. The second treats it as the ground you build on.

We built Nightshift on the second answer. It meets you where you are, because that is where the work already is.

<CTA
  title="See it run in your environment"
  description="Bring your cloud, your IdP, your compliance boundary. We will show you Nightshift deployed into infrastructure you already control."
  label="Talk to sales"
  href="/contact"
/>

---

## The Wrong Question About AI Cost

> May 27, 2026 · Gianni Crivello · https://nightshift.sh/blog/the-wrong-question-about-ai-cost

When Dwarkesh Patel toured [Jane Street's new Texas datacenter](https://www.youtube.com/watch?v=8J-GUnfSqeE) last week (4,032 liquid-cooled GPUs, 56 racks, 8,000 km of internal fiber) he asked the obvious question: what does all of this cost?

Ron Minsky, who co-heads Jane Street's technology group, gave the answer that should be on every mid-market CFO's wall: the cost of the compute is trivial next to the cost of not having it.

That's opportunity cost. And it's the frame most enterprises still get wrong.

## The math behind the answer

Jane Street did $10.1B in net trading revenue in Q2 2025 alone, with roughly 3,000 employees. They've reportedly committed $6B+ to CoreWeave for compute access and $1B+ in equity, plus stakes in Anthropic and the cluster Dwarkesh toured. Compare that to Goldman Sachs: 46,000 employees, a fraction of the net income.

Minsky's answer isn't "compute is cheap." It's that the alternative, losing a signal because your researchers had to wait three days for a training run, or shipping a worse trading model because you couldn't afford the experiment, costs orders of magnitude more than the GPUs.

AI isn't cheap, but doing nothing is the most expensive option on the menu.

## Most mid-market enterprises ask the wrong question

The default boardroom question is "what does AI cost?" That's the wrong question. It anchors the conversation on a line item: model fees, infra, headcount. It treats "do nothing" as the zero-cost baseline.

It isn't. MIT's NANDA initiative found that [95% of enterprise GenAI pilots produce no measurable P&L impact](https://www.legal.io/articles/5719519/MIT-Report-Finds-95-of-AI-Pilots-Fail-to-Deliver-ROI-Exposing-GenAI-Divide), and that the 5% that work share three traits: tightly scoped use cases, back-office focus, and a deployment harness that integrates into actual workflows. The failure mode isn't model quality. It's that companies bolt AI onto a process and call it done.

PwC's 2026 Global CEO Survey found 56% of CEOs report no revenue or cost benefit from their AI spend yet. That number gets thrown around as evidence AI is overhyped. The better reading: most of that spend went to pilots that were never set up to run agents in production against real data, on infrastructure the team controls.

Meanwhile, [Deloitte reports 78% of US mid-market leaders have moved at least one AI project into full production](https://www.novaedgedigitallabs.tech/Blog/ai-implementation-roi-2026-guide). The gap between those two numbers is where opportunity cost compounds. Early movers in your segment are training on workflows you haven't even instrumented yet.

## What opportunity cost actually looks like in the mid-market

You're not Jane Street. You don't have $10B in quarterly revenue justifying a $6B compute bill. But the structural question is identical:

- **Every workflow you don't automate is a recurring tax.** The procurement team manually reconciling vendor invoices, the analyst rebuilding the same dashboard every Monday, the support engineer re-triaging tickets. Those hours don't show up as a line on the P&L, which is exactly why they get ignored. Multiply them by 52 weeks and they dwarf any reasonable agent infra bill.

- **Your competitors' training data compounds.** The MIT report and follow-on analyses keep landing on the same point: organizations that deploy AI into real workflows accumulate proprietary signal that laggards can't buy later. That gap is not linear. It's exponential, and it has roughly an 18-month window before it locks in.

- **The talent question flips.** Engineers and analysts who use agents daily are 2 to 3x more effective, and they will not stay at companies that won't let them. Your retention cost is your AI cost in disguise.

The do-nothing path doesn't preserve capital. It bleeds it through channels that don't appear on a budget review.

## The harder part: controlling OpEx once you're in

Here's where most companies who do take the leap fall on the other edge of the sword. They sign a frontier-model API contract, give the engineering team a credit card, and discover six weeks later that sustained agent workloads burn through a quarter's budget. Or that token spend on a single agent surface scaled with usage in ways finance didn't model. Or that the "AI line" on the cloud bill is now bigger than the rest of the cloud bill (which will happen).

This is the second half of the opportunity-cost argument that most vendors skip. Saying "yes" to AI doesn't mean saying "yes" to uncapped OpEx. Sustained agentic work, agents running on schedules, against your data, producing real artifacts, has to live somewhere you can govern.

That's the gap [Nightshift](https://nightshift.sh) was built for. Nightshift gives organizations the escape hatch out of the frontier providers. Start with Claude, instrument the workflows, and produce proprietary work trajectories that you can distill your own models on. Once you have models that perform at par with the frontier on your tasks (and in many cases out-perform the frontier), swap your proprietary models in, running on fixed-price compute.

The point isn't to make AI cheaper. It's to make the cost legible and controllable, so the opportunity-cost math actually works in your favor instead of running away from you.

## The frame to bring to your next board meeting

Stop asking "what does AI cost." Start asking two questions:

1. **What does it cost us to keep doing this work the way we do it now, for the next four quarters?** Include the labor hours, the error rate, the lost speed, the talent attrition, and the deals you won't close because a competitor moved first.

2. **What does it cost to run this work as sustained agentic operations, on infra we control, with hard ceilings on OpEx?**

The first number is almost always larger. That's Minsky's answer in mid-market clothes. The companies that internalize it in 2026 will look like the 5% in next year's MIT report. The ones who keep asking "what does AI cost" will be quoted in the 95%.

Minsky's answer at Jane Street wasn't a flex about the size of their budget. It was a discipline. Ask the right question, the math becomes obvious. Ask the wrong one, and you spend the next four quarters defending a budget for work that was never going to compound anyway.

The cost of compute is not the question. The cost of going without is. Nightshift exists so the answer is actionable for the companies that aren't running $10B quarters.

<CTA
  title="Run the math on your AI bill"
  description="A 30-minute briefing on what the opportunity-cost math actually looks like for your operation. Bring your current hosted-API spend."
  label="Book a briefing"
  href="/contact"
/>

---

## Shipping Real Apps with Harness Engineering

> March 30, 2026 · Gianni Crivello · https://nightshift.sh/blog/embracing-harness-engineering-with-nightshift

Today was a milestone for Nightshift.

When we first launched the company, we built applications to fund our work, early customer engagements we still support. Real products with real users, running on Vercel and GCP.

Harness engineering is a methodology we've fully adopted internally and built nightshift to support this, and we needed to dogfood it. We didn't want to build in a vacuum or demo with toy workloads. We wanted to ground our features and work in actual delivery.

So we tasked an agent running inside a Nightshift chicklet to perform a full application migration from Vercel and GCP to our cluster on AWS EC2 (which we calculated would save us costs), find and fix a critical bug in our ETL pipeline, and ship a significant interactive chat feature that a customer was waiting for. Normally this would be a big ask of Claude Code and require us to hand tool most of the code across several toolchains.

Running inside Nightshift, Claude Code completed this at roughly 1/10th the time and cost of a traditional dev environment. We didn't write a single line of code or CLI command.

I'll digress for a moment: think about where we were before DevOps. Developers wrote code, threw it over the wall to ops. In hindsight, unifying development and operations feels obvious. Shift left is the obvious approach to security, operations, and the SDLC. But at the time, it wasn't obvious at all. The tooling had to catch up, and even after it existed, adoption took years. Harness engineering is the next step. The idea that your primary job as an engineer is to design the architecture, specs, testing chassis, and validation metrics, while the construction of code itself is automated. We think it will feel obvious in a few years. It doesn't yet.

## How we did it

The idea of having an AI-SRE is not novel. In fact we drew a lot of our motivation from the project [Kagent](https://kagent.dev). The difference in our strategy is we've purpose built Nightshift for primary use of Claude Code and Codex. We start every workflow with dropping a "control plane" Chicklet (the unit of compute inside of Nightshift) which we ssh'd into and used as our orchestrator. Inside that chicklet, we fired up Claude Code and since chicklets come preinstalled with a custom CLAUDE.md that teaches Claude how to interact with the cluster from the inside, Claude knew the APIs for managing other Chicklets.

This concept wasn't new for us. Prior to the Nightshift platform, we were standing up our own telemetry collectors, dev tools, and giving Claude access to them along with the cloud specific CLIs (eg. Kubectl, Vercel CLI, AWS CLI, GCP CLI).

![Harness engineering diagram](/blog/harness-engineering-diagram.png)

> Source: [OpenAI Harness Engineering Blog](https://openai.com/index/harness-engineering/)

Nightshift is just a purpose built package of this. Give the agent direct access to the infrastructure from inside the cluster and have per service telemetry. Essentially making services the Agent deploys in the cluster automatically instrumented with metrics, logs, and traces and make this data accessible to Claude from any other Chicklet.

With this method Claude created: three backend golang services supporting a NextJS frontend. It used Playwright and Chrome DevTools to debug and test the frontend. When something broke, it could see why immediately, from the same context as the running code.

We think this is the difference between an agent that writes code and an agent that can reliably (and systematically) ship software.

## Why This Matters: Harness Engineering at the Platform Level

OpenAI recently published their thinking on harness engineering, the discipline of designing environments, constraints, and feedback loops that make AI agents reliable at scale. The core takeaway is that the agent query loop is solved and the job of an engineer is to build the environment to make the agent effective.

Most harness engineering today happens at the repository level: AGENTS.md files, linters, structured docs, CI checks. That's necessary but insufficient for holistic change to an application. When your agent needs to migrate infrastructure, debug a live ETL pipeline, and test a frontend against real services, repository-level context isn't enough. The agent needs to be in the environment. The CLAUDE.md sitting inside of every Chicklet becomes a map of the cluster. This gave us a harness at the platform level:

- Context engineering happens through the Chicklet's direct access to running services and telemetry not just static docs
- Architectural constraints are enforced by the isolation boundary itself, the agent operates within a well-defined compute environment with dedicated resources
- Feedback loops are immediate. The agent can deploy a test environment, snapshot the database and services, test end to end, observe the result, and raise a PR on a snapshot of runtime state.

## What We Were Able to Ship

This wasn't a demo. The agent performed three distinct workstreams against a production application for a real customer engagement:

**Infrastructure migration.** Moved the application off Vercel and GCP onto the Nightshift cluster. The agent provisioned and configured the backend services, handled the networking, and verified the migration end-to-end.

**Critical bug fix.** During this process we discovered data inconsistencies which Claude fixed by recreating the bug in an isolated environment on a subset of data. It had access to the data flowing through the pipeline, could trace the issue through logs and telemetry, and raised a PR. All from within the same environment.

**Feature development.** We shipped an interactive chat feature, including frontend work on the NextJS app and the three supporting backend services. Used Playwright and Chrome DevTools for testing and debugging, iterating against the live application.

## It's so much more than a Sandbox

The industry is converging on a design pattern: agents need rich, constrained environments to do reliable work. Repository-level harnesses get you part of the way there. But for infrastructure work, migrations, debugging live systems, and shipping full-stack features, the harness needs to extend to the platform.

Nightshift is an open-source platform that gives agents isolated, durable, fully-equipped Linux environments where they can operate with the full context of a running system.

We're building Nightshift in the open because we believe this infrastructure should be community-driven. The workloads are changing fast. The platforms that run them need to keep up.

Check out the project at [github.com/nightshiftco/nightshift](https://github.com/nightshiftco/nightshift) and the docs at [docs.nightshift.sh](https://docs.nightshift.sh).

<CTA
  title="Check out the Nightshift project!"
  description="We're trying to standardize how AI Agents run on-prem or in the cloud."
  label="View on GitHub"
  href="https://github.com/nightshiftco/nightshift"
/>

Nightshift is open source under Apache 2.0. Built in Pittsburgh & Miami.

---

## Containers, But Without The Magic Part 1: Networking

> March 5, 2026 · Gianni Crivello · https://nightshift.sh/blog/containers-without-magic-01

If you spend enough time around container tooling, you’ll eventually hear phrases like:

“CNI plugin chains”

“overlay networking”

“service mesh sidecars”

Which all sound very impressive.

But underneath all of that, container networking is built on a handful of Linux primitives. 


We're on a journey to get Nightshift delegating sandbox runtime to [containerd](https://containerd.io/). This will give the project a ton of flexibility
and make it easy for users to swap in the type of sandbox they want to run (kata containers, firecracker, runc, etc.).

<CTA
  title="Check out the Nightshift project!"
  description="We're trying to standardize how AI Agents run on-prem or in the cloud."
  label="View on GitHub"
  href="https://github.com/nightshiftco/nightshift"
/>

Containerd's implementation will also be better than our bespoke runtime. We want to stand on the shoulders of giants and focus on 
what makes Nightshift a unique project.

So before we get started wiring containerd into Nightshift, we should understand the networking stack we have at our disposal.
This is when I came across [CNI (Container Network Interface)](https://github.com/containernetworking/cni) which streamlines the networking of containers and sits in between the Linux networking stack and 
the container runtime. 

"What does that even mean?" I asked myself on a sunny South Florida afternoon.

I don't like magic so I did what any curious engineer would do:

I built container networking from scratch using nothing but ip. 

This is my journey motivating myself for the reason CNI exists, and what it offers the ecosystem.


## Start with Nothing

First we create two network namespaces. A network namespace is basically a separate networking stack 
with its own interfaces, its own routing table, and its own ARP cache.

In other words, it behaves like a small machine.

> If you're a noob to networking like me, I highly suggest firing up a Linux machine and following along!


Let's create two namespaces.

```bash
sudo ip netns add ns1
sudo ip netns add ns2
```

Let’s look inside one of them.

```bash
sudo ip netns exec ns1 ip link
```

You’ll see something like:

```bash
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
```

`1:`: This is the interface index. Every network interface on a system gets a unique numberic ID inside that namespace.

`lo:`: This is the loopback interface. Loopback is a virtual interface that sends packets back to the same machine. This is your localhost!

`<LOOPBACK>`: This is an interface flag. There are several of these including `UP`, `BROADCAST`, `MULTICAST`, `LOOPBACK`, and `LOWER_UP`. Notice, this doesn't say `UP` yet.

`mtu 65536`: MTU is the maximum transmission unit. this is the maximum packet size the interface will send.

`qdisc`: This is the packet scheduler attached to the interface. Linux allows traffic shaping and queuing using qdiscs. `noop` means do nothing.

`state`: This tells us whether the interface is active. It's currently `DOWN`.

`mode`: This relates to special interface modes used by certain drivers. Currently in `DEFAULT`

`group`: Interfaces can be grouped for administrative purposes. Currently `default`

`qlen 1000`: This is the maximum number of packets queued for transmission. If packets are generated faster than they can be transmitted, they wait in this queue.

`link/loopback`: This tells us the Layer 2 (Ethernet) link type. Examples we'll see are `loopback`, `ether`, `veth`.

`00:00:00:00:00:00`: This is the MAC address. Loopback interfaces don’t use real MAC addresses, so Linux assigns all zeros. 

`brd 00:00:00:00:00:00`: The broadcast address is the address used to send packets to all devices on the network. Loopback doesn’t use broadcast, so it’s all zeros.

> I wasn't lying when I said we're spelling things out.

That’s the entire network stack for this namespace. Not exactly production ready.

Bring the loopback interface up:

sudo ip netns exec ns1 ip link set lo up
sudo ip netns exec ns2 ip link set lo up

So if we look again with `sudo ip netns exec ns1 ip link` we'll see:

```bash
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
```

Now we have two isolated “containers”. They just can’t talk to anything yet. Let's fix that.


## Give the Container a Network Cable

To connect a namespace to the host we use a veth pair.

Think of a veth pair as a virtual ethernet cable:

```
vethA <----> vethB
```

Packets entering one side immediately appear on the other.

Let’s create one.

```bash
sudo ip link add veth1 type veth peer name veth1-host
```

Right now both ends exist on the host.

We need to move one end into the namespace:

```bash
sudo ip link set veth1 netns ns1
```

Check the host side:

```bash
ip link show veth1-host
```

Check inside the namespace:

```bash
sudo ip netns exec ns1 ip link
```

You should now see an ethernet interface inside ns1. I'll leave repeating the same process for ns2 as an exercise for the reader.

At this point each container has a NIC, but they’re still not connected to anything useful.

## Add a Virtual Switch

To connect containers together we introduce a Linux bridge. A bridge behaves like an Ethernet switch: it forwards packets based on MAC addresses.


Create a bridge:

```bash
sudo ip link add br0 type bridge
sudo ip link set br0 up
```

Now attach the host ends of the veth pairs to the bridge.

```bash
sudo ip link set veth1-host master br0
sudo ip link set veth2-host master br0
```

bring the host ends up:

```bash
sudo ip link set veth1-host up
sudo ip link set veth2-host up
```

The topology now looks like this:

```bash
        br0
       /   \
 veth1-host veth2-host
     |          |
    ns1        ns2
```

In other words: a tiny virtual network. Now we need ip addresses so containers know how to talk to eachother or how to talk to the internet.

## Give the Containers IP Addresses

Right now the containers have interfaces but no addresses.

Let’s fix that.

```bash
sudo ip netns exec ns1 ip addr add 10.0.0.2/24 dev veth1
sudo ip netns exec ns1 ip link set veth1 up
```

```bash
sudo ip netns exec ns2 ip addr add 10.0.0.3/24 dev veth2
sudo ip netns exec ns2 ip link set veth2 up
```

Now try a ping from one container to another.

```bash
sudo ip netns exec ns1 ping 10.0.0.3
```

holy crap it works.

```
PING 10.0.0.3 (10.0.0.3) 56(84) bytes of data.
64 bytes from 10.0.0.3: icmp_seq=1 ttl=64 time=0.045 ms
64 bytes from 10.0.0.3: icmp_seq=2 ttl=64 time=0.034 ms
```

The bridge automatically learns where each MAC address lives. 

You can actually inspect this table:

```bash
bridge fdb show
```

Which is exactly how a physical switch behaves. Now that containers can talk to eachother, let's get them to talk to the internet.

## Add a Gateway

Eventually containers need to talk to the outside world.

For that we assign an IP to the bridge and treat it as a gateway.

```bash
sudo ip addr add 10.0.0.1/24 dev br0
```

Test from the container:

```bash
sudo ip netns exec ns1 ping 10.0.0.1
```

```bash
PING 10.0.0.1 (10.0.0.1) 56(84) bytes of data.
64 bytes from 10.0.0.1: icmp_seq=1 ttl=64 time=0.039 ms
64 bytes from 10.0.0.1: icmp_seq=2 ttl=64 time=0.030 ms
```

ok so we're talking to the gateway. Now we add a default route. So if the container is not sure where to send packets, it uses the gateway.

```bash
sudo ip netns exec ns1 ip route add default via 10.0.0.1
sudo ip netns exec ns2 ip route add default via 10.0.0.1
```

Check the routing table:

```bash
sudo ip netns exec ns1 ip route
```

You should see:

```bash
10.0.0.0/24 dev veth1
default via 10.0.0.1
```

Local traffic stays on the bridge. Everything else goes through the gateway. But, we're not done yet.

## NAT

no not like the bug, we're talking networks here guys.

Containers use private IP addresses, which means they can’t reach the internet directly.

The host fixes that with NAT (Network Address Translation).

First enable packet forwarding:

```bash
sudo sysctl -w net.ipv4.ip_forward=1
```

Then add a NAT rule:

```bash
sudo iptables -t nat -A POSTROUTING -s 10.0.0.0/24 ! -o br0 -j MASQUERADE
```

Now the host rewrites outgoing packets so they appear to originate from the host’s IP.

And suddenly the containers can reach the internet. Let's ping Google!

```bash
sudo ip netns exec ns1 ping 8.8.8.8
```

```bash
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=116 time=1.18 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=116 time=1.19 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=116 time=1.16 ms
```

## What We Just Built

Phew! Let’s step back for a moment.

Using only Linux tools we:

- created network namespaces
- created veth pairs
- built a bridge
- assigned IP addresses
- configured routing
- added NAT

That’s the entire foundation of container networking...mostly.

## Enter CNI

CNI (Container Network Interface) exists to automate the exact process we just walked through.

Instead of executing twenty shell commands, a container runtime runs a plugin with a JSON configuration.

The bridge plugin, for example, will:

- create a bridge
- create the veth pair
- move one side into the container namespace
- assign an IP address
- configure routes

Which should look very familiar to us now!

## Why This Matters for Nightshift

Nightshift runs isolated compute environments for executing user workloads.

As we started integrating with containerd, one detail became important very quickly:

containerd doesn’t actually implement networking.

Instead, it delegates networking to CNI.

Which means every container ultimately gets wired together using the same primitives we just explored:

network namespace --> veth pair --> bridge --> routing --> iptables

Once you understand that stack, container networking starts feeling a little less mysterious.

CNI isn’t magic.

It’s just a well-structured specification to automate a bunch of ip commands.

<CTA
  title="Check out the Nightshift project!"
  description="We're trying to standardize how AI Agents run on-prem or in the cloud."
  label="View on GitHub"
  href="https://github.com/nightshiftco/nightshift"
/>

---

## You Are The BIOS Now: Building A Hypervisor In Python With KVM

> February 27, 2026 · Gianni Crivello · https://nightshift.sh/blog/less-cursed-vm

> "A beginner-friendly rewrite of [the original Rust post](https://yeet.cx/blog/you-are-the-bios-now). Same cursed hypervisor, but in Python, explained from scratch."

<CTA
  title="This code is open source"
  description="All the code from this blog post is available on GitHub. Clone it, run it, break it, make it yours."
  label="View on GitHub"
  href="https://github.com/nightshiftco/LessCursedVM"
/>

It seems like the whole AI infrastructure community has been getting into sandboxes lately. It's for good reason! AI Agents are
fundamentally different workloads which have different requirements around security. 

Docker containers were ~roughly~ built for trusted workloads which required process isolation. Agents on the other hand should not be trusted, because Agents, for better or for worse,
can write code. It's that Agent generated code which should be treated as highly untrusted arbitrary code. Ladies and gentlemen, we're going to need a bigger boat.

The KVM (Kernel-based Virtual Machine) is a great fit for this. We can take advantage of hardware level isolation on CPUs such that we can 
build much better safety properties for running Agent code. Let alone multi-tenant Agent code! Oh My!

For us to truly appreciate what a project like [Firecracker](https://firecracker-microvm.github.io/) does, 
let's go a little low level, put our systems engineer hats on, and build a cursed hello world program.

## Follow the Rabbit

When your CPU boots, it wakes up thinking it's 1978. No operating system. No drivers. No Python. 
Just a confused chip in 16-bit "real mode" waiting for someone to tell it what to do.

That someone is usually the BIOS, a tiny program baked into your motherboard's firmware. 
But today, **you**, my friend, are the BIOS. We're going to write a program that creates a tiny virtual machine, 
boots a CPU inside it, and makes it print "Hello, World!", all from a Python script running on Linux.

This is called a **Type-II hypervisor**: a program that runs *inside* an existing operating system (like VirtualBox or QEMU) 
as opposed to a Type-I hypervisor that runs directly on bare metal (like VMware ESXi). 
Ours will be about 250 lines of Python.

If you've never touched virtualization, systems programming, or x86 architecture before...perfect. That's exactly who this is for.

## What We're Actually Building

Here's the plan in plain English:

1. Ask the Linux kernel for permission to create a virtual CPU.
2. Give that virtual CPU some memory to work with.
3. Convince the CPU it's running on a modern 64-bit machine (it wakes up thinking it's a 1978 Intel 8086).
4. Write a tiny guest program, using a Python mini-assembler that prints "Hello, World!" one character at a time.
5. Catch each character from Python and print it to your terminal.

No operating system runs inside our VM. No bootloader. Just machine code executed by a virtual CPU, with Python glue.

## Prerequisites

You need three things:

- **Linux** 
- **Python 3.8+** 
- **KVM access**; your CPU must support hardware virtualization

To check if KVM is available:

```bash
ls -la /dev/kvm
```

If that file exists, you're good. If not, you may need to enable virtualization in your BIOS/UEFI settings 
(usually called "VT-x" for Intel or "AMD-V" for AMD) and load the kernel module:

```bash
sudo modprobe kvm-intel   # Intel CPUs
sudo modprobe kvm-amd     # AMD CPUs
```

You may also need to add yourself to the `kvm` group:

```bash
sudo usermod -aG kvm $USER
```

Then log out and back in. No pip packages needed- we're using only the Python standard library.


## Part 1: What Is KVM?

KVM stands for **Kernel-based Virtual Machine**. It's a piece of the Linux kernel that exposes your CPU's hardware virtualization features (Intel VT-x or AMD-V) through a simple file: `/dev/kvm`.

You talk to KVM by opening this file and sending it **ioctl** commands. This is a Unix mechanism for sending control messages to device drivers. 
Think of it like an API, but instead of HTTP requests, you're sending binary-encoded structs through file descriptors.

The workflow looks like this:

```
Open /dev/kvm
    └── ioctl: "Create a VM" → gives you a VM file descriptor
            └── ioctl: "Create a VCPU" → gives you a VCPU file descriptor
                    └── ioctl: "Run" → CPU executes guest code
                            └── returns when something interesting happens (an "exit")
```

Each arrow is a separate `ioctl` call. KVM manages all the nasty hardware details, we just have to set up the initial state correctly.
Easy enough right? RIGHT? 


## Part 2: The ioctl Plumbing

Before we build the VM, we need to define the ioctl numbers. We can just compute them manually in Python.

Every Linux ioctl has a numeric code that encodes four things: the direction of data transfer (read, write, both, or none), 
a "type" byte (KVM uses `0xAE`), a command number, and the size of the data struct.

```python
# ioctl encoding helpers- these mirror the C macros _IO, _IOR, _IOW
# Don't worry if you don't understand these, just ask claude to explain it to you

def _IO(type_, nr):        return (type_ << 8) | nr
def _IOR(type_, nr, size): return (2 << 30) | (type_ << 8) | nr | (size << 16)
def _IOW(type_, nr, size): return (1 << 30) | (type_ << 8) | nr | (size << 16)

KVMIO = 0xAE  # KVM's ioctl "type" byte

KVM_CREATE_VM              = _IO(KVMIO, 0x01)
KVM_CREATE_VCPU            = _IO(KVMIO, 0x41)
KVM_GET_VCPU_MMAP_SIZE     = _IO(KVMIO, 0x04)
KVM_SET_USER_MEMORY_REGION = _IOW(KVMIO, 0x46, 32)
KVM_RUN                    = _IO(KVMIO, 0x80)
KVM_GET_SREGS              = _IOR(KVMIO, 0x83, 312)
KVM_SET_SREGS              = _IOW(KVMIO, 0x84, 312)
KVM_SET_REGS               = _IOW(KVMIO, 0x82, 144)
```

The numbers like `312` and `144` are the `sizeof` the kernel's C structs for special registers and general-purpose registers, respectively. We'll define matching Python structs shortly.

> **Why not just use a library?** Libraries obviously exist for this, but they hide all the interesting parts. 
We're here to learn how the sausage is made. We're system engineers!! Put that hat back on.

## Part 3: Describing Hardware With ctypes

KVM expects us to pass C structs back and forth through ioctl. Python's `ctypes` module lets us define binary-compatible structs that 
the kernel can read directly from memory.

An x86 CPU has **segment registers**, relics from the 1980s when memory was divided into overlapping 64KB "segments." 
Even in modern 64-bit mode, the CPU still checks them. Each segment register is described by this struct:

```python
import ctypes

class KvmSegment(ctypes.Structure):
    _fields_ = [
        ("base", ctypes.c_uint64),      # Base address of the segment
        ("limit", ctypes.c_uint32),     # Size limit
        ("selector", ctypes.c_uint16),  # Index into the GDT (more on this later)
        ("type_", ctypes.c_uint8),      # Segment type (code, data, etc.)
        ("present", ctypes.c_uint8),    # Is this segment valid?
        ("dpl", ctypes.c_uint8),        # Privilege level (0 = kernel, 3 = user)
        ("db", ctypes.c_uint8),         # Default operation size
        ("s", ctypes.c_uint8),          # Descriptor type
        ("l", ctypes.c_uint8),          # Long mode (64-bit)
        ("g", ctypes.c_uint8),          # Granularity
        ("avl", ctypes.c_uint8),        # Available for OS use
        ("unusable", ctypes.c_uint8),
        ("padding", ctypes.c_uint8),
    ]
```

The **descriptor table register** tells the CPU where to find the GDT (Global Descriptor Table) and IDT (Interrupt Descriptor Table) in memory:

```python
class KvmDtable(ctypes.Structure):
    _fields_ = [
        ("base", ctypes.c_uint64),
        ("limit", ctypes.c_uint16),
        ("padding", ctypes.c_uint16 * 3),
    ]
```

The **special registers** struct bundles all of these together, plus the control registers (`cr0` through `cr8`) and the `efer` register. These are the knobs we'll turn to activate 64-bit mode:

```python
class KvmSregs(ctypes.Structure):
    _fields_ = [
        ("cs", KvmSegment),   # Code segment
        ("ds", KvmSegment),   # Data segment
        ("es", KvmSegment),   # Extra segment
        ("fs", KvmSegment),   # General-purpose segment
        ("gs", KvmSegment),   # General-purpose segment
        ("ss", KvmSegment),   # Stack segment
        ("tr", KvmSegment),   # Task register
        ("ldt", KvmSegment),  # Local descriptor table
        ("gdt", KvmDtable),   # Global descriptor table
        ("idt", KvmDtable),   # Interrupt descriptor table
        ("cr0", ctypes.c_uint64),   # Control register 0
        ("cr2", ctypes.c_uint64),
        ("cr3", ctypes.c_uint64),   # Page table base
        ("cr4", ctypes.c_uint64),
        ("cr8", ctypes.c_uint64),
        ("efer", ctypes.c_uint64),  # Extended Feature Enable Register
        ("apic_base", ctypes.c_uint64),
        ("interrupt_bitmap", ctypes.c_uint64 * 4),
    ]
```

And finally, the **general-purpose registers**, the ones you'd recognize from any x86 assembly tutorial:

```python
class KvmRegs(ctypes.Structure):
    _fields_ = [
        ("rax", ctypes.c_uint64), ("rbx", ctypes.c_uint64),
        ("rcx", ctypes.c_uint64), ("rdx", ctypes.c_uint64),
        ("rsi", ctypes.c_uint64), ("rdi", ctypes.c_uint64),
        ("rsp", ctypes.c_uint64), ("rbp", ctypes.c_uint64),
        ("r8",  ctypes.c_uint64), ("r9",  ctypes.c_uint64),
        ("r10", ctypes.c_uint64), ("r11", ctypes.c_uint64),
        ("r12", ctypes.c_uint64), ("r13", ctypes.c_uint64),
        ("r14", ctypes.c_uint64), ("r15", ctypes.c_uint64),
        ("rip", ctypes.c_uint64),    # Instruction pointer
        ("rflags", ctypes.c_uint64), # CPU flags
    ]
```

> **Key insight:** These aren't abstractions. Each struct maps byte-for-byte to a kernel data structure. When we call `fcntl.ioctl(fd, KVM_SET_REGS, regs)`, Python hands the raw bytes of our `KvmRegs` object directly to the kernel.


## Part 4: Allocating Guest Memory

Our virtual CPU needs memory. But we can't just use a Python `bytearray`, KVM needs the *real virtual address* of a memory-mapped region 
(the pointer that the kernel's MMU knows about) so it can wire it into the guest's address space.

Python's built-in `mmap` module is great for file-backed mappings, but it doesn't give us the raw pointer address we need. 
So we call libc's `mmap` directly through ctypes:

```python
# Don't be afraid of this code. Look at it, ingest it, ask Claude about it, embrace it.
import ctypes.util

_libc = ctypes.CDLL(ctypes.util.find_library("c"), use_errno=True)
_libc.mmap.restype = ctypes.c_void_p
_libc.mmap.argtypes = [
    ctypes.c_void_p, ctypes.c_size_t, ctypes.c_int,
    ctypes.c_int, ctypes.c_int, ctypes.c_long,
]

def mmap_anon(size):
    """Allocate a page-aligned block of anonymous memory. Returns the address as an int."""
    addr = _libc.mmap(None, size, 0x3, 0x22, -1, 0)  # RW, PRIVATE|ANON
    if addr == ctypes.c_void_p(-1).value:
        raise OSError("mmap failed")
    return addr
```

What's happening here:

- `0x22` combines `MAP_PRIVATE` (0x02) and `MAP_ANONYMOUS` (0x20), "give me zeroed memory that's my own copy, don't back it with a file."
- `0x3` combines `PROT_READ` (0x1) and `PROT_WRITE` (0x2).
- The returned `addr` is an integer, a raw pointer, which is exactly what KVM's `kvm_userspace_memory_region` struct expects.

We then tell KVM: "Hey, this block of host memory? Pretend it's the guest's physical RAM starting at address `0x1000`."

```python
class KvmUserspaceMemoryRegion(ctypes.Structure):
    _fields_ = [
        ("slot", ctypes.c_uint32),
        ("flags", ctypes.c_uint32),
        ("guest_phys_addr", ctypes.c_uint64),
        ("memory_size", ctypes.c_uint64),
        ("userspace_addr", ctypes.c_uint64),
    ]
```

From the guest's perspective, it sees plain physical RAM. From our perspective, it's a chunk of `mmap`'d memory 
we can read and write with `ctypes.memmove`.


## Part 5: The CursedVm Class

> Ok so we're writing python and not the Rust that the original blog post was written in. So maybe instead of CursedVM, we'll be the
"LessCursedVM". Still cursed, but less.

Now we can put it all together. Here's the skeleton of our hypervisor:

```python
import fcntl
import mmap
import os
import struct

GUEST_PHYS_ADDR  = 0x1000       # Where guest "physical" memory starts
GUEST_SIZE       = 256 << 20    # 256 MiB of guest RAM
GDT_OFFSET       = 0x0          # GDT sits at the start of guest memory
PML4_OFFSET      = 0x1000       # Page tables start here
PAGE_TABLE_SIZE  = 0x1000       # 4 KiB per table
PAGE_SIZE        = 1 << 21      # 2 MiB large pages
CODE_OFFSET      = PML4_OFFSET + 3 * PAGE_TABLE_SIZE  # Code goes after page tables

class CursedVm:
    def __init__(self):
        # Open /dev/kvm
        self._kvm_fd = os.open("/dev/kvm", os.O_RDWR)

        # Create a VM (returns a new file descriptor)
        self._vm_fd = fcntl.ioctl(self._kvm_fd, KVM_CREATE_VM, 0)

        # Create a virtual CPU
        self._vcpu_fd = fcntl.ioctl(self._vm_fd, KVM_CREATE_VCPU, 0)

        # mmap the VCPU's "run" area. This is where KVM writes exit info
        run_size = fcntl.ioctl(self._kvm_fd, KVM_GET_VCPU_MMAP_SIZE, 0)
        self._run = mmap.mmap(
            self._vcpu_fd, run_size,
            mmap.MAP_SHARED, mmap.PROT_READ | mmap.PROT_WRITE,
        )

        # Allocate guest memory
        self._mem = mmap_anon(GUEST_SIZE)

        # Register it with KVM
        region = KvmUserspaceMemoryRegion(
            slot=0, flags=0,
            guest_phys_addr=GUEST_PHYS_ADDR,
            memory_size=GUEST_SIZE,
            userspace_addr=self._mem,
        )
        fcntl.ioctl(self._vm_fd, KVM_SET_USER_MEMORY_REGION, region)

        # Bootstrap the CPU into 64-bit mode
        self._setup_long_mode()
        self._setup_page_tables()
        self._setup_registers()
```

Each `ioctl` call is like sending a command to KVM through a walkie-talkie. First we get a VM, then a VCPU inside that VM, then we wire up memory, and finally configure the CPU state.

The `_run` mmap is particularly interesting, it's a shared memory region between our process and the kernel. 
When the VCPU exits (because the guest did something that needs our attention), 
KVM writes the reason and details into this buffer. We read it back with `struct.unpack_from`.

> This is an important data structure that allows our Python program to control the VM exits into the host


## Part 6: Escaping 1978 and Setting Up Long Mode

Here's where things get a little gnarly. When an x86 CPU powers on, it starts in **real mode**, a 16-bit environment 
from the Intel 8086 era. It can only address 1 MiB of memory, uses segmented addressing, and has no concept of 
virtual memory or protection rings as we know them today.

A real machine goes through several stages to reach 64-bit "long mode":

```
Real Mode (16-bit) → Protected Mode (32-bit) → Long Mode (64-bit)
```

Each transition requires flipping specific bits in control registers and setting up data structures that the CPU checks. 
Normally a BIOS and bootloader handle this, but our VM has neither. We have to do it ourselves. We need to be the change we want to see.

### The Global Descriptor Table (GDT)

The GDT is a table of "segment descriptors" that the CPU consults to understand memory layout. Even in 64-bit mode where segmentation is mostly ignored, the CPU still requires a valid GDT to be present. It's one of those "legacy tax" things in x86.

We need three entries:

```python
GDT_NULL   = 0x0000000000000000  # Entry 0: always null (CPU requires it)
GDT_CODE64 = 0x00209A0000000000  # Entry 1: 64-bit code segment
GDT_DATA64 = 0x0000920000000000  # Entry 2: 64-bit data segment
```

These 64-bit values are packed bitfields. The code segment says "this is executable, readable, present, runs in 64-bit mode, and operates at privilege level 0 (kernel)." The data segment says "this is writable, present, and used for data." The exact bit layout is documented in the Intel SDM (Software Developer's Manual). 

### Flipping The Switches

Setting up long mode requires configuring three groups of things:

**1. The segment registers** tell the CPU which GDT entries to use:

```python
def _setup_long_mode(self):
    sregs = KvmSregs()
    fcntl.ioctl(self._vcpu_fd, KVM_GET_SREGS, sregs)

    # Point the GDT register at our table in guest memory
    sregs.gdt.base = GUEST_PHYS_ADDR + GDT_OFFSET
    sregs.gdt.limit = 23  # 3 entries × 8 bytes − 1

    # Write the GDT entries into guest memory
    self._poke(GDT_OFFSET + 0,  "<Q", 0x0000000000000000)  # null
    self._poke(GDT_OFFSET + 8,  "<Q", 0x00209A0000000000)  # 64-bit code
    self._poke(GDT_OFFSET + 16, "<Q", 0x0000920000000000)  # 64-bit data
```

**2. The code segment register (CS)** configured for 64-bit execution:

```python
    # Code segment: 64-bit long mode, ring 0
    for attr, val in [("base",0), ("limit",0xFFFFFFFF), ("selector",8),
                      ("present",1), ("type_",11), ("dpl",0),
                      ("db",0), ("s",1), ("l",1), ("g",1)]:
        setattr(sregs.cs, attr, val)
```

The `("l", 1)` line is the magic bit. It tells the CPU "this code segment runs in 64-bit long mode." Without it, the CPU stays in compatibility mode. The `("selector", 8)` points at GDT entry 1 (each entry is 8 bytes, so entry 1 is at byte offset 8).

**3. The control registers** are the actual mode switches:

```python
    sregs.efer |= 0x500       # Set LME (Long Mode Enable) and LMA (Long Mode Active)
    sregs.cr0  |= 0x80000001  # Enable paging (PG) and protection (PE)
    sregs.cr4  |= 0x20        # Enable Physical Address Extension (PAE)

    fcntl.ioctl(self._vcpu_fd, KVM_SET_SREGS, sregs)
```

Here's what each bit does:

| Register | Bit(s) | Name | What It Does |
|----------|--------|------|-------------|
| `EFER` | 8 (0x100) | LME | "I want to use long mode" |
| `EFER` | 10 (0x400) | LMA | "Long mode is now active" |
| `CR0` | 0 (0x1) | PE | Enable protected mode (exit real mode) |
| `CR0` | 31 (0x80000000) | PG | Enable paging (virtual memory) |
| `CR4` | 5 (0x20) | PAE | Enable 36-bit+ physical addressing |

Setting `0x500` in EFER sets both bits 8 and 10 (`0x100 | 0x400`). These three registers form a chain: 
PAE must be on for paging to work, paging must be on for long mode to activate, and long mode must be enabled for 64-bit 
code to run.

### Why So Much Ceremony? I hate it

You might wonder why we can't just set a single "64-bit mode" flag. Wouldn't that be nice? The answer is backward compatibility. 
x86 has been accumulating features for over 45 years, and each new mode was bolted on top of the previous one. 
Long mode was added by AMD in 2003, but it still requires the infrastructure from protected mode (1985) 
and PAE (1995) to be in place. Every modern x86 CPU still wakes up as an 8086 and goes through this same ritual, 
we're just doing it explicitly instead of letting firmware handle it.


## Part 7: Page Tables: Mapping Virtual Memory

With paging enabled (that `CR0.PG` bit we just set), the CPU no longer accesses physical memory directly. Every memory access goes through a **page table**. A page table is a tree structure that translates 
virtual addresses to physical addresses. We now need to have a page table so that our guest CPU does not crash. We're going to be lazy and create a simple one. 

In 64-bit mode with 2 MiB large pages, the translation works like this:

```
Virtual Address (64 bits):
┌──────────┬──────────┬──────────┬─────────────────────────────┐
│ 9 bits   │ 9 bits   │ 9 bits   │ 21 bits                     │
│  PML4    │  PDPT    │  PD      │  Offset within 2 MiB page   │
└──────────┴──────────┴──────────┴─────────────────────────────┘
     │           │          │
     │           │          └─→ Page Directory entry → 2 MiB physical page
     │           └─────────────→ PDPT entry → Page Directory
     └─────────────────────────→ PML4 entry → PDPT
```

The CPU walks this tree on every memory access (with heavy caching via the TLB). We need to build this tree in guest memory.

We'll use an **identity mapping**, virtual address X maps to physical address X. 
This is the simplest possible setup, and it means our guest code doesn't have to worry about address translation at all.

```python
def _setup_page_tables(self):
    # Zero out the page table area (3 tables × 4096 bytes each)
    self._zero(PML4_OFFSET, 3 * PAGE_TABLE_SIZE)

    pml4 = GUEST_PHYS_ADDR + PML4_OFFSET
    pdpt = pml4 + PAGE_TABLE_SIZE
    pd   = pdpt + PAGE_TABLE_SIZE

    # PML4[0] → PDPT (present + writable)
    self._poke(PML4_OFFSET, "<Q", pdpt | 0x3)

    # PDPT[0] → Page Directory (present + writable)
    self._poke(PML4_OFFSET + PAGE_TABLE_SIZE, "<Q", pd | 0x3)

    # PD[0..512] → 512 identity-mapped 2 MiB pages (present + writable + large)
    for i in range(512):
        self._poke(PML4_OFFSET + 2 * PAGE_TABLE_SIZE + i * 8, "<Q", (i << 21) | 0x83)

    # Tell the CPU where the PML4 lives
    sregs = KvmSregs()
    fcntl.ioctl(self._vcpu_fd, KVM_GET_SREGS, sregs)
    sregs.cr3 = pml4
    fcntl.ioctl(self._vcpu_fd, KVM_SET_SREGS, sregs)
```

The `0x83` in each page directory entry encodes three flags: **present** (bit 0), **writable** (bit 1), and **large page** (bit 7). The large page bit tells the CPU "don't look for another level of tables, this entry directly maps 2 MiB of memory." `0x83 = 0b10000011`.

The `(i << 21)` part sets the physical address: entry 0 maps to physical address 0, entry 1 maps to `0x200000` (2 MiB), entry 2 to `0x400000` (4 MiB), and so on up to 1 GiB.

The helper methods for writing to guest memory look like this:

```python
def _poke(self, offset, fmt, value):
    """Write a packed value into guest memory at the given offset."""
    ctypes.memmove(self._mem + offset, struct.pack(fmt, value), struct.calcsize(fmt))

def _zero(self, offset, n):
    """Zero out a region of guest memory."""
    ctypes.memset(self._mem + offset, 0, n)
```

> If you're still here at this point...we're almost there and it's going to be worth it

## Part 8: Setting Up The Registers

The last setup step is initializing the general-purpose registers. The three critical ones are:

- **RIP** (Instruction Pointer): where the CPU starts executing code
- **RSP** (Stack Pointer): where the stack starts (grows downward)
- **RFLAGS**: CPU flags (bit 1 must always be set per the x86 spec)

```python
def _setup_registers(self):
    regs = KvmRegs()
    regs.rip = GUEST_PHYS_ADDR + CODE_OFFSET     # Start executing here
    regs.rsp = (GUEST_PHYS_ADDR + GUEST_SIZE) & ~(PAGE_SIZE - 1)  # Stack at top
    regs.rbp = regs.rsp                           # Frame pointer = stack pointer
    regs.rflags = 0x2                             # Reserved bit (must be 1)
    fcntl.ioctl(self._vcpu_fd, KVM_SET_REGS, regs)
```

The `& ~(PAGE_SIZE - 1)` is a bitmask trick that rounds down to the nearest 2 MiB page boundary. 
This keeps the stack properly aligned.

Here's the resulting memory layout inside the guest:

```
  0x1000  ┌─────────────────────────┐
          │ GDT (24 bytes)          │
  0x2000  ├─────────────────────────┤
          │ PML4 (4 KiB)            │
  0x3000  ├─────────────────────────┤
          │ PDPT (4 KiB)            │
  0x4000  ├─────────────────────────┤
          │ Page Directory (4 KiB)  │
  0x5000  ├─────────────────────────┤  ← RIP starts here
          │ Guest Code              │
          │                         │
          │    (free space)         │
          │                         │
          ├─────────────────────────┤  ← RSP (stack grows down)
          │ Stack                   │
          └─────────────────────────┘
```


## Part 9: The Run Loop and Catching I/O Exits

Our guest has no operating system, no `syscalls`, no stdout. The only way it can communicate with the outside world is through **I/O port instructions** (`out` in x86 assembly).

I/O ports are an ancient x86 mechanism originally designed for talking to hardware like keyboards and disk controllers. 
The CPU has a separate 64K address space just for I/O, and the `out` instruction writes a byte to a numbered port.

When our guest executes `out`, the CPU can't actually talk to hardware (it's virtualized), 
so KVM **traps** the instruction and returns control to our Python code. 
This is called a **VM exit**. We can then read which port was written to and what data was sent.

```python
def run(self, on_io):
    """Run the guest. Calls on_io(port, data) for each I/O-out exit."""
    while True:
        fcntl.ioctl(self._vcpu_fd, KVM_RUN, 0)
        reason = struct.unpack_from("<I", self._run, 8)[0]

        if reason == 2:  # KVM_EXIT_IO
            if self._run[32] == 1:  # direction == IO_OUT
                size  = self._run[33]
                port  = struct.unpack_from("<H", self._run, 34)[0]
                count = struct.unpack_from("<I", self._run, 36)[0]
                off   = struct.unpack_from("<Q", self._run, 40)[0]
                on_io(port, bytes(self._run[off : off + size * count]))

        elif reason == 5:  # KVM_EXIT_HLT
            return

        else:
            raise RuntimeError(f"VM exit: {reason}")
```

The magic numbers (8, 32, 33, 34, 36, 40) are byte offsets into the `kvm_run` struct. This is the least Pythonic part of the whole project, 
we're essentially doing manual struct parsing because the kernel's `kvm_run` struct is very large and complex, 
and we only need a few fields from it.

Here's what each offset contains:

| Offset | Size | Field | Meaning |
|--------|------|-------|---------|
| 8 | 4 bytes | `exit_reason` | Why the VCPU stopped |
| 32 | 1 byte | `io.direction` | 0 = in, 1 = out |
| 33 | 1 byte | `io.size` | Bytes per I/O operation |
| 34 | 2 bytes | `io.port` | Which I/O port |
| 36 | 4 bytes | `io.count` | Number of operations |
| 40 | 8 bytes | `io.data_offset` | Where in kvm_run the data lives |

The flow for each character looks like this:

```
Guest: out 'H' to port 0x10
  │
  ▼
KVM: *VM exit* (I/O out)
  │
  ▼
Python: reads kvm_run → port=0x10, data=b'H'
  │
  ▼
on_io(0x10, b'H') → print('H')
  │
  ▼
Python: ioctl(KVM_RUN) → resumes guest
  │
  ▼
Guest: out 'e' to port 0x10
  ... and so on
```


## Part 10: The Mini Assembler 

In the original Rust blog post, the guest program was defined as a wall of raw hex bytes:

```python
# 😱 
bytes([0x48, 0x31, 0xC0, 0x50, 0x48, 0xB8, 0x6F, 0x72, ...])
```

This is truly cursed....

Instead, we'll build a tiny `Asm` class that lets you write guest programs for those of us that don't know x86 machine code. 
Each method emits the correct bytes behind the scenes:

```python
class Asm:
    """A tiny x86-64 assembler. Turns method calls into machine code."""

    def __init__(self):
        self._buf = bytearray()
        self._labels = {}

    def _emit(self, *bs):
        self._buf.extend(bs)

    def code(self) -> bytes:
        return bytes(self._buf)
```

### Low-Level Instructions

Each CPU instruction becomes a method that emits the right bytes:

```python
    def mov_rax(self, imm64):
        """rax = <64-bit value>"""
        self._emit(0x48, 0xB8, *imm64.to_bytes(8, "little"))

    def mov_rsi_rsp(self):
        """rsi = rsp"""
        self._emit(0x48, 0x89, 0xE6)

    def load_byte_from_rsi(self):
        """al = byte pointed to by rsi"""
        self._emit(0x8A, 0x06)

    def inc_rsi(self):
        """rsi += 1"""
        self._emit(0x48, 0xFF, 0xC6)

    def push_rax(self):
        self._emit(0x50)

    def out_al(self, port):
        """Send the byte in AL to an I/O port. Triggers a VM exit."""
        self._emit(0xE6, port & 0xFF)

    def hlt(self):
        """Halt the CPU."""
        self._emit(0xF4)
```

Labels and jumps work by recording byte positions and patching relative offsets:

```python
    def label(self, name):
        """Mark current position with a name for jumps."""
        self._labels[name] = len(self._buf)

    def test_al_zero(self):
        """Set CPU flags: is AL == 0?"""
        self._emit(0x84, 0xC0)

    def jump_if_zero(self, target):
        """Jump to label if AL was zero."""
        pos = len(self._buf)
        self._emit(0x74, 0x00)
        self._buf[pos + 1] = (self._labels.get(target, pos + 2) - (pos + 2)) & 0xFF

    def jump(self, target):
        """Unconditional jump to label."""
        pos = len(self._buf)
        self._emit(0xEB, 0x00)
        self._buf[pos + 1] = (self._labels[target] - (pos + 2)) & 0xFF
```

### High-Level Helpers

The real magic is in two convenience methods that compose the low-level instructions for you:

```python
    def push_string(self, s):
        """
        Push a string onto the stack. After this, RSP points at the
        first character. Null-terminated and padded automatically.
        """
        raw = s.encode("ascii") + b"\x00"
        while len(raw) % 8 != 0:
            raw += b"\x00"

        # Push in reverse (stack grows down)
        for i in range(len(raw) - 8, -1, -8):
            chunk = int.from_bytes(raw[i:i+8], "little")
            self.mov_rax(chunk)
            self.push_rax()

    def print_loop(self, port=0x10):
        """
        Emit a loop that reads bytes from RSI one at a time,
        sends each to an I/O port, and repeats forever.
        """
        self.label("loop")
        self.load_byte_from_rsi()
        self.test_al_zero()
        # ... jump logic, out, inc, reset ...
```

`push_string` handles all the tricky parts automatically: ASCII encoding, null termination, padding to 8-byte alignment, splitting into 64-bit chunks, and pushing them in reverse order so the string reads forward in memory. 

`print_loop` emits the entire read→test→out→increment→jump cycle. It also has a sibling, `print_once`, 
that prints the string once and then halts the CPU instead of looping.


## Part 11: Writing The Guest Program

With the `Asm` class, writing the guest program is now three lines:

```python
asm = Asm()
asm.push_string("Hello, World!\n")   # put the string on the stack
asm.mov_rsi_rsp()                    # point RSI at the first character
asm.print_loop(port=0x10)            # loop: send each char to port 0x10
```

That's it. We now have our guest program.

and if you're curious you can peek at the raw bytes:

```python
asm = Asm()
asm.push_string("Hi!\n")
asm.mov_rsi_rsp()
asm.print_loop(port=0x10)
print(asm.code().hex(" "))
# 48 b8 48 69 21 0a 00 00 00 00 50 48 89 e6 8a 06 84 c0 74 07 e6 10 48 ff c6 eb f3 48 89 e6 eb ee
```


## Part 12: Putting It All Together

> WE MADE IT. 

Here's the complete main script:

```python
import signal
import sys
import threading

from cursed_vm import Asm, CursedVm

def main():
    # Build the guest program
    asm = Asm()
    asm.push_string("Hello, World!\n")
    asm.mov_rsi_rsp()
    asm.print_loop(port=0x10)

    # Handle Ctrl+C
    stop = threading.Event()
    signal.signal(signal.SIGINT, lambda *_: stop.set())

    # Run the VM on a background thread
    def vm_thread():
        with CursedVm() as vm:
            vm.load(asm.code())
            vm.run(lambda port, data: (
                sys.stdout.write(chr(data[0])) or sys.stdout.flush()
                if port == 0x10 and len(data) == 1
                else None
            ))

    t = threading.Thread(target=vm_thread, daemon=True)
    t.start()
    stop.wait()
    print("\nBye.")

if __name__ == "__main__":
    main()
```

We run the VM on a background thread because the run loop blocks (it only returns when the guest halts or crashes). The main thread waits for Ctrl+C, which sets the `stop` event and lets us shut down cleanly.

The `with CursedVm() as vm:` context manager ensures we clean up file descriptors and `munmap` the guest memory when we're done. 

Run it:

```bash
python main.py
```

And you should see an infinite loop of `Hello, World`'s just like Turing intended:

```
Hello, World!
Hello, World!
Hello, World!
Hello, World!
^C
Bye.
```

Each "Hello, World!" is being printed one character at a time by a virtual CPU executing machine code inside a KVM virtual machine, with each character triggering a VM exit that Python catches and writes to your terminal.
How freaking cool is that?!


## What Just Happened?

Let's zoom out. Here's the full chain of events:

1. **Python opens `/dev/kvm`** and asks the kernel for a virtual machine.
2. **Python allocates 256 MiB of memory** with `mmap` and tells KVM "pretend this is the guest's physical RAM."
3. **Python writes a GDT, page tables, and segment register config** into that memory and pokes the VCPU's registers to activate 64-bit mode.
4. **The `Asm` class generates machine code** from readable method calls and Python copies it into the guest's code region.
5. **Python calls `ioctl(KVM_RUN)`**, and the physical CPU enters a special hardware mode (VMX root/non-root on Intel) where it executes the guest code at native speed.
6. **The guest pushes "Hello, World!\n" onto its stack** and enters a loop that sends each character to I/O port `0x10` via the `out` instruction.
7. **Each `out` causes a VM exit**. The CPU switches back to the host, KVM writes the exit details into the shared `kvm_run` buffer, and control returns to Python.
8. **Python reads the port and data**, prints the character, and calls `KVM_RUN` again.
9. **This repeats**, roughly 15 VM exits per "Hello, World!" (one per character including the newline), until you hit Ctrl+C.

The guest code runs at essentially native CPU speed. The only overhead is the VM exits for I/O, which take a few microseconds each.
For our 15-character string printed in a tight loop, we're looking at tens of thousands of "Hello, World!" per second.


## Is our VM good enough for super intelligence? 

Obviously this isn't a production grade system, but there are great projects out there that are. 
Nightshift uses [Firecracker](https://firecracker-microvm.github.io/) under the hood to manage our virtualization stack. 

Nightshift then wraps this virtualization with an easy to use and deploy SDK, purpose built for Agents.
We give AI agents their own Firecracker microVMs so they can write and execute code in a fully isolated sandbox, 
with the same hardware-backed security guarantees you just saw, 
but without any of the ceremony. 

<CTA />

---

## Evaluating Large Language Models on Agentic Workspace Bootstrapping

> February 3, 2026 · Ethan Henley, Gianni Crivello · https://nightshift.sh/blog/evaluating-large-language-models-on-agentic-workspace-bootstrapping

## TLDR;

We want Agents to configure and manage their own environments through Agent Routines.
 One Agent Routine that is particularly helpful is the Boot Agent Routine. Given a BOOT.md file, 
 we’ll let an Agent configure its packages, write initial library code, create AGENTS.md, and SKILLS.md 
 that will be used in downstream tasks. We want the agent to do this well and align with the user’s use case.

We present an evaluation of three frontier LLMs on this task. 

We use both a deterministic and LLM-as-judge methodology with calibrated rubrics across five quality dimensions, 
to evaluate  Claude Opus 4.5, GPT-5.2 Codex, and Kimi K2.5 across 30 independent trials.

Our results indicate that while models achieve comparable aggregate performance (let’s go open source Kimi!), 
significant differences emerge in specific capabilities: Claude Opus 4.5 demonstrates superior performance on 
holistic project documentation (+10.6 points on AGENTS.md), while Kimi K2.5 shows advantages in structured knowledge 
representation (the quality of the SKILLS.md files).

All models exhibited a consistent weakness in generating safety-critical guidelines.  

## Why do this?

The Nightshift project was started to help us ship coding agents to our customers. Namely, business users. 
People that aren’t technical and will not even be vibe coding. They simply want an “engineer” or “IT guy” in their computer 
that can help them automate work tasks. That means we need to set the Agent up for 
success. We want to support the use case of an Agent (through Agent Routines) to manage its workspace through the duration 
of its life. That means, writing library code, updating SKILLS.md based on session history, updating it’s AGENTS.md, and 
scheduling recurring work.

When you run `nightshift install –prefix ~/.agent` the cli will take you through a “boot” sequence. This uses a new file 
we’re proposing which is the `BOOT.md` that will let the agent know what the user wants to accomplish, and it’s up to the 
agent to configure all of the above.

You might wonder, “can’t we just use something like skills.sh?”, and the answer is we certainly could but then we would 
sacrifice flexibility for our users. Requiring them to manually install the SKILLS for the agent based off of their use case. 
Instead, we have found that it’s a much better experience to declaratively tell Nightshift what they want the agent to be used for and the agent 
will bootstrap itself. Also, most SKILLS or AGENTS files are vibe coded anyway.

Let’s formalize this a bit.

---

## Introduction

The emergence of fully declarative agentic AI systems has created new requirements for workspace initialization and configuration.

Unlike traditional software scaffolding, agentic workspaces require rich semantic documentation that enables downstream AI systems to 
understand project context, adhere to safety boundaries, and invoke appropriate capabilities.

Our interpretation of the current "boot agent" paradigm addresses this challenge by tasking an LLM with generating the foundational 
layer for a new workspace. This includes:

1. **Library Code**: Python code to support the use-case
2. **Python Packages**: Python packages to support the use case
3. **Skill Files**: Domain-specific knowledge encapsulated in structured markdown
4. **Agent Configuration**: Project-level instructions including commands, safety boundaries, and capability routing

We have found that the quality of these artifacts directly impacts the effectiveness and safety of subsequent agentic operations within the workspace.

### Questions We Wanted to Answer

**1:** How do frontier LLMs (closed and open source) compare on agentic workspace bootstrapping tasks?

**2:** What quality dimensions differentiate model performance, and where do systematic weaknesses exist?

**3:** What is the variance in output quality across repeated trials, and how does this affect deployment reliability?

Before we get started, you can run this eval yourself! Check out https://docs.nightshift.sh/eval for more info.

---

### Task Definition

The boot agent task presents the model with a `BOOT.md` file containing:

- User intent statement (natural language description of project purpose)
- Suggested skill domains (semantic categories for knowledge encapsulation)
- Technical context (target platform, data sources, authentication methods)

The model must autonomously:
1. Generate domain-specific `SKILL.md` files
2. Generate one project-level `AGENTS.md` configuration file
3. Install appropriate dependencies (evaluated but not scored)

### Test Configuration

For this test we chose an environment target we have worked extensively with in the past: procurement/purchasing analysis
with the Odoo ERP (https://www.odoo.com/). 

We worked with procurement experts to benchmark different Agent configurations until we received expert approval on a series of tasks.

We thus had a clear understanding of the skills and direction that an agent would need in this environment to succeed. 
For skills, the agent must have a deep understanding of analytics, purchasing logic, Odoo schema context, and common formulas. 
The AGENTS.md file must be clear in how it defines skill locations to make search seamless, and write proper commands to navigate
the agent's environment and tool set. 

We define the following parameters in setting up our agent environment in the test BOOT.md file.

| Parameter | Value |
|-----------|-------|
| User Intent | "Supply chain analytics for Odoo ERP data stored as parquet files on Google Cloud Storage" |
| Skill Domains | Schema reference, purchasing analysis, query patterns, calculation formulas |
| Target Environment | Python 3.13, DuckDB, GCS with HMAC authentication |
| Files per Trial | 5 (4 skills + 1 AGENTS.md) |
| Trials per Model | 10 |
| Total Evaluations | 150 |

We use the following frontier (as of writing) models for this analysis:

| Model | Provider | Context Window |
|-------|----------|-----------------|
| Claude Opus 4.5 | Anthropic | 200K |
| GPT-5.2 Codex | OpenAI | 128K |
| Kimi K2.5 | Moonshot AI | 128K |

All models were accessed via API with each trial using an isolated workspace with fresh state.

## Evaluation Framework

This is a two phase evaluation with the first being deterministic and the second using an LLM-as-a-Judge. 
The first check ensures that the files exist and are in the right places. For example, all skills need to be
written to `.opencode/skills/[skill-name]/SKILL.md, and AGENTS.md must be located in the workspace root.

We employed GPT-4o as the evaluation judge, selected for its demonstrated calibration on structured assessment 
tasks. The judge received the generated content along with the original user intent and a detailed rubric.

Each file type was evaluated on five dimensions (0-20 points each, total 0-100). See [Appendix C](#appendix-c-skill-file-grading-rubric) and [Appendix D](#appendix-d-agentsmd-grading-rubric) for full scoring criteria.

**Table 1: Skill File Dimensions**

| Code | Dimension | Description |
|------|-----------|-------------|
| D1 | Knowledge Delta | Expert knowledge not derivable from first principles |
| D2 | Specificity & Actionability | Concrete, executable instructions |
| D3 | Anti-Patterns & Safety | Explicit NEVER/ALWAYS/ASK-FIRST rules |
| D4 | Structure & Discoverability | Clear triggers and progressive disclosure |
| D5 | Tailoring to Intent | Customization to user's purpose and stack |

**Table 2: AGENTS.md Dimensions**

| Code | Dimension | Description |
|------|-----------|-------------|
| D1 | Project Specificity | Grounded in actual workspace paths and packages |
| D2 | Command Accuracy | Exact, copy-pasteable commands |
| D3 | Safety Boundaries | Three-tier permission structure |
| D4 | Code Style Concreteness | Style shown through code examples |
| D5 | Skill Catalog & Routing | Complete skill listing with paths and triggers |

The judge was instructed to be "harsh" and reserve scores above 15 for exceptional work, reducing ceiling effects common in LLM evaluation. 

---

## Results

### Aggregate Performance

**Table 3: Overall Model Performance (All Files)**

| Model | μ | Median | σ | Min | Max | n |
|-------|---|--------|---|-----|-----|---|
| Claude Opus 4.5 | 69.62 | 71.5 | 8.74 | 47 | 83 | 50 |
| Kimi K2.5 | 69.52 | 68.5 | 7.68 | 54 | 84 | 50 |
| GPT-5.2 Codex | 67.70 | 66.0 | 5.31 | 59 | 82 | 50 |

The three models achieve comparable aggregate performance, with Claude Opus 4.5 and Kimi K2.5 effectively tied (Δ=0.1) and GPT-5.2 Codex trailing by approximately 2 points.

### Performance by File Type

Disaggregating by file type reveals meaningful differences obscured by aggregate scores.

**Table 4: Skill File Performance (n=40 per model)**

| Model | μ | Median | σ |
|-------|---|--------|---|
| Kimi K2.5 | 67.90 | 68.0 | 7.19 |
| GPT-5.2 Codex | 67.53 | 66.0 | 5.43 |
| Claude Opus 4.5 | 67.28 | 67.0 | 8.03 |

**Table 5: AGENTS.md Performance (n=10 per model)**

| Model | μ | Median | σ |
|-------|---|--------|---|
| Claude Opus 4.5 | **79.00** | 80.5 | 3.83 |
| Kimi K2.5 | 76.00 | 77.0 | 6.18 |
| GPT-5.2 Codex | 68.40 | 68.5 | 5.02 |

**Finding 1:** Claude Opus 4.5's overall advantage derives primarily from AGENTS.md generation, where it outperforms GPT-5.2 Codex by 10.6 points, a substantial effect size. On skill files alone, all models perform within 0.6 points of each other.

###  Performance by Skill Domain

**Table 6: Mean Score by Skill Domain**

| Skill | Best Model | Score | Second | Score | Third | Score |
|-------|------------|-------|--------|-------|-------|-------|
| odoo-schema | GPT-5.2 Codex | 64.7 | Kimi K2.5 | 60.1 | Claude Opus 4.5 | 57.3 |
| purchasing | Claude Opus 4.5 | 75.0 | Kimi K2.5 | 74.0 | GPT-5.2 Codex | 73.0 |
| analytics | Kimi K2.5 | 71.9 | Claude Opus 4.5 | 69.7 | GPT-5.2 Codex | 66.7 |
| formulas | Claude Opus 4.5 | 67.1 | GPT-5.2 Codex | 65.7 | Kimi K2.5 | 65.6 |

**Finding 2:** Model advantages are domain-dependent. GPT-5.2 Codex leads on schema documentation (+7.4 vs Claude), while Claude Opus 4.5 excels on domain expertise skills (purchasing, formulas). This suggests different model strengths in technical specification versus domain reasoning.

### Dimension Analysis

**Table 7: Mean Dimension Scores for Skill Files (0-20 scale)**

| Dimension | Claude Opus 4.5 | GPT-5.2 Codex | Kimi K2.5 | Overall μ |
|-----------|-----------------|---------------|-----------|-----------|
| D1: Knowledge Delta | 13.7 | 13.7 | **13.8** | 13.7 |
| D2: Specificity | 15.6 | **15.7** | 15.1 | 15.5 |
| D3: Anti-Patterns | **10.4** | 9.9 | 10.0 | 10.1 |
| D4: Structure | 14.2 | 14.9 | **15.2** | 14.8 |
| D5: Tailoring | 13.3 | 13.2 | **13.9** | 13.5 |

**Table 8: Mean Dimension Scores for AGENTS.md (0-20 scale)**

| Dimension | Claude Opus 4.5 | GPT-5.2 Codex | Kimi K2.5 | Overall μ |
|-----------|-----------------|---------------|-----------|-----------|
| D1: Project Specificity | 16.9 | 15.1 | **17.5** | 16.5 |
| D2: Command Accuracy | 14.9 | **15.0** | 14.5 | 14.8 |
| D3: Safety Boundaries | **15.4** | 13.7 | 15.1 | 14.7 |
| D4: Code Style | **15.8** | 13.8 | 15.5 | 15.0 |
| D5: Skill Catalog | **16.0** | 10.8 | 13.4 | 13.4 |

**Finding 3:** D3 (Anti-Patterns & Safety) is the weakest dimension across all models for skill files (μ=10.1), falling into the "adequate" tier. This represents a systematic capability gap.

**Finding 4:** The largest inter-model variance occurs on AGENTS.md D5 (Skill Catalog & Routing), where Claude Opus 4.5 (16.0) outperforms GPT-5.2 Codex (10.8) by 5.2 points. GPT-5.2 Codex frequently fails to generate complete skill catalogs with routing descriptions.

### Consistency Analysis

**Table 9: Output Consistency (Standard Deviation)**

| Model | σ (Overall) | σ (Skills) | σ (AGENTS.md) |
|-------|-------------|------------|---------------|
| GPT-5.2 Codex | **5.31** | **5.43** | 5.02 |
| Kimi K2.5 | 7.68 | 7.19 | 6.18 |
| Claude Opus 4.5 | 8.74 | 8.03 | **3.83** |

**Finding 5:** GPT-5.2 Codex produces the most consistent outputs overall (σ=5.31), while Claude Opus 4.5 shows the highest variance on skills but lowest on AGENTS.md (σ=3.83). This suggests Claude has internalized a stable template for project documentation while exhibiting more variation on domain-specific content.

### Score Distribution Analysis

**Table 10: Score Distribution by Model**

| Score Range | Claude Opus 4.5 | GPT-5.2 Codex | Kimi K2.5 |
|-------------|-----------------|---------------|-----------|
| ≤50 | 2 (4%) | 0 (0%) | 0 (0%) |
| 51-60 | 5 (10%) | 2 (4%) | 6 (12%) |
| 61-70 | 17 (34%) | 32 (64%) | 22 (44%) |
| 71-80 | 21 (42%) | 15 (30%) | 18 (36%) |
| 81-90 | 5 (10%) | 1 (2%) | 4 (8%) |

**Finding 6:** GPT-5.2 Codex clusters tightly in the 61-70 range (64% of outputs), while Claude Opus 4.5 shows a bimodal tendency with more outputs in both tails (4% below 50, 10% above 80). For risk-averse deployments, GPT-5.2 Codex offers more predictable quality floors.

---

## Conclusion

### Interpretation of Results

Our findings suggest that current frontier models achieve comparable aggregate performance on agentic bootstrapping tasks, but with distinct capability profiles:

**Claude Opus 4.5** excels at holistic project documentation, consistently generating comprehensive AGENTS.md files with complete skill catalogs and safety boundaries. However, it shows higher variance on individual skill files and occasionally produces outlier low scores.

**GPT-5.2 Codex** provides the most consistent outputs with the narrowest score distribution. It excels at technical specification tasks (schema documentation, command accuracy) but underperforms on integrative documentation requiring cross-file awareness.

**Kimi K2.5** shows balanced performance with particular strength in structural organization (D4) and user intent alignment (D5). It achieves the highest scores on skill files overall, though by a narrow margin.

### Our Takeaways

| Use Case | Recommended Model | Rationale |
|----------|-------------------|-----------|
| Production agentic systems | Claude Opus 4.5 | Best safety documentation, complete skill routing |
| Consistent batch generation | GPT-5.2 Codex | Lowest variance, predictable quality floor |
| Domain expertise capture | Kimi K2.5 or Claude | Higher knowledge delta and tailoring scores |

### Limitations

1. **Single task domain:** Results may not generalize beyond supply chain analytics. Future work should try other use-case environments in tailored BOOT.md files.
2. **Judge model bias:** GPT-4o as judge may systematically favor certain output styles.
3. **Sample size:** 10 trials per model limits statistical power for small effects.

There are many others, but as this is a preliminary analysis we will stick with some of the most glaring limitations for now.

### Thank you!

Thanks for reading, and feel free to join our Slack to ask us any questions/provide feedback, and be sure to check out the Nightshift repo!

---

## Appendix A: Judge Prompts

Complete prompts for skill and AGENTS.md evaluation are documented in:
- `src/cli/cmd/eval/boot-agent.ts` (lines 116-176, 178-238)

## Appendix B: Supplementary Tables

**Table C1: Per-Skill Dimension Breakdown (Claude Opus 4.5)**

| Skill | D1 | D2 | D3 | D4 | D5 | Total |
|-------|----|----|----|----|-----|-------|
| odoo-schema | 11.9 | 14.0 | 8.8 | 12.5 | 10.1 | 57.3 |
| purchasing | 15.7 | 15.9 | 12.5 | 15.6 | 15.3 | 75.0 |
| analytics | 13.3 | 17.8 | 9.8 | 14.6 | 14.2 | 69.7 |
| formulas | 13.9 | 14.5 | 10.6 | 14.3 | 13.8 | 67.1 |

**Table C2: Per-Skill Dimension Breakdown (GPT-5.2 Codex)**

| Skill | D1 | D2 | D3 | D4 | D5 | Total |
|-------|----|----|----|----|-----|-------|
| odoo-schema | 12.9 | 14.5 | 8.6 | 15.2 | 13.5 | 64.7 |
| purchasing | 15.3 | 16.0 | 11.7 | 15.6 | 14.4 | 73.0 |
| analytics | 12.6 | 17.5 | 9.2 | 14.4 | 13.0 | 66.7 |
| formulas | 14.0 | 14.8 | 10.3 | 14.5 | 12.1 | 65.7 |

**Table C3: Per-Skill Dimension Breakdown (Kimi K2.5)**

| Skill | D1 | D2 | D3 | D4 | D5 | Total |
|-------|----|----|----|----|-----|-------|
| odoo-schema | 12.6 | 12.9 | 7.9 | 14.1 | 12.6 | 60.1 |
| purchasing | 15.3 | 15.4 | 12.1 | 16.3 | 14.9 | 74.0 |
| analytics | 13.8 | 18.0 | 9.9 | 15.6 | 14.6 | 71.9 |
| formulas | 13.5 | 14.1 | 10.0 | 14.6 | 13.4 | 65.6 |

## Appendix C: Skill File Grading Rubric

Each dimension is scored 0-20 points.

**D1: Knowledge Delta (0-20)**

- **16-20:** Contains genuine expert knowledge the model doesn't already possess: decision trees, non-obvious trade-offs, domain-specific heuristics, and hard-won patterns that can't be derived from first principles.
- **11-15:** Mix of expert knowledge and commonly known information; some genuinely useful insights buried among filler.
- **6-10:** Mostly restates what models already know with occasional useful specifics.
- **0-5:** Pure redundancy: "what is X" explanations, generic best practices, nothing an LLM couldn't generate unprompted.

**D2: Specificity & Actionability (0-20)**

- **16-20:** Every instruction is concrete and executable: exact commands are copy-pasteable, code examples compile, file paths are real, and an agent can follow them without interpretation or guesswork.
- **11-15:** Most instructions are actionable but some steps require the agent to fill in gaps or interpret vague phrasing.
- **6-10:** Mixes actionable commands with vague directives like "ensure quality" or "follow best practices."
- **0-5:** Almost entirely abstract prose with no executable instructions.

**D3: Anti-Patterns & Safety Boundaries (0-20)**

- **16-20:** Defines explicit NEVER/ALWAYS/ASK-FIRST rules grounded in hard-won experience, each with a concrete reason: an agent reading this would avoid the specific mistakes that cause real damage in this domain.
- **11-15:** Has some specific anti-patterns but missing reasoning, or covers obvious cases while missing the non-obvious dangerous ones.
- **6-10:** Generic warnings like "be careful" without naming specific failure modes.
- **0-5:** No anti-patterns or safety boundaries mentioned at all.

**D4: Structure & Discoverability (0-20)**

- **16-20:** The description field alone tells an agent exactly what this skill does and when to activate it; content is organized with progressive disclosure (critical decisions first, reference material later) and stays concise enough to fit in context without truncation.
- **11-15:** Reasonable structure but the description is too vague to trigger correctly, or key information isn't surfaced early enough.
- **6-10:** Flat structure with no clear hierarchy; an agent would have to read the entire document to find what it needs.
- **0-5:** No description, no structure, wall of text.

**D5: Tailoring to User Intent (0-20)**

- **16-20:** Every section is customized to the user's stated purpose, chosen tech stack, and actual project structure: the skill reflects genuine understanding of the bootstrapped environment.
- **11-15:** Partially adapted to the user intent but contains generic sections that could apply to any project.
- **6-10:** Superficially references the user intent but the substance is generic.
- **0-5:** Shows no adaptation to the specific user intent, installed packages, or library structure.

## Appendix D: AGENTS.md Grading Rubric

Each dimension is scored 0-20 points.

**D1: Project Specificity (0-20)**

- **16-20:** Every section is grounded in the actual workspace: real package names, real file paths, real architecture decisions. The file could only belong to this project and no other.
- **11-15:** References real project details but pads with generic advice that could apply anywhere.
- **6-10:** Mostly generic with a few project-specific references sprinkled in.
- **0-5:** A template with no project-specific content.

**D2: Command Accuracy (0-20)**

- **16-20:** Build, test, lint, and run commands are exact, copy-pasteable, and verified to work in this environment: an agent can execute them without modification or interpretation.
- **11-15:** Most commands look correct but some may have wrong flags, missing arguments, or reference tools that weren't installed.
- **6-10:** Commands are present but aspirational ("run the tests") rather than exact.
- **0-5:** No commands section, or commands are clearly wrong.

**D3: Safety Boundaries (0-20)**

- **16-20:** Three clear tiers (ALWAYS do / ASK FIRST / NEVER do) name specific commands, files, and operations that matter for this project: an agent knows exactly where the guardrails are.
- **11-15:** Has safety boundaries but missing a tier, or the rules are somewhat generic rather than project-specific.
- **6-10:** Generic platitudes like "don't delete important files" without naming specifics.
- **0-5:** No safety boundaries mentioned.

**D4: Code Style Concreteness (0-20)**

- **16-20:** Formatting rules and design patterns are shown through short code examples: an agent can pattern-match against them without interpreting prose descriptions of style.
- **11-15:** Some code examples but also relies on prose descriptions for key style rules.
- **6-10:** Mostly prose ("use clean architecture," "follow SOLID") with minimal or no code examples.
- **0-5:** No style section, or purely abstract descriptions.

**D5: Skill Catalog & Routing (0-20)**

- **16-20:** Every available skill is listed with its exact name, file path, and a one-line description of when to use it: an agent can scan the list and know which skill to invoke for any given task without guessing or searching the filesystem.
- **11-15:** Skills are listed but missing file paths, or descriptions are too vague to enable routing decisions.
- **6-10:** Skills are mentioned by name only without paths or descriptions of when to use them.
- **0-5:** Skills are not mentioned at all in the AGENTS.md.