Summaries > Technology > Software > Software factories are missing this...

Software Factories Are Missing This

https://www.youtube.com/watch?v=XUEfIxkdT2Q

TLDR Software factories are measurable, code-defined workflows that use agents to take work from idea to launch and improve through recorded runs, benchmarks, and testing. Their success depends on defining meaningful business outcomes, starting with proven workflows, keeping humans involved in high-risk decisions, and building a culture of shared ownership and psychological safety—rather than automating everything at once.

Key Insights

Define the Factory Before Automating It

Treat a software factory as a measurable workflow that takes work from an idea or issue through implementation, review, testing, release, and monitoring. Make its agents, tools, permissions, runtime, and models explicit in code so the team can inspect and change them. Start with the workflow your team actually uses rather than designing an idealized system from scratch. This creates a clear baseline and makes later improvements testable.

Choose Outcomes That Reflect Business Value

Define what “yield” means for your product before judging whether the factory is improving. Measures can range from pull requests and cycle time to features shipped, customer outcomes, or business impact; choose metrics that match your product’s maturity and goals. Teams with limited users or data may not yet be able to measure customer impact reliably, while mature products can track outcomes and regressions more directly. Pair delivery metrics with human judgment, since shipping more code does not necessarily mean creating more value.

Bring Work Into a Shared, Tracked Loop

Start by recording agent-assisted work in a cloud-based workflow, even if developers still interact with agents directly. A shared process creates visibility into task progress, costs, decisions, and human involvement, and can make collaboration easier through tools such as Slack. Once the workflow is understood, automate repeatable steps gradually. Avoid trying to remove humans or redesign the entire development process before the team knows which parts work well.

Capture Runs and Turn Them Into Evaluations

Store traces of coding-agent runs so the team can investigate failures, unnecessary tests, repeated work, dead ends, cost, and the amount of human steering required. Turn representative completed tasks into benchmarks, then compare models or factory configurations against the same tasks. For reliable comparisons, replay each task from a clean starting state and ensure the agent cannot see future commits or completed pull requests. Use online evaluations and statistically meaningful tests to validate whether a change actually improves performance.

Measure Efficiency, Quality, and Cost Together

Use a balanced set of measures rather than relying on a single productivity metric. DORA metrics, work volume, cycle time, cost per engineer or workflow stage, automation levels, and human interactions per pull request can reveal different strengths and bottlenecks. Track friction too, such as time spent figuring out test commands or repeatedly correcting an agent. Review these measures regularly and improve the system against a clear definition of “better.”

Adopt Risk-Based Review and Keep Changes Small

Let low-risk changes move quickly while requiring explicit human approval for changes that could affect security, databases, dependencies, persisted state, or user data. Encode ownership and approval rules so the required review is consistent and auditable. Small, focused pull requests are easier to evaluate and can help keep risky work separate from routine changes. Better specifications and automated checks can reduce review effort, while screenshots or recordings can help verify user-facing behavior.

Build Adoption Through Visibility and Psychological Safety

A shared factory changes how engineers work, so adoption requires more than installing tools. Make work visible early to catch misalignment before a large, difficult-to-merge pull request is complete, but create a culture where people feel safe sharing unfinished work and learning publicly. Pairing can help engineers learn agent workflows together and stay focused. Avoid running so many agents in parallel that coordination costs and quality problems outweigh the speed gains.

Plan Enough to Reduce Rework, Then Learn by Building

Use specifications to clarify important goals and reduce costly rework, but do not overplan details that are cheap to change. A practical cycle is to build a prototype, learn from it, revise the specification, and then implement a cleaner version. Software factories also depend on product and engineering teams sharing ownership of outcomes and decision-making. Aligning those teams helps ensure the factory solves real organizational and customer needs rather than merely automating code production.

Questions & Answers

What is a software factory?

A software factory is a cloud-based, measurable agentic workflow that takes work from an idea or issue through triage, optional specification, implementation, code review, QA, CI/CD, launch, and monitoring. Its workflow, agents, skills, tools, runtime, and models should be explicitly defined so teams can measure and improve how work gets done.

How does a software factory differ from the traditional software development lifecycle?

Rather than treating development as a fixed sequence of human-run steps, a software factory makes the workflow shared, cloud-based, observable, and increasingly automated. Teams can record agent runs, evaluate results, test changes to the factory, and improve the process over time.

Why have software factories not yet become reliably effective?

Many organizations lack clearly defined workflows, measurable customer outcomes, and a shared way to evaluate engineering productivity. They may also attempt to automate too much too soon or build a system before confirming that it solves a real need. Adoption is another challenge because moving from private local workflows to a shared factory changes how engineers work.

How should companies introduce a software factory?

Start by bringing existing work—even interactive, prompt-driven work—into a shared, tracked cloud workflow. Define and measure the process, then automate parts of it gradually. Another viable path is to automate proven workflows used by strong engineers. Both approaches recommend avoiding a full redesign or complete automation before the workflow and its value are understood.

How should a company measure a software factory’s yield?

Yield should reflect useful outcomes, but the right definition varies by business. Possible measures include features shipped, customer or business outcomes, DORA metrics, work volume, cycle time, cost, degree of automation, human interactions or steering per PR, and time wasted using the system. More meaningful outcome measures require judgment and can be difficult to attribute.

When does it make sense to invest in a software factory?

The need depends more on product maturity and constraints than on team size. A pre-product-market-fit team may not have enough users or data to measure improvements, while a mature product can track outcomes and regressions. Regulated businesses may have an earlier need for safeguards such as review, compliance, and auditing.

How can teams use coding-agent traces to improve a factory?

Recording every agent run lets teams evaluate patterns such as redundant tests, dead ends, verbosity, relearning, cost, and the amount of human guidance required. Teams can compare models and configurations using evidence rather than impressions, then use agents to inspect failed runs and propose changes to the factory definition.

How can teams reliably evaluate and compare models or factory configurations?

Teams can turn completed tasks into benchmarks and run controlled A/B tests. Evaluations must prevent a model from seeing future commits or completed pull requests and must reproduce a clean, consistent starting environment. With these safeguards, results can be compared statistically across models or configurations.

What makes engineers adopt an agent-driven development system?

Adoption requires making shared, public workflows useful and psychologically safe. Engineers may need to shift from continuous local work to responding intermittently to remote agents, with work visible to teammates. Public progress can improve alignment and collaboration, but leaders should support experimentation, pairing, and shared learning rather than making visibility feel like surveillance.

What kinds of code changes still need human review?

Humans should focus review on high-risk changes, especially those involving persisted or serialized state, databases, dependencies, security, or user data. Programmable ownership policies can require explicit approval for affected files while allowing low-risk changes to move faster. Small, focused PRs and stronger specifications can also reduce review burden.

How much planning and specification should teams do before implementation?

Plan enough to reduce expected rework by weighing how likely requirements are to change against how costly a change would be. Avoid overplanning low-cost details. A useful iterative loop is to build a prototype, revise the specification based on what was learned, and then rebuild with cleaner code and architecture.

What organizational conditions are needed for a software factory to succeed?

Engineering and product need aligned ownership of customer outcomes, along with the willingness to share decisions and work openly. Teams that protect separate domains or lack psychological safety may struggle to use a shared factory effectively, regardless of the technology.

Summary of Timestamps

Zach, Warp’s founder and CEO, defines a software factory as a cloud-based, measurable loop that moves work from an idea or issue through triage, specification, implementation, review, testing, deployment, and monitoring. The central idea is that developers must improve the system that produces software—not only build products. A key open question is how to define “yield,” since useful output and business value vary by company.

The group discusses why software factories have not yet consistently delivered high-throughput, high-value engineering work. One proposal is to define the factory in code, much like infrastructure: specify its workflow, agents, skills, tools, runtime, and models. This makes it possible to test and branch factory configurations and even have coding agents suggest improvements.

Measuring yield is difficult because simple measures, such as pull request counts, may not reflect customer or business value. More meaningful measures include features shipped, customer outcomes, and business impact, although these require more judgment and are harder to attribute. The speakers argue that engineers should understand users’ problems and take ownership of outcomes, rather than treating shipped code as the end goal.

Many companies still use coding agents as individual developer tools, while others are building internal factories with cloud-based agents, portals or Slack access, and automated triggers. These systems can improve speed, consistency, cost control, security, and auditing. The speakers compare two paths: gradually automating proven workflows, or first bringing interactive agent work into a shared, measurable loop and automating it over time.

The speakers caution against attempting to automate everything or redesigning processes before understanding them. Many organizations have not clearly defined their workflows or how to measure customer outcomes, so a factory built from scratch may solve the wrong problem. Whether a factory makes sense depends not just on company size, but also on product maturity and constraints.

Software factories may be premature for pre-product-market-fit teams, which often lack enough users and outcome data to guide optimization. More mature products can track regressions and assess whether automation improves results. In regulated industries, safeguards such as code review and compliance checks may be essential. Suggested measures include human interactions per pull request, DORA metrics, cycle time, work volume, platform costs, and factory performance evaluations.

Recording every coding-agent run creates a foundation for improvement. Teams can examine traces for redundant tests, dead ends, excessive verbosity, repeated learning, cost, and the amount of human guidance required. Comparing these results across models and factory configurations makes evaluations more objective than relying on impressions about how a model behaves.

When factory definitions and run histories are stored in code, agents can inspect failed tasks, find recurring problems, and suggest changes to workflows or agent skills. Completed tasks can also become benchmarks for evaluating models on real company work. Reliable comparisons require clean starting environments and safeguards that prevent models from seeing future commits or completed pull requests.

Once tasks can be replayed consistently, teams can run meaningful A/B tests of models and factory configurations. Useful evaluation signals include cost, DORA metrics, automation levels, human steering, and time wasted figuring out how to use the system. Repeated failure patterns can reveal where agent skills, documentation, or memory need improvement.

A major adoption challenge is moving developers from private, local workflows to a shared, visible factory. Work may happen through remote agents that engineers check intermittently, with tasks initiated in Slack using prompts, screenshots, or design links. This can help teammates collaborate and catch misalignment early, but it also changes the day-to-day experience of engineering.

Working in public can prevent teams from spending weeks on work that misses the goal, and it lets people learn how experienced engineers guide agents. However, it requires psychological safety: developers need to feel comfortable sharing unfinished work and receiving feedback. Pairing can support focus and learning, while running too many agents in parallel may reduce quality and slow progress.

Even when agents write much of the code, humans remain important for steering and reviewing high-risk changes involving databases, persisted state, dependencies, security, or user data. Programmable ownership rules can require explicit approval for sensitive files. Allowing low-risk changes to merge without human review reportedly raised the share of pull requests meeting the policy from about 10% to 80%, encouraging smaller, more focused changes.

Clear specifications and automated checks for spec adherence may reduce review effort. For user interfaces, screenshots and computer-use videos can help verify behavior without requiring reviewers to inspect every implementation detail. The discussion then turns to balancing alignment with the risk of writing specifications that are too detailed.

Planning is useful when it reduces expected rework, but the amount of planning should depend on how likely a requirement is to change and how costly that change would be. The speakers describe a cycle of specification, implementation, revised specification, and implementation: build a prototype, learn from it, then refine the requirements and architecture.

A software factory depends on organizational alignment as well as engineering tools. Product and engineering teams need shared ownership, which can be difficult when each group protects its own authority. The conversation closes with a story about turning away a customer whose organization was not ready to adopt a product; after addressing those issues, the customer returned and succeeded. The broader lesson is that factories work best when teams are prepared to collaborate openly and share decision-making.