Ivo on r/legaltech

There’s a concept in software development called the ninety-ninety rule. The first 90% of the code accounts for the first 90% of development time. The remaining 10% of the code accounts for the other 90% of development time.

There is a similar variation of this rule in our domain. It’s easy to build a redlining tool that works okay and looks good in a demo. But there is an enormous long tail of work that goes into covering the last 10%. That’s everything from making sure the redlines are high quality (e.g. surgical, makes minimal changes), to handling difficult legal concepts, to an endless amount of formatting edge cases. Often these are algorithmic improvements (e.g. we had to build our own coalescing algorithm to generate high quality, surgical redlines) rather than model improvements.

There is a hand-crafted, artisanal quality that goes into fixing these “edge” cases. We see this in blind results: we significantly outperform Claude for Word in redlining tasks, and perform comparably to experienced human lawyers.

There's a bunch of other things we do too that allow us to be useful beyond what's easy to do via a generic tool - e.g. making sure we are accessing the right legal and business context in your organization to inform recommendations - but that's secondary to the main point above.

We divide output quality in three categories:

Issue Spotting - e.g. have we identified that there is an MFN provision? Location accuracy - e.g. are we inserting comments or recommendations in appropriate parts of the agreements * Redlining quality - e.g. are the redlines surgical? Are they as good as what a competent human would do in that situation?

Part of the way we capture this is through tracking thumbs downs from our users. Our threshold for what qualifies as a "bug" is very low. Basically, any time our software does something worse than a human, it's a bug - even if it hasn't done anything technically incorrect. There's a long-tail of improvements that can be made in this way.

The quality of our redlines are better than everyone else's. We know this because we track data on competitive head-to-head trials very closely. It's not uncommon for us to enter an evaluation with 30+ lawyers, and for the evaluation to conclude with literally unanimous votes in our favor. There are other things we do well: Our outputs are informed by contextual information, such as your historically negotiated contracts (we can track this very granularly, down to specific issues and sub-issues), complex playbooks, deal information, and market data. Our Intelligence tool allows users to extract information at scale across existing agreements, generate playbooks based on past agreements, benchmarks agreements against comparable precedents, visualize relationships and collapse multiple amendments into a single synthetic amended and restated version of your contracts, and more. * Our support is excellent. We take a very white glove approach to supporting our customers through complex use cases and implementations into existing systems.

It was really important we started in Microsoft Word because we needed to meet lawyers where they worked. When we launched, nobody took AI contracting tools seriously - we figured that maybe we could persuade lawyers to make one major change to the way they worked (adopting AI for reviewing contracts). But asking them to make two major changes (moving off Microsoft Word 😱) was a step too far.

In saying that, I feel pretty confident that Microsoft Word is not the best form factor for contracting. Microsoft Word is a very skeuomorphic product in that it's meant to be a digital reproduction of the pre-computing pen and paper process. What does an opinionated, AI-first approach to contracting look like? I don't know the answer yet, but it's something I spend too much thinking about.

(We have a Google Dogs plugin as well!)

Probably Ironclad - I thought they were such a groundbreaking company (and I still do!) I even applied for a job there as a PM many years ago. Sadly I was rejected.

I've posted about this somewhere else, but we recently did a blind study of Ivo's outputs against an experienced human lawyer and Claude. Ivo significantly outperformed Claude and performed comparably to the human lawyer.

Other than the quality of the outputs, there's a lot of work we do to embed ourselves into contracting workflows end-to-end. In general, our product experience is much more opinionated than Claude, which trades flexibility for a superior contracting experience.

There is no simple answer - every day we hammer down an endless number of edge cases, and our accuracy score gradually inches up higher. You need to care very deeply about output quality over a sustained period of time before you start seeing measurable results.

We use a combination of OpenAI, Google, and Anthropic's models. We have our own internal evals to evaluate the performance of the models against a variety of dimensions, and then we use the best model for each task. A single contract review could be chaining together over 400 model calls, and we find that different models are better suited for different parts of our pipeline. When a new model outperforms an existing model at any step in our pipeline against our evals, we replace them for that step.

Ivo is first and foremost a contracting company. I have a romantic view of contracts as one of the oldest and most important of business technologies, and I see the opportunity there being broader than legal. In a way, I see our main competitor as DocuSign - DocuSign found incredible product/market fit by helping people sign contracts. This turned out to be a valuable enough problem to support a $10B company. But, signing contracts is actually not that big of a problem! It's maybe 20 minutes of clerical admin work - it pales in comparison to every other part of the contracting process, e.g. getting the contract to a point where it's ready to sign in the first place. This is where we see the most exciting opportunities.

We're built from the ground up with AI. This means that Ivo can extract useful information from your agreements without requiring a protracted implementation process or ongoing maintenance. You just connect Ivo to your data sources, and Ivo will automatically start surfacing information and building relationship trees at scale across your existing agreements.

In practice we haven't seen this change much. The cost per unit of intelligence is going down, but demand for more intelligence is endless, so it evens out. There is a bigger question around per-seat pricing models. I don't think that can last.

I have a lot of admiration for CLM companies because they solve a really important set of problems - they were just too ahead of their time. It's really hard to turn contracts into structured data without AI. Contracts are perhaps the perfect application LLMs, and it gives us an opportunity to rethink how to solve these problems.

The way to build trust is pretty simple: you tell someone you're going to do something, and then you do it. Then you repeat this over and over again. Eventually, your customers and potential customers start trusting you. There are no shortcuts - I've relied on this since we were a seed stage company in New Zealand that nobody had ever heard of, and it's something we'll continue to rely on for as long as the company exists.

Implementation is incredibly easy. You simply upload your playbook (in whatever format you want), and connect into your existing systems of record (e.g. CLM, Sharepoint, Salesforce, GDrive). Ongoing maintenance is also straightforward because we can continuously update your playbooks based on our knowledge of your historical contracts. In fact, you might not even need playbooks at all, since we can reference your previous positions to inform our recommendations.

Implementation gets more complicated for larger companies - a roll out to 1000 users at a Fortune 500 company is obviously very different to small in-house legal teams.

It'll almost certainly be the system of record that owns the full contract lifecycle. I find it hard to see how you can have real defensibility as a business otherwise. You also want your contract review tool to be interoperable with your other systems, because the additional context can be used to improve the quality of the recommendations, and it's easier to do this if you own the full stack.

The challenge is that you can't do all of this from day one. You need to win trust from your customers and gradually earn the right to compete in adjacent areas.

Yes, of course, at least a little bit - I'm surprised to see none of the other founders are! Maybe I'm just particularly neurotic :D

In saying that, in practice, we haven’t seen Claude's foray into legal impact our business (either positively or negatively).

Part of the misconception people often have is they assume that the quality of the outputs from AI tools are determined solely (or mostly) by the underlying AI model. In our testing, the underlying AI models only account for roughly half of the accuracy of AI contracting tools, which leaves room for significant variance in performance.

We recently conducted a blind study where the Ivo significantly outperformed Claude across all metrics, and even performed comparably to an experienced human lawyer (a special counsel at an Am Law 25 firm). You should be able to find the report on our website if you’re interested.

A few: We surface recommendations based on analogous previously negotiated contracts We automatically understand relationships between your different historical agreements, and compile synthetic amended and restated versions of your amendments so we understand what your actual live obligations with a counterparty are at any given point in time