Automated Review of Contracts: Some Thoughts on LawGeex’s AI-Versus-Humans Study

I’m a contract-drafting guy, but I have to acknowledge that drafting contracts might not be the most annoying part of the day-to-day contracts process.

Assume that Acme does ten deals with ten different companies in which it drafts the contracts using its templates. Then assume that it does those deals using the other guy’s drafts. Odds are that in the second scenario Acme would end up doing a lot more work than it would in the first scenario—producing a draft using a template you’re familiar with is likely to take less time than reviewing a draft produced by the other side.

So it’s not surprising that software aimed at reviewing the other side’s drafts should now be attracting attention. The two names I’m familiar with are LegalSifter and LawGeex. They’re welcome innovations.

LawGeex recently disseminated this study that made the following claim:

In a landmark study, 20 experienced US-trained lawyers were pitted against the LawGeex Artificial Intelligence algorithm. The 40-page study details how AI has overtaken top lawyers for the first time in accurately spotting risks in everyday business contracts.

That prompted a bunch of hyperventilating articles, including this one.

I’m sure the reported results reflect what happened. I know something about confidentiality agreements, having spent around a year building an automated one. (It’s described in this LinkedIn article.) So I had a look at the LawGeex study, and it prompted the following thoughts. My intention isn’t to criticize, but to offer some context.

Humans Are Fallible

Yes, humans are fallible. In terms of contract drafting, my presumption is that everything is bad, and I’ve offered on this blog many examples of that. When that’s not the case, I’m pleasantly surprised. So I would expect a comparable dynamic to apply when it comes to review. But the circumstances of LawGeex’s study are a worst-case scenario. If a company is paying attention, it would give those reviewing confidentiality agreements some sort of checklist against which to measure what they’re reviewing. By contrast, those taking part in LawGeex’s test had to rely only in their experience and native wit.


Second, the issues flagged by LawGeex are very broad. For example, one issue was presence of a no-soliciting provision. In my automated confidentiality agreement, the no-soliciting provision is customizable up the wazoo, starting with whether it covers just hiring or both hiring and soliciting. Simply flagging no-soliciting provisions doesn’t get one very far.

Necessarily Limited Scope

Third, inevitably, LawGeex’s list of issues wasn’t comprehensive. To select an example at random, it doesn’t include flagging instances of the word proprietary, something I wrote about in this 2010 post. And their list doesn’t cover general drafting issues, such as whether something should be expresses as a condition and not as an obligation, and I doubt it ever will. That’s why review by software should support review by a person, not replace it.

What Comes Next

Fourth, spotting issues is the first part of what the technology does. Then LawGeex suggests edits based on a company’s pre-defined legal policies. I’d be interested to know the level of detail it offers, in terms of both what it reads and the suggestions it offers. But that would require a demo. For purposes of this post, I’m just looking at their study.

Whom Do You Trust?

And fifth, my biggest question about the new crop of “AI” technologies isn’t the technology per se, it’s the humanoid expertise it incorporates. That concern applies to all services that address contract content. I’m toying with the slogan “Editorial expertise is the new black box.” In the case of services that offer contract templates, if I don’t know who prepared a template, I’m not going to trust it. Even if I do know, I’ll be skeptical unless given good reason not to be. Relying on someone’s contract language requires a leap of faith, so I know that I have to not only be an expert but also appear to be an expert. The same goes for services that assist with review.

To its credit, LawGeex identifies the “team of prestigious law professors and veteran lawyers” that prepared the list of issues that forms the basis for the test. But I happened to spot that the heading for one of the identified issues was “Exclusion—Public Domain.” That’s a little worrisome: as I note in this 2010 post, the phrase in the public domain “has no bearing on how widely available any given information is. Instead, it means that the information isn’t protected by intellectual-property rights and so can be used by anyone free of charge. That would represent an irrationally narrow exclusion from the definition of ‘Confidential Information’ ….” So LawGeex’s team flubbed by using that phrase, albeit just in a heading. Might they have missed other stuff? Just as those performing an old-fashioned review are likely to be fallible, the experts giving instructions to AI might be fallible too.

Most people shouldn’t find that sort of problem disconcerting, as most of us would be grateful to have the benefit of the collective expertise of LawGeex’s team, even if they’re fallible.

Other Kinds of Contracts

It’s not surprising that LawGeex’s report features confidentiality agreements. They’re the cockroach of the contract world—ubiquitous, annoying, and apparently indestructible. And you see the same issues in contract after contract. It will be interesting to see how LawGeex and its competitors do when it comes to reviewing more fluid kinds of contracts.

This category of product has the potential to make contract review quicker and more effective. Let’s see whether the technology and the underlying expertise are up to the job. And, to quote this post, let’s see whether the intended users give a ****.

The Stormy Daniels Contract Is a Dumpster Fire

I looked at the Stormy Daniels confidentiality agreement, the one intended to keep under wraps her fling with Trump—excuse me, her alleged fling. I figured it would have been remiss of me not to look at it. (It’s exhibit 1 to her complaint, here. If you’re not familiar with her and the saga of this confidentiality agreement, go here for one article among zillions.)

I can confidently say it’s the most flamboyantly dreadful contract I can recall seeing. Trust me, I’ve read my share of bad contracts, but bad business contracts tend to relentlessly repeat standard forms of dysfunction. By contrast, on display in the Stormy Daniels contract are myriad standard and wildly original forms of suboptimal prose and layout. It’s a dumpster fire.

The first thing that caught my eye was use of and/or in the introductory clause. That prompted the following tweets:

That issue might actually have a bearing on the dispute. Otherwise, my only concern was the quality of the drafting, and what I saw established that the drafter was semiliterate, transcendently careless, and not a transactional person. You’ll find below some random examples.

But before that, why did I bother looking at this contract? Well, first, everyone loves a train wreck. Second, and more seriously, I, like everyone else, have been in the peanut gallery during Trump’s reign. Here, for the first time, something related to Trump has come within my sphere of competence, and it’s sorely lacking. And third, if anyone comes away from this with a sense of what’s required for clear contract language, that would be a bonus.

Now, on to the examples:

  1. Why is the first page given the page number zero?
  2. Hey, let’s misspell nondisparagement in the title!
  3. Why are the introductory clause and the recitals presented as if they were part of the body of the contract?
  4. The recitals appear to contain obligations. That’s not what recitals are for.
  5. The contract is full of idiosyncractic blather such as “This Agreement is entered into with reference to the facts and circumstances in the following recitals.” Recitals do what they do: they don’t require any fanfare.
  6. Similarly pointless and idiosyncratic is “the Parties adopt the foregoing recitals as a statement of their intent.”
  7. The heading 3.0 should be followed by section 3.1. Instead it’s followed by the heading That’s like no multiple-numeration scheme I’ve ever encountered.
  8. Everything under 3.0 and before 3.1 uses hanging indents. The rest of the contract uses first-line indents. WTF.
  9. In section (crazy enumeration!), an obligation is imposed on PP’s lawyer, although they’re not party to the contract.
  10. Section 3.1 (finally!) is given the heading “Undertakings & Obligations by PP.” What’s the difference between an undertaking and an obligation?
  11. One of those undertakings and obligations is “PP shall execute this Agreement.” So if PP hadn’t signed the contract, she would have been in breach of the contract! Hahahahahaha.
  12. Here are the verbs used with respect to PP’s handling of the “Property”: sell, transfer, turn-over, assign, deliver, divestconvey. Hey, why not add a few more?
  13. Blocks of text (a) through (g) in section 3.1 start as tabulated enumerated clauses and end as subsections. That’s messed up.
  14. Tabulated enumerated clause 3.1(c) somehow acquired a solitary second-level tabulated enumerated clause, clause (1), but it’s not introduced as a tabulated enumerated clause should be. I assume it should have been given the enumeration (d).
  15. Section 3.2 says “all of PP’s respective rights, title, and interest in and to the Property.” The “respective” would make sense only if this sentence applied to more than one party.
  16. Section 3.3 says “PP represents and warrants … that upon such delivery to DD, PP shall not maintain ….” In other words, with respect to that language of prohibition, “represents and warrants” constitutes what I call “throat-clearing”—a redundant verb structure preceding what should be a stand-alone provision.
  17. Section 4.1 uses both means and includes as definitional verbs. See A Manual of Style for Contract Drafting ¶ 6.4 for why that doesn’t make sense.
  18. Section 4.2 goofily states tangible and intangible in italics.
  19. Gotta love heading 4.3, “Representations & Warranties and Agreements.” Note that section 3.3 and section 4.2 contain, ahem, representations and warranties, so structure wouldn’t see a strong point of the drafter. (See this article for why represents and warrants and its variants are pointless and confusing. But that’s a nuanced discussion; we’re not in the realm of nuance with this contract.)
  20. More throat-clearing in 4.3.2(a): “PP agrees and warrants and represents that PP will ….” And why switch the order to “warrants and represents”?
  21. In 4.3.2(a) will is used to express an obligation; elsewhere other verb structures are used to express obligations. It goes without saying that the verb structures are chaotic throughout this contract; they don’t come close to complying with MSCD‘s categories-of-contract-language framework. For more about that, go here.
  22. It looks that the heading to section 4.3.3 should have included “Tangible and/or Intangible Confidential information created by or relating to DD.”
  23. Section 4.3.3 contains the defined-term parenthetical “(‘Third Party’ and/or Third Parties’)”. It’s always silly to state a defined term in both the singular and the plural when you’re defining it; furthermore, this isn’t even the way to do that. Note in passing the weird and/or and the missing opening quotation marks.
  24. Section 4.3.5 says “PP separately and further warrants and represent ….” The “separately and further” is redundant. And what’s a botched subject-verb agreement among friends?
  25. The drafter has an uncertain and shifting understanding of the role of punctuation in parentheticals.

OK, I’ve had enough, and I’m only halfway through. Over and out.

My Updated Preempt-the-Traditionalists Cover Note: Get Your Retaliation in First

MSCD guidelines mostly fly under the radar. Apart from a couple of notorious recommendations—I’m talking about you, this agreement with a small a and states instead of represents and warrants—the guidelines don’t attract attention to themselves.

But sometimes traditionalists gonna traditional, so you might get a draft back with all sorts of unhelpful traditionalist comments. As I note in this 2013 post, starting in 2009 I’ve suggested that the best way to make that less likely is to tell the other side that you’re following MSCD guidelines and that you politely recommend they not waste everyone’s time by trying to make things more traditional. When I think about this, the phrase “Get your retaliation in first” comes to mind. It was coined by the great Irish rugby player Willie John McBride, in connection with the Lions 1974 tour to South Africa. (Those circumstances were rather more brutal than what I’m talking about!)

Well, the fourth edition of MSCD contains a revised version of that cover note. It starts at the bottom of page xxxix of the introduction, which is available here. If you’ve tried my cover note or one of your own, I’d like to hear how it worked out.

If you’d like to know more about the context of “Get your retaliation in first,” there’s this article and the following video:

Alternative Defined Terms + “And/Or” = !?!

Behold the following, all culled from EDGAR and all featuring and/or used between alternative defined terms:

… and EdgeWave, Inc., a Delaware corporation (“Employer” and/or “Company”) …


… Michael J. Friedman (hereinafter the “Executive” and/or “Employee”) …

Now, we know that and/or is has got problems (see this 2017 post). And we know that providing alternative defined terms for something is a bad idea (see this 2007 post). But combining the two, well that’s …

On Twitter, some suggested that this practice is a hedge against copy-and-paste problems. That by itself would be unimpressive—what’s so hard about Ctrl + H? But it goes beyond that, in that it seems that the drafter is signalling that they don’t know whether one or the other defined term, or both, are used in the contract.

But it’s actually a little more subtle than that. Usually alternative defined terms are linked with or. It wouldn’t make sense to interpret that as meaning that the contract uses only one or the other defined term. Instead, the drafter is in effect saying that a given instance of use of a defined term for that definition might be one or the other defined term. Applying that logic consistently, use of and/or to link defined terms would suggest that a given instance might consist of … both defined terms?

This is what I get for doing a blog post late on Friday afternoon. Bring on the weekend!

The Case of the Elusive “Inclusion Rider”

During the Academy Awards show last Sunday, the “Best Actress” winner Frances McDormand unleashed on the world the phrase “inclusion rider.” That prompted a tsunami of chatter on the subject, including this by the Washington Post and this by Vanity Fair.

But I was interested in the rider itself, not explanations. So I asked around, and I asked on Twitter, but no one had a copy. I emailed one of the people responsible, Kalpana Kotagal, a civil-rights employment lawyer at Cohen Milstein, but I didn’t hear back.

I ended up having an exchange on Twitter with the Annenberg Inclusion Initiative (@inclusionists), a think tank at the University of Southern California. One of the people behind it is communications professor Stacy Smith, the other person responsible for the inclusion rider. What they told me was puzzling. They said “Lots of folks have the language,” that it’s available for industry use, and anyone who wants a copy can ask for it by emailing them, but they were unwilling to send me a copy. If they’re so keen to have the entertainment industry adopt this approach, and if they’re sending copies of the rider to “lots of folks,” why not send it to me?

The answer is to be found in this article by entertainment lawyer Jonathan Handel in The Hollywood Reporter. Here’s the relevant bit:

Smith first introduced the idea in a 2014 Hollywood Reporter guest column, but it lay mostly dormant until this year’s Oscar night. The media scholar says she’s not aware of any actors having used the rider, and entertainment attorneys say they’re unfamiliar with it. “Seen none. Have none,” says a top talent lawyer.

Yet Smith and Kotagal aren’t sharing the clauses they’ve crafted. “The language is for attorneys, actors and content creators — we don’t give it out,” says a colleague of Smith’s. “We want to avoid public negotiation,” says Kotagal, but the Washington-based attorney may also see secrecy as a ticket to Hollywood legal work. “Civil rights lawyers have a right to make money,” she notes.

So much for “lots of folks.”

I see two problems with Smith and Kotagal’s approach. First, it’s at odds with the way people create contracts in the information age. Once deal language starts being used, it starts being copied. Anyone who prepares distinctive contract language can claim copyright protection, but all that’s required to avoid a claim of copyright violation is to copy the idea without copying too much of the wording. (See my 2006 article on the subject.)

So it’s perhaps naive to think you can build a revenue stream by keeping contract language close to your chest. Instead, you get business through your expertise. Marty Lipton’s highly lucrative poison-pill legal work wasn’t based on his having access to the relevant Word templates—the documents in question quickly became public. Instead, clients sought him out for his expertise.

And second, there would seem to be a conflict between advocating that the industry adopt inclusion riders and wanting to limit distribution of inclusion-rider text. Getting the text out there might help prevent the inclusion rider from disappearing from discussion as rapidly as it arrived.

By the way, I have no reason to think that Smith and Kotagal’s inclusion rider is a model of contract drafting. In our Twitter exchange, @inclusionists said, “We had a civil rights attorney and expert on hiring and discrimination craft the language. Then, it was vetted by multiple entertainment attorneys and business affairs folks.” But what seems to be missing is any input from a contract-drafting specialist.

Another Source of Inertia in Contract Drafting: “We Don’t Give a ****”

You don’t have to be a physicist to experience the force of gravity. Similarly, I’m only a casual student of inertia, but I’ve long grappled with it.

Casey Flaherty is a serious observer of inertia. I’ve returned several times to the following in this post:

My most obvious error has been predicting the pace of change. While there is sufficient demand for change to keep me and a cadre of fellow travelers occupied … , the overall pace of change keeps being far slower than I imagine even when I update my priors to incorporate the observation that the pace of change is far slower than I imagine.

This should not surprise me.

Massive passive resistance. Agency dilemmas. Institutional inertia. Status quo bias. Loss aversion. Endowment effects. Lack of urgency. KAP gaps. The Chasm. System justification. Institutional isomorphism. Reams of academic literature explain the Planckian notion that progress does not occur when its opponents see the light but only when they lose their power to oppose—that is, funeral by funeral. I’ve not only read my Rogers, I’ve read Bill Henderson’s masterful series applying the Rogers Diffusion Curve to innovation, or lack thereof, in the legal ecosystem.

I’m inclined to add to Casey’s list another source of inertia. It’s encapsulated in my mind by the phrase We don’t give a ****. (Fill in the blank as you see fit!) Allow me to explain.

Recently I saw online an extract from a contract drafted by a global company. It exhibited traditional contract drafting (albeit a slightly modernized version), in that the prose was both clumsy and unhelpfully legalistic. This extract wasn’t an exception: it was broadly comparable with another contract drafted by that company that I had had occasion to examine.

The company in question has the resources to do things right. And they have at least a passing acquaintance with what “right” looks like. But they’re not interested. In my one interaction with them, they made it clear that they didn’t wish to hear about shortcomings in their contracts.

Why? Here’s an all-purpose explanation Casey offers in this post:

Inside counsel are really busy doing work that is mission critical to the enterprises they serve. They are overburdened and do not have the time or other resources to pursue transformative change within or outside the law department.

In this case, I don’t think that captures the dynamic, and neither do any of the more nuanced explanations. I simply concluded that the company in question doesn’t give a ****.

Studies of inertia tend to seek to explain why rational economic actors opt to stick with suboptimal results. But perhaps we’re not dealing with rational economic actors. Maybe we’re dealing with people for whom the suboptimal is acceptable because achieving the optimal is just too much of a drag.

I’m hard-wired to give a ****. My family refers to me as “the society policeman.” Waste and selfishness in the public sphere makes me uncomfortable. When I think of, say, what it took to land on the moon, I think, They gave a ****! When I think of the Swiss train system, I think, They give a ****!

Currently the zeitgeist seems to contain a big helping of We don’t give a ****. A random example: whenever I return from my travels, take an AirTrain from JFK, and find myself at Jamaica Station, I survey the grubbiness and find myself thinking, We don’t give a ****.

When it comes to my corner of the contract-drafting world, I like to assume that people would prefer to avoid wasting time and money, hurting their competitiveness, and assuming unnecessary risk, so like Casey I find a bunch of explanations for continued dysfunction. But let’s be realistic—it’s inevitable that some proportion of the inertia that afflicts contract drafting is attributable to We don’t give a ****.

Tackling dysfunction requires putting yourself out for the common good. It requires challenging the status quo. If you’re mainly concerned with doing the minimum and not rocking the boat, you’ll stay clear of tackling dysfunction, and you’ll probably do just fine.

I’m not inclined to get hung up on this. If I thought the don’t-give-a-**** contingent had the upper hand throughout the industry, I’d throw in the towel. Instead, I’m constantly reminded of the people and organizations out there trying to do better. I’ll continue to do my thing for those who do give a ****.