Suppose you ask an AI tool to summarize the termination provisions in a contract. The answer is clear, the table looks useful, and one notice period is wrong. Someone still has to find the error before the summary informs a decision.
That is where a useful comparison between legal AI and ChatGPT starts: the work you need done, the information the tool uses, and the review needed before anyone relies on the result.
ChatGPT and a specialist legal AI tool may both be candidates for a particular task. Assess each deployment against the same work. A legal label is not evidence of accuracy, and a general-purpose label does not tell you which data protections apply.
Start with a defined legal task
Choose a task small enough to check properly. That might be extracting notice requirements, producing a first-pass summary of approved material, or comparing a revised clause with an agreed position. Define the source documents and what the reviewer must receive.
For a notice-provision summary, a useful output would identify the triggering event, the notice period, the required delivery method and the relevant clause. The reviewer should be able to trace each entry to the document.
Decide in advance what makes the result unacceptable. An invented notice period should fail the test. So should a correct period attached to the wrong triggering event. Those are more useful acceptance criteria than asking whether the answer sounds convincing.
Compare the actual tools on the same work
Legal AI covers products with different sources, integrations and workflows. The label alone does not tell you whether a product can manage a matter, maintain a repository or support a particular review process. A specialist tool may offer relevant legal sources or a configured review workflow. Confirm those features in the product you are testing. Ask which models and sources it uses, how it keeps those sources current, and what the user can inspect. The underlying model's name is only part of the answer.
Use the same document set and review standard for each candidate. Record the product, plan, model or configuration where available, and test date so the results can be interpreted later. Confirm that the features in a demonstration are available in your intended deployment; a preview or early-access feature should be identified as such.
| Evaluation point | What to check in ChatGPT | What to check in a specialist legal tool |
|---|---|---|
| Source coverage | Which uploaded files or connected sources did this answer use? | Which documents and legal sources are included, and are they relevant to the matter? |
| Traceability | Can each material statement be checked against an exact source passage? | Do citations support the statement and identify the applicable source version? |
| Review standard | Does the output follow the instructions and positions you supplied? | Does the configured review reflect your actual positions, fallbacks and escalation rules? |
| Missing information | Does the output distinguish absent information from an answer? | Does the tool flag gaps or conflicts, and can the reviewer inspect them? |
| Data handling | Which account, settings, terms and connected services apply? | Which deployment, provider terms and downstream services apply? |
| Review effort | How much work is needed to verify and correct the result? | How much work is needed to verify and correct the result? |
Treat the answer as a draft until it passes those checks. Open the sources: a convincing citation format does not establish that an authority exists or supports the conclusion.
Check data handling before uploading documents
Identify the account and workspace people will actually use. A personal account, a business workspace and a tool connected to other services can involve different settings and contractual terms.
OpenAI says it does not use ChatGPT Business or Enterprise business data to train its models by default, with exceptions for data a customer explicitly opts to share. Its enterprise privacy information also describes security and retention controls. A no-training commitment does not by itself establish that information is never retained.
OpenAI's data controls guidance explains the distinction between model-training preferences and saved conversations. Turning off model training does not delete chat history. That setting alone is not an organizational approval to upload client material.
Before using a provider, resolve these questions:
- What information may users enter, and who approves exceptions?
- Who can access the input, output, shared links and connected sources?
- What is retained, for how long, and under whose control?
- Does any information go to another service, and which terms apply there?
- Can the team record the source material, output and review decision appropriately?
Use the answers to set the scope of a pilot. Start with approved material, such as a fictional agreement or an internally cleared test set. Redaction is useful only if the remaining material is also appropriate for that use.
Make the output easy to verify
Ask for an output structure that exposes the evidence. For the contract example, a starting instruction could be:
Using only the supplied agreement, list its notice requirements in a table. Include the triggering event, notice period, delivery method, clause reference and exact supporting words. Mark missing information as not found. Flag conflicting provisions rather than resolving them without explanation.
This makes the result easier to inspect; it does not establish that the output is complete or correct. The reviewer still needs to check the agreement, relevant definitions, schedules and any supplied amendments.
If the tool reports a 30-day notice period, locate the supporting text. Check which event it governs, whether a definition changes its meaning, and whether another relevant provision was omitted. Ask the tool to identify uncertainties, then make the legal assessment through the team's normal review process.
Apply the same discipline to research. Open a cited authority, confirm that it exists and supports the proposition, and check the relevant jurisdiction and current status. A generated quotation needs to match the source.
Keep legal judgment and professional responsibility with the reviewer
The ABA's Formal Opinion 512 discusses how existing professional duties apply to generative AI, including competence, confidentiality, communication and reasonable fees. It is guidance under the ABA Model Rules; lawyers must assess the rules and requirements that apply to their own practice.
Set a named reviewer for each approved use case. That reviewer needs enough subject knowledge and access to the sources to identify a material error. Software availability should not become permission for everyone to use every document in every task.
Before entering client information, have the responsible lawyer assess confidentiality, applicable privilege considerations, and any required client communication or consent. A provider's product category or marketing language does not settle those questions. Access controls, audit records and approval steps each need to be checked in the intended deployment; their presence does not by itself establish a separate privilege architecture or preservation of privilege.
The review record should show what was checked and what changed. If an output cannot be verified, record the gap and escalate or complete that part of the work manually.
Measure the work that remains after generation
Include preparation, review, corrections and exceptions when measuring a trial. Compare that total with the time needed for the existing process; the speed of generation alone is a poor measure of useful work.
Use a small test set that includes ordinary work and a few known edge cases. For contract extraction, those might include a missing field, conflicting dates or an amendment that changes a relevant term. Have a qualified reviewer prepare the expected findings before the test.
Track:
- Material errors and omissions, including their consequences for the task.
- Whether each important statement can be traced to a source.
- Material differences when the same task is repeated with the same documents, instructions and configuration.
- Review and correction time, compared with the existing process.
- Cases requiring escalation or manual completion.
For example, run the same review five times and compare the findings, supporting passages and escalation decisions. Assess whether each result meets the review standard, rather than requiring identical wording. A configured workflow is not evidence that every generated answer will be correct.
If the task includes Word redlines, check numbering, cross-references, tables and tracked changes after exporting or handing off the document. Include formatting repairs in the review time.
Set acceptance criteria before expanding use. One successful demonstration is not enough to establish that the tool is suitable for every matter. Recheck the approved use when the product, configuration, source set or task changes.
Recognize when the question becomes a workflow decision
A tool may produce a useful clause summary while leaving the team responsible for routing an exception, collecting approval and updating the contract record. Those process requirements deserve their own evaluation.
For a wider buying decision, compare how copilots, legal AI assistants and contract platforms support the full contract workflow. For an individual task, keep the test concrete. Approve its use only when the reviewer can verify the result and the team has a workable process for corrections and exceptions.
Start with one approved contract task. Write down the source set, review standard and failure criteria, then run the same test in each candidate tool. Use the findings and the work left for the reviewer to decide whether to proceed.
