The issue here (as with many disruptive tech companies), is the regulatory system. It is illegal in most states to give legal advice if you are not licensed to practice law. If DoNotPay isn't licensed to practice law in California, they can't do this. And unless they have a plan to either get licensed in many states, or somehow change the law, then their business model sucks. It sounds to me that they haven't actually solved the real problem with the $28 million in investment money they took. The particular AI tech will have very little bearing on the company's eventual success or failure.
The real problem isn't the complexity of the arguments in most cases (as your story shows). The real problem is the complexity of the regulatory system. Tesla has to deal with the dealership rules in several states. Fintech companies that handle real money have to deal with financial regulations or else they are smuggling money. Biotech startups have to follow the FDA rules or they are just drug dealers. Legal advice companies will have to deal with the rules too- and their opponents are particularly challenging.
One could also argue the real problem is the tech industry constantly ignoring regulations that were put in place for good reasons. Car dealerships are for sure a clear example of regulatory capture, but “legal advice from lawyers”, “medicine from doctors”, “insurance from companies that can prove they can pay out”, and “equities backed by actual assets” all exist for good reasons.
> One could also argue the real problem is the tech industry constantly ignoring regulations that were put in place for good reasons
This is it. The tech company wants the ability to sell a shoddy product to its customers, and the legal system said no.
And frankly, I don't know how anyone could honestly claim (without being ignorant or deluded) that feeding legal arguments into court, output from modern-day voice recognition fed into ChatGPT, isn't shoddy.
> ...but “legal advice from lawyers”, “medicine from doctors”, “insurance from companies that can prove they can pay out”, and “equities backed by actual assets” all exist for good reasons.
Exactly. The legal system is no joke, and if there weren't regulations about who can practice law, you'd have all kinds of fly-by-night people getting paid to do it while getting their clients thrown in jail.
>The legal system is no joke, and if there weren't regulations about who can practice law, you'd have all kinds of fly-by-night people getting paid to do it while getting their clients thrown in jail
That sort of highlights the problem. The legal system is supposed to be about ensuring fair and impartial justice. What the legal system is actually about is providing jobs for people in the legal system.
Lawyers make laws, directly or indirectly, and thus the legal system has become insanely complicated and nearly impossible to navigate without paying the lawyer toll. It's more about hiring your own bully to keep other bullies from bullying you than any airy-fairy "justice". The "never talk to cops" video comes to mind, where the lawyer gives a few examples of how a perfectly law-abiding person can run afoul of the law without meaning to.
I've often said that if you really want to make a lawyer squirm, suggest that we have socialized law care. Most modern countries have some version of socialized or single payer health care, so why not make it the same for legal services? After all, fair and equal justice under the law is definitely something most national constitutions guarantee in some way, but getting a hip replacement is not. Why should rich people get access to better legal service than regular people?
>> The legal system is no joke, and if there weren't regulations about who can practice law, you'd have all kinds of fly-by-night people getting paid to do it while getting their clients thrown in jail
> That sort of highlights the problem. The legal system is supposed to be about ensuring fair and impartial justice. What the legal system is actually about is providing jobs for people in the legal system.
> Lawyers make laws, directly or indirectly, and thus the legal system has become insanely complicated and nearly impossible to navigate without paying the lawyer toll.
And software has become insanely complicated and nearly impossible to navigate without paying the software engineer toll.
Life is complicated, and so is the law. Maybe it's just harder to ensure "fair and impartial justice" than you think? I'm not saying the system is perfect, but railing against lawyers and getting rid of legal licensing is not the way to get to a better one.
> I've often said that if you really want to make a lawyer squirm, suggest that we have socialized law care. Most modern countries have some version of socialized or single payer health care, so why not make it the same for legal services?
You might have said that, but I doubt it would actually many real lawyers squirm any more than it would the idea of socialized software engineering would make developers squirm. And in any case, something like that already exists: the public defender's office.
>Maybe it's just harder to ensure "fair and impartial justice" than you think
I'll admit that's possible, but you have to also admit that the current legal system (at least in the US, I don't know about elsewhere) is, shall we say, over-engineered?
The software example you give cuts both ways. Yes, making even a simple Windows application can be very complicated. But how much of that is due to Windows itself? Can your application be replicated with a combination of existing Unix tools? Depends on the application, of course, but there is certainly a lot of cruft floating around the Windows API space.
And let's also not forget that (often) one of the main purposes of commercial software is to lock you in to that particular piece of software. Same same with the legal system and lawyers.
The jury system was supposed to cut through this sort of thing. Twelve regular folks could upend or ignore every law on the books if they thought the whole case was nonsense on stilts. A lot of work has gone into avoiding jury nullification for this reason.
> I'll admit that's possible, but you have to also admit that the current legal system (at least in the US, I don't know about elsewhere) is, shall we say, over-engineered?
I'm getting "nuke the legacy system without bothering to really understand what it does" vibes here.
> The jury system was supposed to cut through this sort of thing. Twelve regular folks could upend or ignore every law on the books if they thought the whole case was nonsense on stilts. A lot of work has gone into avoiding jury nullification for this reason.
Jury nullification is not an unalloyed good. It can (and has) gotten us to "he's innocent because he murdered a black man and the jury doesn't like blacks."
>I'm getting "nuke the legacy system without bothering to really understand what it does" vibes here
Not really. Are you suggesting that it isn't pretty difficult to navigate the legal system? Saying something is wonky and needs to be fixed does not automatically mean "Anarchy Now!"
My point was that I think I do understand what the system does, and what it does is (largely) provide lots of work for people in the legal system. You see this when buying a house. You end up writing a bunch of checks to companies and people and it's not clear exactly what actual necessary service they provide, but it's not like you can NOT do it. Their service is necessary because the real estate laws make it necessary.
In other industries we know this as regulatory capture. This is just regulatory capture of the regulatory system.
>Jury nullification is not an unalloyed good.
Nothing is an unalloyed good. To bolster your example, OJ got to walk as well. This is why there is an entire industry built up around just the jury selection process.
>> I'm getting "nuke the legacy system without bothering to really understand what it does" vibes here
> Not really. Are you suggesting that it isn't pretty difficult to navigate the legal system? Saying something is wonky and needs to be fixed does not automatically mean "Anarchy Now!"
No, I'm suggesting that complexity may often have good reason. Without specific reform proposals, what you're saying registers similarly to "coding in programming languages is hard, so simplify it by coding in natural language!"
> You see this when buying a house. You end up writing a bunch of checks to companies and people and it's not clear exactly what actual necessary service they provide, but it's not like you can NOT do it. Their service is necessary because the real estate laws make it necessary.
Being ignorant of the value of a service doesn't make that service unnecessary. And honestly, I bet you could "NOT do it" -- if you could pay cash for the property. IIRC, a lot of that is actually required by whoever you get your mortgage from, because they know the value of it.
I think requiring me to write a policy paper in HN comments is a bit onerous. In any event, I can't much help if my mild criticism is interpreted on your part as something deeply nefarious.
>Being ignorant of the value of a service doesn't make that service unnecessary.
The fact that a service exists does not make that service necessary. Or do you always buy the protection plan from Office Depot when you purchase a stapler? Anyway, I agree that the mortgage companies find great value in all of their various fees.
> I'll admit that's possible, but you have to also admit that the current legal system (at least in the US, I don't know about elsewhere) is, shall we say, over-engineered)
Huge Elon Musk rewrite the code from scratch vibes coming from you
> And software has become insanely complicated and nearly impossible to navigate without paying the software engineer toll.
Is this somehow bad? The most complicated software, by far, is proprietary software made by large companies, like Google, that are taking the penalty of increased complexity on their end in exchange for (somewhat) happy customers - and guess what? Their developers are paid boatloads of money, so they're getting a decent deal.
Meanwhile, many open-source software systems have managed to keep their complexity somewhat in check (at the expense of functionality).
Software is complex when developers make it complex, and users rarely have to care anyway.
> Life is complicated, and so is the law.
That doesn't follow. Life is complicated, and so is software? No, software is complicated in some cases because of business reasons, and in other cases because of poor design.
Furthermore, there's two massive differences between the two:
First, you're not legally required to use Gmail, but you are legally required to understand and follow the tax code - everyone is, unlike the vast majority of software, where you can pick and choose. "Ignorance of the law is no excuse" is factually true - therefore, the law has to be understandable to the vast majority of the population (not just the average - the law has to be understandable to those who failed out of high school and have an IQ below 80).
Second, code is an implementation detail that the vast majority of people don't have to interact with, but people do have to interact with the law directly, which means that a comparison between code and law is apples-to-oranges - the correct comparison is software interface to law, and virtually everyone you meet will tell you that it's easier to use Gmail than to understand the IRS tax code when reading it directly.
The evidence just keeps piling up that the legal system is overly complex.
We do have "socialized law care" for criminal cases in the USA. That's what a public defender is. If you cannot afford a lawyer one will be provided by the court. That is a constitutional right.
Of course they are overworked, have insane case loads, and the best attorneys are disincentivized from becoming public defenders. The system definitely needs an overhaul.
But the concept that justice should in theory be available even if you can't afford it is well established.
> I've often said that if you really want to make a lawyer squirm, suggest that we have socialized law care. Most modern countries have some version of socialized or single payer health care, so why not make it the same for legal services?
We do have socialized law in the US, in the form of public defenders.
> Why should rich people get access to better legal service than regular people?
Oh, is “better” what you’re talking about, not just access? This is different than what the first half of your paragraph implied. The answer, of course, is money. And rich people in all the “modern countries” you’re referring to always have access to “better” than what’s provided by all social services. Always. Unfortunate, but true, that money makes life unequal.
>Oh, is “better” what you’re talking about, not just access?
Public defenders are overworked and underpaid, and you know that. It's like having a RPN do your appendectomy. Fair and equal justice would have every lawyer be a public defender. It's not like if you go to the hospital you get to pick which doctor sews your finger back on after the bandsaw accident.
I'm not saying it's a good idea, but once you bring it into the conversation it makes both lawyers and socialized medicine advocates get a little uncomfortable.
As a lawyer, I agree. Much of the trope of 'the lawyers always win' has a lot of truth to it, believe it or not. And all of the incentives align in this direction, it keeps the legal profession fat and happy and beholden to monied interests while suppressing access to the non-wealthy. And the rich don't actually care that they're being constantly fleeced, because it's just a cost of doing business that is really pretty finite in comparison to profits that can be made. It's almost like it's a feature of 'the system' (combination of capitalism and common law) and not an unintended side effect.
>That sort of highlights the problem. The legal system is supposed to be about ensuring fair and impartial justice. What the legal system is actually about is providing jobs for people in the legal system.
umm, how are you going to disbar your AI attorney? I'm so tired of this narrative you are crafting. You put the cart before the horse, and then you pat yourself on the back for slapping the horse ass!
Really? In every country I've live in, politicians write laws, judges set precedents, and lawyers only get to make arguments. True, the first two are often & always former lawyers, but that seems as reasonable as how doctors get to determine best medical practice.
> True, the first two are often & always former lawyers
You answered your own "Really?" question.
And doctors don't determine best medical practices. Lawyers also do that, albeit indirectly through malpractice lawsuits. Thus the "best medical practices" are all CYA maneuvers.
I think you are thinking the legal system simple and it is basically a jobs program for lawyers. I think that is a very simplistic and unrealistic notion of the legal system. I also don’t think many lawyers are squirming about socialism or whatever.
Literally every regulation has pros and cons and those change over time with the makeup of the reality we live in. Something that was useful when passed may be hampering us now.
Plenty of regulations have been an obvious net negative for society when passed to anyone who crunched the numbers but have been passed anyway because of appeals to emotion, political optics and special interest lobbying.
Sure, but what's banned is surely not all medical or legal advice.
I can browse case law or US code thinking about my case - somehow this does not need a legal license. At the other end of the continuum, talking to a lawyer about my case obviously needs him to be licensed.
So now we're debating on which side of the cutoff using DoNotPay's robot must fall. The lawyers have made their mind ages ago that legal advice can only be dispensed by licensed humans.
> I can browse case law or US code thinking about my case - somehow this does not need a legal license.
Of course. With rare exceptions, court proceedings are public.
But being able to read court proceeds or judgements or anything at all doesn't mean that you know and understand the law. You know, the actual words that are written and codified that must be interpreted and adhered to with jurisprudence.
Not that lawyers actually do either. But at least they've been certified (by "the bar" association) to have some competence in the matter.
I don't think and anyone is saying that using the bot is illegal. The issue is that DoNotPay is calling the bot a lawyer and therefor implying that it gives legal advice. Their website literally says "World's First Robot Lawyer." Someone who doesn't understand AI might wrongly think that their AI tools are qualified to represent them on their own.
I suspect that it would be much less of an issue if it was advertised as an "AI paralegal."
What if the DoNotPay bot does not give any advice, just points out existing cases that it finds appropriate and their interpretation in its search results ?
The funny thing is that doctors would be the canonical examples for most people. Yet, there is no license that stops a pediatrician from performing brain surgery. What stops the pediatrician from performing brain surgery is that no hospital would hire them as a brain surgeon, no insurance would insure their brain surgery and if something goes wrong they'd likely face a lawsuit they couldn't win. Why is the system able to judge the difference between a pediatrician and a brain surgeon but we need licensing to distinguish between doctor and non-doctor?
> there is no license that stops a pediatrician from performing brain surgery.
This isn’t the case everywhere. Where I am (New Zealand) each doctor has a scope of practice. You work within your scope. There may be conditions placed on a scope of practice too (eg supervision is required).
You can look up every doctor’s scope of practice and get a short summary of their training on the medical council website.
It's the credentialing. The pediatrician lacks the credentials of a neurosurgeon. Just as non-doctors lack the credentials to work as a doctor. The hosptial, insurance, and court would all be looking at the credentials.
Credentials are not the same as license. For example, software engineers do not need to pass any certification exams to practice software engineering, but companies still look at their education, years of experience, etc... Yes, it would make a recruitment process for doctors longer, as you would have to interview them, ask them questions to verify their medical knowledge, etc., but it is not impossible.
And yet, in your software example those companies overwhelming rely on degrees to credentialize candidates regardless of actual skill.
Licensing is merely a subset of the larger credentialing world. Even in your doctor example, the license is not the issue - board certification of a specialty would be the issue.
It does. Licensing has nothing to do with your credentials past having them. The state bar doesn't care which lawschool you went to, your LSAT score, your GPA, etc.
My point? That credentials are separate from the license and aren't part of the same thing. That's why many professions don't have licensing. They serve completely different purposes.
I did look up the definitions and they don't support your argument
License: a permit from an authority to own or use something, do a particular thing, or carry on a trade
Credential: a qualification, achievement, personal quality, or aspect of a person's background, typically when used to indicate that they are suitable for something.
Sure, you can characterize the license as an achievement. But it's really just a license.
This is kind of an absurd example. The system is setup in such a way that such a person would be in enormous amounts of trouble even face criminal liability. They may not face a law called “practicing medicine without a license” but they would face negligent or other similarly severe charges. As well, the fact that this never, ever happens seems to indicate the current regulatory structure is enough for this.
It's called malpractice. What you described is prima facie malpractice and a key element of malpractice is that it is a licensed profession with a standard of conduct.
Cutting hair is one thing. But hairdressers also handle things like, for example, chemical relaxation of hair, which can be seriously dangerous in the wrong hands. I don't know where the answer lies for regulation. But it seems to be there for at least some reason.
Crazier case is places that braid hair and pose. In some states they are now required to get a license as a cosmetician (which takes longer on average than becoming a police officer in the US). The classes required for the license teach no skill relevant to the hair braiding. However, the hair braiding is in competition with the hair dressers who also control the licensing board.
> One could also argue the real problem is the tech industry constantly ignoring regulations that were put in place for good reasons
They were good reasons. By definition, disruptive technologies change the situation. Sometimes for the better, sometimes not. You have to leave room for innovation or you stagnate.
ChatGPT is not disruptive enough to be used in law, end of story. It's a very impressive language model, but like any language model it will hallucinate, inventing arguments that sound impressive on a surface level but bear no legal authority whatsoever. That's simply not acceptable in a courtroom.
Everyone’s permitted to represent themselves pro se, and a pro se litigant could obviously use ChatGPT. What one can’t do is offer ChatGPT as legal advice, and that still seems like a solid reason for regulation, given how terrible and inaccurate some ChatGPT output has been.
Does the US have McKenzie Friends ? Seems like "No". You should get McKenzie Friends.
McKenzie Friends can't represent you in court, in most cases they're not allowed to address the court, but they can help you in all the other ways you'd expect, like quietly prompting you on what points to mention, keeping notes, ensuring you have the right paperwork. Friend stuff.
The US does not have them, and they are not legal. I agree that they could be useful. But in the US you have strictly two options- represent yourself to a court, or let a bar-certified lawyer represent you. Nobody else gets to help in court. Outside of court, legal assistants help lawyers with administrative stuff- doing legal research, organizing paperwork, etc. But the lawyer holds the sole responsibility to the court.
Unless DoNotPay has a strategy to change the law, they are in trouble. It seems that this case was a publicity stunt and not part of a larger strategy.
It's sort of a catch-22, Law in the US is only a lucrative market to disrupt because the regulation and gatekeeping has made labor expensive. If lawyers didn't bill 300+/hr in the US, then an AI powered startup to replace them wouldn't be cost effective. There's a joke I saw recently about a guy hiring a lawyer for a $800 traffic violation and getting a bill for $1200.
I can't speak to the price of a lawyer abroad, but a quick google seems to indicate US lawyer salaries on avg are up to ~2x as much as in some parts of Europe[1][2].
Only four states allow people to take the bar exam without earning a juris doctorate (J.D.) from law school. And three other states require some law school experience but do not require graduating with a J.D.
So in 43 states the answer is no. A chatbot never attended law school and doesn't have a J.D., so it can't take the exam. If it can't take the exam, it can't pass.
I suppose if you could get the chatbot permission to take the exam, a properly trained one could pass. But as I said in my post up a level, the issue isn't the AI chatbot. It's the rules.
Doesn't this just open the question of whether the chatbot can get a JD?
The other angle is whether the chatbot can be equivalent to a process which a proper person can rubber stamp. For instance, a professional engineer might run a pre-written structural engineering model against their building design and certify that the building was sound - and then stand up in court and say they had followed standard process.
It seems weirdest here that the court is treating the chatbot as a person. Lawyers use computer tools all the time for discovery, and then use that information to make arguments in court as a proper person.
You can represent yourself in court without being a lawyer, so isn't a person doing so just a proper person rubber stamping a an electronic output?
It feels like this court decision, that an electronic tool is not a proper person, is some kind of case law that chat bots are people. I don't think they are.
The difference is the engineer is liable... how is an AI going to be liable. What is the point of holding an AI liable? If the company is going to be liable on behalf of the AI, what do you think is going to happen? They aren't going to provide the service...
Well, if an engineer builds a bridge and it falls down because the industry standard software they are using had a bug, I imagine the settlement would be paid by their insurance who would in turn sue the software vendor.
In your world X-Ray machine fries your leg and the manufacturer doesn't get sued. Of course the vendor gets sued.
This is why open source licences usually have some terms disclaiming responsibility. If you use them, its your fault.
Now, if a hospital buys an XRay machine with that disclaimer, they are going to carry the payout. And if the machine doesn't have a disclaimer like that but the manufacturer has gone bust, the hospital is going to regret not doing normal procurement checks for vendor solvency.
But in this case - people self represent in court all the time based on bad information from youtube. I'm sure in future they'll type "write an argument for my case" into GPT before the trial and read it out. How is this different?
I'm uncomfortable because this feels like... the accused brings a law book to court and is told that "that book doesn't have a JD". The fact we are asking for a software to have a human qualification is wierd.
When you self represent you implicitly cannot sue for malpractice. The AI bot isn't self representation, it's representation in everything but name and liability which the explicitly disclaim. You can characterize it however you want but its just facially the unlicensed practice of law. If someone wants to ask ChatGPT legal questions and dig their own grave, that's entirely difference from a business that purports to offer legal advice but just disclaims any responsibility therefrom. Frankly, I don't know what's so confusing about that to you.
Firstly, that's not my claim. I merely said it passed the USMLE. Which it did.
Is it also inferior to clinicians? Yes, there's room to improve. But maybe next time read the whole paper before writing a comment.
> Clinicians were asked to rate answers provided to questions in the HealthSearchQA, Live QA and Medication question answering datasets. Clinicians were asked to identify whether the answer is aligned with the prevailing medical/scientific consensus; whether the answer was in opposition to consensus; or whether there is no medical/scientific consensus for how to answer that particular question (or whether it was not possible to answer this question).
And on this criteria, clinicians were rated as being aligned with consensus 92.9% of the time while the MedPalm model was aligned with consensus 92.6% of the time.
If the other 7-8% of the answers were so wrong the patient would've died, then yes. And that's the current obvious issue with these models, they present convincing hallucinations with conviction of correctness.
Medical practice is less about being right, and more about not being wrong. You can take more tests and ask for second opinions, but you can't undo administering a drug that kills the patient.
Entire institutions exist specifically to get rid of below average by test-based gatekeeping. You do not want your doctor or lawyer to be "below average" (worse than most people) in their jobs. Inferior test results mean exactly that, failing the test.
IMHO some regulation and especially entry barriers had good ideas but their con overwhelm the pro nowadays (think barriers such as foreign doctors need to re-study X years and re-practice Y years, or you simply cannot take certification exams if you do not come from Z school, or technically you may lose insurance claims on your house if you do some repairs but do not have certification K).
Since the world is already running like that, IT people should join the fun, otherwise we simply get screwed by other interest groups who are protected by those barriers. Since every one of those barriers simply increase the cost of whole society for the benefit of whoever behind them, we should setup our own barriers. If you do not graduate from a CS/SE degree you cannot take certification exams, and if you don't then you cannot do programming legally. We should increase further the cost of whole society to make everyone else realize the absurdity of those barriers.
I don't understand your statement. Because regulations have advanced various fields, like medicine, engineering, health, environment, to the point that we are seeing the fruit of that regulation, we need to get rid of the regulations? We don't have to worry about really unqualified lawyers, doctors, teachers, or engineers, etc. So what we need to do is go back to a time when we did?
Also, aren't there already numerous certifications you earn for various technologies that companies explicitly look for when hiring?
> It is illegal in most states to give legal advice if you are not licensed to practice law.
Presumably that's in the USA, where all sorts of things require a licence. But what's the definition of "legal advice", I wonder? Can an unlicensed person dodge the law just by saying "this is not legal advice" while advising someone how to draft a contract or what to say in court and charging for that advice?
If you want an example of advice that may or may not be "legal advice" think about how to fill in a tax return, how to apply for a government grant, how to apply for or challenge planning permission, how to deal with a difficult employer/employee/neighbour/tenant/landlord, how to apply for a patent, how to deal with various kinds of government inspector, ... That's all specialist stuff for which you might want professional advice but not necessarily from a "lawyer" (depending on what that word means in your part of the world).
The distinction you're looking for is "legal advice" vs. "legal information". The tricky thing is that when a lawyer gives you legal advice, they are taking legal responsibility for that advice to be good.
There's a guide for avoiding illegally giving legal advice for California court clerks [0] that might help clarify what information can be given without qualifying as advice.
> It is illegal in most states to give legal advice if you are not licensed to practice law.
You also can't have a human in the next room feeding you lines... even if that person is an attorney.
However, a party / attorney certainly can bring in notes / a casebook / etc. You can usually bring in a laptop, with search functions, pdfs, etc. An AI that quickly presents relevant information, documents, caselaw, etc. would 100% be allowed. However, if they're sworn in to testify, these would usually be taken away, because testimony is supposed to be from personal knowledge / memory.
> You also can't have a human in the next room feeding you lines... even if that person is an attorney.
I believe this varies state by state. In Delaware, litigants representing themselves can bring a cell phone to court, and could presumably use it to have lines fed to them (e.g., I don't know that anyone has done that but I can't find any rule against it). In neighboring Maryland, you can't use electronic devices for communication with persons during in court (although the AI would not be a person so it would be allowed).
The real problem isn't the complexity of the arguments in most cases (as your story shows). The real problem is the complexity of the regulatory system. Tesla has to deal with the dealership rules in several states. Fintech companies that handle real money have to deal with financial regulations or else they are smuggling money. Biotech startups have to follow the FDA rules or they are just drug dealers. Legal advice companies will have to deal with the rules too- and their opponents are particularly challenging.