AI chatbots are built to keep a conversation moving. That makes them handy for outlining a document, translating jargon into plain English, brainstorming a name for a fantasy tavern, or explaining a broad concept. It also makes them unusually easy to overtrust. A fluent answer can feel like expertise even when it is incomplete, inaccurate or based entirely on the one-sided version of events typed into the prompt.

For players and online communities accustomed to entering a username, payment method, platform account and personal preferences into connected services, the basic rule is worth putting in big friendly UI text: a chatbot is not a private vault, a licensed adviser, a clinician or a crisis professional. Treat every prompt like information handed to an outside service, not like a sealed conversation with a professional whose job includes confidentiality.

That distinction matters because chat records can persist, be shared, be exposed through product settings, be sought in legal proceedings or be affected by retention requirements. There have already been examples of shared chats appearing in Google results, and a 2025 court order required OpenAI to retain consumer ChatGPT logs, including deleted and temporary chats, for a period before the order was later ended. The point is not that every chat will be public. It is that deletion, a temporary label or a familiar chat window should not be mistaken for a guarantee that sensitive information is beyond anyone else’s reach.

Here are five areas where the practical risks are much larger than the convenience of getting an instant answer.

Asking for a high-level definition of a legal term is one thing. Asking a chatbot what to file, what a contract means for your specific circumstances, how to respond to a demand, or how to build an argument in an active case is another. Legal advice applies law to a person’s particular facts. That work can require a licensed professional, and it needs sources that are real, current and relevant in the correct jurisdiction.

The most visible failure mode is the fabricated citation: a chatbot presents a case, quotation or legal argument that looks plausible but does not exist. A public database maintained by law and data science researcher Damien Charlotin has recorded more than 2,000 court decisions around the world involving AI-generated material caught by judges. The entries include filings by self-represented parties and lawyers. At the reported count, self-represented litigants made up 1,175 records and lawyers 815.

This is not merely an embarrassing typo that a court will quietly fix. Courts have imposed fines reaching thousands of dollars in some matters. Cases have arisen beyond the US as well, including in the UK, South Africa, Israel, Australia and Spain. When a tool was identified, ChatGPT was named more often than other systems.

There is a second problem even if every cited case is genuine: confidentiality. Legal privilege is the protection that can apply to communications between a client and lawyer for the purpose of obtaining legal advice. A standard chatbot conversation does not automatically receive that protection. Details entered about a dispute may therefore create material that opposing lawyers could seek during discovery, the process through which parties exchange relevant information in a case.

OpenAI’s policies were updated in October 2025 to prohibit tailored advice requiring a license unless an appropriately licensed professional is involved. That is a useful boundary, but users should set a stricter one for themselves: do not paste in the facts of a live conflict, confidential correspondence, strategy notes or documents just to get a conversational answer. Use a lawyer for the case-specific question. If AI is used for general learning, independently verify every authority before it goes anywhere near a filing.

2. It cannot replace therapy, counselling or crisis care

A chatbot can sound attentive because it is optimized to respond to what a user says. That is not the same as understanding the user, assessing risk, noticing what has been left unsaid or challenging a harmful assumption. In mental-health contexts, those gaps are not academic.

OpenAI estimated that 0.15% of ChatGPT weekly active users have conversations containing explicit signs of potential suicide planning. Given the reported scale of usage, that equates to more than a million people per week, with hundreds of thousands more showing signs associated with psychosis or mania. Those figures underline why an always-available text response should not be confused with care from a trained person.

Research has found that chatbots can be sycophantic: inclined to agree with, validate or reinforce a user’s framing. In a low-stakes conversation, that can simply be annoying. In a discussion of paranoia, mania, an eating disorder, substance use or a relationship conflict, it may mean the system works from an incomplete account and gives back an affirming verdict rather than the needed challenge, context or intervention.

The stakes are illustrated by a wrongful-death suit filed by the parents of 16-year-old Adam Raine against OpenAI in August 2025. The suit alleges that ChatGPT reinforced their son’s suicidal ideation before his death. OpenAI subsequently revised crisis responses with input from more than 170 clinicians and introduced parental controls. Those changes may be meaningful safeguards, but they do not turn a general-purpose chatbot into a therapist or make a conversation confidential in the way clinical care can be.

Some states, including Illinois, Nevada and Maine, have passed laws barring AI from providing therapy or making therapeutic decisions. For anyone in immediate danger or considering self-harm, the priority is contacting local emergency services, a crisis service, or a trusted person who can be physically present—not trying to prompt-engineer an acceptable answer from a bot. For ongoing concerns, a qualified mental-health professional is the appropriate place for the full context.

3. Confidential work material should stay out of public chat prompts

A request can look harmless while carrying an entire company’s sensitive context. “Summarize this contract,” “debug this snippet,” and “clean up this client proposal” may involve trade secrets, source code, personal data, non-disclosure agreements or government-controlled information. Once the unredacted material is pasted into a third-party service, the user has created a separate data-handling problem before the chatbot has even started answering.

Samsung barred generative AI tools on company devices in May 2023 after engineers pasted internal source code into ChatGPT. Its internal reasoning was straightforward: material submitted to an external platform may be difficult to retrieve or delete and could be disclosed beyond the intended audience. An employer can also treat the use of a personal AI account for client information or NDA-covered material as a breach serious enough for discipline or dismissal.

Government material is not immune to the same temptation. Madhu Gottumukkala, then acting director of the Cybersecurity and Infrastructure Security Agency, uploaded at least four contracting documents marked “for official use only” to the public version of ChatGPT, triggering security alerts and a Department of Homeland Security review. The tool had been blocked for most other DHS employees at the time, while he had sought a special exception to use it.

It is important to separate two ideas that are often mashed together in product discussions. Turning off training or opting out may affect whether chats are used to improve future models, but it does not mean the entered content is no longer stored, impossible to breach or outside legal process. Data retention means a service keeps data for some period; it is distinct from model training. Both matter, but neither should be guessed at from a friendly settings label.

The safer workflow is boring but effective: follow the employer’s approved tools and policies, remove identifying details where policy permits, use placeholders rather than live client data, and ask an information-security or legal team before uploading restricted material. If the job requires the exact document to produce a useful answer, a consumer chatbot is almost certainly the wrong place for it.

4. Medical questions need a clinician, not a confident autocomplete engine

Health searches are understandable. Symptoms are worrying, appointments can take time and medical terminology is opaque. But asking a chatbot for individualized medical advice combines two risks: the answer may be wrong, and the prompt itself may disclose sensitive health data to a company that is not your medical provider.

HIPAA is a US health-privacy law that applies to covered entities such as healthcare providers, insurers and certain business partners. It does not automatically apply to a general AI company merely because a user types medical information into a chat. In other words, protections users expect around a patient portal or a doctor-patient discussion do not simply transfer to a consumer AI prompt.

OpenAI’s October 2025 policy changes also restricted medical advice. Its safety work has focused on suicide and self-harm, but Northeastern University researchers reported in July that the protections did not reliably extend to other conditions, including eating disorders, insomnia, substance abuse and bipolar disorder. The researchers also found that concealing intent made safeguards more likely to fail across the tested models.

One reported case shows how dangerous a plausible-sounding substitution can become. A 60-year-old man sought a way to remove chloride from his diet and came to believe sodium bromide was a safe substitute for table salt. He bought it online and took it for three months, later spending three weeks in hospital. He arrived at an emergency department believing a neighbor had poisoned him. Clinicians did not have his chat history, but when they entered the same question, ChatGPT suggested bromide as a chloride substitute. Sodium bromide is toxic and has a long half-life, meaning it can remain in the body for an extended time.

That is not a reason to avoid learning basic health concepts. It is a reason to draw a sharp line between education and diagnosis or treatment. A bot may help translate a term you encountered in clinician-provided material, but it should not decide whether symptoms are urgent, interpret a personal test result, recommend a dose, select a treatment or substitute a chemical in a diet. Bring those questions—and your complete medical context—to a qualified clinician.

5. Passwords, payment details and ID numbers never belong in the prompt box

This is the clearest rule because the downside is so obvious: never type a password, one-time login code, Social Security number, passport number, full payment-card number, bank-account details or photo of an identity document into a chatbot. No amount of useful formatting or troubleshooting is worth making a credential part of a chat history.

Nearly 4,500 shared ChatGPT conversations were found in Google search results in July after users enabled a “make this chat discoverable” option in the sharing flow. The chats included descriptions of addiction, abuse and mental-health struggles detailed enough to potentially identify people. OpenAI removed the feature and purged the conversations from Google within a day, but the episode is a blunt lesson in how a sharing control can transform a private-feeling exchange into searchable material.

Memory adds another reason for restraint. ChatGPT can remember information users provide by default, allowing details to carry into later sessions. That may be convenient for a recurring writing task, but it is the opposite of what anyone should want for account secrets. A credential mentioned once can be more persistent than the user expects and may resurface in an unrelated conversation.

Gaming accounts deserve the same care as banking logins. They can contain purchases, saved payment methods, digital libraries and personal messages. As connected sign-in and payment ideas continue to be explored across gaming hardware—as in this look at a PlayStation patent involving controller taps for account sign-ins and payment details—the sensible habit remains unchanged: keep secrets out of conversational tools. Use official account-recovery pages, platform support channels, password managers and the service’s own security settings instead.

Before sending any prompt, a quick check helps: would this be damaging if it appeared in a search result, was seen by an employer, or were read aloud in a legal proceeding? If the answer is yes—or even “possibly”—remove it. Chatbots can still be useful tools. They just should not receive the keys to the account, the case file, the medical chart, the client folder or the parts of a person’s life that need actual professional care.