healthcarereimagined

Why AI Cannot Be Allowed to Mediate Medicine Without Accountability

By Katalin K. Bartfai-Walcott, CTO, Synovient Inc

On January 7, 2026, OpenAI announced ChatGPT Health, a new feature that lets users link their actual medical records and wellness data, from EMRs to Apple Health, to get personalized responses from an AI. It is positioned as a tool to help people interpret lab results, plan doctor visits, and understand health patterns. But this initiative is not just another health tech product. It is a dangerous architectural leap into personal medicine with very little regard for patient safety, accountability, or sovereignty.

The appeal is obvious. Forty million users already consult ChatGPT daily about health issues. Yet popularity does not equal safety. Connecting deep personal health data to a probabilistic language model, rather than a regulated medical device, creates a new class of risk.

As a new class of consumer AI products begins to position itself as a companion to healthcare, these systems offer to connect directly to personal medical records, wellness apps, and long-term health data to generate more personalized guidance, explanations, and insights. The promise is familiar and intuitively appealing. Doctor visits are short. Records are fragmented. Long gaps between appointments leave patients wanting to feel informed rather than passive. Into that space steps a conversational system offering continuous synthesis, reassurance, and pattern recognition at any hour. It presents itself as improvement and empowerment, yet it does so by asking patients to trade agency, control, accountability, and sovereignty for convenience.

This is a terrible idea, and it is terrible for reasons that have nothing to do with whether people ask health questions or whether the healthcare system is failing them.

Connecting longitudinal medical records to a probabilistic language model collapses aggregation, interpretation, and influence into a single system that cannot be held clinically, legally, or ethically accountable for the narratives it produces. Once that boundary is crossed, the risk becomes persistent, compounding, and largely invisible to the person whose data is being interpreted, and the results will be dire.

Medical records are not neutral inputs. They are identity-defining artifacts that shape access to care, insurance outcomes, employment decisions, and legal standing. Anyone who has worked inside healthcare systems understands that these records are often fragmented, duplicated, outdated, or simply wrong. Errors persist for years. Corrections are slow. Context is frequently missing. When those imperfections remain distributed across systems, the damage is contained. In that form, errors stop behaving as isolated inaccuracies and begin to shape enduring narratives about a person’s body, behavior, and risk profile.

Language models do not reason the way medicine reasons. They do not weigh uncertainty with caution, surface ambiguity as a first-class signal, or slow down when evidence conflicts. They produce fluent synthesis. That fluency reads as confidence, and confidence is precisely what medical practice treats carefully because it can crowd out questioning, second opinions, and clinical judgment. When such a synthesis is grounded in sensitive personal data, even minor errors cease to be informational. They become formative.

The repeated assurance that these systems are meant to support rather than replace medical care does not hold up under scrutiny. The moment a tool reframes symptoms, highlights trends, normalizes interpretations, or influences how someone prepares for or delays a medical visit, it is already shaping care pathways. That influence does not require diagnosis or prescription. It only requires trust, repetition, and perceived authority. Disclaimers do not meaningfully constrain that effect. Only enforceable architectural boundaries do, and those boundaries are absent.

Medicare is already moving in this direction, and that should give us pause. Algorithmic systems are increasingly used to assess coverage eligibility, utilization thresholds, and the medical necessity of procedures for elderly patients, often with limited transparency and constrained avenues for appeal. When these systems mediate access to care, they do not feel like decision support to the patient. They feel like authority. A recommendation becomes a gate. An inference becomes a delay or a denial. The individual rarely knows how the conclusion was reached, what data shaped it, or how to meaningfully challenge it. When AI interpretation is embedded into healthcare infrastructure without enforceable accountability, it quietly displaces human judgment while preserving the appearance of neutrality, and the people most affected are those with the least power to contest it.

What is missing most conspicuously is patient sovereignty at the data level. There is no object-level consent that limits use to a declared purpose. There is no lifecycle control that allows a patient to revoke access or correct errors in a way that propagates forward. There is no clear separation between information used transiently to answer a question and inference artifacts that may be retained, recombined, or learned from over time. Without those controls, the system recreates the worst failures of modern health IT while accelerating their impact through conversational authority.

The argument that people already seek health advice through AI misunderstands responsibility. Normalized behavior is not a justification for institutionalizing risk. People have always searched for symptoms online, yet that reality never warranted centralizing full medical histories into a single interpretive layer that speaks with personalized authority. Turning coping behavior into infrastructure without safeguards does not empower patients. It exposes them.

If the goal is to help individuals engage more actively in their health, the work must start with agency rather than intelligence. Patients need enforceable control over how their data is accessed, for what purpose, for how long, and with what guarantees around correction, provenance, and revocation. They need systems that preserve uncertainty rather than smoothing it away, and that prevent the silent accumulation of interpretive power.

Health data does not need to be smarter. It needs to remain governable by the person it represents. Until that principle is embedded at the architectural level, connecting medical records to probabilistic conversational systems is not progress. It is a failure to absorb decades of hard lessons about trust, error, and the irreversible consequences of speaking with authority where none can be justified.

If systems like this are going to exist at all, they must be built on a very different foundation. Patient agency cannot be an interface preference. It has to be enforced at the data level. Individuals must be able to control how their medical data is accessed, for what purpose, for how long, and with what guarantees around correction, revocation, and downstream use. Consent cannot be implied or perpetual. It must be explicit, contextual, and technically enforceable.

Data ownership and sovereignty are not philosophical positions in healthcare. They are safety requirements. Medical information must carry its provenance, its usage constraints, and its lifecycle rules with it, so that interpretation does not silently outlive permission. Traceability must extend not only to the source of the data, but to the inferences drawn from it, making it possible to understand how conclusions were reached and what inputs shaped them.

AI can have a role in medicine, but only when its use is managed, bounded, and accountable. That means clear separation between transient assistance and retained interpretation, between explanation and decision-making, and between support and authority. It means designing systems that preserve uncertainty rather than smoothing it away, and that prevent the accumulation of silent power through repetition and scale.

If companies building large AI systems are serious about improving healthcare, they should not be racing to aggregate more data or expand interpretive reach. They should engage with architectures and technologies that already prioritize enforceable consent, data-level governance, provenance, and patient-controlled use. Without those foundations, intelligence becomes the least important part of the system.

Health data does not need to be centralized to be helpful. It needs to remain governable by the person it represents. Until that principle is treated as a design requirement rather than a policy aspiration, tools like this will continue to promise empowerment while quietly eroding the very agency they claim to support.

Article link: https://www.linkedin.com/pulse/chatgpt-health-terrible-idea-katalin-bártfai-walcott-dchzc?