TECHNOLOGY, DATA AND ARTIFICIAL INTELLIGENCE

Every shift you work, you hand your judgment to machines dozens of times without noticing: the scanner that confirms a medication, the sepsis alert that fires at three in the morning, the dispensing cabinet that lets you override it, the phone in your pocket logging where you stood while you did all of it. None of those tools arrived with a philosophy attached, yet each encodes decisions somebody made about what matters, who counts and what may be traded away. This chapter takes those decisions apart. It looks at what privacy actually protects and why consent forms so rarely deliver it, at how a well-intentioned algorithm steered care away from the patients who needed it most, and at who is left holding the blame when an automated system kills someone. The aim is not fear of technology. The aim is the vocabulary, the cases and the questions that let you argue about it competently.

Section 1Why Technology Is Never Neutral

The tool that has already chosen for you

Historian Melvin Kranzberg opened his 1985 presidential address to the Society for the History of Technology, published the following year as “Kranzberg’s Laws”, with the line that has organised the field ever since: “Technology is neither good nor bad; nor is it neutral.” The first clause disarms the technophobe, the second disarms the salesman. A device carries no moral guilt, but it does carry a shape, and that shape makes some actions easy, some laborious and some practically impossible. The psychologist James Gibson called this an affordance; Donald Norman brought the term into design. A flat plate on a door affords pushing, a handle affords pulling, and if you have ever hauled on a push door you have felt a designer’s decision act on your body.

Now apply that to the electronic health record. A required field forces documentation; an optional one invites blank space. A drop-down offering two choices for sex erases every intersex and many transgender patients from the dataset before a single analysis is run. A name field capped at fifteen characters turns a patient’s identity into a truncation error. A default that copies yesterday’s note forward saves you four minutes and propagates yesterday’s mistake for a week. None of this appears in a policy manual, and all of it governs behaviour more reliably than a policy manual does.

Automotive safety makes the point in blood. Crash-test dummies were built to the dimensions of a fiftieth-percentile male; a fifth-percentile female dummy did not enter United States frontal compliance testing until the advanced-airbag rules phased in from 2003, and the government’s consumer star ratings did not add one until the 2011 model year, seated as the passenger while a male dummy drove. Analysing real crashes, Dipan Bose and colleagues reported in the American Journal of Public Health in 2011 that a belted female driver faced roughly 47 percent higher odds of serious injury than a belted male driver in a comparable collision. Nobody set out to injure women. A default body was chosen, the default became the standard, and the standard became a distribution of harm.

Policy vacuums and the Collingridge trap

In 1985 the philosopher James Moor asked what makes computer ethics distinctive and answered that computers are logically malleable. A hammer does what hammers do; a computer does whatever it can be shaped to do, so it keeps generating activities for which no rule yet exists. Moor named the result a policy vacuum, and observed that filling one usually requires first clearing up a conceptual muddle. Is a synthetic video of your face saying words you never said defamation, theft of your likeness, a species of forgery, or none of these? Until we settle what the thing is, no rule about it can be drafted well.

Waiting for clarity carries a price, described by David Collingridge in 1980 and now bearing his name. Early in a technology’s life you could still steer it, but you do not yet know what it will do; by the time the effects are undeniable, the thing is welded into supply chains, habits, budgets and law, and steering has become ruinously expensive. That is the Collingridge dilemma, and social media is its monument — the design choices that made feeds compulsive were cheap to reverse in 2010 and are close to immovable now. Large language models compressed the same arc into about three years.

Notice what this does to the familiar demand for evidence before regulation. In fast-moving technology, the demand is not neutral: it systematically favours deployment, because harm data can only be generated by deploying. Reasonable people disagree about the remedy. Precautionary approaches put the burden of proof on the developer; proactionary ones argue that delay has victims too, and that the deaths prevented by a diagnostic tool held back for five years are just as real for being invisible.

Dual use, defaults and the limits of “I only built it”

Most powerful technologies are dual use. The chemistry behind organophosphate pesticides produced nerve agents; gene editing that cures sickle-cell disease can edit a germline; and a model trained to design useful molecules can be inverted. Fabio Urbina and colleagues showed exactly that in Nature Machine Intelligence in 2022, flipping the reward function of a drug-discovery model and generating some 40,000 candidate toxic molecules in under six hours. The builder’s standard defence — that the tool is neutral and misuse belongs to the user — holds only if the misuse was unforeseeable and no cheap safeguard existed. Both conditions usually fail.

The constructive alternative is value-sensitive design, developed by Batya Friedman and colleagues from the mid-1990s, which treats human values as design requirements investigated empirically alongside speed and cost, rather than as a compliance box added at the end. Its simplest lever is the default. Eric Johnson and Daniel Goldstein reported in Science in 2003 that European countries with presumed-consent organ donation ran effective consent rates near 100 percent while comparable opt-in countries ran far lower — Austria at almost 100 percent against Germany at 12, separated by more than eighty percentage points. The citizens did not differ. The checkbox did.

Section 2Privacy, Data and the Myth of Consent

Three words people use interchangeably and should not

Privacy is a person’s interest in controlling the flow of information about themselves. Confidentiality is the duty you assume once information has been entrusted to you. Security is the set of technical and physical controls that keep information from leaking. They come apart constantly. A hospital can have flawless security and still violate privacy by licensing a de-identified dataset patients never imagined existed. A nurse can breach confidentiality over coffee with no security failure whatever. Naming which of the three is at stake is usually the first honest step in any argument about data.

The legal idea is younger than it feels. Samuel Warren and Louis Brandeis published “The Right to Privacy” in the Harvard Law Review in 1890, provoked by two new technologies: the portable snapshot camera and a gossip press with the means to circulate its pictures. Their argument was that law already protected property and reputation and now had to protect what they called inviolate personality. Every privacy panic since has followed that template — a device makes cheap what used to be costly, and the social norms that depended on the cost quietly collapse.

Helen Nissenbaum turned this into the most useful working tool in the field. Her theory of contextual integrity, developed from 2004 and elaborated in Privacy in Context (2010), holds that information flows are governed by norms attached to the context in which they arise, not by a general rule of secrecy. You tell your nurse how much you drink; that flow is appropriate. The same fact moving to your employer, your insurer or an advertising exchange violates the norm, even if you technically clicked something once. It explains why “but it was already public” is such a weak defence, and why aggregation can be a genuine harm when every ingredient was innocuous.

The consent that is not consent

The dominant model in commercial technology is notice-and-consent: publish a policy, capture a click, treat everything downstream as authorised. It does not work, and the failure is measurable. Aleecia McDonald and Lorrie Cranor calculated in 2008 that reading the privacy policies a typical American encountered in a year would consume roughly 200 hours. Comprehension is worse than the time cost implies, and the interface is often engineered against the reader. Dark patterns — a term coined by the designer Harry Brignull in 2010 — cover the confirm-shaming, the pre-ticked boxes and the cookie banner where accepting everything takes one click and refusing takes eleven.

Even sincere consent erodes through secondary use: data gathered for one purpose migrating to another. The Havasupai Tribe of the Grand Canyon gave blood to Arizona State University researchers in 1990 for a study of their extraordinary rates of diabetes. The samples were later used for work on schizophrenia, inbreeding and population migration, the last of which directly contradicted the tribe’s account of its own origins. The lawsuit was filed in 2004 and settled in April 2010 for 700,000 dollars, the return of remaining samples and an apology. No hacker was involved. Forms had been signed. The context had been shredded.

Institutions repeat the pattern at scale. Between 2015 and 2016 the Royal Free NHS Trust in London transferred identifiable records on about 1.6 million patients to Google DeepMind to test a kidney-injury alerting app; in July 2017 the UK Information Commissioner’s Office ruled that the transfer breached data protection law because patients would not reasonably have expected it. In 2019 the Wall Street Journal revealed Project Nightingale, a partnership moving records on roughly 50 million Ascension patients to Google. That arrangement was very probably lawful under HIPAA’s business-associate provisions. It detonated public trust anyway, which is exactly what contextual integrity predicts: lawful and appropriate are two different tests.

De-identification, re-identification and the HIPAA gap

HIPAA offers two routes to de-identified data: Safe Harbor, which strips eighteen listed identifiers, and Expert Determination, in which a qualified statistician certifies that re-identification risk is very small. Neither makes data anonymous in the ordinary sense of the word. In 1997 Latanya Sweeney, then a graduate student, took a Massachusetts dataset of state-employee hospital visits stripped of names and matched it against a Cambridge voter roll she bought for twenty dollars, extracting the health records of Governor William Weld and mailing them to his office. Her later estimate that 87 percent of Americans are uniquely identified by five-digit ZIP code, date of birth and sex was recomputed by Philippe Golle in 2006 at closer to 63 percent, and the epidemiologist Daniel Barth-Jones has argued that the Weld demonstration was easier than a typical case because a governor’s collapse was front-page news. The corrections lower the number. They do not rescue the assumption that deleting identifiers makes a dataset anonymous.

Two other demonstrations settled the matter. In 2006 AOL published 20 million search queries from 650,000 supposedly anonymous users; the New York Times identified user 4417749 as Thelma Arnold of Lilburn, Georgia within days, and AOL’s chief technology officer resigned. In 2008 Arvind Narayanan and Vitaly Shmatikov de-anonymised Netflix Prize ratings by cross-matching them with public IMDb reviews, exposing viewing histories with political and sexual implications; Netflix cancelled its planned sequel competition in 2010. Sparse, high-dimensional records of human behaviour function as fingerprints, and deleting the name barely smudges them.

Genetic data is harder still, because it is not only about you. Investigators identified Joseph James DeAngelo as the Golden State Killer in April 2018 by uploading crime-scene DNA to the public genealogy site GEDmatch and working down from third cousins who had never heard of him. In October 2023 a credential-stuffing attack on 23andMe exposed data touching roughly 6.9 million profiles, and the company’s 2025 bankruptcy raised a question no consent form had contemplated: what becomes of millions of genomes when the custodian itself is an asset in a sale?

That right-hand column is where most people actually live, so enforcement has had to come from elsewhere. In February 2023 GoodRx paid a 1.5 million dollar penalty in the first action under the Federal Trade Commission’s Health Breach Notification Rule for sharing users’ medication and condition data with Facebook and Google; in March 2023 the therapy platform BetterHelp settled for 7.8 million dollars over similar disclosures. After Dobbs v. Jackson Women’s Health Organization in June 2022, those same uncovered categories — cycle tracking, location history, search queries — acquired criminal-law consequences in some states that nobody had priced in when they installed the app.

Section 3Bias in the Machine

How the bias gets in

A model is a compression of its training data, so it inherits whatever that data recorded, including the recording’s distortions. Harini Suresh and John Guttag separated the sources usefully. Historical bias means the world was unjust and the data faithfully captured it. Representation bias means some groups were undersampled. Measurement bias means the thing you could measure was not the thing you cared about. Aggregation bias means one model was forced onto populations that behave differently. Deployment bias means the tool was used for something it was never built to do. Each has a different fix, which is why “the algorithm is biased” is a symptom, not a diagnosis.

Representation bias has the cleanest illustration. In their 2018 Gender Shades study, Joy Buolamwini and Timnit Gebru tested three commercial gender-classification systems and found error rates of about 0.8 percent for lighter-skinned men and up to 34.7 percent for darker-skinned women. The systems were not built to fail; they were built on face datasets overwhelmingly composed of lighter-skinned men, then reported as accurate on an average that hid the gap. Consequences followed. Robert Williams was arrested outside his home in Detroit in January 2020 on a face-recognition match and held about thirty hours before the case collapsed; Porcha Woodruff, eight months pregnant, was arrested by the same department in February 2023. Detroit settled with Williams in June 2024 and agreed to new limits on how such matches may be used.

Measurement bias is subtler and, in health care, more dangerous. In October 2019 Ziad Obermeyer and colleagues published an analysis in Science of a widely used commercial algorithm that identified patients for extra care management. The vendor could not measure need, so it measured next year’s health-care spending as a stand-in. Because less money is historically spent on Black patients at the same level of illness, the model concluded they were healthier. At any given risk score Black patients had significantly more chronic conditions, and correcting the target variable raised the proportion of Black patients flagged for extra help from 17.7 percent to 46.5 percent. Race was not an input. The proxy variable did all the damage on its own.

Fairness has no single definition

In May 2016 ProPublica reporters Julia Angwin, Jeff Larson, Surya Mattu and Lauren Kirchner examined COMPAS, a recidivism risk score used in Broward County, Florida. Among defendants who did not go on to reoffend, Black defendants were roughly twice as likely as white defendants to have been labelled high risk (about 45 percent against 23 percent), while white defendants who did reoffend were far more often labelled low risk. Northpointe, the vendor, replied that the score was well calibrated: among those given the same score, the actual reoffence rate was about the same regardless of race, which is the property most people intuitively demand of a risk score.

Both were telling the truth. Work by Jon Kleinberg, Sendhil Mullainathan and Manish Raghavan and independently by Alexandra Chouldechova proved that when the base rates of the outcome differ between groups, a risk score cannot simultaneously be calibrated and produce equal false-positive and false-negative rates across those groups, except in degenerate cases. This is an impossibility theorem, not a bug. It means the choice among fairness criteria is a value judgment that engineering cannot settle, and it should be made openly, by people accountable for the consequences, and stated in writing.

Two legal concepts help keep the discussion grounded. Disparate treatment is deliberately using a protected characteristic. Disparate impact, recognised in United States employment law since Griggs v. Duke Power Co. in 1971, is a facially neutral practice that falls unequally and cannot be justified by business necessity. Nearly every algorithmic harm in this section is disparate impact. That matters practically: no one has to prove that anyone intended it.

When the instrument itself is biased

Some of the worst distortions predate machine learning and now feed it. In December 2020 Michael Sjoding and colleagues reported in the New England Journal of Medicine that occult hypoxaemia — true arterial saturation under 88 percent despite a pulse oximeter reading of 92 to 96 percent — affected 11.7 percent of Black patients whose oximeter read in that reassuring range, against 3.6 percent of white patients — roughly a threefold difference caused by how skin pigment absorbs light. During the COVID-19 pandemic that gap plausibly delayed oxygen, admission and treatment eligibility for the patients most likely to need them. The Food and Drug Administration issued a safety communication in 2021 and has been revising its performance expectations since.

Similar reckonings followed elsewhere. In September 2021 a joint task force of the National Kidney Foundation and the American Society of Nephrology recommended a new estimated glomerular filtration rate equation with the race coefficient removed, because the old adjustment systematically reported better kidney function for Black patients and delayed referral and transplant listing. Race-based correction factors in spirometry have come under the same scrutiny. If a biased instrument supplies the labels, no amount of modelling sophistication downstream will produce an unbiased model — it will produce a confident one.

Government-scale examples show what happens when nobody audits at all. In the Netherlands, the tax authority’s childcare-benefit fraud system flagged families using indicators including dual nationality; roughly 26,000 families were wrongly accused, ordered to repay tens of thousands of euros, driven into bankruptcy and, in more than a thousand cases, separated from their children. The Dutch data protection authority fined the tax authority 2.75 million euros in 2021 for unlawfully processing nationality data, and the Rutte cabinet resigned in January 2021. In Australia, the Robodebt scheme ran from 2016 and calculated welfare debts by averaging annual tax income across fortnights, an arithmetic that manufactured debts for people with irregular work. About 470,000 debts were unlawful, a settlement worth roughly 1.8 billion Australian dollars followed, and a Royal Commission reported in July 2023. In both, the automation did not invent the cruelty; it industrialised it and stripped out the human who might have said this cannot be right.

Section 4Machines That Decide, and the People Who Answer for Them

The responsibility gap and the problem of many hands

Helen Nissenbaum carried Dennis Thompson’s problem of many hands into computing in 1996: when a system is built by dozens of teams over years, every contributor can truthfully say their part worked as specified while the whole kills someone. Andreas Matthias added the responsibility gap in 2004 — as systems learn from data after release, the manufacturer no longer fully controls behaviour and the operator does not fully understand it, so there is a real risk that nobody is properly answerable. Neither is an excuse. Both are a warning that responsibility must be assigned in advance by design and contract, because it will not assemble itself after the fact.

The canonical software case is the Therac-25, a radiation therapy machine that between June 1985 and January 1987 delivered massive overdoses to six patients in the United States and Canada, at least three of whom died. Nancy Leveson and Clark Turner’s 1993 analysis found a race condition triggered when an experienced operator edited the treatment screen quickly, compounded by the removal of the hardware interlocks that had silently caught the same software fault in the earlier Therac-20, by a cryptic error message operators had learned to bypass, and by a manufacturer who initially insisted an overdose was impossible. Every element there is organisational as much as technical.

The Boeing 737 MAX repeated it at scale. The Maneuvering Characteristics Augmentation System pushed the nose down using input from a single angle-of-attack sensor, was not described in flight manuals, and after a faulty sensor reading, contributed to the loss of Lion Air flight 610 on 29 October 2018 and Ethiopian Airlines flight 302 on 10 March 2019 — 346 deaths. The design pressure was commercial: avoid a new type rating so airlines would not have to retrain crews. Regulators had delegated much of the certification work to the manufacturer. No single engineer chose this outcome, which is exactly the point.

Then there is where the blame actually lands. When an Uber test vehicle struck and killed Elaine Herzberg in Tempe, Arizona on 18 March 2018, the National Transportation Safety Board found the perception system had cycled through classifying her as a vehicle, a bicycle and an unknown object, that the software had no concept of a pedestrian crossing outside a crosswalk, and that Uber had disabled the factory automatic emergency braking. The safety driver, streaming a television show, was prosecuted; the company was not charged. The researcher Madeleine Clare Elish named this the moral crumple zone: like the crumple zone of a car, the human operator absorbs the impact of a systemic failure, protecting the institution that built the system.

Automation bias, alert fatigue and the tired human

Automation bias is the well-documented tendency to over-trust an automated cue. Linda Skitka and colleagues showed in the late 1990s that it produces two distinct failures: errors of commission, where a person follows a wrong recommendation against good contrary evidence, and errors of omission, where a person misses a problem because the system did not flag it. Both increase when workload is high, when the automation is usually right, and when the alternative is doing the work yourself at the end of a twelve-hour shift.

Its mirror image is alert fatigue. The Joint Commission’s Sentinel Event Alert 50, issued in April 2013, reported 98 alarm-related events over roughly three and a half years, 80 of them deaths, and noted the underlying driver: depending on setting, something like 85 to 99 percent of clinical alarms do not require intervention. A clinician who silences ninety-five false alarms has been trained by the system to silence the ninety-sixth. This is not a character flaw to be corrected by exhortation; it is a design failure, and it responds to design fixes such as tuned thresholds, tiered escalation, actionable-only alerting and default settings owned by a committee rather than by the vendor.

The evidence that vendor claims need local checking is now unambiguous. In June 2021 Andrew Wong and colleagues published an external validation of the widely deployed Epic Sepsis Model across nearly 40,000 hospitalisations in JAMA Internal Medicine. The model achieved an area under the curve of 0.63 against the 0.76 to 0.83 the vendor reported, missed about two thirds of patients with sepsis, and generated alerts on 18 percent of all hospitalised patients. Deployed nationally, that is a machine manufacturing alert fatigue while under-detecting the condition it exists to catch. Ironically, as Lisanne Bainbridge argued in her 1983 essay on the ironies of automation, the better automation becomes, the rustier and less prepared its human supervisor is on the rare occasion it fails.

Opacity, confabulation and the limits of explanation

People say “black box” loosely. Jenna Burrell distinguished three kinds of opacity in 2016: deliberate corporate secrecy, technical illiteracy on the part of the user, and a genuine mismatch between how high-dimensional statistical models operate and how humans reason. Only the first is straightforwardly a moral fault, but all three block accountability. Cynthia Rudin has argued forcefully since 2019 that for high-stakes decisions we should stop trying to explain opaque models after the fact — post-hoc explanations can be plausible and unfaithful at once — and instead build models that are interpretable to begin with, since the accuracy penalty on structured data is often negligible. Not everyone agrees, and the debate is live.

Generative systems add a distinct failure mode. A large language model predicts fluent continuations; it has no internal representation of what is true, so it produces confabulations — assertions delivered with the same confident register as its correct output. In June 2023 a federal judge in New York sanctioned two attorneys who filed a brief in Mata v. Avianca citing six judicial decisions that did not exist, generated by ChatGPT and then, when challenged, confirmed by asking the same tool whether they were real. In October 2024 the Associated Press reported that a widely used speech-recognition model was inserting fabricated sentences into transcripts, including in medical settings. Fluency is not accuracy, and in health care the difference is measured in patients.

Mental-health applications have supplied the sharpest cautionary tales. In January 2023 the peer-support platform Koko disclosed that roughly 4,000 users had received replies composed with GPT-3. Its co-founder maintained that the feature was opt-in for the volunteers writing those replies and that the exercise was product testing rather than human-subjects research; critics answered that the people receiving the messages had agreed to nothing of the sort. Whether it required ethics review is genuinely disputed. That the recipients were a vulnerable population is not. In June 2023 the National Eating Disorders Association suspended its chatbot Tessa after it gave calorie-restriction and weight-loss advice to people seeking help for eating disorders. Neither was malicious. Both illustrate deployment bias: a tool that behaves acceptably in general is not thereby safe for the specific people it will meet.

Section 5Governing the Machine: Rules, Institutions and You

The rules that now exist

The European Union’s General Data Protection Regulation, applicable since 25 May 2018, did something more interesting than authorise large fines. It codified principles that translate directly into ethical practice: lawfulness and transparency, purpose limitation (data collected for one stated purpose may not drift into another), data minimisation (collect what you need, not what you can), accuracy, storage limitation, integrity and confidentiality, and accountability, meaning you must be able to demonstrate compliance rather than assert it. Article 22 restricts decisions producing legal or similarly significant effects made solely by automated processing. Whether it also creates an enforceable right to an explanation is genuinely contested — Sandra Wachter, Brent Mittelstadt and Luciano Floridi argued in 2017 that the binding text supports only a limited right to information, while Andrew Selbst and Julia Powles argued the opposite. The disagreement is a live scholarly one, not a settled fact.

The EU AI Act, in force since 1 August 2024, is the first broad statute to regulate AI by risk tier. A small set of practices is prohibited outright, including social scoring by public or private actors, untargeted scraping of facial images to build recognition databases, and emotion inference in workplaces and schools. A larger high-risk tier — covering medical devices, employment, education, credit and essential services — carries obligations for risk management, data governance, logging, human oversight and post-market monitoring. Limited-risk systems owe transparency; everything else is unregulated. The prohibitions applied from 2 February 2025 and general-purpose model obligations from 2 August 2025. The high-risk duties have already slipped: a simplification package agreed in 2026 moved stand-alone high-risk systems to 2 December 2027 and high-risk AI embedded in regulated products to 2 August 2028. Check the current text before relying on any date.

In the United States the picture is a patchwork. The Food and Drug Administration has authorised well over a thousand AI-enabled medical devices, most through the 510(k) pathway. The landmark took a different route: the De Novo authorisation of IDx-DR in April 2018, the first system permitted to make a screening diagnosis of diabetic retinopathy without a physician reading the image. The agency’s 2024 guidance on predetermined change control plans finally addressed the awkward fact that a learning model changes after clearance. The National Institute of Standards and Technology published its voluntary AI Risk Management Framework in January 2023 around four functions: govern, map, measure and manage. Illinois has policed biometrics aggressively since its 2008 Biometric Information Privacy Act, which produced a 650 million dollar Facebook settlement approved in 2021 and a 2022 settlement sharply restricting Clearview AI. Colorado passed the first broad state law on algorithmic discrimination in 2024, postponed its start date twice, then in May 2026 repealed and replaced it with a narrower transparency-and-disclosure regime for automated decision-making that takes effect in January 2027 — a fair measure of how unsettled this ground still is. The World Health Organization’s 2021 guidance on ethics and governance of AI for health remains the clearest short statement of principle: protect autonomy, promote wellbeing and safety, ensure transparency, foster accountability, ensure inclusiveness and equity, and promote sustainability.

What an institution owes before it turns the thing on

Documentation practices developed in research have become the practical backbone of institutional ethics. A model card, proposed by Margaret Mitchell and colleagues in 2019, states a model’s intended use, out-of-scope uses, training data, and performance disaggregated by subgroup. A datasheet for a dataset, proposed by Timnit Gebru and colleagues, records how data was collected, by whom, with what consent and for what purpose. An algorithmic impact assessment requires an organisation to state in advance who could be harmed, how the harm would be detected and what would trigger withdrawal. None of these is a guarantee. All of them convert vague optimism into claims someone can be held to.

Local validation is the non-negotiable one. The Epic sepsis case shows that a vendor’s reported performance is a claim about the vendor’s population, not yours, and models drift as coding practices, case mix and staffing change. Serious institutions validate on their own data before go-live, stratify results by race, primary language, insurance status, age and disability, publish thresholds and false-alarm rates to the clinicians who will live with them, monitor continuously, and name a person with authority to switch the system off. That last item is the tell. If nobody can stop it, nobody is governing it.

What you owe, on your own shift

Professional codes already cover most of this if you read them as written. The American Nurses Association’s Code of Ethics for Nurses makes protecting privacy and confidentiality a duty of the nurse, not a courtesy of the institution, and obliges you to act when practice conditions endanger patients. The Association for Computing Machinery’s 2018 code obliges computing professionals to avoid harm, to be honest about system limitations and to take special care with systems that become integrated into infrastructure. The American Medical Association has deliberately used the phrase augmented intelligence since 2018 to insist the tool assists rather than replaces clinical judgment. These converge on a short list of concrete duties.

Do not access a record you have no clinical reason to open; curiosity about a celebrity, a neighbour or an ex-partner has ended many careers, and audit logs are read routinely. Do not post about patients, and understand that removing the name does not de-identify a story containing a rare diagnosis, a date and a room layout; four nurses at an Atlanta hospital lost their jobs in December 2022 over one sixty-second video that named no patient at all. Do not paste identifiable patient information into a consumer chatbot, which is a disclosure to a third party under most institutional policies and possibly a reportable breach. Verify anything AI-generated before it enters a record, and document what the tool contributed so the next clinician can weigh it. When you override a system, write down why, because that note is the only evidence that a human was actually in the loop.

When something is wrong, escalate in order: the colleague, the charge nurse or manager, the safety-event reporting system, the compliance or privacy officer, the ethics committee. Do this in writing, and keep dates. External disclosure is a genuine option but a last one, ethically defensible when the harm is serious, internal routes have failed or are futile, and you have evidence rather than suspicion — the pattern Frances Haugen followed in 2021 when she took internal Facebook research to regulators and Congress after concluding the company would not act. Protection for whistleblowers is real in some jurisdictions and thin in others, so knowing your actual legal position before you act is part of acting responsibly.

Words worth knowing