The Problem Is Already Here

71% of healthcare workers using personal AI accounts for work. 81% of data policy violations in healthcare involve regulated data including PHI. Consumer tools (ChatGPT, Gemini) do not sign BAAs and are not HIPAA compliant. Primary evidence for shadow use claim — EMS is a covered entity; findings apply directly.

EMS1 actively encouraged consumer LLM use by paramedics for operational tasks as early as February 2023 with no HIPAA or data governance guidance — validating that the shadow use dynamic was normalized in trade media before any policy infrastructure existed.

Fire service trade media discussing AI governance need (NIST AI RMF, DHS principles) while acknowledging AI policies are necessary to "ensure you are not violating federal law, betraying public trust, or committing a crime by sharing sensitive information." Written by a practitioner, not a technology advocate.

DHS S&T confirms AI for public safety is still in requirements-gathering and pilot phases. First responders "do not want to turn it all over to AI yet." Law enforcement concerns specifically named: deepfakes, swatting/false calls, AI-assisted targeting. Current AI pilots supplement — not replace — human judgment.

Since April 2025, malicious actors have used AI-generated voice messages to impersonate senior U.S. officials. Technique combines smishing and vishing. Once accounts are compromised, scammers target contacts in cascade — directly applicable to incident commander and medical director impersonation risk.

Names three adversarial AI threat vectors specific to 911 systems: (1) Swatting via AI-generated ghost calls to draw resources and create coverage gaps; (2) Data poisoning of dispatch AI training data to deprioritize call types; (3) CAD ransomware — cites Change Healthcare Feb 2024 attack as analogous documented incident.

Peer-reviewed analysis arguing AI-driven synthetic voice presents a more immediate threat than deepfake video. Accessible tools now mimic speech with precision and minimal resources. Examines role of agentic AI systems that blur distinctions between human and synthetic callers — directly relevant to 911 and PSAP threat landscape.

One deepfake attack occurred every five minutes in 2024. 49% of companies surveyed experienced audio/video deepfake fraud. 70% of people unable to distinguish cloned voices. Vishing attempts in Netherlands tripled in 2024. Over 7,500 fraudulent calls intercepted in one Europol operation, preventing €10M+ in losses.

LLM developers and vendors become HIPAA business associates when processing PHI on behalf of covered entities. Analyzes FTC enforcement actions against health AI companies (GoodRx, BetterHelp). Documents 8 FTC complaint categories including HIPAA compliance misrepresentation — establishes organizational liability exposure from shadow use.

"The current free version of ChatGPT does not support (nor does it intend to support) services covered under HIPAA through accessing PHI." Covered entities must enter BAAs before implementing any technology potentially accessing patient data. Three deidentification pathways required under HIPAA for compliant AI use.

Five documented HIPAA violation categories from improper AI use: uploading PHI to unsecured platforms; chatbot unauthorized data sharing; violations of the minimum necessary standard; inadequate risk assessments; medical device data exposure. 66% of physicians reported AI use in 2025 (vs 38% in 2023).

"Generic ChatGPT services are not HIPAA compliant and cannot be used in a HIPAA-compliant manner." Most ChatGPT services cannot support HIPAA-standard access controls, activity logs, or audit trails. Consumer services may use inputs to improve model accuracy unless users opt out or subscribe to paid tier. Updated January 2026 to reflect current product status including ChatGPT for Healthcare.

Foundational systematic review of 74 studies across aviation, transport, and healthcare. Automation bias documented in 6–11% of cases as negative consultations: correct pre-AI decisions changed to incorrect ones following AI recommendations. Workload, time constraint, and task complexity amplify automation bias. The standard reference for automation bias definition in all subsequent research in this evidence base.

Quantitative intervention study (n=210). Non-specialists are most susceptible to automation bias — precisely those who stand to gain most from AI-decision support. Higher perceived benefit of the AI system significantly associated with promoting false agreement. Directly applicable: first responders using AI tools without formal AI literacy training face the highest automation bias risk.

PRISMA 2020 review of 35 peer-reviewed studies (2015–2025). XAI (explainable AI) approaches may both mitigate and exacerbate automation bias — overly technical or overly simplified explanations may inadvertently reinforce misplaced trust among less experienced professionals. "User engagement emerges as the most feasible and impactful point of intervention." Cites Goddard et al. 2012 as foundational reference.

Bowtie analysis of automation bias in AI-driven clinical decision support. Proposes preventive measures during the AI model design phase and mitigation strategies post-deployment. Conclusion: a systems approach integrating technological advancements, regulatory frameworks, and collaborative efforts between AI developers and healthcare practitioners is imperative.

"By and large, when it comes to public agency use of AI, these systems remain untested in real-world conditions due to challenges ranging from a lack of consensus standards for evaluation to a lack of agency capacity to conduct testing. Yet state and local public safety agencies are using AI systems now." Most directly relevant source for the procurement gap framing.

IACP panelists agreed "overarching federal regulation is lacking." Most AI tools used in law enforcement are developed by third-party vendors "with limited visibility into how they function." Zoufal: "You're not buying AI — you're buying a product that has AI in it or will have AI in the future." Generative AI systems "remain largely untested and poorly understood."

Public entities are deploying AI systems "at various administrative levels without robust due diligence, monitoring, or transparency." Critically maps procurement challenges necessitating "AI-specific procurement guidelines and processes." Conclusion: "AI-specific public procurement guidelines are urgently needed to protect fundamental rights and due process."

None of the required sector AI risk assessments fully addressed the six foundational risk assessment activities. None fully evaluated level of risk by combining magnitude of harm with probability. None fully mapped mitigation strategies to risks. DHS guidance template issued August 2024 still did not fully address likelihood of occurrence. GAO recommendation accepted by DHS. Highest-authority source in this evidence set.

DHS first-of-its-kind cross-sector AI risk guidelines per EO 14110. Three risk categories: (1) Attacks Using AI — using AI to automate or enhance attacks on critical infrastructure; (2) Attacks on AI — targeting AI systems supporting critical infrastructure; (3) AI Design and Implementation Failures — deficiencies leading to malfunctions or unintended consequences. Emergency response planning requirements explicitly name "emergency responders and law enforcement personnel." Compliance is advisory for state and local agencies — itself a gap Signal & Response addresses.