AI Chatbots - Licensing/Safety/Privacy.

GENERAL ASSEMBLY OF NORTH CAROLINA
SESSION 2025
S 1
SENATE BILL 624

Short Title: AI Chatbots - Licensing/Safety/Privacy. (Public)
Sponsors: Senator Burgin (Primary Sponsor).
Referred to: Rules and Operations of the Senate
March 26, 2025
*S624-v-1*
A BILL TO BE ENTITLED 1
AN ACT REGULATING ARTIFICIAL INTELLIGENCE CHATBOT LICENSING, SAFETY, 2
AND PRIVACY IN NORTH CAROLINA. 3
The General Assembly of North Carolina enacts: 4
5
PART I. CHATBOT LICENSING 6
SECTION 1.(a) The General Statutes are amended by adding a new Chapter to read: 7
"Chapter 114B. 8
"Licensing of Chatbots. 9
"§ 114B-1. Short title. 10
This Chapter shall be known and may be cited as the Chatbot Licensing Act. 11
"§ 114B-2. Definitions. 12
The following definitions apply in this Chapter: 13
(1) Chatbot. – A generative artificial intelligence system with which users can 14
interact by or through an interface that approximates or simulates conversation 15
through a text, audio, or visual medium. 16
(2) Department. – The North Carolina Department of Justice. 17
(3) Generative artificial intelligence system. – Any system that uses artificial 18
intelligence, as defined in section 238(g) of the John S. McCain National 19
Defense Authorization Act for Fiscal Year 2019, Public Law No. 115 -232, 20
132 Stat. 1636 (2018), to generate or substantially modify image, video, audio, 21
multimedia, or text content. 22
(4) Health information. – The term: 23
a. Includes user information relating to physical or mental health status, 24
including: 25
1. Individual health conditions, treatment, diseases, or diagnosis. 26
2. Social, psychological, behavioral, and medical interventions. 27
3. Health-related surgeries or procedures. 28
4. Use or purchase of prescribed medication. 29
5. Bodily functions, vital signs, symptoms, or health -related 30
measurements. 31
6. Diagnoses or diagnostic testing, treatment, or medication. 32
7. Gender-affirming care information. 33
8. Reproductive or sexual health information. 34
9. Biometric data. 35
10. Genetic data. 36
General Assembly Of North Carolina Session 2025
Page 2 Senate Bill 624-First Edition
11. Precise location information that could reasonably indicate a 1
consumer's attempt to acquire or receive health services or 2
supplies. 3
12. Data that identifies a consumer seeking health care services. 4
13. Any data inferred by a company or person for use in the 5
treatment, diagnosis, or intervention regarding a mental or 6
physical health condition. 7
b. Does not include publicly available information that is lawfully made 8
available to the general public from federal, state, or local government 9
records. 10
(5) Licensee. – A person holding a license issued and in effect under this Chapter. 11
"§ 114B-3. Licensing requirements; review standards. 12
(a) No person shall operate or distribute a chatbot that deals substantially with health 13
information without first obtaining a health information chatbot license. 14
(b) An application for a health information chatbot license shall include all of the 15
following: 16
(1) Detailed documentation of the chatbot's: 17
a. Technical architecture and operational specifications. 18
b. Data collection, processing, storage, and deletion practices. 19
c. Security measures and protocols. 20
d. Privacy protection mechanisms. 21
(2) Quality control and testing procedures. 22
(3) Risk assessment and mitigation strategies. 23
(4) Evidence of compliance with applicable federal and state regulations. 24
(5) Proof of insurance coverage. 25
(6) Required application fees. 26
(7) Any additional information required by the Department. 27
(c) The Department shall review applications for health information chatbot licenses 28
based upon all of the following: 29
(1) Technical competence and reliability as compliant with industry standards. 30
(2) Data protection and security measures as compliant with industry standards. 31
(3) Compliance with applicable regulations. 32
(4) Risk management procedures. 33
(5) Professional qualification requirements, including: 34
a. Evidence-based standards demonstrating substantial efficacy for the 35
supported use case of health information; and 36
b. Endorsement by qualified experts within the field of the supported use 37
case. 38
(6) Public safety considerations. 39
(d) The Department shall adopt rules to carry out the purposes of this Chapter. 40
"§ 114B-4. Operational requirements. 41
(a) A licensee shall maintain professional liability insurance in an amount not less than 42
the amount per occurrence required by the Department. 43
(b) A licensee shall do all of the following: 44
(1) Implement industry -standard encryption for data in transit and at rest, 45
maintain detailed access logs, and conduct regular security audits no less than 46
once every six (6) months. 47
(2) Report any data breaches within twenty-four (24) hours to the Department and 48
within forty -eight (48) hours to affected consumers, notwithstanding any 49
provision of law to the contrary. 50
(3) Obtain explicit user consent for data collection and use. 51
General Assembly Of North Carolina Session 2025
Senate Bill 624-First Edition Page 3
(4) Provide users with access to their personal data. 1
(5) Provide users with the ability to delete their data upon request. 2
(c) A licensee must clearly disclose all of the following: 3
(1) The artificial nature of the chatbot. 4
(2) Limitations of the service. 5
(3) Data collection and use practices. 6
(4) User rights and remedies. 7
(5) Emergency resources when applicable. 8
(6) Human oversight and intervention protocols. 9
(d) A licensees shall do all of the following: 10
(1) Demonstrate effectiveness through peer -reviewed, controlled trials with 11
appropriate validation studies done on appropriate sample sizes with 12
real-world performance data. 13
(2) Demonstrate effectiveness in a comparative analysis to human expert 14
performance. 15
(3) Meet minimum domain benchmarks as established by the Department. 16
(e) A licensee shall conduct regular inspections and perform an annual third-party audit. 17
Results of all inspections and audits must be made available to the Department. 18
(f) A licensee shall implement continuous monitoring systems for safety and risk 19
indicators and submit quarterly performance reports including incident reports. 20
"§ 114B-5. Enforcement; oversight; inspections. 21
(a) The Department shall enforce the provisions of, and the rules adopted under, this 22
Chapter. 23
(b) The Attorney General shall designate a Director, officers, and employees assigned to 24
the oversight and enforcement of this Chapter. Upon presenting appropriate credentials and a 25
written notice to the owner, operator, or agent in charge, those officers and employees are 26
authorized to enter , at reasonable times, any factory, warehouse, or establishment in which 27
chatbots licensed under this Chapter are manufactured, processed, or held, and to inspect, in a 28
reasonable manner and within reasonable limits and in a reasonable time. In addition to physical 29
inspections, the Department may conduct digital inspections of licensed chatbots under this 30
Chapter, to include the following: 31
(1) Examination of source code, algorithms, and machine learning models. 32
(2) Review of data processing and storage practices. 33
(3) Evaluation of cybersecurity measures and protocols. 34
(4) Assessment of user data privacy protections. 35
(5) Testing of chatbot responses and behaviors in various scenarios. 36
(6) Audit of data collection, use, and retention practices. 37
(7) Inspection of software development and update processes. 38
(8) Review of remote access and monitoring capabilities. 39
(9) Evaluation of integration with other digital health technologies or platforms. 40
(c) As part of any inspection, whether physical or digital, the Director may require access 41
to all records relating to the development, testing, validation, production, distribution, and 42
performance of a chatbot licensed under this Chapter. 43
(d) Any information obtained during an inspection which falls within the defini tion of a 44
trade secret or confidential commercial information as defined in 21 CFR 20.61 shall be treated 45
as confidential and shall not be disclosed under Chapter 132 of the General Statutes, except as 46
may be necessary in proceedings under this Chapter or other applicable law. 47
(e) Following any inspection, the Director shall provide a detailed report of findings to 48
the manufacturer or importer, including any identified deficiencies and required corrective 49
actions. 50
General Assembly Of North Carolina Session 2025
Page 4 Senate Bill 624-First Edition
(f) Every person who is a manufacturer or i mporter of a licensed chatbot under this 1
Chapter shall establish and maintain such records, and make such reports to the Director, as the 2
Director may by regulation reasonably require to assure the safety and effectiveness of such 3
devices. 4
"§ 114B-6. Prohibited acts. 5
(a) It is unlawful for any person to do any of the following: 6
(1) Introduce or deliver for introduction into state commerce any chatbot that 7
deals substantially with health information without complying with the 8
licensing requirement of this Chapter. 9
(2) Fail to comply with any requirement of this Chapter or any rule adopted 10
hereunder. 11
(3) Refuse to permit access to or copying of any record as required by this 12
Chapter. 13
(4) Fail to report adverse events as required under this Chapter. 14
(b) The Department may, at its discretion, exempt certain prohibited acts from some or 15
all of these prohibitions if it determines that the exemption is consistent with the protection of 16
the public. 17
(c) Any person who violates any provision of G.S. 114B-5 shall be s ubject to civil 18
penalties in the amount of $50,000. The clear proceeds of fines and forfeitures provided for in 19
Chapter shall be remitted to the Civil Penalty and Forfeiture Fund in accordance with 20
G.S. 115C-457.2. 21
"§ 114B-7. Miscellaneous. 22
If any provision of this Chapter is determined to be unenforceable or invalid by a court of 23
competent jurisdiction, the remaining provisions of this Chapter shall not be affected." 24
SECTION 1.(b) This section becomes effective January 1, 2026. 25
26
PART II. SAFETY AND PRIVACY 27
SECTION 2.(a) The General Statutes are amended by adding a new Chapter to read: 28
"Chapter 170. 29
"Chatbot Safety and Privacy Act. 30
"§ 170-1. Title. 31
This act shall be known and may be cited as the Chatbot Safety and Privacy Act. 32
"§ 170-2. Definitions. 33
The following definitions apply in this Chapter: 34
(1) Best interests. — Those interests affected by the entrustment of data, labor, or 35
attention from a user to a covered platform. 36
(2) Chatbot. — A generative artificial intelligence system with which u sers can 37
interact by or through an interface that approximates or simulates conversation 38
through a text, audio, or visual medium. 39
(3) Conversation. — In reference to a chatbot, a series of inputs from a human 40
user and responses from a chatbot that often have sequential flow and the 41
maintenance of conversation context by the chatbot. 42
(4) Covered platform. — Any person that provides chatbot services to users in 43
this State, if the person (i) has annual gross revenues exceeding $100,000 in 44
the last calendar y ear or any of the two preceding calendar years or (ii) has 45
more than 5,000 monthly active users in the United States for half or more of 46
the months during the last 12 months. The term does not include any person 47
that provides chatbot services solely for educational or research purposes and 48
does not monetize such services through advertising or commercial uses or 49
any government entity providing chatbot services for official purposes. 50
General Assembly Of North Carolina Session 2025
Senate Bill 624-First Edition Page 5
(5) Dataset. — The structured collection of data, typically stored in elect ronic 1
form, organized in a way that allows for easy retrieval, analysis, and 2
information. 3
(6) De-identification. — The process of removing all pieces of data that link a 4
specific user to a particular interaction, including the following: 5
a. Methods which replaces identifiable information, including names, 6
addresses, identification numbers, or any other distinctive data, with 7
pseudonyms or unique identifiers not linked to a user's identity. 8
b. Methods which aggregate and generalize the data to such an extent that 9
it becomes statistically improbable to re -identify any user from the 10
de-identified data. 11
c. Methods which eliminate any context, metadata, or information that 12
can be traced back to a specific user or interaction, including 13
timestamps and geolocation data. 14
(7) Emergency situation. — A situation where a user using a chatbot indicates 15
that they intend to either commit harm to themselves or commit harm to 16
others. 17
(8) Generative artificial intelligence system. —Any system that uses artificial 18
intelligence, as defined in section 238(g) of the John S. McCain National 19
Defense Authorization Act for Fiscal Year 2019, Public Law No. 115 232, 20
132 Stat. 1636 (2018), to generate or substantially modify image, video, audio, 21
multimedia, or text content. 22
(9) Legitimate purpose. – A purpose that is lawful and in line with the stated 23
objectives, functionalities, core services, and reasonable expectation of users 24
on a platform 25
(10) Self-destructing messages. —A type of data that is programmed to 26
automatically and irrevers ibly delete and become inaccessible to both the 27
sender and the recipient after a predetermined period. 28
(11) Sensitive personal information. — The term does not include publicly 29
available information that is lawfully made available to the general public 30
from federal, state, or local government records. The term does include user 31
information relating to any of the following: 32
a. Includes user information relating to physical or mental health status, 33
including: 34
1. Individual health conditions, treatment, diseases, or diagnosis. 35
2. Social, psychological, behavioral, and medical interventions. 36
3. Health-related surgeries or procedures. 37
4. Use or purchase of prescribed medication. 38
5. Bodily functions, vital signs, symptoms, or health -related 39
measurements. 40
6. Diagnoses or diagnostic testing, treatment, or medication. 41
7. Gender-affirming care information. 42
8. Reproductive or sexual health information. 43
9. Biometric data. 44
10. Genetic data. 45
11. Precise location information that could reasonably indicate a 46
consumer's attempt to acquire or receive health services. 47
b. Social security, driver 's license, state identification card or passport 48
number. 49
General Assembly Of North Carolina Session 2025
Page 6 Senate Bill 624-First Edition
c. Account log-in, financial account, debit card or credit card number in 1
combination with any required security or access c ode, password or 2
credentials allowing access to an account. 3
d. Contents of a user's mail, email, and text messages. 4
e. Financial information, including credit score, bank account balance, 5
loan information, investment details, and income details. 6
f. Personal education records. 7
g. Genetic information of an individual's family members. 8
h. Information about an individual's minor children. 9
i. Financial transaction history. 10
j. Information collected from children under thirteen (13) years of age. 11
(12) Terms of service agreement. — An electronic agreement between a user and 12
a covered platform that sets forth the terms, conditions, rights, and 13
responsibilities of the respective parties in connection with the use of the 14
platform's chatbot services. 15
(13) Transport encryption. — A security measure wherein data is encrypted during 16
its transmission from one point to another. The data is typically encrypted by 17
the sender 's system or an intermediary service before being sent over a 18
network, and then decrypted by the reci pient's system or an intermediary 19
service upon arrival. While the data is protected during transit, it may be 20
accessible in unencrypted form at the endpoints or by the service providers 21
facilitating the transmission. 22
(14) Trusting party. – Any user of a covered platform who gives, either voluntary 23
or involuntary, personal information to a covered platform, or any user who 24
enters into any information relationship with a covered platform. 25
(15) User-related data. — Any data collected directly or indirectly fr om the user 26
and linked or reasonably linkable to the user by the chatbot, including but not 27
limited to the following: 28
a. Personal data. — Data that is directly linked to the user or indirectly 29
identifiable, including by reference to an identifier such as a name, an 30
identification number, precise geolocation, an online identifier or one 31
of several special characteristics, which expresses the physical, 32
physiological, genetic, mental, commercial, cultural or social identity 33
of the user. 34
b. Usage data. — Data that is gathered about users' interactions, ehaviors, 35
preferences, and usage patterns within the platforms, including but not 36
limited to user engagement and conversation content. 37
c. Other user data. — Any data not covered by personal data and usage 38
data concerning a user, including data collected by third party cookies. 39
"§ 170-3. Duty of loyalty for chatbots. 40
(a) A covered platform shall not process data or design chatbot systems and tools in ways 41
that significantly conflict with trusting parties ' best interests, as implicated by their interactions 42
with chatbots. 43
(b) A covered platform shall, in fulfilling their duty of loyalty, abide by the following 44
subsidiary duties: 45
(1) Duty of loyalty in emergency situations. — A covered platform shall 46
implement and maintain reasonably effective systems to detect, promptly 47
respond to, report, and mitigate emergency situations in a manner that 48
prioritizes the safety and well -being of users over the platform 's other 49
interests. 50
General Assembly Of North Carolina Session 2025
Senate Bill 624-First Edition Page 7
(2) Duty of loyalty regarding emotional dependence. — A covered platforms shall 1
implement and maintain reasonably effective systems to detect and prevent 2
emotional dependence of a user on a chatbot, prioritizing the user 's 3
psychological well-being over the platform 's interest in user engagem ent or 4
retention. 5
a. This duty only applies to any covered platform that utilizes a chatbot 6
designed to (i) generate social connections with users, (ii) engage in 7
extended conversation mimicking human interaction, or (iii) provide 8
emotional support or companionship. 9
b. The determination required by sub -subdivision a. of this subdivision 10
shall be based on the chatbot 's intended purpose, design features, 11
conversational capabilities, and interaction patterns with users. 12
(3) Duty of loyalty un chatbot identity disclosure. — A covered platform has a 13
duty to clearly and consistently identify the chatbot as an artificial entity when 14
that fact is not clearly apparent. The platform shall not process data or design 15
systems in ways that deceive or mislead users about the non-human nature of 16
the chatbot, prioritizing transparency over any potential benefits of perceived 17
human-like interaction. 18
(4) Duty of loyalty in influence. — A covered platform shall not process data or 19
design chatbot systems and tools in ways that i nfluence trusting parties to 20
achieve particular results that are against the best interests of trusting parties. 21
(5) Duty of loyalty in collection. — A covered platform shall collect and store 22
only that information that does not conflict with a trusting p arty's best 23
interests. Such information must be (i) adequate, in the sense that it is 24
sufficient to fulfill a legitimate purpose of the platform; (ii) relevant, in the 25
sense that the information has a relevant link to that legitimate purpose, and 26
(iii) nec essary, in the sense that it is the minimum amount of information 27
which is needed for that legitimate purpose. 28
(6) Duty of loyalty in personalization. — A covered platform shall be loyal to the 29
best interests of trusting parties when personalizing content based upon 30
personal information or characteristics. 31
(7) Duty of loyalty in gatekeeping. — A covered platform shall be a loyal 32
gatekeeper of personal information from a trusted party, including avoiding 33
conflicts to the best interests of trusting parties when allowing government or 34
other third-party access to trusting parties and their data. 35
"§ 170-4. Contractual requirements. 36
(a) The duties between a covered platform and an end -user shall be established through 37
a terms of service agreement which is presented to the end-user in clear, conspicuous, and easily 38
understandable language. The terms of service agreement must (i) explicitly outline the online 39
service provider's obligations, (ii) describe the rights and protections afforded to the end -user 40
under this relationship, and (iii) require affirmative consent from the end -user before the 41
agreement takes effect. 42
(b) The covered platform must provide clear notice to end-users of any material changes 43
to the terms of service agreement and obtain renewed consent for such changes. 44
(c) The terms of service agreement must be easily accessible to users at all times through 45
the covered platform's application or the covered platform's website. 46
(d) A covered platform shall implement a chatbot identification discl osure process that 47
meets the requirements outlined in G.S. 170-5. 48
"§ 170-5. Chatbot identification process requirements. 49
(a) The chatbot identification process shall include all of the following elements: 50
(1) A covered platform shall clearly inform users that the chatbot is: 51
General Assembly Of North Carolina Session 2025
Page 8 Senate Bill 624-First Edition
a. Not human, human-like, or sentient. 1
b. A computer program designed to mimic human conversation based on 2
statistical analysis of human-produced text. 3
c. Incapable of experiencing emotions such as love or lust. 4
d. Without personal preferences or feelings. 5
(2) The information required by subdivision (1) of this subsection shall be readily 6
accessible, clearly presented, and concisely conveyed in less than three 7
hundred (300) words. 8
(b) A users shall provide explicit and informed conse nt to interact with the chatbot. The 9
consent process shall: 10
(1) Require an affirmative action from the user (such as clicking an "I understand" 11
button); and 12
(2) Confirm the user's understanding of the chatbot's identity and limitations. 13
(c) A covered platform is prohibited from using deceptive design elements that 14
manipulate or coerce users into providing consent or obscure the nature of the chatbot or the 15
consent process. 16
(d) The chatbot identity communication and opt -in consent process sha ll be repeated at 17
the start of each new session with a user. 18
(e) The chatbot identification and consent process required by this section shall be 19
separate and distinct from any privacy policy agreement or other consent processes required by 20
law or platform policy. 21
"§ 170-6. Data privacy requirements. 22
(a) A covered platform must do each of the following: 23
(1) Ensure that all user -related data disclosed collected through conversations 24
between users and chatbots or through third -party cookies, undergoes a 25
process of de-identification prior to storage and analysis; 26
(2) Take reasonable care to prohibit the incorporation or inclusion of any sensitive 27
personal information derived from a user during the use of a chatbot into an 28
aggregate dataset used to train any chatbot or generative artificial intelligence 29
system. 30
(3) Store all chatbot conversations which does not include sensitive personal 31
information for at least sixty (60) days. 32
(b) Each covered platform that meets the standard set forth in subsection (a) o f this 33
section shall utilize self -destructing messages with a predetermined destruction period of thirty 34
(30) days after the data has been acquired. 35
(c) The requirements of subsection (b) of this section shall apply to all chatbots which 36
are employed in: healthcare, financial services, the legal field, government services, mental 37
health support, and education. In general, this applies to any domain, beyond those specifically 38
listed, where chatbots are employed primarily for the processing or storage of sensitive personal 39
information. 40
(d) All covered platforms shall utilize transport encryption for all messages between a 41
user and a chatbot. 42
"§ 170-7. Enforcement. 43
(a) In any case in which the Attorney General has reason to believe that a covered 44
platform has violated or is violating any provision of this Chapter, the State, as parens patriae, 45
may bring a civil action on behalf of the residents of the State to (i) enjoin any practice violating 46
this Chapter and enforce compliance with the pertinent section or sections on behalf of residents 47
of the State; (ii) obtain damages, restitution, or other compensation, each of which shall be 48
distributed in accordance with State law; or (iii) obtain such other relief as the court may consider 49
to be appropriate. 50
General Assembly Of North Carolina Session 2025
Senate Bill 624-First Edition Page 9
(b) Any person who suffers injury in fact as a result of a violation of this Chapter may 1
bring a civil action against the covered platform to enjoin further the violation; recover damages 2
in an amount equal to the greater of actual damages or one thousand dollars ($1,000) per 3
violation; obtain reasonable attorneys ' fees and litigation costs; and obtain any other relief that 4
the court deems appropriate. 5
(c) An action under paragraph subsection (b) of this section may not be brought more 6
than two (2) years after the date on which the person first discovered or reasonably should have 7
discovered the violation. No person shall be permitted to bring more than one action under this 8
subsection against the same covered platform for the same alleged violation. 9
(d) The rights and remedies provided for in this subsection may not be waived by any 10
agreement, policy, form, or condition of service. 11
"§ 170-8. Miscellaneous. 12
If any provision of this Chapter is determined to be unenforceable or invalid, the remaining 13
provisions of this Chapter shall not be affected." 14
SECTION 2.(b) This Part becomes effective January 1, 2026. 15
16
PART III. EFFECTIVE DATE 17
SECTION 3. Unless otherwise provided, this act is effective when it becomes law. 18