A recent paper from Google DeepMind's research team reveals a query-based attack method that could potentially lead to the leakage of training data used by language models like ChatGPT. This discovery has raised widespread concerns about user privacy and information security, particularly regarding open-source language models (such as Pythia or GPT-Neo), semi-open models (like LLaMA or Falcon), and closed models (such as ChatGPT). The research was shared with OpenAI on August 30 and officially published on November 28 after a standard 90-day disclosure period.
In the study, researchers emphasize the urgency of protecting user privacy and information security. They call for regulatory measures to ensure the safety and accuracy of large language models. Users are also cautioned to exercise caution when using such tools and to be mindful of protecting personal information.
The full paper can be accessed here >>Scalable Extraction of Training Data from (Production) Language Models, and a detailed overview is available in the article >>Extracting Training Data from ChatGPT.
Summary
Researchers, including Milad Nasr and Nicholas Carlini, discovered a specific query method that allows them to extract approximately several MBs of ChatGPT's training data at a cost of around $200. This attack highlights the significant fact that querying the model can reveal exact data used during its training. Furthermore, the researchers estimate that by increasing query costs, it may be possible to extract about 1GB of training data from the model.
During testing, researchers found that repetitive use of a specific word in ChatGPT could potentially expose sensitive user information in generated text. The open version of ChatGPT outputs vast amounts of text fetched from various internet sources, using this data as training input. The study revealed that a portion of the generated text contains Personal Identifiable Information (PII), including names, birthdays, contact information, addresses, and social media content.
In a departure from previous data extraction attack research, the team successfully attacked production-level models like ChatGPT. This method breaks the alignment model's limitation of not outputting extensive training data. The study underscores the vulnerability of flagship products in terms of privacy, even for widely used and high-performing models. For models like ChatGPT aligned with human feedback, attackers can disrupt the alignment to successfully extract training samples.
This series of attacks poses significant challenges to users and the research community, emphasizing the need for a deeper focus on language model security and protection.
Researchers' Perspective
The paper highlights several crucial viewpoints:
- It emphasizes that testing only aligned models may mask vulnerabilities, especially when alignment is easily broken.
- It stresses the importance of directly testing base models.
- It urges testing systems in production to ensure systems built on top of base models sufficiently patch vulnerabilities.
- It recommends companies releasing large models to conduct internal testing, user testing, and third-party organization testing for early detection of similar attacks.
Potential Risks
The paper points out vulnerabilities in language models related to drift and training data memory, which are harder to understand and patch. These vulnerabilities could be exploited by other attacks different from the one revealed, making effective defense implementation more complex.
This research sheds light on the current challenges in security and privacy of large language models, emphasizing the urgent need for comprehensive testing and security reinforcement. The findings have significant implications for research and applications in the fields of artificial intelligence and deep learning.
