Government Data Management for the AI Era

"The federal government transparently discloses more than $4 trillion in spending data every year. Yet even its own audit agency says that data is hard to trust."

The United States has long been a global leader in open government data.

• Data.gov : Has made more than 300,000 public datasets openly available.

• USAspending.gov : Has transparently disclosed every federal expenditure to the public.

Ironically, however, the first to raise questions about the quality of this data was not citizens or the press, but the government itself.

The U.S. Government Accountability Office (GAO) regularly audits the quality of federal government data at the request of Congress and publishes the results in public reports. And the issues those reports point out are by no means trivial.

• According to a June 2026 GAO report, there are more than 100 data sources that can be used to verify eligibility for federal programs.

• When GAO audited nine of them, data quality issues (missing values, errors, and duplicates) were found in all nine, and seven of them contained data that did not match one another.

• The audit even revealed that no governance framework exists to connect these data sources with one another in the first place.

The U.S. government has emphasized data openness and transparency for more than 30 years. Yet when it comes to the fundamental question of "how much can this data be trusted," a question mark still remains.

So why does this keep happening? And what kind of approach is actually needed to solve this problem?

How Can We Overcome the Limitations of Government Data Quality Management?

1. Balance must be maintained between privacy protection and data usability.

As AI technology advances rapidly, the importance of considering both privacy protection and data usability has grown significantly. One of the most widely used solutions in this process is de-identification.

What is de-identification? It is a technique that replaces or removes information that can directly identify an individual, such as names, phone numbers, and addresses, allowing data to be utilized while protecting personal information.

However, there is something to watch out for here.

1) De-identifying data can erase the essential context within it.

• What happens when removing or de-identifying personal information in data also erases the essential context within it? The value of the data is compromised. In the case of AI development, it becomes difficult for AI to learn accurate patterns.

• Problems can also arise when personal information is removed, but information that is unnecessary for AI training or that undermines reliability remains intact.

In other words, the key is to manage quality while balancing privacy protection and data usability together.

2) Even with de-identification, the risk of re-identification remains.

• Major state laws, including the CCPA, are already aware of this re-identification risk. That is why they impose strict standards: for data to qualify as "de-identified information," companies themselves must demonstrate that the data cannot be re-identified.

2. Data governance management must keep the entire data lifecycle healthy.

In truth, we believe those responsible for data in the U.S. already know the answer to this problem: managing data continuously and keeping it up to date.

The problem is not a lack of awareness, but an organizational environment that makes continuous management difficult.

• The United States regulates AI through a patchwork of state laws and executive orders without a unified federal AI law, and these standards themselves have kept changing. When even what must be complied with keeps shifting, confusion among practitioners on the ground inevitably grows.

• There is one more irony here. The United States has the largest cumulative number of AI users in the world. This means the realities of data quality problems are also being exposed there faster than anywhere else. As the country that uses AI most frequently, it is also the first to pay the price when data is not ready.

Starting from that point, we propose the 'Data Greenhouse' operating system, which autonomously diagnoses and manages the data lifecycle on its own. And at the heart of this system, its core engine, is 'Data Clinic', a government data quality management tool powered by agentic AI.

You can find the strategic background and detailed vision of this system at the link below.

In particular, to meet the advanced security requirements of the government and defense sectors, Data Greenhouse supports an 'On-Premise Package' that operates fully in air-gapped, closed-network environments isolated from external networks. This allows agencies to manage their valuable data without any concern about external leakage, while fully protecting their data sovereignty (Sovereign AI).

• Pebblous is currently preparing to obtain accreditation from KOLAS (Korea Laboratory Accreditation Scheme). KOLAS is an accreditation body based on the international standard ISO/IEC 17025, and under the International Laboratory Accreditation Cooperation (ILAC) Mutual Recognition Arrangement (MRA), its accreditation is recognized as equivalent to that of accreditation bodies in more than 90 countries worldwide.

• Pebblous aims to realize its vision of taking the lead in setting international standards by issuing a 'Global Passport' for government data quality. This will be the pinnacle of a quality assurance framework that enables the quality of any country's government data to be recognized anywhere in the world.

• Pebblous currently maintains B2G services with public institutions that have the most demanding security requirements, including the Daegu Digital Innovation Promotion Agency (DIP), a regional government data promotion agency in Korea, as well as defense sector institutions such as the Republic of Korea Marine Corps and the Defense Agency for Technology and Quality, and the Army Test and Evaluation Group.

• The fact that Pebblous has already built real-world references in the defense and national security sectors, where data leakage is absolutely unacceptable, means its security level has been validated for deployment in any public sector environment.

The Daegu Digital Innovation Promotion Agency's Case: Improving the Quality of Daegu Metropolitan City's Traffic Complaint Data

So how are actual public institutions solving these problems? Here, we introduce a case in which the Daegu Digital Innovation Promotion Agency (DIP) improved the quality of Daegu Metropolitan City's traffic complaint data.

Removing personal information from data while preserving the context and meaning of the sentences is a challenge we believe public institutions in every country face. In this case, we present a solution through Korea’s Daegu Digital Innovation Promotion Agency (DIP) and its improvement of Daegu Metropolitan City's traffic complaint data quality.

Limitations of Conventional Pseudonymization

The Daegu Digital Innovation Promotion Agency had already been pseudonymizing data containing personal information on its own, but was facing limitations at the same time. Conventional text pseudonymization primarily relies on NER (Named Entity Recognition) to identify and remove only the personal information.

What is NER? It is an AI technology that automatically detects specific entities in a sentence, such as personal names, addresses, phone numbers, and vehicle license plate numbers.

Here is an example using a complaint. In the example below, the bolded text is personal information, and NER is the method that automatically recognizes and removes only this personal information from the full sentence.

A vehicle with plate number 12GA3456 is illegally parked every day on the road in front of 160 Alpha City 1-ro, Suseong-gu, Daegu, making it difficult to pass. It is inconvenient for children on their way to school every morning, so please enforce the parking rules. (Reported by Hong Gildong, 010-1234-5678)

NER is an excellent method, but some side effects were unavoidable in the process of pseudonymizing unstructured traffic complaint data. The first problem to emerge was that even after pseudonymization, profanity, hateful language, and emotionally charged expressions remained intact, as in the example below.

"Every evening around 7 PM, a white SUV (45NA6789) parks illegally at the bus stop in front of Daegu Apartment. Residents have complained multiple times, but the driver ignores them and rudely tells passersby, 'What would someone like you know?' It is very unpleasant, as elderly residents have difficulty passing through."

Furthermore, in the process of removing personal information, the meaning of the sentences was damaged, and the usability of the data actually declined. As this case shows, removing personal information does not necessarily lead to high-quality data.

The Daegu Digital Innovation Promotion Agency's New Approach

In this case, DIP applied an approach that differs from conventional methods. Simply put, rather than merely deleting personal information, it first understands 'what is this complaint trying to say?' and then generates a new sentence.

1. The title of each complaint was defined as an 'Intent'.

2. The method involves extracting key nouns and verbs from the body text and reconstructing the sentence.

For example, given a complaint such as "There is an illegally parked vehicle in front of ○○ Road in Suseong-gu, Daegu, making it difficult to pass," personal information such as the address and license plate number is removed, while the core intent of the complaint, that it is hard to get through because of the illegal parking, is preserved through LLM-based synthetic data generation that reconstructs the sentence.

Through this approach, DIP was able to not only protect personal information and preserve meaning, but also cleanse profanity and aggressive expressions, transforming the data into a form suitable for AI training.

Evaluation Results

In this way, DIP analyzed and improved the quality of a total of 4,889 traffic complaint records from Daegu Metropolitan City. In addition, to obtain more objective evaluation results, the improved data was verified using the following four methods.

• NER, the conventional personal information detection method

• Qwen 2.5 and EXAONE 3.5, leading LLMs for synthetic data generation

• The Intent Combination approach, Pebblous' method of intent-based data reconstruction 

1) Sentence Meaning Preservation

All four methods recorded high sentence embedding similarity scores of 0.93 or above. The score is expressed as a value between 0 and 1, and the closer it is to 1, the better the meaning has been preserved.

In other words, the results show that the core meaning of the original text was relatively well preserved regardless of which method was applied. The NER method in particular showed the highest semantic similarity.

2) Privacy Protection Performance

In personal information removal performance, the Intent Combination method recorded the lowest residual personal information rate. Among all models, it was confirmed to remove personal information most effectively while still preserving the meaning of the original text.

3) Profanity and Hateful Expression Removal Performance

In removing profanity and hateful expressions as well, the Intent Combination method demonstrated the best performance, while the other three methods also showed low detection rates. This confirmed that not only personal information but also harmful expressions unnecessary for AI training can be cleansed together.

If you are interested in more government data quality management cases, check out the links below!

For deeper insights into AI data quality, we recommend the following resources.

• Everything About AI Data Quality Management (A Complete Guide to the Concepts)

• Data Clinic Hub: From Assessment to Certification at a Glance

Build a systematic framework with a government data quality management tool so that public sector staff can manage data quality efficiently and accurately, and so that open data can be created for all citizens to use conveniently. Pebblous anticipates the direction of rapidly evolving technology and applies it to its data quality management solutions.

If you would like to receive key insights on data quality management on an ongoing basis, beyond the blog, click the button below to subscribe to the Data Clinic newsletter.