Curation is the art of selecting quality items for a maintained collection. It’s a very broad term — encompassing music, art, historical artifacts and anything else with lasting value. Just like a museum depends on well-curated artifacts, Artificial Intelligence (AI) models depend upon well-curated data. In most AI projects, everyone in your data pipeline is responsible for curation.
Today, we’ll walk through the data curation process step by step.
Before and during AI training, curation paves the road to success. When we curate data, we need to collect, process and enrich our data before using it to train our AI models. At a higher level, this is a simple process. In reality, each step requires a healthy amount of nuance — you need to make attentive, case-by-case decisions.
The quality of your data directly determines the quality of your AI model. You’ll soon learn that bad data creates a bad model — regardless of architecture.
What Is Data Curation?
Data curation is the process of selecting, cleaning and managing datasets to meet the needs of your team or AI model. Our curation process actually begins with sourcing. Next, we need to clean it and handle any errors or inconsistencies. After we’ve cleaned and formatted our data, we can optionally add annotations and labels for supervised learning — this helps AI models learn relationships and better understand data. Sometimes, you might even use people or software to annotate incoming data as soon as it comes in. With automated data pipelines, you can streamline your curation process — and even allow AI models to help with your data curation!
Data curation ensures that your AI models aren’t training on messy or skewed datasets.
The Importance of Data Curation
There is no one size fits all method to data curation. Every team is different. Every project is different. Even every AI model is different. On top of that, new breakthroughs happen every day. These breakthroughs happen so fast that institutions struggle to keep up and set clear standards.
That said, most developers, analysts and data scientists can agree on one thing: Garbage in = Garbage out. Over the past decade, lack of data curation has resulted in some of the most spectacular AI fails in history.
- Tay: Microsoft’s first real foray into the world of modern AI. It wasn’t built on transformer architecture like ChatGPT and Grok today, but the real weakness was training data. Tay was trained on live social media, which quickly turned it into a firehose of toxicity and hate speech.
- Amazon’s AI Recruiting Tool: In 2018, reports surfaced alleging that Amazon created a recruiting bot exhibiting gender bias. As it turned out, the bot was trained on historical data. Amazon’s previous hires had skewed male — the bot picked up on this and inferred that new hires should be male too.
These are just a few examples where poor data led to model failure. Bad training data creates a bad model.
The Data Curation Lifecycle
- Data Sourcing and Collection: Data needs to come from reputable and relevant sources. The best data sources can often be right under your nose — you just need to know where to look. For some AIs, this might be through an API or scraping pipeline. Other models might use historical datasets or multimodal formats. Each potential source comes with its own set of quirks, risks and perks. You can find free APIs on GitHub. There are many free datasets available from sites like Kaggle. Online libraries even give you access to free books.
- Processing and Preparation: You can use raw data for training. You can also drink water from a storm drain, but it’s not a good idea. Your data needs to be processed and prepared for proper training. You should handle missing values, duplicates and outliers accordingly. Approach suspicious records like leftovers in your fridge: “When in doubt, throw it out.”
- Enhancement for AI: Once your data’s been properly cleaned and formatted, it’s time to enhance the data. Here, you would add things like labels and metadata (categories, tags, timestamps etc.) to assist in the training process. The model will find relationships quicker using your guidance.
Data Sourcing and Collection
When building a data curation pipeline, you need to identify and select your sources. Data sources are all over the place, but they need to be both relevant and reputable. To put it plainly, your data should represent the real-world facts you want the model to learn.
If you’re training a model on historical weather data, you could use a site like Weather Underground. Theoretically, you could also use pictures from the local kindergarten class. Both of these sources might tell you, “Yesterday was rainy.” The kindergarten class likely won’t give any historical data past the beginning of the school year and the data will likely be unstructured. However, a weather site or Application Programming Interface (API) will give you consistent data with a structure built for pattern recognition.
Raw data sources can include:
- API Datafeeds: These give structured or semi-structured data on demand. API feeds are excellent for minimizing the processing and preparation step. You can find a collective list of public APIs here on GitHub.
- Web Scraping: You can also extract fully raw data from websites themselves. This process can be a little tricky, but it allows you to retrieve any and all public data — whenever you need. Bright Data and Oxylabs provide excellent tools for real-time scraping.
- Historical Datasets: AI models can analyze vast historical datasets to learn both micro and macro level trends. This technique is often used in financial and weather predictions. Kaggle and Common Crawl both provide open source datasets with permissive licensing.
- Free Text Data: Most LLMs are trained on an ocean of books. In terms of scale, it’s like every public domain book in the world. Project Gutenberg, Open Library and even Amazon are all great places to find free books for your model to learn from. However, even if a book is free, check its licensing.
- Images/Video: You’ve probably heard that a picture is worth a thousand words. This holds true even in machine learning. You can train models on multimodal data using tools like this dataset from Hugging Face or the Web Data Archive from Bright Data.
Choose a source relevant to your needs and reputable enough to cite when necessary.
Processing and Preparing Your Data
Cleaning the Data
Next, you need to process and prepare your data. There are a variety of tools and solutions from developer focused libraries like Pandas and PyJanitor to GUI tools like Excel and Google Sheets. You can even use specialized tools like OpenRefine and KNIME.
- Duplicates: Remove any records that might skew results. Ideally, your data should reflect the real world relationships that you want your model to recognize.
- Outliers: Almost every dataset holds outliers. When a record doesn’t fit the pattern, it doesn’t always need to be removed — you should investigate these outliers and make decisions on a case by case basis.
Real-World vs Synthetic Datasets
Depending on your use case, you might want to create a synthetic dataset based on your actual data. Real world datasets hold the actual relationships we want the model to learn. However, synthetic data gives us the ability to mask or even expand the original data.
- Real-World Data: If your dataset is large enough and doesn’t hold sensitive information, it’s almost always the better choice.
- Synthetic Data: If your data holds sensitive information or less permissive licensing, you may wish to generate a synthetic dataset. Synthetic data can also be used to expand datasets in cases where training data is sparse.
Synthetic data generators will learn your real world structure and create new data matching the structure. You can use tools like Mostly AI and Gretel to generate comprehensive synthetic data.
Not all datasets need to be synthetic. If you’re training on customer data, synthetic is often the responsible choice. If you’re working with stock market data, real world datasets will do the job. Duplicates and outliers always need to be handled. Skewed data creates biased models.
Data Enhancement: Labeling, Annotation and Metadata
After cleaning and preparing the data, you could technically pass it into the training environment. That said, if you’re willing to enhance your data, it can greatly improve your training speed and overall learning quality. When we enhance our data, we’re essentially adding little notes that the model can use to find relationships faster and reinforce them.
- Labeling: Labeling is a tedious but rewarding process. Imagine your model is training on pet pictures. If you label every picture in the dataset as a “cat” or “dog” — the model won’t need to spend as much time or resources figuring it out.
- Annotation: Annotation actually includes labeling. However, it also includes finer details — think timestamps and sentiment tagging for free text, images and videos.
- Metadata: When adding metadata, you’ll add columns like data type, data source and confidence scores. Metadata helps your model filter and compartmentalize the data. In some cases, this allows models to trust sources and even cite them in conversation later on — this is dependent on model capabilities.
There are all sorts of enhancement tools you can use. Label Studio is probably the most well known open source option. Enterprise tools like Prodigy and Amazon SageMaker Groundtruth are also widely available.
Enhancement is an optional best practice for training AI models. Time and care during this step can make a world of difference in your actual training and deployment.
How To Implement Data Curation Effectively
- Choosing the Right Tool: Choose tools that match your team’s needs. Pandas gives granular control while PyJanitor offers a similar level of detail but with less direct control.
- Manual vs. Automated Curation: Automated tools have small quirks (such as lower case columns) that we need to account for. You’re doing less work with an automated tool but you have less control over your data. Automation can drastically speed things up but it needs to be done carefully.
- Common Points of Integration: In most cases, curation isn’t a one stop process. It happens throughout the entire pipeline. From selection to fine tuning, the entire team needs to care about the quality of the data.
There is no universal method to implement data curation, but you need to make smart, responsible decisions with curation in mind. Everyone on your team is responsible for data curation — whether they know it or not.
Conclusion
Now, you understand data curation. It’s not just a technical task, it’s a responsibility and your entire team owns it. Whether you’re a developer using code-based tools, or you’re performing your curation tasks with GUI based tools, you’ve got a hand in this process. It’s your job to help curate it.
Data curation isn’t just a workflow, it’s a mindset. You choose what goes into your AI model and its output after training depends entirely on that choice.
Remember: Garbage in = Garbage out.