BigSet: An Automated Dataset Construction Framework Driven by Natural Language
Recently, the BigSet project, developed and open-sourced by the tinyfish-io team, has gained widespread attention, accumulating over 1.1K stars on GitHub. The project is positioned as an innovative data solution that can automatically transform a user’s single natural language request into a high-quality, structured dataset that is continuously maintained and can be refreshed on a schedule. Its design philosophy is considered a powerful open-source alternative to the commercial tool Clay AI, aiming to aggregate scattered information from the web into usable data assets.
Technical Architecture: A Multi-Agent Collaborative Workflow
The core of BigSet is an automated Multi-Agent research system with a highly integrated and automated workflow, which can be broken down into the following key steps:
Requirement Understanding and Schema Inference: When a user inputs a natural language description, such as “get the API prices for mainstream LLM models,” the system first uses a Large Language Model (LLM) to infer the schema for the required dataset. This includes determining the columns the table should contain (e.g., “Model ID,” “Vendor,” “Input Price,” “Output Price,” “Price Source Link”) and defining a primary key (e.g., “Model ID”), laying the foundation for subsequent data population.
Entity Discovery: An Orchestrator Agent is initiated to discover core entities related to the task through web searches. In the example above, it would identify major large model providers like OpenAI, Google, Anthropic, DeepSeek, and Alibaba.
Parallel Data Collection: The Orchestrator Agent then decomposes the task, launching a separate Sub-agent for each discovered entity. These sub-agents work in parallel, each responsible for researching a specific entity. For example, one sub-agent would focus on visiting OpenAI’s official pricing page to scrape its model price data.
Data Validation and Consolidation: After fetching the data, each sub-agent validates it to ensure accuracy and then inserts the result as a row into the predefined table. This “divide and conquer” parallel architecture balances both the breadth and depth of data collection.
Ultimately, the system generates a structured data table that users can browse in the interface and export in CSV or XLSX format. Additionally, users can set a refresh frequency (e.g., every 30 minutes, 6 hours, daily, or weekly), and the system will automatically repeat the entire workflow on schedule to ensure the dataset remains up-to-date.
Deployment and Practice: Local Environment Setup Guide
The deployment process for BigSet is straightforward, and developers can follow the official documentation to set it up in their local environment.
First, you need to clone the project repository from GitHub and prepare the environment file:
bash
git clone https://github.com/tinyfish-io/bigset.git
cd bigset
cp .env.example .env
Next, you need to configure three types of essential API keys in the .env file:
- TinyFish API Key: Used to call its underlying web Search and Fetch services. The
Fetch service, in particular, uses a real Chromium engine to render pages, effectively removing ads and pop-ups, and returns clean Markdown-formatted content.
- OpenRouter API Key: Used to power the Large Language Models, supporting core AI capabilities like schema inference and agent decision-making.
- Clerk Credentials: Includes the
Publishable Key, Secret Key, and JWT Issuer URL, used to implement user login and authentication features.
Once configured, start the entire application with the following command:
bash
make dev
After a successful start, three access points will be available locally: the main BigSet application (http://localhost:3500), the Convex database dashboard (http://localhost:6791), and Mastra Studio (http://localhost:4111). In practice, when deploying in a Windows Subsystem for Linux (WSL2) environment, it is recommended to store the project files in the native Linux file system to avoid potential compatibility issues.
Core Strengths and Limitations
BigSet's core strength lies in its innovative workflow, which transforms one-off information retrieval tasks into a sustainable process of building data assets. For users who need to continuously track industry trends, monitor market prices, or conduct in-depth research, such an automated, maintainable data table is far more valuable than scattered web pages or single query results.
However, the project currently has notable limitations. Its data collection capabilities are primarily restricted to public web information and cannot access content that requires login authentication or paid subscriptions. This somewhat limits its application in high-value, proprietary data collection scenarios.
Industry Impact: Beyond Instant Answers to Building Maintainable Data Assets
In the current AI technology landscape, search engines provide an entry point to information, while technologies like Retrieval-Augmented Generation (RAG) aim to deliver precise, instant answers. BigSet points in another direction: using AI capabilities to build and maintain persistent, dynamic, and structured data assets.
This paradigm shift from “one-time answers” to “maintainable data tables” is significant for promoting the application of AI in real-world business scenarios. In many enterprise applications, having an accurate, real-time, and updated structured dataset is a core requirement in itself. BigSet, with its agent-based architecture, provides a promising open-source solution to achieve this goal.
