When building RAG (Retrieval-Augmented Generation) applications or LLM agents, document ingestion is almost always the biggest bottleneck. Standard web scrapers and PDF loaders constantly pull in non-essential content: header navigation, f...
When building RAG (Retrieval-Augmented Generation) applications or LLM agents, document ingestion is almost always the biggest bottleneck.
Standard web scrapers and PDF loaders constantly pull in non-essential content: header navigation, footer links, cookie banners, ads, and broken DOM trees. When fed directly into vector databases, this clutter wastes context window tokens and degrades embedding quality.
To solve this, I spent the last few weeks building Brookiro—a dedicated parsing API that cleans web pages and PDFs into structured JSON or clean Markdown before chunking.
The Core Ingestion Problem
When chunking raw HTML or poorly parsed PDFs using basic loaders, two main issues break retrieval:
Token Bloat: Blog posts or documentation pages often contain 50%+ irrelevant boilerplate (script tags, sidebar nav, promotional banners).
Table & Structural Loss: Standard PDF parsers frequently flatten multi-column layouts and tables into unstructured sentences, causing text splitters to cut right through tabular data.
How Brookiro Works
Instead of running custom BeautifulSoup scripts or heavy local PDF engines for every project, Brookiro exposes a single endpoint that handles extraction asynchronously.
URL & PDF Parsing: Handles both web URLs and uploaded PDF files.
Noise Elimination: Automatically strips non-content DOM elements, ads, and scripts.
Markdown & Structured JSON: Outputs semantically clean Markdown (preserving # headers, lists, and tables) or typed JSON extractions.
Backend Architecture
Here is the tech stack powering the backend:
Framework: Python & Django REST Framework for API endpoint handling and parsing logic.
Authentication & Rate Limiting: Integrated with Unkey for edge-based API key management.
Infrastructure: Deployed on Render.
Code Example: Integrating with Python & RAG Workflows
You can drop a request straight into your Python ingestion script:
import requests
url = "[https://brookiro.com/api/v1/extract](https://brookiro.com/api/v1/extract)"
headers = {
"Authorization": "Bearer YOUR_API_KEY",
"Content-Type": "application/json"
}
payload = {
"url": "[https://example.com/target-page](https://example.com/target-page)"
}
response = requests.post(url, json=payload, headers=headers)
data = response.json()
# Access clean markdown ready for RecursiveCharacterTextSplitter
clean_markdown = data.get("markdown_content")
print(clean_markdown)
Try It Out
I set up a free tier with 100 requests every month (no credit card required) so developers can benchmark the extraction quality on their own data pipelines.
👉 Test the API:
https://brookiro.com
I’d love to hear your thoughts! What specific document formats or ugly PDF layouts give your RAG pipelines the most headache?