Skip to content
TechIndexSubmit
Back to directory

llm-dataset-pipeline

Lightweight, containerized data ingestion and sanitization pipeline formatting raw web/API extractions into deduplicated JSONL instruction datasets.

Similar tools

  • Local-first text and LLM dataset curation studio for JSONL, SFT, DPO, conversations, agent traces, quality review, versioning, exports, MCP, and modular AI plugins.

    Datasets
  • LLM-Driven Extraction of Unstructured Data — Built for API Deployments & ETL Pipeline Workflows

    MCP Servers
  • Unified framework for building enterprise RAG pipelines with small, specialized models

    Models
  • Ready-to-run cloud templates for RAG, AI pipelines, and enterprise search with live data. 🐳Docker-friendly.⚡Always in sync with Sharepoint, Google Drive, S3, Kafka, PostgreSQL, real-time data APIs, and more.

    Models
  • 🦙 Integrating LLMs into structured NLP pipelines

    Prompts
  • Prepare the Kleister NDA dataset for LLM-based extraction. Validates labels against a Pydantic schema and delivers partitioned Parquet with co-located PDFs

    Datasets