Data engineering / AI
Codebeamer Documentation Scraping for AI-Ready Data
We turned PTC's Codebeamer Developer Guide into a clean, structured JSON dataset, chunked and ready for AI and LLM applications.
- Client
- Software company
- Our role
- Data engineering
- Year
- 2025
- Category
- Data engineering / AI
Where the client started
The client wanted the content of PTC's Codebeamer Developer Guide as structured data it could use in AI and LLM-based applications.
The problem
Documentation web pages mix content with navigation and layout, while AI tools need that content clean, structured and in its original hierarchy.
The goal
An automated process that turns the guide into clean, structured, AI-ready JSON.
What we built
We built a Python scraping workflow with Selenium and Playwright. It reads the Developer Guide's pages, extracts headings, paragraphs, lists, images and links while keeping the document's hierarchy, cleans and organises the content, splits it into chunks suited to AI processing, and writes the result as a structured JSON dataset.
How the system works
Each stage in the order the work happens, with what it does and screens from the system.
Source documentation
The Codebeamer Developer Guide is read page by page through browser automation.
Automated page collection
Python scripts use Selenium and Playwright to open the Developer Guide's pages in a browser and read their content.
The source documentation: the Codebeamer Developer Guide in PTC's help center.Open full size: Codebeamer Developer Guide page in PTC's help center, with the guide's table of contents and a diagram of the Codebeamer application architecture (opens in a new tab) Extraction
Everything useful on a page is pulled out, in the order and hierarchy it appears.
Content extraction
Headings, paragraphs, lists, images, links and other relevant HTML content are extracted from each page.
Structure preserved
The document's structure and heading hierarchy are preserved during extraction.
Cleaning and chunking
The raw content is cleaned, organised and split into pieces an AI model can work with.
Cleaning and organising
Scraped content is cleaned and organised into structured data.
AI-ready chunks
The documentation is split into meaningful chunks optimised for AI and LLM processing.
JSON dataset
The result is a structured JSON dataset, ready for further ingestion and analysis.
Structured JSON output
A structured JSON dataset is generated from the cleaned, chunked content.
Automated end to end
The extraction and transformation process is automated using Python.
What it runs on
The products behind the system, and the job each one does.
Python
The extraction, cleaning, chunking and JSON generation scripts.
Selenium
Browser automation for loading and reading documentation pages.
Playwright
Browser automation for loading and reading documentation pages.
JSON
Format of the structured output dataset.
Challenges
- Keeping the documentation's heading hierarchy intact while extracting content
- Separating useful content from page navigation and layout
- Splitting long documentation into chunks that still make sense on their own
Results
- The Codebeamer Developer Guide available as a structured JSON dataset
- Content chunked and ready for AI and LLM-based applications
- Extraction and transformation automated in Python
Building something similar?
Tell us about the problem and we will come back with how we would approach it.