This is the story of one of them — a PDF-to-Excel data extraction pipeline that turned a 3–4 day manual process into something that runs in minutes.

The Problem

Companies deals with enormous volumes of technical documentation. Project specifications, measurement sheets, bills of quantities, material schedules — all of it arrives as dense, multi-page PDFs. Some documents run hundreds of pages. Some run over a thousand.

Before this tool existed, an engineer would open the PDF, scan through pages manually, identify the relevant data, and copy it cell by cell into a pre-formatted Excel sheet. For a 1,200-page document, that could take an entire working week for one person.

The data existed. The Excel template existed. The problem was purely the gap between them.

My job was to close that gap.

Why This Is Harder Than It Sounds

If you've never worked with real-world engineering PDFs, you might think this is straightforward — just extract the text and parse it. It isn't.

Here's what makes it genuinely difficult:

1. PDFs are not structured data. A PDF is essentially a rendering instruction set. Text that looks like a table to your eyes is often a flat list of strings with no inherent row/column relationship. Spatial positioning has to be inferred.

2. Document formats vary. No two specification documents are identical. Column headers appear in different positions, section titles use different phrasing, and sometimes the same data point is described three different ways across different sheets.

3. Accuracy vs. recall tradeoff. Extracting everything aggressively means flooding the output with noise. Being too conservative means missing critical data. Finding the right balance for engineering specifications is non-trivial.

4. Scale. A tool that works on 10 pages needs to work reliably on 1,200. Memory management, processing time, and error handling all become real concerns.

The Architecture

After a week of experimentation, I settled on a hybrid retrieval pipeline with a 3-pass verification architecture.

Step 1 — PDF Text Extraction

I used two libraries in combination:

pythonimport fitz # PyMuPDF import pdfplumber

def extract_text_blocks(pdf_path): doc = fitz.open(pdf_path) blocks = [] for page_num in range(len(doc)): page = doc[page_num] # Extract with positional metadata block_data = page.get_text("blocks") for block in block_data: blocks.append({ "page": page_num + 1, "text": block[4], "bbox": block[:4] # x0, y0, x1, y1 }) return blocks

PyMuPDF handles positional metadata well. pdfplumber handles table detection. Together they cover most document structures I encountered.

Step 2 — Hybrid Retrieval

This is the core of the pipeline. For each target field in the Excel template, I needed to find the most relevant text block in the document.

I combined two retrieval methods:

BM25 — a classical information retrieval algorithm that scores text blocks based on keyword overlap with the target field name. Fast, interpretable, and surprisingly effective for domain-specific terminology.

FAISS — Facebook's vector similarity search library. I embedded both the query (field name) and all text blocks using SentenceTransformers, then retrieved the most semantically similar blocks.

Neither method alone was reliable enough. BM25 misses semantic similarity. FAISS misses exact keyword matches. The solution was Reciprocal Rank Fusion — combining the ranked results from both systems:

pythondef reciprocal_rank_fusion(bm25_results, faiss_results, k=60): scores = {} for rank, doc_id in enumerate(bm25_results): scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1) for rank, doc_id in enumerate(faiss_results): scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank + 1) return sorted(scores.items(), key=lambda x: x[1], reverse=True)

This consistently outperformed either method alone.

Step 3 — 3-Pass Verification

A single extraction pass isn't reliable enough for production. I implemented a 3-pass system:

Pass 1: Extract candidate values for each field Pass 2: Cross-reference candidates against known domain patterns and synonyms Pass 3: Flag low-confidence extractions for human review rather than silently inserting wrong data

This last point matters. A tool that confidently outputs wrong data is worse than a tool that admits uncertainty. The flagging system meant engineers knew exactly which cells to double-check.

Step 4 — Parallel Processing

Processing 1,200 pages sequentially is slow. I used Python's ThreadPoolExecutor to process multiple page ranges concurrently:

pythonfrom concurrent.futures import ThreadPoolExecutor

def process_document(pdf_path, sheet_configs): with ThreadPoolExecutor(max_workers=4) as executor: futures = { executor.submit(extract_sheet_data, pdf_path, config): config for config in sheet_configs } results = {} for future in futures: config = futures[future] results[config['sheet_name']] = future.result() return results

This reduced processing time significantly compared to sequential execution.

The Results

Pages processed per run: 1,200+ Extraction accuracy: ~65–70% across all defined sheet formats Time saved: 3–4 days of manual work reduced to minutes Deployment: Production useable within L&T Constructions.

65–70% accuracy sounds modest until you consider what the alternative was — 0% automation. Engineers now spend their time reviewing and correcting a structured output rather than starting from a blank sheet. The cognitive load reduction is enormous.

What I Learned

Domain knowledge matters more than algorithm sophistication. The biggest accuracy improvements didn't come from better models — they came from understanding what "compressive strength" means in a civil engineering context versus a materials context, and encoding that knowledge into the retrieval system.

Fail loudly, not silently. Early versions of the tool would fill in a best guess even when confidence was low. This caused more problems than it solved. Flagging uncertain outputs and leaving them blank for human review was the right call.

Production is different from prototypes. A Jupyter notebook that works on a sample PDF is not a production tool. Error handling, logging, memory management, and graceful degradation under unexpected input formats all had to be built explicitly.

The problem definition is the hardest part. It took two weeks just to fully understand what the tool needed to do — what "correct" meant, what edge cases existed, what the engineers actually needed versus what they said they needed.

What's Next...

This tool handles text-based PDFs. A second tool I built at L&T handles a different problem entirely — AutoCAD-exported electrical single-line diagrams with zero embedded text, where all data exists only as visual labels. That required a completely different approach using Tesseract OCR/GPT vision and custom image pre-processing.

That's a story for another post.

The Bigger Picture

I'm a third-year undergraduate student. Six months ago I didn't know what FAISS was. I built this tool by reading documentation, failing repeatedly, and being honest with myself about what wasn't working.

If you're a student reading this — the gap between classroom ML and production data engineering is real, but it's crossable. Find a problem that actually exists, build something that actually works, and write down what you learned.

That's what I did. And somehow it ended up being a useful tool at one of India's largest infrastructure companies.

S. Varun Vaibhav is a B.Tech CSE (AI & Data Analytics) student at SRIHER Chennai and a former Data Analyst Intern at Larsen & Toubro Construction. Find me on GitHub and LinkedIn.

Tags: Python, Data Engineering, PDF Extraction, Machine Learning, Internship, FAISS, BM25, PyMuPDF, NLP, Real World ML