Almost every AI project we hear about starts with the same sentence.
"We have a lot of documents. Can we chat with them?"
It sounds simple. You pick a PDF library, pull out the text, send it to an LLM, and build a chat box on top. You test it on a 20-page sample file, and it works beautifully. Everyone is happy.
Then the client sends the real file. It's a 10,000-page policy manual.
The server slows down, the memory graph shoots up, and a few minutes later the whole thing crashes.
Nothing was wrong with the LLM. Nothing was wrong with the prompt. The problem was a few lines of very ordinary Python.
This is the first post in our AI & LLM series at CreoWis. Before we get into embeddings, RAG, and agents, we want to start with the Python ideas that quietly decide whether an AI app survives real-world data. The first one is how your code handles large data.
Why Start an AI Series With Python?
Because almost all AI work happens in Python.
The models, the embedding libraries, the vector database clients, and the agent frameworks are mostly Python-first. Calling an LLM is usually the easy part. The hard part is everything around it:
Reading thousands of files
Calling APIs that return huge responses
Creating and storing embeddings (and there are a lot of them)
Doing all of this on a server that doesn't have unlimited memory
If your Python handles data carelessly, no model will save you.
The Real Problem: Loading Everything at Once
Here's how most people first write the "read the PDF" step:
def extract_all_pages(pdf):
all_text = []
for page in pdf.pages:
all_text.append(page.extract_text() or "")
return all_text
There's nothing wrong with this code. For a small file, it's perfectly fine.
But look at what it does. It reads every single page and keeps all of that text in memory before returning anything.
Now think about the real situation:
A 10,000-page PDF can easily be 1 to 2 GB on disk.
Once that data is inside Python, it often takes up much more space than it did on disk, because Python stores extra information with every object it creates.
Your server isn't only running this script. It's also running your API, maybe a database, maybe a local model.
On a server with 4 or 8 GB of RAM, this is how you get a crash. And many real deployments do run on servers that size, because bigger servers cost more every month.
A Quick Look at How Python Handles Memory
Before the fix, it helps to know one thing about Python variables. They are , not boxes.
labels
a = [1, 2, 3]
b = a
del a
print(b) # [1, 2, 3]
When you write b = a, you don't get a copy. You get a second label pointing to the same list.
When you write del a, you remove the label a. The list itself is still there, because b still points to it. Python only cleans up an object once nothing points to it anymore.
Python also has a garbage collector that runs in the background and tidies up objects that are no longer used. You can ask it to run right away:
import gc
gc.collect()
You rarely need this. It's useful on a small server after you finish a heavy job, like a big batch of embeddings, and want to free memory before the next one.
A small trap if you use Jupyter notebooks: if you display a big object as a cell's output, Jupyter keeps a copy in its output history. So even after del, the data stays in memory. Avoid printing huge objects in notebooks.
All of this is useful, but it's cleanup after the damage is done. The better idea is never to load everything in the first place.
The Fix: Give Me One Page at a Time
Think about the difference between a bucket and a tap.
A bucket gives you all the water at once. You need somewhere big enough to hold it.
A tap gives you water only when you open it, as much as you need, when you need it.
Python's version of a tap is called a generator. You create one by using yield instead of return.
def count_up(numbers):
for n in numbers:
yield n
That's it. Here's what it does:
gen = count_up([10, 20, 30])
next(gen) # 10
next(gen) # 20
next(gen) # 30
next(gen) # StopIteration: nothing left
Each time you call next(), the function runs until it hits yield, hands you one value, and pauses. It remembers exactly where it stopped. The next call picks up from there.
In practice, you rarely call next() yourself. A normal for loop does it for you and stops cleanly when the values run out:
for n in count_up([10, 20, 30]):
print(n)
Three things that surprise people about generators
1. Calling the function doesn't run it.count_up([10, 20, 30]) doesn't print or process anything. It just gives you a generator object, ready to go. The work only starts when you ask for the first value.
2. You can only go through it once. Once a generator has given you all its values, it's finished. If you need the data again, create a new generator.
3. You can't jump ahead. With a list, you can grab item 60 directly with my_list[60]. With a generator, you have to go through items 1 to 59 first. That's the price of keeping memory low.
Reading a Huge PDF the Right Way
Now let's apply this to the original problem. Here's the same PDF reader, rewritten as a generator using the pypdf library:
from pypdf import PdfReader
def read_pages(path):
reader = PdfReader(path)
for page_number, page in enumerate(reader.pages, start=1):
text = page.extract_text() or ""
yield page_number, text
for page_number, text in read_pages("policy_manual.pdf"):
# clean it, chunk it, embed it, send it to the LLM...
print(page_number, len(text))
The code barely changed. But now only one page of text is held in memory at a time. Whether the file has 20 pages or 20,000, memory stays roughly flat.
This also fits how LLM pipelines actually work. You process page 1, send it off, wait for the result, then move to page 2. You never need the whole book in your hands at once.
The hidden requirement nobody mentions
While you're reading page by page, you'll notice something: some pages come back empty.
That's usually because those pages aren't text at all. They're scanned images that only look like text. A normal PDF library can't read pixels, so it returns nothing.
To read those pages, you need OCR (Optical Character Recognition), the same technology your phone uses when you copy text from a photo. Options range from open-source tools like Tesseract and PaddleOCR to cloud services like Amazon Textract and Azure Document Intelligence.
The page-by-page approach makes this easy to handle:
for page_number, text in read_pages("policy_manual.pdf"):
if not text.strip():
text = run_ocr_on_page(page_number) # your OCR step
process(text)
Here's the part we care about most as a team that builds these systems: clients almost never tell you about scanned pages. They don't think about their documents that way. If you don't ask, you ship a chatbot that silently ignores 30% of the content, and nobody notices until a user asks about something that's "definitely in the manual."
So we ask early:
Are some of your documents scanned copies?
Do they contain tables, charts, or images that matter?
How big is the largest file you'll send us?
Those three questions have saved us a lot of rework.
The Catch: A Generator Can't Undo a Big Download
There's one mistake that's easy to make once you fall in love with generators.
Say you're pulling data from an API that returns a huge response:
import requests
def get_items(url):
data = requests.get(url).json() # the whole response is now in memory
for item in data:
yield item
This looks memory-friendly because it uses yield. It isn't.
By the time the first yield runs, requests.get(url).json() has already downloaded the entire response and turned it into Python objects. The memory is already used. Handing out items one at a time afterwards saves nothing.
A generator only helps if the source itself is read lazily. A file, a PDF reader that goes page by page, or a network stream.
For network data, the fix is stream=True:
import requests
url = "https://example.com/large-dataset.zip"
with requests.get(url, stream=True, timeout=30) as response:
response.raise_for_status()
with open("large-dataset.zip", "wb") as f:
for chunk in response.iter_content(chunk_size=1024 * 1024): # 1 MB at a time
f.write(chunk)
With stream=True, requests doesn't download the whole body up front. You pull it in pieces, 1 MB at a time here, and write each piece to disk. Your program's memory stays small whether the file is 50 MB or 5 GB.
A few small details in that code matter in real projects:
timeout=30: without a timeout, a slow or dead server can leave your program waiting forever.
raise_for_status(): turns error responses like 404 or 500 into proper errors, instead of you saving an error page as if it were your file.
with: makes sure the connection is closed when you're done.
Where You'll See This in Everyday LLM Work
Once you know the pattern, you'll spot it everywhere in AI apps.
That "typing" effect in chat apps. When an LLM answer appears word by word, that's streaming. The model sends small pieces as they're generated, and the app shows each piece as it arrives. Many LLM APIs send these pieces as lines of text, which you can read with response.iter_lines() in the same way as above. The user sees an answer start in under a second instead of staring at a spinner.
Creating embeddings in batches. Embeddings are lists of numbers that represent your text, and you'll create thousands of them for a large document set. Sending everything in one request is slow and often hits API limits. A small generator that hands out batches keeps things under control:
def in_batches(items, size=100):
batch = []
for item in items:
batch.append(item)
if len(batch) == size:
yield batch
batch = []
if batch:
yield batch
for batch in in_batches(read_chunks("policy_manual.pdf"), size=100):
vectors = create_embeddings(batch) # one API call per 100 chunks
save_to_vector_db(vectors)
Notice how generators chain together. read_chunks hands out chunks one by one, in_batches groups them, and nothing is ever loaded all at once.
Reading logs, CSVs, and data exports.for line in open("app.log") already reads one line at a time, because Python file objects work like generators. f.read() and f.readlines() load the whole file. On a multi-gigabyte log file, that choice decides whether your script finishes or crashes.
Is Streaming Always Better?
Not always. There's a trade-off, and it's worth being honest about it.
Pulling data in pieces adds a little overhead. If your data is small, your server has plenty of memory, and every millisecond counts, loading everything at once can be slightly faster and simpler.
But for AI apps, the data almost always grows. A client's "small" document set today becomes thousands of files six months later. Rewriting your pipeline at that point costs far more than writing it the lazy way from day one.
So our default is simple: if it's a file, a document or a network response, read it in pieces.
Key Takeaways
Python variables are labels. del removes a label, not the data, and memory is only freed when nothing points to it anymore.
Loading a huge file or response all at once is the most common reason AI pipelines crash on real data.
A generator (yield) hands out one item at a time and pauses in between, so memory stays flat.
Generators only save memory when the source is also read lazily. Use stream=True for network data.
Reading page by page also makes it easy to catch scanned pages and send them to OCR.
Ask clients about scanned documents and file sizes early. It's the requirement they'll never mention.
What's Next in This Series
In the next post, we'll look at what happens when you need to call an LLM API not once, but hundreds of times. We'll cover why your script waits on each call one by one, and how async Python lets you run many calls at the same time without buying a bigger server.
If you're building an AI product and running into problems like these, whether it's documents your pipeline can't read, servers that keep running out of memory, or a prototype that fell apart on real data, let's talk. This is exactly the kind of work we do at CreoWis.
We at CreoWis believe in sharing knowledge publicly to help the developer community grow. Let’s collaborate, ideate, and craft passion to deliver awe-inspiring product experiences to the world.
This article is crafted by Syket Bhattachergee, a passionate developer at CreoWis. You can reach out to him on X/Twitter, LinkedIn, and follow his work on the GitHub.
