That was the first message in the team channel at 9:12 on a Tuesday morning. The support team had just logged in and asked the RAG assistant about a policy change that went live the day before. The assistant gave them the old answer.
The bot wasn't down. It was still reading from Monday's index, because Tuesday's re-index job hadn't finished. It had started at midnight and was on document 23,000 of 40,000.
The job wasn't complicated. Open each policy PDF, split it into chunks, send the chunks to an embedding API, save the vectors. One document at a time, inside a for loop. Parsing took about 0.3 seconds per document. The embedding call took about 0.8 seconds, and the script spent almost all of that time waiting for the API to reply. Multiply by 40,000, and the job finishes around lunchtime.
The quick fix that afternoon was threads. Parsing got no faster at all. Someone pushed the thread count to 500, and the embedding provider answered with a wall of 429 Too Many Requests. By Friday, finance wanted to know why the token count on the client's invoice looked too low.
Three bugs, one cause. Nobody had asked whether the slow part of the code was waiting or working.
This is Part 4 of our AI & LLM series at CreoWis. Part 1 covered working with large data without running out of memory. Part 2 covered handling errors and files so your pipeline doesn't crash. Part 3 used OOP so switching LLM providers doesn't break your app. This one is about threads, processes, and pool executors: how to get more work done in less time without breaking things along the way.
Why AI Engineers Should Care About This
Because AI apps spend most of their time in one of two states:
Waiting: for an LLM to reply, for an embedding API, for a vector database, for a file download
Working: parsing PDFs, chunking text, resizing images, running a local model on the CPU
If you treat both the same way, you'll either waste hours or make things worse. Threads help with waiting. They do almost nothing for working. Processes help with working, but they cost more to start.
There's one more reason. When you ask an AI coding assistant to "make this faster," it often adds threads or processes without asking what kind of work it is. If you can't tell the difference, you can't tell whether the change actually helped.
Two Kinds of Slow: Waiting vs Working
Before you choose a tool, label the work.
I/O-bound (waiting)
CPU-bound (working)
What's happening
Your code sends a request and sits idle until the answer comes back
The CPU is busy the whole time, with no waiting gaps
AI examples
LLM calls, embedding calls, vector DB queries, downloads, uploads
PDF parsing, chunking, tokenizing, image processing, local inference on CPU
Right tool
Threads (or asyncio)
Processes
Here's a simple test: "If I had a faster CPU, would this get much faster?"
Yes → it's CPU-bound.
No, because it's waiting on the network or the disk → it's I/O-bound.
A faster laptop won't make OpenAI or Anthropic answer you sooner. That's how you know an LLM call is I/O-bound.
The GIL in Plain English
Standard Python has something called the GIL (Global Interpreter Lock). It means that inside one Python process, only one thread runs Python code at a time.
That sounds like it makes threads useless, but it doesn't. When a thread starts waiting (for the network, a file, or time.sleep), it gives up the GIL, and another thread can run. So threads are great at overlapping waits. They're bad at sharing heavy calculation, because only one of them can calculate at a time.
Processes get around this completely. Each process is a separate Python program with its own GIL, so separate processes can run on separate CPU cores at the same moment.
Newer Python versions also offer an optional "free-threaded" build without the GIL. It's not the default yet, and most production setups still run the standard build, so everything in this post assumes the GIL is there.
Two words you'll hear a lot:
Concurrency: switching between tasks so all of them make progress. One chef, many pans.
Parallelism: tasks running at the exact same instant, on different cores. Many chefs.
Threads in Python give you concurrency. Processes give you parallelism. They aren't opposites, just different tools.
The Baseline: One Thing at a Time
Let's start with the slow version. We'll use time.sleep to stand in for an embedding API call. Just imagine a real request happening there.
import time
def embed_chunk(chunk_id):
print(f"Embedding chunk {chunk_id}...")
time.sleep(2) # stand-in for a 2-second embedding API call
print(f"Chunk {chunk_id} embedded")
start = time.perf_counter()
for chunk_id in [1, 2, 3]:
embed_chunk(chunk_id)
print(f"Total time: {time.perf_counter() - start:.2f} seconds")
Python runs line by line, so the times add up: 2 + 2 + 2 = 6 seconds.
time.sleep is a blocking call. The code that runs it is stuck there until the time is up. A real requests.post(...) to an embedding API blocks in exactly the same way. For most of those 6 seconds, your CPU is doing nothing.
Threads: Doing Something While You Wait
A thread is a separate line of work inside the same program. Here's the smallest example:
import threading
import time
def embed_chunk(chunk_id):
print(f"Embedding chunk {chunk_id}...")
time.sleep(2)
print(f"Chunk {chunk_id} embedded")
thread = threading.Thread(target=embed_chunk, args=(1,))
thread.start() # the thread starts running in the background
thread.join() # wait here until the thread is finished
print("Done")
Three things to notice:
threading.Thread(...) only creates the thread. Nothing runs yet.
start() actually runs it.
join() means "don't move forward until this thread is finished."
One small mistake catches almost everyone:
threading.Thread(target=embed_chunk, args=(1)) # wrong: (1) is just the number 1
threading.Thread(target=embed_chunk, args=(1,)) # right: (1,) is a tuple
args must be a tuple. A single value needs a trailing comma.
Now let's run three embeddings at once:
start = time.perf_counter()
threads = []
for chunk_id in [1, 2, 3]:
thread = threading.Thread(target=embed_chunk, args=(chunk_id,))
threads.append(thread)
thread.start() # start all of them first...
for thread in threads:
thread.join() # ...then wait for all of them
print(f"Total time: {time.perf_counter() - start:.2f} seconds")
Six seconds became two. The three waits overlapped. Thread 1 started waiting, so thread 2 got a turn and started waiting, then thread 3. All three were waiting at the same time.
Notice the order of the "embedded" lines. It can change from run to run. Once you use threads, you don't control the exact order things finish in.
Threads Share Memory: The Token Counter Bug
Threads share the same memory. That's convenient, and it's also where the billing bug from the intro came from.
Say every embedding call reports how many tokens it used, and we add them to one global counter so we can bill the client:
import threading
import time
total_tokens = 0
def record_usage(tokens_per_call, calls):
global total_tokens
for _ in range(calls):
current = total_tokens # 1. read
time.sleep(0) # 2. a tiny pause (a real API call would be here)
total_tokens = current + tokens_per_call # 3. write
t1 = threading.Thread(target=record_usage, args=(100, 1000))
t2 = threading.Thread(target=record_usage, args=(100, 1000))
t1.start(); t2.start()
t1.join(); t2.join()
print(f"Expected: 200000, Got: {total_tokens}")
Expected: 200000, Got: 104300
Run it again and you'll get a different wrong number.
Here's what happens:
Step
Thread
What it does
total_tokens
1
A
Reads 500
500
2
B
Reads 500 (the same old value)
500
3
A
Writes 500 + 100
600
4
B
Writes 500 + 100
600 ← A's update is lost
This is called a race condition. Two threads read the same old value before either one writes, so one update disappears. The result depends on timing, which is why it changes every run.
In a demo, that's a wrong number. In production, it's a client being under-billed, or a rate-limit budget that says you have room when you don't.
The fix is a lock. Think of it as the only key to a room: only the thread holding the key can go in, and everyone else waits at the door.
total_tokens = 0
usage_lock = threading.Lock()
def record_usage(tokens_per_call, calls):
global total_tokens
for _ in range(calls):
with usage_lock: # take the key
current = total_tokens
time.sleep(0)
total_tokens = current + tokens_per_call
# the key is released automatically here
Expected: 200000, Got: 200000
Every run, the right answer.
The with block is the same context manager idea from Part 2's with open(...). The lock is released even if something inside raises an error.
A few rules for locks:
Lock only the small piece that touches shared data, not the whole function. Locked code runs one thread at a time, so a big lock throws away your speedup.
Never hold a lock while making the API call itself. Make the call, then lock just long enough to update the counter.
Local variables inside a function are not shared, so they don't need a lock.
When a Thread Fails, Nobody Tells You
In Part 2, we spent a lot of time on errors. Threads add a new problem:
Exception in thread Thread-1 (embed_batch):
Traceback (most recent call last):
...
RateLimitError: Batch 7: rate limit reached
All batches done!
The error was printed, but the thread just died. The main program carried on and announced "All batches done!" Nobody can try/except that error from the main thread.
In a nightly job with thousands of threads, that message scrolls past in a log nobody reads, and batch 7 is silently missing from your vector database.
Keep this in mind. Pool executors, which we'll see shortly, fix it.
Processes: Using All Your CPU Cores
Now let's look at the other half of the pipeline: parsing.
Parsing a PDF, cleaning the text, and splitting it into chunks is CPU-bound. There's no waiting gap for another thread to fill. And because of the GIL, only one thread can calculate at a time. So threads can't help here at all.
Let's prove it with a stand-in for heavy parsing work:
import threading
import time
def parse_document(doc_id):
# stand-in for PDF parsing: pure CPU work, no waiting
total = sum(i * i for i in range(5_000_000))
return total
start = time.perf_counter()
for doc_id in range(4):
parse_document(doc_id)
print(f"Sequential: {time.perf_counter() - start:.2f} s")
start = time.perf_counter()
threads = [threading.Thread(target=parse_document, args=(d,)) for d in range(4)]
for t in threads: t.start()
for t in threads: t.join()
print(f"4 threads: {time.perf_counter() - start:.2f} s")
Sequential: 1.62 s
4 threads: 1.65 s
(Your numbers will be different, but the pattern will be the same.) The threads took just as long, sometimes a little longer, because switching between them costs something too.
This is where multiprocessing comes in. A process is a separate running copy of Python, with its own memory and its own GIL. The operating system can put each process on a different core, so they truly run at the same time.
import multiprocessing
def parse_document(doc_id):
total = sum(i * i for i in range(5_000_000))
print(f"Document {doc_id} parsed")
if __name__ == "__main__":
process = multiprocessing.Process(target=parse_document, args=(1,))
process.start()
process.join()
print("Main process: done")
Same pattern as threads: create, start(), join().
But there's a new line: if __name__ == "__main__":. Leave it out, and on Windows or macOS you'll get this:
RuntimeError: An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.
Here's why. A new process starts a fresh Python that imports your file again. Without the guard, that import would run your process-creating code again, which would start more processes, and so on. The guard means "only run this when this file is the one being run directly, not when it's imported."
Older tutorials say Linux doesn't need the guard. That used to be mostly true, but since Python 3.14, Linux no longer uses the old "fork" method by default either. Always add the guard. It makes your code work everywhere.
Getting Results Back From a Process
Threads share memory, so they can write to the same list. Processes don't. A value created inside a child process is gone when that process ends.
To get results back, you need a special container that both sides can use. The simplest one is multiprocessing.Queue:
import multiprocessing
def parse_document(doc_name, result_queue):
chunks = ["chunk 1", "chunk 2", "chunk 3"] # pretend we parsed the PDF
result_queue.put({"doc": doc_name, "chunks": len(chunks)})
if __name__ == "__main__":
results = multiprocessing.Queue()
process = multiprocessing.Process(
target=parse_document,
args=("policy_0042.pdf", results),
)
process.start()
print(results.get()) # take the result out
process.join()
{'doc': 'policy_0042.pdf', 'chunks': 3}
A queue is first in, first out, like people in a line. Each get() takes out the oldest item.
Notice that we call get()beforejoin(). If a child puts a lot of data in the queue, it can wait until someone reads it before it fully exits. If the main process is stuck in join() waiting for the child, and the child is waiting for the main process to read, both wait forever. Read first, then join.
Managing processes and queues by hand works for a few tasks. For 40,000 documents, there's a much better way.
Pool Executors: A Fixed Team Instead of One Worker per Job
Should we start 40,000 processes for 40,000 documents? Or 40,000 threads for 40,000 API calls?
No. Every thread and process costs memory. Processes take time to start. And the embedding provider will rate-limit you long before you reach 40,000 calls in flight. That's exactly what happened in the intro with 500 threads.
A factory doesn't hire one worker per product. It has a fixed team that handles many products, one after another. That team is called a pool.
Python's concurrent.futures module gives you two pools with the same API:
ThreadPoolExecutor: a team of threads, for I/O-bound work
ProcessPoolExecutor: a team of processes, for CPU-bound work
ThreadPoolExecutor With map
from concurrent.futures import ThreadPoolExecutor
import time
def embed_chunk(chunk):
time.sleep(2) # stand-in for the embedding API call
return f"vector for {chunk}"
chunks = ["chunk A", "chunk B", "chunk C", "chunk D", "chunk E"]
start = time.perf_counter()
with ThreadPoolExecutor(max_workers=3) as executor:
results = executor.map(embed_chunk, chunks)
for result in results:
print(result)
print(f"Total time: {time.perf_counter() - start:.2f} s")
vector for chunk A
vector for chunk B
vector for chunk C
vector for chunk D
vector for chunk E
Total time: 4.01 s
What happened:
max_workers=3 means at most 3 threads at a time. The first 3 chunks start right away. D and E wait until a worker is free.
That's two rounds of 2 seconds: about 4 seconds instead of 10.
executor.map returns results in the same order as the inputs, even if they finish in a different order.
Leaving the with block waits for all the work and shuts the pool down cleanly. No start(), no join().
Futures and submit: Results as Soon as They're Ready
map is simple, but it has two limits. You get results in input order, so one slow chunk at the front holds up everything behind it. And if one call raises an error, you'll hit it while looping and lose your place.
For more control, use submit. It starts one job and immediately gives you back a Future: an object that represents the job's result, which will be ready at some point.
from concurrent.futures import ThreadPoolExecutor
with ThreadPoolExecutor(max_workers=3) as executor:
future = executor.submit(embed_chunk, "chunk A")
print("Finished yet?", future.done()) # doesn't wait
print(future.result()) # waits, then gives the value
print("Finished now?", future.done())
Finished yet? False
vector for chunk A
Finished now? True
Method
What it does
executor.submit(func, arg)
Starts the job and returns a Future right away
future.done()
True if the job has finished, False if it's still waiting or running. Doesn't wait.
future.result()
Waits for the job and returns its value. If the job raised an error, it raises that same error here.
That last line is the fix for the "silent thread" problem. With a pool, the error isn't lost. It travels back to you, and you can catch it with a normal try/except.
Now combine submit with as_completed to handle each result the moment it's ready:
from concurrent.futures import ThreadPoolExecutor, as_completed
import time
class RateLimitError(Exception):
pass
def embed_chunk(chunk, seconds):
time.sleep(seconds)
if chunk == "chunk C":
raise RateLimitError("rate limit reached")
return f"vector for {chunk}"
jobs = [("chunk A", 3), ("chunk B", 1), ("chunk C", 2)]
with ThreadPoolExecutor(max_workers=3) as executor:
futures = {executor.submit(embed_chunk, chunk, sec): chunk for chunk, sec in jobs}
for future in as_completed(futures):
chunk = futures[future] # which chunk was this?
try:
print(future.result())
except RateLimitError as error:
print(f"{chunk} failed: {error}")
vector for chunk B
chunk C failed: rate limit reached
vector for chunk A
Fastest first, not input order. And one failure didn't stop the others.
The dictionary {future: chunk} is a small trick worth remembering. Futures don't tell you which input they came from, so you keep that mapping yourself. It's how you know what to write in your failures log.
The rule to remember:
map = results in input order. Simple.
submit+as_completed = results in finishing order, with error handling per job. Better for production.
ProcessPoolExecutor: Same API, Real Parallelism
Here's the best part. To switch from threads to processes, you change one word:
from concurrent.futures import ProcessPoolExecutor
import time
def parse_document(doc_id):
# stand-in for PDF parsing: pure CPU work
sum(i * i for i in range(5_000_000))
return f"document {doc_id} parsed"
if __name__ == "__main__":
start = time.perf_counter()
with ProcessPoolExecutor(max_workers=4) as executor:
for result in executor.map(parse_document, range(4)):
print(result)
print(f"4 processes: {time.perf_counter() - start:.2f} s")
The work that took about 1.6 seconds with one process, and about 1.6 with four threads, now takes about half a second.
No Queue this time. The pool sends results back to you through map or Futures.
Two things trip people up with process pools.
First, startup has a cost. Starting a process takes time. If each job is tiny, that cost can be bigger than the work itself. Process pools pay off when each job does real work: parsing a big PDF, processing an image, not adding two numbers.
Second, everything you send to a process has to be packed up and copied. Python uses a tool called pickle for this. Functions defined at the top level of your file can be packed. Lambdas and functions defined inside other functions can't:
with ProcessPoolExecutor() as executor:
executor.map(lambda doc: doc.upper(), docs)
# PicklingError: Can't pickle <function <lambda> ...>
The fix is to write a normal function at the top level of the file. Also, send each process what it needs (a file path, a document ID), not huge objects. Copying a 200 MB object to every process is slow and eats RAM.
How Many Workers?
There's no magic number, but there are good starting points.
For CPU-bound work (processes): start with the number of CPU cores. ProcessPoolExecutor() does this by default. More processes than cores won't make anything faster. They'll just take turns.
For I/O-bound work (threads): the real limit is usually the API, not your machine. This is where the intro team went wrong.
Say your embedding provider allows about 10 requests per second, and each request takes about 0.5 seconds. Then:
workers ≈ requests per second × seconds per request
workers ≈ 10 × 0.5 = 5
Five workers keep you right at the limit. 500 workers just get you 429 errors faster.
Then measure. Time the run with time.perf_counter(), watch CPU and memory, watch for 429s, and adjust. Start small and go up.
Putting It All Together: A Nightly Indexing Pipeline
Here's the intro problem solved properly. Processes do the parsing. Threads do the embedding. A lock protects the token counter. Errors are caught per chunk. And results are written to disk as they arrive, using the JSONL pattern from Part 2.
import json
import threading
import time
from concurrent.futures import ProcessPoolExecutor, ThreadPoolExecutor, as_completed
class LLMProviderError(Exception):
"""Base error for anything that goes wrong with the LLM provider."""
class RateLimitError(LLMProviderError):
pass
# --- Step 1: CPU-bound work, runs in processes ---
def parse_and_chunk(doc):
# stand-in for real PDF parsing: pure CPU work
sum(i * i for i in range(1_000_000))
words = doc["text"].split()
return [
{"doc_id": doc["id"], "chunk_no": n, "text": " ".join(words[i:i + 50])}
for n, i in enumerate(range(0, len(words), 50))
]
# --- Step 2: I/O-bound work, runs in threads ---
class UsageTracker:
"""Counts tokens across threads, safely."""
def __init__(self):
self.total_tokens = 0
self._lock = threading.Lock()
def add(self, tokens):
with self._lock: # lock only the update, not the API call
self.total_tokens += tokens
def call_embedding_api(text):
# the real HTTP call goes here (see Part 2 for timeouts and error mapping)
time.sleep(0.5)
return [0.12, 0.48, 0.91], len(text.split())
def embed_chunk(chunk, usage, retries=3):
for attempt in range(retries):
try:
vector, tokens = call_embedding_api(chunk["text"])
usage.add(tokens)
return vector
except RateLimitError:
if attempt == retries - 1:
raise
time.sleep(2 ** attempt) # wait 1 s, then 2 s...
# --- The pipeline ---
def run_pipeline(documents, embed_workers=5):
usage = UsageTracker()
# Parse in parallel, one process per CPU core
with ProcessPoolExecutor() as pool:
chunk_lists = pool.map(parse_and_chunk, documents)
chunks = [chunk for chunk_list in chunk_lists for chunk in chunk_list]
failed = 0
# Embed concurrently, within the provider's rate limit
with (
open("embeddings.jsonl", "a", encoding="utf-8") as results,
open("failures.log", "a", encoding="utf-8") as failures,
ThreadPoolExecutor(max_workers=embed_workers) as pool,
): # this form needs Python 3.10+
futures = {pool.submit(embed_chunk, chunk, usage): chunk for chunk in chunks}
for future in as_completed(futures):
chunk = futures[future]
try:
vector = future.result()
except LLMProviderError as error:
failures.write(f"{chunk['doc_id']}#{chunk['chunk_no']}\t{error}\n")
failed += 1
else:
results.write(json.dumps({
"doc_id": chunk["doc_id"],
"chunk_no": chunk["chunk_no"],
"vector": vector,
}) + "\n")
results.flush() # push it to disk now, not at the end
return len(chunks), failed, usage.total_tokens
if __name__ == "__main__":
documents = [
{"id": f"policy_{n:04d}", "text": "coverage terms and conditions " * 40}
for n in range(20)
]
start = time.perf_counter()
total, failed, tokens = run_pipeline(documents)
print(f"{total} chunks, {failed} failed, {tokens} tokens, "
f"{time.perf_counter() - start:.2f} s")
Why it's built this way:
Parsing usesProcessPoolExecutor because it's CPU-bound. Each core parses its own documents at the same time.
Embedding usesThreadPoolExecutor because it's I/O-bound. While one thread waits for the API, others are waiting too, so the waits overlap.
embed_workers=5 comes from the rate-limit math, not from guessing.
UsageTrackeruses a lock so the billing number is correct. The lock covers only the counter update, not the slow API call.
as_completed+try/except means one failed chunk is logged and skipped. It doesn't kill the night's run, and it doesn't fail silently either.
Only the main thread writes to the files. Worker threads return values; they don't touch the files. That's why the file writes don't need a lock.
results.flush() means that if the job dies at chunk 30,000, the first 29,999 are already safe on disk. That's the lesson from Part 2.
With 40,000 documents, this is the difference between a job that ends at lunchtime and one that's done long before anyone types "Is the bot down?"
Quick Reference
Question
Answer
My code waits on APIs, databases, or files
ThreadPoolExecutor
My code does heavy calculation
ProcessPoolExecutor
Many threads update the same variable
Protect it with threading.Lock
I need results in input order
executor.map
I need results as they finish, with per-job errors
submit + as_completed
I'm using multiprocessing
Always add if __name__ == "__main__":
How many process workers?
Start with the number of CPU cores
How many thread workers for an API?
Requests per second × seconds per request, then measure
Common Mistakes to Watch For
When you review code (yours or an AI assistant's), look for these:
Threads used for CPU-heavy work like parsing or image processing. They won't help.
Hundreds of threads calling an LLM API with no max_workers limit. Expect 429s.
Shared counters, lists, or dictionaries updated from many threads with no lock.
Raw threading.Thread with no way to see errors. Prefer a pool and future.result().
Multiprocessing code without the __main__ guard.
Lambdas passed to ProcessPoolExecutor.
Results collected in a list and written to disk only at the very end.
Wrapping Up
Most "my AI pipeline is too slow" problems don't need a bigger server or a different model. They need one question answered first: is this code waiting, or is it working?
If it's waiting, use threads, and size the pool to your API's limits. If it's working, use processes, and size the pool to your CPU cores. If threads share data, lock it. And let a pool executor manage the workers for you, so errors come back to you instead of disappearing.
In the next post in this series, we'll look at asyncio: a way to run thousands of API calls at once on a single thread. It's the model behind FastAPI and most modern LLM SDKs, and it needs a slightly different way of writing code.
If you're building an AI product and running into problems like these, whether it's an indexing job that doesn't finish before your users log in, an LLM pipeline that hits rate limits the moment you try to speed it up, or AI-generated code that's full of threads but no faster, let's talk. This is exactly the kind of work we do at CreoWis.
We at CreoWis believe in sharing knowledge publicly to help the developer community grow. Let’s collaborate, ideate, and craft passion to deliver awe-inspiring product experiences to the world.
This article is crafted by Syket Bhattachergee, a passionate developer at CreoWis. You can reach out to him on LinkedIn,X/Twitter and follow his work on the GitHub.