[Solved] Loading many images and running out of memory when using NativeJpg

Question

I don’t know for sure if this is your problem, but there is a huge design flaw with your current code. You are creating 1 thread per image. Assuming that you have hundreds or thousands of threads this design cannot scale.

For a start there is a significant overhead associated with creating, starting and terminating threads. You don’t want to pay that overhead time and time again.

But more problematic is the resource overhead for a thread. Each thread has its own private stack space. The address space (1MB) for that stack is reserved (but not committed) when the thread is created. With enough threads you will exhaust your address space, even though your actual memory commit level is still low.

I strongly urge you to abandon that code and start again. You should use one of the established threading libraries. Threading is hard to do well and you need a lot of knowledge and expertise to do it well. Use either OmniThreadLibrary or AsyncCall.

What you are looking for is a simple thread pool with a small number of threads. You should then simply feed tasks (i.e. image file names) to the thread pool and let it manage the processing of those tasks.

Accepted Answer

I don’t know for sure if this is your problem, but there is a huge design flaw with your current code. You are creating 1 thread per image. Assuming that you have hundreds or thousands of threads this design cannot scale.

For a start there is a significant overhead associated with creating, starting and terminating threads. You don’t want to pay that overhead time and time again.

But more problematic is the resource overhead for a thread. Each thread has its own private stack space. The address space (1MB) for that stack is reserved (but not committed) when the thread is created. With enough threads you will exhaust your address space, even though your actual memory commit level is still low.

I strongly urge you to abandon that code and start again. You should use one of the established threading libraries. Threading is hard to do well and you need a lot of knowledge and expertise to do it well. Use either OmniThreadLibrary or AsyncCall.

What you are looking for is a simple thread pool with a small number of threads. You should then simply feed tasks (i.e. image file names) to the thread pool and let it manage the processing of those tasks.