Featured post
multithreading - Python threads all executing on a single core -
i have python program spawns many threads, runs 4 @ time, , each performs expensive operation. pseudocode:
for object in list: t = thread(target=process, args=(object)) # if fewer 4 threads running, t.start(). otherwise, add t queue
but when program run, activity monitor in os x shows 1 of 4 logical cores @ 100% , others @ 0. can't force os i've never had pay attention performance in multi-threaded code before wondering if i'm missing or misunderstanding something.
thanks.
note in many cases (and virtually cases "expensive operation" calculation implemented in python), multiple threads not run concurrently due python's global interpreter lock (gil).
the gil interpreter-level lock. lock prevents execution of multiple threads @ once in python interpreter. each thread wants run must wait gil released other thread, means multi-threaded python application single threaded, right? yes. not exactly. sort of.
cpython uses what’s called “operating system” threads under covers, each time request make new thread made, interpreter calls operating system’s libraries , kernel generate new thread. same java, example. in memory have multiple threads , operating system controls thread scheduled run. on multiple processor machine, means have many threads spread across multiple processors, happily chugging away doing work.
however, while cpython use operating system threads (in theory allowing multiple threads execute within interpreter simultaneously), interpreter forces gil acquired thread before can access interpreter , stack , can modify python objects in memory willy-nilly. latter point why gil exists: gil prevents simultaneous access python objects multiple threads. not save (as illustrated bank example) being lock-sensitive creature; don’t free ride. gil there protect interpreters memory, not sanity.
see global interpreter lock section of jesse noller's post more details.
to around problem, check out python's multiprocessing module.
multiple processes (with judicious use of ipc) are[...] better approach writing apps multi-cpu boxes threads.
- Get link
- X
- Other Apps
Comments
Post a Comment