multiprocessing.Pool报pickling error

最新推荐文章于 2026-05-20 08:38:34 发布

原创最新推荐文章于 2026-05-20 08:38:34 发布 · 1.4w 阅读

10 ·

本内容遵循CC 4.0 BY-SA版权协议

标签

#python #多线程 #多进程

Python基础专栏收录该内容

18 篇文章

订阅专栏

本文探讨了在使用Python的multiprocessing模块时遇到的PicklingError问题，并提供了几种有效的解决方案，包括使用线程替代进程、利用copy_reg模块进行方法序列化及采用dill或pathos.multiprocessing替代标准库。

Python3.8

Conda

Python

Python 是一种高级、解释型、通用的编程语言，以其简洁易读的语法而闻名，适用于广泛的应用，包括Web开发、数据分析、人工智能和自动化脚本

现象

multiprocessing.Pool传递一个普通方法(不在class中定义的)时, 能正常工作.

from multiprocessing import Pool

p = Pool(3)
def f(x):
     return x*x

p.map(f, [1,2,3])

但在class中定义的方法使用multiprocessing.Pool会报pickling error错误.

报错代码

# coding: utf8
import multiprocessing


class MyTask(object):
    def task(self, x):
        return x*x

    def run(self):
        pool = multiprocessing.Pool(processes=3)

        a = [1, 2, 3]
        pool.map(self.task, a)


if __name__ == '__main__':
    t = MyTask()
    t.run()

会出现如下异常:

cPickle.PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup __builtin__.instancemethod failed

原因:

stackoverflow上的解释:
Pool methods all use a queue.Queue to pass tasks to the worker processes. Everything that goes through the queue.Queue must be pickable. So, multiprocessing can only transfer Python objects to worker processes which can be pickled. Functions are only picklable if they are defined at the top-level of a module, bound methods are not picklable.

pool方法都使用了queue.Queue将task传递给工作进程。multiprocessing必须将数据序列化以在进程间传递。方法只有在模块的顶层时才能被序列化，跟类绑定的方法不能被序列化，就会出现上面的异常。

解决方法:

用线程替换进程
可以使用copy_reg来规避上面的异常.
dill 或pathos.multiprocesssing ：use pathos.multiprocesssing, instead of multiprocessing. pathos.multiprocessing is a fork of multiprocessing that uses dill. dill can serialize almost anything in python, so you are able to send a lot more around in parallel.

正确代码1

 # coding: utf8
from multiprocessing.pool import ThreadPool as Pool


class MyTask(object):
    def task(self, x):
        return x*x

    def run(self):
        pool = Pool(3)

        a = [1, 2, 3]
        ret = pool.map(self.task, a)
        print ret


if __name__ == '__main__':
    t = MyTask()
    t.run()

正确代码2:

# coding: utf8
import multiprocessing
import types
import copy_reg


def _pickle_method(m):
    if m.im_self is None:
        return getattr, (m.im_class, m.im_func.func_name)
    else:
        return getattr, (m.im_self, m.im_func.func_name)


copy_reg.pickle(types.MethodType, _pickle_method)


class MyTask(object):
    def __init__(self):
        self.__result = []

    def task(self, x):
        return x * x

    def result_collector(self, result):
        self.__result.append(result)

    def run(self):
        pool = multiprocessing.Pool(processes=3)

        a = [1, 2, 3]
        ret = pool.map(self.task, a)
        print ret


if __name__ == '__main__':
    t = MyTask()
    t.run()