Repository navigation
Importing numba is slow #4927
Description
Hello,
on my Windows workstation with >1200 pip installed packages, import numba is very slow:
py -3.7-64 -X importtime -c "import numba"
import time: self [us] | cumulative | imported package
import time: 3069020 | 3235357 | pkg_resources
import time: 2526 | 3237883 | numba.entrypoints
import time: 31084 | 4448066 | numba
The culprit is
Line 3 in 310fe84
Importing pkg_resources is known to be excruciatingly slow. See pypa/setuptools#510
How about replacing the use of pkg_resources for entrypoints by importlib_metadata or entrypoints if possible?
Reporting a bug
- I am using the latest released version of Numba (most recent is visible in
the change log (https://github.com/numba/numba/blob/master/CHANGE_LOG). - I have included below a minimal working reproducer (if you are unsure how
to write one see http://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports).
Activity
stuartarchibald commentedon Dec 5, 2019
Thanks for the report. IIRC this approach mirrors that taken in pandas for the plotting backends https://github.com/pandas-dev/pandas/blob/0c2b1db198ce628ca889ad25c1179fff4ab3337c/pandas/plotting/_core.py#L1613-L1615 but it would appear they delay this scanning until needed. It seems like the suggested fixes involve taking on dependencies. CC @seibert, thoughts?
seibert commentedon Dec 5, 2019
Adding a dependency on
import_metadataI think would be fine. It is pure Python, and already packaged in Anaconda as well. It also appears in the standard library as of Python 3.8:https://docs.python.org/3/library/importlib.metadata.html
so this seems like heading toward where Python is already going.
stuartarchibald commentedon Dec 5, 2019
Task here is to remove the use of
pkg_resourcesand replace withimportlib_metadataand fix up the builds for that dependency chain.