Tutorial :How can I get a list of locally installed Python modules?



Question:

I would like to get a list of Python modules, which are in my Python installation (UNIX server).

How can you get a list of Python modules installed in your computer?


Solution:1

Solution

My 50 cents for getting a pip freeze-like list from a Python script:

import pip  installed_packages = pip.get_installed_distributions()  installed_packages_list = sorted(["%s==%s" % (i.key, i.version)       for i in installed_packages])  print(installed_packages_list)  

As a (too long) one liner:

sorted(["%s==%s" % (i.key, i.version) for i in pip.get_installed_distributions()])  

Giving:

['behave==1.2.4', 'enum34==1.0', 'flask==0.10.1', 'itsdangerous==0.24',    'jinja2==2.7.2', 'jsonschema==2.3.0', 'markupsafe==0.23', 'nose==1.3.3',    'parse-type==0.3.4', 'parse==1.6.4', 'prettytable==0.7.2', 'requests==2.3.0',   'six==1.6.1', 'vioozer-metadata==0.1', 'vioozer-users-server==0.1',    'werkzeug==0.9.4']  

Scope

This solution applies to the system scope or to a virtual environment scope, and covers packages installed by setuptools, pip and (god forbid) easy_install.

My use case

I added the result of this call to my flask server, so when I call it with http://example.com/exampleServer/environment I get the list of packages installed on the server's virtualenv. It makes debugging a whole lot easier.

Caveats

I have noticed a strange behaviour of this technique - when the Python interpreter is invoked in the same directory as a setup.py file, it does not list the package installed by setup.py.

Steps to reproduce:

Create a virtual environment
$ cd /tmp  $ virtualenv test_env  New python executable in test_env/bin/python  Installing setuptools, pip...done.  $ source test_env/bin/activate  (test_env) $   
Clone a git repo with setup.py
(test_env) $ git clone https://github.com/behave/behave.git  Cloning into 'behave'...  remote: Reusing existing pack: 4350, done.  remote: Total 4350 (delta 0), reused 0 (delta 0)  Receiving objects: 100% (4350/4350), 1.85 MiB | 418.00 KiB/s, done.  Resolving deltas: 100% (2388/2388), done.  Checking connectivity... done.  

We have behave's setup.py in /tmp/behave:

(test_env) $ ls /tmp/behave/setup.py  /tmp/behave/setup.py  
Install the python package from the git repo
(test_env) $ cd /tmp/behave && python setup.py install  running install  ...  Installed /private/tmp/test_env/lib/python2.7/site-packages/enum34-1.0-py2.7.egg  Finished processing dependencies for behave==1.2.5a1  

If we run the aforementioned solution from /tmp

>>> import pip  >>> sorted(["%s==%s" % (i.key, i.version) for i in pip.get_installed_distributions()])  ['behave==1.2.5a1', 'enum34==1.0', 'parse-type==0.3.4', 'parse==1.6.4', 'six==1.6.1']  >>> import os  >>> os.getcwd()  '/private/tmp'  

If we run the aforementioned solution from /tmp/behave

>>> import pip  >>> sorted(["%s==%s" % (i.key, i.version) for i in pip.get_installed_distributions()])  ['enum34==1.0', 'parse-type==0.3.4', 'parse==1.6.4', 'six==1.6.1']  >>> import os  >>> os.getcwd()  '/private/tmp/behave'  

behave==1.2.5a1 is missing from the second example, because the working directory contains behave's setup.py file.

I could not find any reference to this issue in the documentation. Perhaps I shall open a bug for it.


Solution:2

help('modules')  

in a Python shell/prompt.


Solution:3

Now, these methods I tried myself, and I got exactly what was advertised: All the modules.

Alas, really you don't care much about the stdlib, you know what you get with a python install.

Really, I want the stuff that I installed.

What actually, surprisingly, worked just fine was:

pip freeze  

Which returned:

Fabric==0.9.3  apache-libcloud==0.4.0  bzr==2.3b4  distribute==0.6.14  docutils==0.7  greenlet==0.3.1  ipython==0.10.1  iterpipes==0.4  libxml2-python==2.6.21  

I say "surprisingly" because the package install tool is the exact place one would expect to find this functionality, although not under the name 'freeze' but python packaging is so weird, that I am flabbergasted that this tool makes sense. Pip 0.8.2, Python 2.7.


Solution:4

  • In ipython you can type "importTab".

  • In the standard Python interpreter, you can type "help('modules')".

  • At the command-line, you can use pydoc modules.

  • In a script, call pkgutil.iter_modules().


Solution:5

Since pip version 1.3, you've got access to:

pip list  

Which seems to be syntactic sugar for "pip freeze". It will list all of the modules particular to your installation or virtualenv, along with their version numbers. Unfortunately it does not display the current version number of any module, nor does it wash your dishes or shine your shoes.


Solution:6

In normal shell just use

pydoc modules  


Solution:7

I just use this to see currently used modules:

import sys as s  s.modules.keys()  

which shows all modules running on your python.

For all built-in modules use:

s.modules  

Which is a dict containing all modules and import objects.


Solution:8

If we need to list the installed packages in the Python shell, we can use the help command as follows

>>help('modules package')  


Solution:9

I ran into a custom installed python 2.7 on OS X. It required X11 to list modules installed (both using help and pydoc).

To be able to list all modules without installing X11 I ran pydoc as http-server, i.e.:

pydoc -p 12345  

Then it's possible to direct Safari to http://localhost:12345/ to see all modules.


Solution:10

Very simple searching using pkgutil.iter_modules

from pkgutil import iter_modules  a=iter_modules()  while True:      try: x=a.next()      except: break      if 'searchstr' in x[1]: print x[1]  


Solution:11

on windows, Enter this in cmd

c:\python\libs>python -m pip freeze  


Solution:12

Aside from using pip freeze I have been installing yolk in my virtual environments.


Solution:13

  1. to get all available modules, run sys.modules
  2. to get all installed modules (read: installed by pip), you may look at pip.get_installed_distributions()

For the second purpose, example code:

import pip  for package in pip.get_installed_distributions():      name = package.project_name # SQLAlchemy, Django, Flask-OAuthlib      key = package.key # sqlalchemy, django, flask-oauthlib      module_name = package._get_metadata("top_level.txt") # sqlalchemy, django, flask_oauthlib      location = package.location # virtualenv lib directory etc.      version = package.version # version number  


Solution:14

This solution is primary based on modules importlib and pkgutil and work with CPython 3.4 and CPython 3.5, but has no support for the CPython 2.


Explanation

  1. sys.builtin_module_names - names all built-in modules (look my answer here)
  2. pkgutil.iter_modules() - returns an information about all available modules
  3. importlib.util.find_spec() - returns an information about importing module, if exists
  4. BuiltinImporter - an importer for built-in modules (docs)
  5. SourceFileLoader - an importer for a standard Python module (by default has extension *.py) (docs)
  6. ExtensionFileLoader - an importer for modules as shared library (written on the C or C++)

Full code

import sys  import os  import shutil  import pkgutil  import importlib  import collections    if sys.version_info.major == 2:      raise NotImplementedError('CPython 2 is not supported yet')      def main():        # name this file (module)      this_module_name = os.path.basename(__file__).rsplit('.')[0]        # dict for loaders with their modules      loaders = collections.OrderedDict()        # names`s of build-in modules      for module_name in sys.builtin_module_names:            # find an information about a module by name          module = importlib.util.find_spec(module_name)            # add a key about a loader in the dict, if not exists yet          if module.loader not in loaders:              loaders[module.loader] = []            # add a name and a location about imported module in the dict          loaders[module.loader].append((module.name, module.origin))        # all available non-build-in modules      for module_name in pkgutil.iter_modules():            # ignore this module          if this_module_name == module_name[1]:              continue            # find an information about a module by name          module = importlib.util.find_spec(module_name[1])            # add a key about a loader in the dict, if not exists yet          loader = type(module.loader)          if loader not in loaders:              loaders[loader] = []            # add a name and a location about imported module in the dict          loaders[loader].append((module.name, module.origin))        # pretty print      line = '-' * shutil.get_terminal_size().columns      for loader, modules in loaders.items():          print('{0}\n{1}: {2}\n{0}'.format(line, len(modules), loader))          for module in modules:              print('{0:30} | {1}'.format(module[0], module[1]))      if __name__ == '__main__':      main()  

Usage

For the CPython3.5 (truncated)

$ python3.5 python_modules_info.py   ------------------------------------------------------------------------------------------------------------------------------------------------------------------------  30: <class '_frozen_importlib.BuiltinImporter'>  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------  _ast                           | built-in  _codecs                        | built-in  _collections                   | built-in  _functools                     | built-in  _imp                           | None  _io                            | built-in  _locale                        | built-in  _operator                      | built-in  _signal                        | built-in  _sre                           | built-in  _stat                          | built-in  _string                        | built-in  _symtable                      | built-in  _thread                        | built-in  (****************************truncated*******************************)  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------  227: <class '_frozen_importlib_external.SourceFileLoader'>  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------  __future__                     | /usr/local/lib/python3.5/__future__.py  _bootlocale                    | /usr/local/lib/python3.5/_bootlocale.py  _collections_abc               | /usr/local/lib/python3.5/_collections_abc.py  _compat_pickle                 | /usr/local/lib/python3.5/_compat_pickle.py  _compression                   | /usr/local/lib/python3.5/_compression.py  _dummy_thread                  | /usr/local/lib/python3.5/_dummy_thread.py  _markupbase                    | /usr/local/lib/python3.5/_markupbase.py  _osx_support                   | /usr/local/lib/python3.5/_osx_support.py  _pydecimal                     | /usr/local/lib/python3.5/_pydecimal.py  _pyio                          | /usr/local/lib/python3.5/_pyio.py  _sitebuiltins                  | /usr/local/lib/python3.5/_sitebuiltins.py  (****************************truncated*******************************)  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------  64: <class '_frozen_importlib_external.ExtensionFileLoader'>  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------  _bisect                        | /usr/local/lib/python3.5/lib-dynload/_bisect.cpython-35m-x86_64-linux-gnu.so  _bz2                           | /usr/local/lib/python3.5/lib-dynload/_bz2.cpython-35m-x86_64-linux-gnu.so  _codecs_cn                     | /usr/local/lib/python3.5/lib-dynload/_codecs_cn.cpython-35m-x86_64-linux-gnu.so  _codecs_hk                     | /usr/local/lib/python3.5/lib-dynload/_codecs_hk.cpython-35m-x86_64-linux-gnu.so  _codecs_iso2022                | /usr/local/lib/python3.5/lib-dynload/_codecs_iso2022.cpython-35m-x86_64-linux-gnu.so  (****************************truncated*******************************)  

For the CPython3.4 (truncated)

$ python3.4 python_modules_info.py  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------  54: <class '_frozen_importlib.BuiltinImporter'>  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------  _ast                           | built-in  _bisect                        | built-in  _codecs                        | built-in  _collections                   | built-in  _datetime                      | built-in  _elementtree                   | built-in  _functools                     | built-in  _heapq                         | built-in  _imp                           | None  _io                            | built-in  _locale                        | built-in  _md5                           | built-in  _operator                      | built-in  _pickle                        | built-in  _posixsubprocess               | built-in  _random                        | built-in  (****************************truncated*******************************)  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------  246: <class '_frozen_importlib.SourceFileLoader'>  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------  __future__                     | /usr/lib/python3.4/__future__.py  _bootlocale                    | /usr/lib/python3.4/_bootlocale.py  _collections_abc               | /usr/lib/python3.4/_collections_abc.py  _compat_pickle                 | /usr/lib/python3.4/_compat_pickle.py  _dummy_thread                  | /usr/lib/python3.4/_dummy_thread.py  _markupbase                    | /usr/lib/python3.4/_markupbase.py  _osx_support                   | /usr/lib/python3.4/_osx_support.py  _pyio                          | /usr/lib/python3.4/_pyio.py  (****************************truncated*******************************)  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------  44: <class '_frozen_importlib.ExtensionFileLoader'>  ------------------------------------------------------------------------------------------------------------------------------------------------------------------------  _bz2                           | /usr/lib/python3.4/lib-dynload/_bz2.cpython-34m-x86_64-linux-gnu.so  _codecs_cn                     | /usr/lib/python3.4/lib-dynload/_codecs_cn.cpython-34m-x86_64-linux-gnu.so  _codecs_hk                     | /usr/lib/python3.4/lib-dynload/_codecs_hk.cpython-34m-x86_64-linux-gnu.so  _codecs_iso2022                | /usr/lib/python3.4/lib-dynload/_codecs_iso2022.cpython-34m-x86_64-linux-gnu.so  _codecs_jp                     | /usr/lib/python3.4/lib-dynload/_codecs_jp.cpython-34m-x86_64-linux-gnu.so  _codecs_kr                     | /usr/lib/python3.4/lib-dynload/_codecs_kr.cpython-34m-x86_64-linux-gnu.so  _codecs_tw                     | /usr/lib/python3.4/lib-dynload/_codecs_tw.cpython-34m-x86_64-linux-gnu.so  _crypt                         | /usr/lib/python3.4/lib-dynload/_crypt.cpython-34m-x86_64-linux-gnu.so  (****************************truncated*******************************)  


Solution:15

In case you have an anaconda python distribution installed, you could also use

$conda list  

in addition to solutions described above.


Solution:16

pip freeze does it all finding packages however one can simply write the following command to list all paths where python packages are.

>>> import site; site.getsitepackages()  ['/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']  


Solution:17

As of pip 10, the accepted answer will no longer work. The development team has removed access to the get_installed_distributions routine. There is an alternate function in the setuptools for doing the same thing. Here is an alternate version that works with pip 10:

import pkg_resources  installed_packages = pkg_resources.working_set  installed_packages_list = sorted(["%s==%s" % (i.key, i.version)       for i in installed_packages])  print(installed_packages_list)  

Please let me know if it will or won't work in previous versions of pip, too.


Solution:18

From the shell

ls site-packages  

If that's not helpful, you can do this.

import sys  import os  for p in sys.path:      print os.listdir( p )  

And see what that produces.


Note:If u also have question or solution just comment us below or mail us on toontricks1994@gmail.com
Previous
Next Post »