httpimport 0.5.15

Example - In a Nutshell

>>> import httpimport
>>> httpimport.__all__
['HttpImporter', 'add_remote_repo', 'remove_remote_repo', 'remote_repo', 'github_repo', 'bitbucket_repo']

>>> with httpimport.remote_repo(['package1','package2','package3'], 'http://my-codes.example.com/python_packages'):
...     import package1
...

>>> with httpimport.github_repo('operatorequals', 'covertutils', branch = master):
...     import covertutils
... # Also works with 'bitbucket_repo'

>>> # A depends to B and B depends to C (A, B, C : Python modules/packages in different domains):
>>> # A exists in "repo_a.my-codes.example.com" |
>>> # B exists in "repo_b.my-codes.example.com" | <-- Different domains
>>> # C exists in "repo_c.my-codes.example.com" |
>>> with httpimport.remote_repo(['C'], 'http://repo_c.my-codes.example.com/python_packages'):
...  with httpimport.remote_repo(['B'], 'http://repo_b.my-codes.example.com/python_packages'):
...     with httpimport.remote_repo(['A'], 'http://repo_a.my-codes.example.com/python_packages'):
...     import A
... # Asks for A, Searches for B, Asks for B, Searches for C, Asks for C --> Resolves --> Imports A
>>>

>>> module_object = httpimport.load('package1', 'http://my-codes.example.com/python_packages')
>>> module_object
<module 'package1' from 'http://my-codes.example.com/python_packages/package1/__init__.py'>

Example - The Whole Picture

Using the SimpleHTTPServer, a whole directory can be served through HTTP as follows:

user@hostname:/tmp/test_directory$ ls -R
.:
test_package

./test_package:
__init__.py  __init__.pyc  module1.py  module2.py
user@hostname:/tmp/test_directory$
user@hostname:/tmp/test_directory$ python -m SimpleHTTPServer &
[1] 9565
Serving HTTP on 0.0.0.0 port 8000 ...

user@hostname:/tmp/test_directory$
user@hostname:/tmp/test_directory$
user@hostname:/tmp/test_directory$ curl http://localhost:8000/test_package/module1.py
127.0.0.1 - - [22/Aug/2017 17:42:49] "GET /test_package/module1.py HTTP/1.1" 200 -


def dummy_func() : return 'Function Loaded'


class dummy_class :

    def dummy_method(self) : return 'Class and method loaded'


dummy_str = 'Constant Loaded'

user@hostname:/tmp/test_directory$
user@hostname:/tmp/test_directory$ curl http://localhost:8000/test_package/__init__.py
127.0.0.1 - - [22/Aug/2017 17:45:20] "GET /test_package/__init__.py HTTP/1.1" 200 -
__all__ = ["module1", "module2"]

Using this simple built-in feature of Py2/3, a custom importer can been created, that given a base URL and a list of package names, it fetches and automatically loads all modules and packages to the local namespace.

Usage

Making the HTTP repo

user@hostname:/tmp/test_directory$ ls -R
.:
test_package

./test_package:
__init__.py  __init__.pyc  module1.py  module2.py

user@hostname:/tmp/test_directory$
user@hostname:/tmp/test_directory$ python -m SimpleHTTPServer
Serving HTTP on 0.0.0.0 port 8000 ...

Importing Remotely

add_remote_repo() and remove_remote_repo()

These 2 functions will add and remove to the default sys.meta_path custom HttpImporter objects, given the URL they will look for packages/modules and a list of packages/modules its one can serve.

>>> import test_package
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named test_package
>>>
>>> from httpimport import add_remote_repo, remove_remote_repo
>>> # In the given URL the 'test_package/' is available
>>> add_remote_repo(['test_package'], 'http://localhost:8000/') #
>>> import test_package
>>>
>>> remove_remote_repo('http://localhost:8000/')
>>> import test_package.module1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named module1

The remote_repo() context

Adding and removing Remote Repos can be a pain, specially if there are packages that are available in more than one repos. So the with keyword does the trick again:

>>> from httpimport import remote_repo
>>>
>>>
>>> with remote_repo(['test_package'], 'http://localhost:8000/') :
...     from test_package import module1
...
>>>
>>> from test_package import module2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name module2

>>> module1.dummy_str
'Constant Loaded'
>>> module1.dummy_func
<function dummy_func at 0x7f7a8a170410>

The Tiny Test for your amusement

The test.py file contains a minimal test. Try changing working directories and package names and see what happens…

$ python test.py
serving at port 8000
127.0.0.1 - - [22/Aug/2017 17:36:44] code 404, message File not found
127.0.0.1 - - [22/Aug/2017 17:36:44] "GET /test_package/module1/__init__.py HTTP/1.1" 404 -
127.0.0.1 - - [22/Aug/2017 17:36:44] "GET /test_package/module1.py HTTP/1.1" 200 -
Constant Loaded
Function Loaded
Class and method loaded

Here is an example with my beloved covertutils project:

>>>
>>> import covertutils
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named covertutils
>>> # covertutils is not available through normal import!
>>>
>>> covertutils_url = 'https://raw.githubusercontent.com/operatorequals/covertutils/master/'
>>>
>>> from httpimport import remote_repo
>>>
>>> with remote_repo(['covertutils'], covertutils_url) :
...     import covertutils
...
>>> print covertutils.__author__
John Torakis - operatorequals

The dedicated github_repo() context:

>>> from httpimport import github_repo
>>> with github_repo( 'operatorequals', 'covertutils', ) :
...     import covertutils
...
>>> covertutils.__author__
'John Torakis - operatorequals'
>>>

What about branches?

>>> from httpimport import github_repo
>>> with github_repo( 'operatorequals', 'covertutils', branch='py3_compatibility' ) :
...     import covertutils
...
>>> covertutils.__author__
'John Torakis - operatorequals'
>>>

And ad-hoc commits too?

What if you need to stick to a fixed -known to work- commit?

>>> from httpimport import github_repo
>>> with github_repo( 'operatorequals', 'covertutils', commit='cf3f78c77c437edf2c291bd5b4ed27e0a93e6a77' ) :
...     import covertutils
...
>>> covertutils.__author__
'John Torakis - operatorequals'
>>>

The newer sibling bitbucket_repo() (as of 0.5.9)

>>> with bitbucket_repo('atlassian', 'python-bitbucket', module='pybitbucket'):
...     import pybitbucket
...
>>>

>>> from httpimport import remote_repo
>>> with remote_repo(['B'],"http://example.com/b_repo/") :
...     with remote_repo(['A'],"http://example.com/a_repo/") :
...             import A
...
[!] 'B' not found in HTTP repository. Moving to next Finder.
>>>
>>> A
<module 'A' from 'http://example.com/a_repo/A/__init__.py'>
>>> B
<module 'B' from 'http://example.com/a_repo/B.py'>
>>>

>>> import httpimport
>>> pack1 = httpimport.load('random-package','http://localhost:8000/')
>>> pack1
<module 'random-package' from 'http://localhost:8000//random-package/__init__.py'>
>>>
>>> # Trying to load 'os' module from the URL will fail, as it won't delegate to to other Finders/Loaders.
>>> httpimport.load('os','http://localhost:8000/')
[!] 'non-existent-package' not found in HTTP repository. Moving to next Finder.
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "httpimport.py", line 287, in load
    raise ImportError("Module '%s' cannot be imported from '%s'" % (module_name, url) )
ImportError: Module 'os' cannot be imported from 'http://localhost:8000/'

And no data touches the disk, nor any virtual environment. The import happens just to the running Python process!

Life suddenly got simpler for Python module testing!!!

Imagine the breeze of testing Pull Requests and packages that you aren’t sure they will work for you!

>>> from httpimport import *
>>>
>>> import logging
>>> logging.getLogger('httpimport').setLevel(logging.DEBUG)
>>>
>>> with github_repo('operatorequals','covertutils') :
...     import covertutils
...
FINDER=================
[!] Searching covertutils
[!] Path is None
[@] Checking if connection is HTTPS secure >
[@] Checking if in declared remote module names >
[@] Checking if built-in >
[@] Checking if it is name repetition >
[*]Module/Package 'covertutils' can be loaded!
LOADER=================
[+] Loading covertutils
[+] Trying to import as package from: 'https://raw.githubusercontent.com/operatorequals/covertutils/master//covertutils/__init__.py'
[+] Importing 'covertutils'
[+] Ready to execute 'covertutils' code
[+] 'covertutils' imported succesfully!
>>>

Preventing the disaster (setting httpimport.INSECURE flag):

>>> import httpimport
>>>
>>> # Importing from plain HTTP ...
>>> httpimport.load('test_module', 'http://localhost:8000//')
[!] Using non HTTPS URLs ('http://localhost:8000//') can be a security hazard!
[-] 'httpimport.INSECURE is not set! Aborting...
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "httpimport.py", line 302, in load
    raise ImportError("Module '%s' cannot be imported from URL: '%s'" % (module_name, url) )
ImportError: Module 'test_module' cannot be imported from URL: 'http://localhost:8000/'
>>> # ... Throws Error!
>>>
>>> # Importing from plain HTTP has to be DELIBERATELY enabled!
>>> httpimport.INSECURE = True
>>> httpimport.load('test_module', 'http://localhost:8000//')
[!] Using non HTTPS URLs ('http://localhost:8000//') can be a security hazard!
<module 'test_module' from 'http://localhost:8000//test_module.py'>
>>> # Succeeded!

You have been warned! Use HTTPS URLs with httpimport!

Did I hear you say “Staging protocol for covertutils backdoors”?

Technique documentation on using ``httpimport` to stage covertutils backdoor code <http://covertutils.readthedocs.io/en/latest/staging_exec.html>`__, making EXE packed and unreadable code load non-included module dependencies.