Ultra fast python binding for curl-impersonate via cython
Project description
curl_cffi
Python binding for curl-impersonate via cffi.
Documentation | 中文 README | Discuss on Telegram
Unlike other pure python http clients like httpx
or requests
, curl_cffi
can
impersonate browsers' TLS signatures or JA3 fingerprints. If you are blocked by some
website for no obvious reason, you can give this package a try.
Features
- Supports JA3/TLS and http2 fingerprints impersonation.
- Much faster than requests/httpx, on par with aiohttp/pycurl, see benchmarks.
- Mimics requests API, no need to learn another one.
- Pre-compiled, so you don't have to compile on your machine.
- Supports
asyncio
with proxy rotation on each request. - Supports http 2.0, which requests does not.
library | requests | aiohttp | httpx | pycurl | curl_cffi |
---|---|---|---|---|---|
http2 | ❌ | ❌ | ✅ | ✅ | ✅ |
sync | ✅ | ❌ | ✅ | ✅ | ✅ |
async | ❌ | ✅ | ✅ | ❌ | ✅ |
fingerprints | ❌ | ❌ | ❌ | ❌ | ✅ |
speed | 🐇 | 🐇🐇 | 🐇 | 🐇🐇 | 🐇🐇 |
Install
pip install curl_cffi --upgrade
This should work on Linux(x86_64/aarch64), macOS(Intel/Apple Silicon) and Windows(amd64).
If it does not work on you platform, you may need to compile and install curl-impersonate
first and set some environment variables like LD_LIBRARY_PATH
.
To install beta releases:
pip install curl_cffi --pre
Usage
requests-like
from cycurl import requests
# Notice the impersonate parameter
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome110")
print(r.json())
# output: {..., "ja3n_hash": "aa56c057ad164ec4fdcb7a5a283be9fc", ...}
# the js3n fingerprint should be the same as target browser
# http/socks proxies are supported
proxies = {"https": "http://localhost:3128"}
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome110", proxies=proxies)
proxies = {"https": "socks://localhost:3128"}
r = requests.get("https://tls.browserleaks.com/json", impersonate="chrome110", proxies=proxies)
Sessions
# sessions are supported
s = requests.Session()
# httpbin is a http test website
s.get("https://httpbin.org/cookies/set/foo/bar")
print(s.cookies)
# <Cookies[<Cookie foo=bar for httpbin.org />]>
r = s.get("https://httpbin.org/cookies")
print(r.json())
# {'cookies': {'foo': 'bar'}}
Supported impersonate versions, as supported by curl-impersonate:
- chrome99
- chrome100
- chrome101
- chrome104
- chrome107
- chrome110
- chrome99_android
- edge99
- edge101
- safari15_3
- safari15_5
asyncio
from cycurl.requests import AsyncSession
async with AsyncSession() as s:
r = await s.get("https://example.com")
More concurrency:
import asyncio
from cycurl.requests import AsyncSession
urls = [
"https://googel.com/",
"https://facebook.com/",
"https://twitter.com/",
]
async with AsyncSession() as s:
tasks = []
for url in urls:
task = s.get("https://example.com")
tasks.append(task)
results = await asyncio.gather(*tasks)
curl-like
Alternatively, you can use the low-level curl-like API:
from cycurl import Curl, CurlOpt
from io import BytesIO
buffer = BytesIO()
c = Curl()
c.setopt(CURLOPT_URL, b'https://tls.browserleaks.com/json')
c.setopt(CURLOPT_WRITEDATA, buffer)
c.impersonate("chrome110")
c.perform()
c.close()
body = buffer.getvalue()
print(body.decode())
See the docs for more details.
If you are using scrapy, check out this middleware: tieyongjie/scrapy-fingerprint
Acknowledgement
- Originally forked from multippt/python_curl_cffi, which is under the MIT license.
- Headers/Cookies files are copied from httpx, which is under the BSD license.
- Asyncio support is inspired by Tornado's curl http client.
Sponsor
Project details
Release history Release notifications | RSS feed
Download files
Download the file for your platform. If you're not sure which to choose, learn more about installing packages.
Source Distribution
Built Distributions
Hashes for cycurl-0.5.10-cp312-cp312-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 2f8f4c9f2cafde620cbbbbcfa12686d706b9dde199bc7a54f849a67c27f0d7bb |
|
MD5 | bbee30bc83ecd31ba19f659e3a3fa048 |
|
BLAKE2b-256 | b0459ab9b4bc2078d73ab04e55d3f92f4321b9514873838cb4ec9ba30331196e |
Hashes for cycurl-0.5.10-cp312-cp312-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec9653a0d483b8be1a45308ce1f4591ce6717f3b05b299aac57fc1b67fc7a318 |
|
MD5 | 614a43858bef3f37be58334ab571c9e2 |
|
BLAKE2b-256 | 52025cb6075a55fb8720379bd57f3c9231ac8a84b2c1f3a279a4250abe775c2b |
Hashes for cycurl-0.5.10-cp312-cp312-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | ec6dec15e79ee0179ff6a0fb07bf4da304cd15990dbb38b217ed3a3a7e6e155f |
|
MD5 | a152f9f0e419e84bfbe855bd28c71d1a |
|
BLAKE2b-256 | ef2379c94e616d17e7fdfd513e7b8efa256328cf94236c0e62ba3d20e8e231c9 |
Hashes for cycurl-0.5.10-cp311-cp311-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9adf53dbda5069e1baf765c97d8c2b27b28860120a5d3d8ac995f5da589b3e10 |
|
MD5 | 7e72ef8753155f70f2eafdbf03dd40b3 |
|
BLAKE2b-256 | 1c43f2704ad1e16b2f755653cff9a554a74ef2fe4c2d0d63f2a551699863609d |
Hashes for cycurl-0.5.10-cp311-cp311-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0e43e0efc2826b1d350a21fc31eca4877815419579f5de3c5f37ab5be0a7fe01 |
|
MD5 | 4e2aa5f98cd048d646bb7f8a837115bf |
|
BLAKE2b-256 | 8de25c7cd74a84c16d990b15d369aae35235804d40b0a0a2ce3306ce0cb61ece |
Hashes for cycurl-0.5.10-cp311-cp311-macosx_10_9_universal2.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 9672e8eb218d17d267ed799b9ff6e30392c1493f8dd8195d67673a88dc622c2a |
|
MD5 | a39db939945342a807611dd0e162ce33 |
|
BLAKE2b-256 | 06cb227679a52c69fe50591fc83134d60c28372a1ad8b9ae46cf51039a9d4e65 |
Hashes for cycurl-0.5.10-cp310-cp310-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 85b1e72affdd31817d4217015f0bbc406c3b2839135b33bbc3747e6c196a6faf |
|
MD5 | f28168b3e195e4e2d2a2e8b9f5b98f3b |
|
BLAKE2b-256 | 7119d3f7f9da36b1dec2d2cf2947bdabaf96017895508a8ac03b14f6094b3456 |
Hashes for cycurl-0.5.10-cp310-cp310-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 257613dae9e3f1b3ce1f1d156a14b207902c7b79f116c4ae0186215c82842b94 |
|
MD5 | eafb298bd687203033f15cd67661f9df |
|
BLAKE2b-256 | 26935bd5a1e82900ac8cb6d7afe0f3b9f69f8d9f9c9843843243d130d29d7511 |
Hashes for cycurl-0.5.10-cp310-cp310-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3225cde7f3d440493da9e9e2a2822781590f7e4f79f77eeedfa5ab68df11441a |
|
MD5 | 873b517a9112205dcdbec2cc513a1c52 |
|
BLAKE2b-256 | b17eff3dd765bb890b89185c925c2c94250128b8b608ee81f3ffc3376b808460 |
Hashes for cycurl-0.5.10-cp39-cp39-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 3872e32a617d0540011c51a9f11281c976622e1a8a8e832277b9f71eb191947b |
|
MD5 | 1298157e361660046c09c43074a5cbd8 |
|
BLAKE2b-256 | 02eebaca7a1963eaeacc3997a1c010ebe6a0755d15210cd1999c359da5dfd53c |
Hashes for cycurl-0.5.10-cp39-cp39-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | fb195a8499011b0f2d8a164f4bc3e28f34d1bc31d829c915eaf18d901524248b |
|
MD5 | b12481a87ceadeb3adca99fd4cf32280 |
|
BLAKE2b-256 | 26bc4b60cdf2e0018ee6c89bef1b5481cd05ccb7015472c474dfcd5862d5e094 |
Hashes for cycurl-0.5.10-cp39-cp39-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0e9da232fb8fa70fade03c25f5e2776896cc9fe59acbb2a12fd1f5210934ddbd |
|
MD5 | ad8a7958e3f512ce2fc24bf8fe7a5e08 |
|
BLAKE2b-256 | c23dddb45d1a8ce4ca006537eca54d3dd2e4a4fe49e64de85a0b8528635383e9 |
Hashes for cycurl-0.5.10-cp38-cp38-win_amd64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 7e2a902128bc1fcaa0df84fd8695871f1a77470eec89df92a4d144676dc63306 |
|
MD5 | 04f00dbe418f7f691611e24a63969142 |
|
BLAKE2b-256 | 90e160b2e36a04993e29a0e8b1cd10811cc3e3d09ea318249d499fa598d828f6 |
Hashes for cycurl-0.5.10-cp38-cp38-manylinux2014_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 6b1289194636aa3272193f559de6462e4c0430c0230ac36e952faef40f2bbfa3 |
|
MD5 | 8f4de04415f58a221affa286137bea40 |
|
BLAKE2b-256 | c2b00c359cc5f608a5931bb56cbc15201cc9de462627869304b2492b4919cf67 |
Hashes for cycurl-0.5.10-cp38-cp38-macosx_11_0_x86_64.whl
Algorithm | Hash digest | |
---|---|---|
SHA256 | 0b1a7106e53c5d389058faf12a026348932d34054cc45e0758c790ecd76548dd |
|
MD5 | 1cd08153737a331d942439479dd61b69 |
|
BLAKE2b-256 | 76a548ac538bd7f3b15e768dcb73f1de966b85c7d5df437cc3a78754b0e847a7 |