[Revert me] Experiment with Python-Ubuntu combinations

boto3 depends on botocore which in turn depends on urllib3.

.github/dependabot.yml

@@ -0,0 +1,11 @@
+version: 2
+
+updates:
+  - package-ecosystem: pip
+    directory: "/"
+    schedule:
+      interval: weekly
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: weekly

.github/workflows/main.yml

@@ -0,0 +1,59 @@
+name: Test
+
+on:
+  push:
+  pull_request:
+  workflow_dispatch:
+
+jobs:
+  build:
+    runs-on: ${{ matrix.runs-on }}
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - python-version: "3.7"
+            runs-on: ubuntu-20.04
+          - python-version: "3.8"
+            runs-on: ubuntu-20.04
+          - python-version: "3.9"
+            runs-on: ubuntu-20.04
+          - python-version: "3.10"
+            runs-on: ubuntu-22.04
+          - python-version: "3.11"
+            runs-on: ubuntu-22.04
+          - python-version: "pypy-3.7"
+            runs-on: ubuntu-20.04
+          - python-version: "pypy-3.8"
+            runs-on: ubuntu-20.04
+          - python-version: "pypy-3.9"
+            runs-on: ubuntu-20.04
+          - python-version: "pypy-3.10"
+            runs-on: ubuntu-22.04
+          - python-version: "pypy-3.11"
+            runs-on: ubuntu-22.04
+
+    steps:
+      - uses: actions/checkout@v3.5.2
+
+      - name: Set up Python ${{ matrix.python-version }}
+        uses: actions/setup-python@v4
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - name: Install project dependencies
+        run: |
+          pip install --upgrade pip
+          pip install codecov tox tox-gh-actions
+
+      - name: "Debug OpenSSL version used with Python ${{ matrix.python-version }}"
+        run: |
+          which python
+          python --version
+          python -c 'import ssl; print(ssl.OPENSSL_VERSION_INFO)'
+
+      - name: Run tests with tox
+        run: tox
+
+      - name: Run coverage
+        run: codecov

.travis.yml

@@ -1,26 +0,0 @@
-dist: xenial
-language: python
-matrix:
-  include:
-    # Only run lint on a single 3.x
-    - env: TOX_SUFFIX="lint"
-      python: "3.7"
-
-python:
-  - "3.5"
-  - "3.6"
-  - "3.7"
-  - "3.8"
-  - "pypy3"
-
-before_install:
-  - openssl version
-  - sudo apt-get install libgnutls28-dev
-
-install:
-  - pip install tox-travis codecov
-  - if [[ $TOX_SUFFIX != 'lint' ]]; then python setup.py install ; fi
-script:
-  - tox
-after_success:
-  - codecov

README.rst

@@ -62,8 +62,8 @@ more details
    :target: https://pypi.python.org/pypi/vcrpy
 .. |Python versions| image:: https://img.shields.io/pypi/pyversions/vcrpy.svg
    :target: https://pypi.python.org/pypi/vcrpy
-.. |Build Status| image:: https://secure.travis-ci.org/kevin1024/vcrpy.svg?branch=master
-   :target: http://travis-ci.org/kevin1024/vcrpy
+.. |Build Status| image:: https://github.com/kevin1024/vcrpy/actions/workflows/main.yml/badge.svg
+   :target: https://github.com/kevin1024/vcrpy/actions
 .. |Gitter| image:: https://badges.gitter.im/Join%20Chat.svg
    :alt: Join the chat at https://gitter.im/kevin1024/vcrpy
    :target: https://gitter.im/kevin1024/vcrpy?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge

docs/advanced.rst

@@ -271,7 +271,7 @@ You can also do response filtering with the
 similar to the above ``before_record_request`` - you can
 mutate the response, or return ``None`` to avoid recording
 the request and response altogether. For example to hide
-sensitive data from the request body:
+sensitive data from the response body:
 
 .. code:: python
 
@@ -404,3 +404,25 @@ the Cassette ``allow_playback_repeats`` option.
         for x in range(10):
             response = urllib2.urlopen('http://www.zombo.com/').read()
         assert cass.all_played
+
+Discards Cassette on Errors
+---------------------------
+
+By default VCR will save the cassette file even when there is any error inside
+the enclosing context/test.
+
+If you want to save the cassette only when the test succeeds, set the Cassette
+``record_on_exception`` option to ``False``.
+
+.. code:: python
+
+    try:
+        my_vcr = VCR(record_on_exception=False)
+        with my_vcr.use_cassette('fixtures/vcr_cassettes/synopsis.yaml') as cass:
+            response = urllib2.urlopen('http://www.zombo.com/').read()
+            raise RuntimeError("Oops, something happened")
+    except RuntimeError:
+        pass
+
+    # Since there was an exception, the cassette file hasn't been created.
+    assert not os.path.exists('fixtures/vcr_cassettes/synopsis.yaml')

docs/changelog.rst

@@ -1,12 +1,24 @@
 Changelog
 ---------
 
-For a full list of triaged issues, bugs and PRs and what release they are targetted for please see the following link.
+For a full list of triaged issues, bugs and PRs and what release they are targeted for please see the following link.
 
 `ROADMAP MILESTONES <https://github.com/kevin1024/vcrpy/milestones>`_
 
 All help in providing PRs to close out bug issues is appreciated. Even if that is providing a repo that fully replicates issues. We have very generous contributors that have added these to bug issues which meant another contributor picked up the bug and closed it out.
 
+-  4.2.1
+    - Fix a bug where the first request in a redirect chain was not being recorded with aiohttp
+    - Various typos and small fixes, thanks @jairhenrique, @timgates42
+-  4.2.0
+    - Drop support for python < 3.7, thanks @jairhenrique, @IvanMalison, @AthulMuralidhar
+    - Various aiohtt bigfixes (thanks @pauloromeira and boechat107)
+    - Bugfix: filter_post_data_parameters not working with aiohttp.  Thank you @vprakashplanview, @scop, @jairhenrique, and @cinemascop89
+    - Bugfix: Some random misspellings (thanks @scop)
+    - Migrate the CI suite to Github Actions from Travis (thanks @jairhenrique and @cclauss)
+    - Various documentation and code misspelling fixes (thanks @scop and @Justintime50)
+    - Bugfix: httpx support (select between allow_redirects/follow_redirects) (thanks @immerrr)
+    - Bugfix: httpx support (select between allow_redirects/follow_redirects) (thanks @immerrr)
 -  4.1.1
     - Fix HTTPX support for versions greater than 0.15 (thanks @jairhenrique)
     - Include a trailing newline on json cassettes (thanks @AaronRobson)
@@ -100,7 +112,7 @@ All help in providing PRs to close out bug issues is appreciated. Even if that i
     - decode_compressed_response option and filter (thanks @jayvdb).
 -  1.7.4 [#217] 
     - Make use_cassette decorated functions actually return a value (thanks @bcen). 
-    - [#199] Fix path transfromation defaults.
+    - [#199] Fix path transformation defaults.
     - Better headers dictionary management.
 -  1.7.3 [#188] 
     - ``additional_matchers`` kwarg on ``use_cassette``.
@@ -203,7 +215,7 @@ All help in providing PRs to close out bug issues is appreciated. Even if that i
 -  0.3.4
     - Bugfix: close file before renaming it. This fixes an issue on Windows. Thanks @smallcode for the fix.
 -  0.3.3
-    - Bugfix for error message when an unreigstered custom matcher was used
+    - Bugfix for error message when an unregistered custom matcher was used
 -  0.3.2
     - Fix issue with new config syntax and the ``match_on`` parameter. Thanks, @chromy!
 -  0.3.1

docs/conf.py

@@ -94,7 +94,7 @@ version = release = find_version("..", "vcr", "__init__.py")
 #
 # This is also used if you do content translation via gettext catalogs.
 # Usually you set "language" from the command line for these cases.
-language = None
+language = "en"
 
 # There are two options for replacing |today|: either, you set today to some
 # non-false value, then it is used:

docs/configuration.rst

@@ -42,7 +42,8 @@ This can be configured by changing the ``match_on`` setting.
 The following options are available :
 
 -  method (for example, POST or GET)
--  uri (the full URI.)
+-  uri (the full URI)
+-  scheme (for example, HTTP or HTTPS)
 -  host (the hostname of the server receiving the request)
 -  port (the port of the server receiving the request)
 -  path (the path of the request)

docs/contributing.rst

@@ -96,11 +96,11 @@ The test suite is pretty big and slow, but you can tell tox to only run specific
 
     tox -e {pyNN}-{HTTP_LIBRARY} -- <pytest flags passed through>
 
-    tox -e py36-requests -- -v -k "'test_status_code or test_gzip'"
+    tox -e py37-requests -- -v -k "'test_status_code or test_gzip'"
     tox -e py37-requests -- -v --last-failed
 
 This will run only tests that look like ``test_status_code`` or
-``test_gzip`` in the test suite, and only in the python 3.6 environment
+``test_gzip`` in the test suite, and only in the python 3.7 environment
 that has ``requests`` installed.
 
 Also, in order for the boto tests to run, you will need an AWS key.
@@ -130,10 +130,10 @@ in this example::
     pip install tox tox-pyenv
 
     # Install supported versions (at time of writing), this does not activate them
-    pyenv install 3.5.9 3.6.9 3.7.5 3.8.0 pypy3.6-7.2.0
+    pyenv install 3.7.5 3.8.0 pypy3.8
 
     # This activates them
-    pyenv local 3.5.9 3.6.9 3.7.5 3.8.0 pypy3.6-7.2.0
+    pyenv local 3.7.5 3.8.0 pypy3.8
 
     # Run the whole test suite
     tox

docs/installation.rst

@@ -9,7 +9,7 @@ with pip::
 Compatibility
 -------------
 
-VCR.py supports Python 3.5+, and `pypy <http://pypy.org>`__.
+VCR.py supports Python 3.7+, and `pypy <http://pypy.org>`__.
 
 The following HTTP libraries are supported:
 

docs/usage.rst

@@ -4,10 +4,10 @@ Usage
 .. code:: python
 
     import vcr
-    import urllib2
+    import urllib
 
     with vcr.use_cassette('fixtures/vcr_cassettes/synopsis.yaml'):
-        response = urllib2.urlopen('http://www.iana.org/domains/reserved').read()
+        response = urllib.request.urlopen('http://www.iana.org/domains/reserved').read()
         assert 'Example domains' in response
 
 Run this test once, and VCR.py will record the HTTP request to
@@ -25,7 +25,7 @@ look like this:
 
     @vcr.use_cassette('fixtures/vcr_cassettes/synopsis.yaml')
     def test_iana():
-        response = urllib2.urlopen('http://www.iana.org/domains/reserved').read()
+        response = urllib.request.urlopen('http://www.iana.org/domains/reserved').read()
         assert 'Example domains' in response
 
 When using the decorator version of ``use_cassette``, it is possible to
@@ -35,7 +35,7 @@ omit the path to the cassette file.
 
     @vcr.use_cassette()
     def test_iana():
-        response = urllib2.urlopen('http://www.iana.org/domains/reserved').read()
+        response = urllib.request.urlopen('http://www.iana.org/domains/reserved').read()
         assert 'Example domains' in response
 
 In this case, the cassette file will be given the same name as the test

pyproject.toml

@@ -1,2 +1,9 @@
 [tool.black]
 line-length=110
+
+[tool.isort]
+line_length = 110
+known_first_party = "vcrpy"
+multi_line_output = 3
+use_parentheses = true
+include_trailing_comma = true

setup.py

@@ -5,7 +5,7 @@ import os
 import re
 import sys
 
-from setuptools import setup, find_packages
+from setuptools import find_packages, setup
 from setuptools.command.test import test as TestCommand
 
 long_description = open("README.rst", "r").read()
@@ -46,8 +46,7 @@ install_requires = [
     "PyYAML",
     "wrapt",
     "six>=1.5",
-    'yarl; python_version>="3.6"',
-    'yarl<1.4; python_version=="3.5"',
+    "yarl",
 ]
 
 setup(
@@ -60,7 +59,7 @@ setup(
     author_email="me@kevinmccarthy.org",
     url="https://github.com/kevin1024/vcrpy",
     packages=find_packages(exclude=["tests*"]),
-    python_requires=">=3.5",
+    python_requires=">=3.7",
     install_requires=install_requires,
     license="MIT",
     tests_require=["pytest", "mock", "pytest-httpbin"],
@@ -70,10 +69,11 @@ setup(
         "Intended Audience :: Developers",
         "Programming Language :: Python",
         "Programming Language :: Python :: 3",
-        "Programming Language :: Python :: 3.5",
-        "Programming Language :: Python :: 3.6",
         "Programming Language :: Python :: 3.7",
         "Programming Language :: Python :: 3.8",
+        "Programming Language :: Python :: 3.9",
+        "Programming Language :: Python :: 3.10",
+        "Programming Language :: Python :: 3.11",
         "Programming Language :: Python :: 3 :: Only",
         "Programming Language :: Python :: Implementation :: CPython",
         "Programming Language :: Python :: Implementation :: PyPy",

tests/integration/aiohttp_utils.py

@@ -2,7 +2,6 @@
 import asyncio
 
 import aiohttp
-from aiohttp.test_utils import TestClient
 
 
 async def aiohttp_request(loop, method, url, output="text", encoding="utf-8", content_type=None, **kwargs):

tests/integration/cassettes/test_httpx_test_test_behind_proxy.yml

@@ -13,7 +13,7 @@ interactions:
       user-agent:
       - python-httpx/0.12.1
     method: GET
-    uri: https://httpbin.org/headers
+    uri: https://mockbin.org/headers
   response:
     content: "{\n  \"headers\": {\n    \"Accept\": \"*/*\", \n    \"Accept-Encoding\"\
       : \"gzip, deflate, br\", \n    \"Host\": \"httpbin.org\", \n    \"User-Agent\"\

tests/integration/conftest.py

@@ -0,0 +1,17 @@
+import pytest
+
+
+@pytest.fixture(params=["https", "http"])
+def scheme(request):
+    """Fixture that returns both http and https."""
+    return request.param
+
+
+@pytest.fixture
+def mockbin(scheme):
+    return scheme + "://mockbin.org"
+
+
+@pytest.fixture
+def mockbin_request_url(mockbin):
+    return mockbin + "/request"

tests/integration/test_aiohttp.py

@@ -8,6 +8,7 @@ asyncio = pytest.importorskip("asyncio")
 aiohttp = pytest.importorskip("aiohttp")
 
 import vcr  # noqa: E402
+
 from .aiohttp_utils import aiohttp_app, aiohttp_request  # noqa: E402
 
 
@@ -33,14 +34,9 @@ def post(url, output="text", **kwargs):
     return request("POST", url, output="text", **kwargs)
 
 
-@pytest.fixture(params=["https", "http"])
-def scheme(request):
-    """Fixture that returns both http and https."""
-    return request.param
+def test_status(tmpdir, mockbin_request_url):
+    url = mockbin_request_url
 
-
-def test_status(tmpdir, scheme):
-    url = scheme + "://httpbin.org"
     with vcr.use_cassette(str(tmpdir.join("status.yaml"))):
         response, _ = get(url)
 
@@ -51,8 +47,8 @@ def test_status(tmpdir, scheme):
 
 
 @pytest.mark.parametrize("auth", [None, aiohttp.BasicAuth("vcrpy", "test")])
-def test_headers(tmpdir, scheme, auth):
-    url = scheme + "://httpbin.org"
+def test_headers(tmpdir, auth, mockbin_request_url):
+    url = mockbin_request_url
     with vcr.use_cassette(str(tmpdir.join("headers.yaml"))):
         response, _ = get(url, auth=auth)
 
@@ -67,8 +63,9 @@ def test_headers(tmpdir, scheme, auth):
         assert "yarl.URL" not in cassette.data[0]
 
 
-def test_case_insensitive_headers(tmpdir, scheme):
-    url = scheme + "://httpbin.org"
+def test_case_insensitive_headers(tmpdir, mockbin_request_url):
+    url = mockbin_request_url
+
     with vcr.use_cassette(str(tmpdir.join("whatever.yaml"))):
         _, _ = get(url)
 
@@ -79,8 +76,9 @@ def test_case_insensitive_headers(tmpdir, scheme):
         assert cassette.play_count == 1
 
 
-def test_text(tmpdir, scheme):
-    url = scheme + "://httpbin.org"
+def test_text(tmpdir, mockbin_request_url):
+    url = mockbin_request_url
+
     with vcr.use_cassette(str(tmpdir.join("text.yaml"))):
         _, response_text = get(url)
 
@@ -90,8 +88,8 @@ def test_text(tmpdir, scheme):
         assert cassette.play_count == 1
 
 
-def test_json(tmpdir, scheme):
-    url = scheme + "://httpbin.org/get"
+def test_json(tmpdir, mockbin_request_url):
+    url = mockbin_request_url
     headers = {"Content-Type": "application/json"}
 
     with vcr.use_cassette(str(tmpdir.join("json.yaml"))):
@@ -103,8 +101,8 @@ def test_json(tmpdir, scheme):
         assert cassette.play_count == 1
 
 
-def test_binary(tmpdir, scheme):
-    url = scheme + "://httpbin.org/image/png"
+def test_binary(tmpdir, mockbin_request_url):
+    url = mockbin_request_url + "/image/png"
     with vcr.use_cassette(str(tmpdir.join("binary.yaml"))):
         _, response_binary = get(url, output="raw")
 
@@ -114,23 +112,23 @@ def test_binary(tmpdir, scheme):
         assert cassette.play_count == 1
 
 
-def test_stream(tmpdir, scheme):
-    url = scheme + "://httpbin.org/get"
+def test_stream(tmpdir, mockbin_request_url):
+    url = mockbin_request_url
 
     with vcr.use_cassette(str(tmpdir.join("stream.yaml"))):
-        resp, body = get(url, output="raw")  # Do not use stream here, as the stream is exhausted by vcr
+        _, body = get(url, output="raw")  # Do not use stream here, as the stream is exhausted by vcr
 
     with vcr.use_cassette(str(tmpdir.join("stream.yaml"))) as cassette:
-        cassette_resp, cassette_body = get(url, output="stream")
+        _, cassette_body = get(url, output="stream")
         assert cassette_body == body
         assert cassette.play_count == 1
 
 
 @pytest.mark.parametrize("body", ["data", "json"])
-def test_post(tmpdir, scheme, body, caplog):
+def test_post(tmpdir, body, caplog, mockbin_request_url):
     caplog.set_level(logging.INFO)
     data = {"key1": "value1", "key2": "value2"}
-    url = scheme + "://httpbin.org/post"
+    url = mockbin_request_url
     with vcr.use_cassette(str(tmpdir.join("post.yaml"))):
         _, response_json = post(url, **{body: data})
 
@@ -151,13 +149,14 @@ def test_post(tmpdir, scheme, body, caplog):
     ), "Log message not found."
 
 
-def test_params(tmpdir, scheme):
-    url = scheme + "://httpbin.org/get"
+def test_params(tmpdir, mockbin_request_url):
+    url = mockbin_request_url + "?d=d"
     headers = {"Content-Type": "application/json"}
-    params = {"a": 1, "b": False, "c": "c"}
+    params = {"a": 1, "b": 2, "c": "c"}
 
     with vcr.use_cassette(str(tmpdir.join("get.yaml"))) as cassette:
         _, response_json = get(url, output="json", params=params, headers=headers)
+        assert response_json["queryString"] == {"a": "1", "b": "2", "c": "c", "d": "d"}
 
     with vcr.use_cassette(str(tmpdir.join("get.yaml"))) as cassette:
         _, cassette_response_json = get(url, output="json", params=params, headers=headers)
@@ -165,10 +164,10 @@ def test_params(tmpdir, scheme):
         assert cassette.play_count == 1
 
 
-def test_params_same_url_distinct_params(tmpdir, scheme):
-    url = scheme + "://httpbin.org/get"
+def test_params_same_url_distinct_params(tmpdir, mockbin_request_url):
+    url = mockbin_request_url
     headers = {"Content-Type": "application/json"}
-    params = {"a": 1, "b": False, "c": "c"}
+    params = {"a": 1, "b": 2, "c": "c"}
 
     with vcr.use_cassette(str(tmpdir.join("get.yaml"))) as cassette:
         _, response_json = get(url, output="json", params=params, headers=headers)
@@ -184,8 +183,8 @@ def test_params_same_url_distinct_params(tmpdir, scheme):
             get(url, output="text", params=other_params)
 
 
-def test_params_on_url(tmpdir, scheme):
-    url = scheme + "://httpbin.org/get?a=1&b=foo"
+def test_params_on_url(tmpdir, mockbin_request_url):
+    url = mockbin_request_url + "?a=1&b=foo"
     headers = {"Content-Type": "application/json"}
 
     with vcr.use_cassette(str(tmpdir.join("get.yaml"))) as cassette:
@@ -249,8 +248,8 @@ def test_aiohttp_test_client_json(aiohttp_client, tmpdir):
     assert cassette.play_count == 1
 
 
-def test_redirect(aiohttp_client, tmpdir):
-    url = "https://mockbin.org/redirect/302/2"
+def test_redirect(tmpdir, mockbin):
+    url = mockbin + "/redirect/302/2"
 
     with vcr.use_cassette(str(tmpdir.join("redirect.yaml"))):
         response, _ = get(url)
@@ -273,9 +272,9 @@ def test_redirect(aiohttp_client, tmpdir):
     assert cassette_response.request_info.real_url == response.request_info.real_url
 
 
-def test_not_modified(aiohttp_client, tmpdir):
+def test_not_modified(tmpdir, mockbin):
     """It doesn't try to redirect on 304"""
-    url = "https://httpbin.org/status/304"
+    url = mockbin + "/status/304"
 
     with vcr.use_cassette(str(tmpdir.join("not_modified.yaml"))):
         response, _ = get(url)
@@ -290,13 +289,13 @@ def test_not_modified(aiohttp_client, tmpdir):
         assert cassette.play_count == 1
 
 
-def test_double_requests(tmpdir):
+def test_double_requests(tmpdir, mockbin_request_url):
     """We should capture, record, and replay all requests and response chains,
     even if there are duplicate ones.
 
     We should replay in the order we saw them.
     """
-    url = "https://httpbin.org/get"
+    url = mockbin_request_url
 
     with vcr.use_cassette(str(tmpdir.join("text.yaml"))):
         _, response_text1 = get(url, output="text")
@@ -399,3 +398,19 @@ def test_cookies_redirect(scheme, tmpdir):
                 assert cookies["Cookie_1"].value == "Val_1"
 
     run_in_loop(run)
+
+
+def test_not_allow_redirects(tmpdir, mockbin):
+    url = mockbin + "/redirect/308/5"
+    path = str(tmpdir.join("redirects.yaml"))
+
+    with vcr.use_cassette(path):
+        response, _ = get(url, allow_redirects=False)
+        assert response.url.path == "/redirect/308/5"
+        assert response.status == 308
+
+    with vcr.use_cassette(path) as cassette:
+        response, _ = get(url, allow_redirects=False)
+        assert response.url.path == "/redirect/308/5"
+        assert response.status == 308
+        assert cassette.play_count == 1

tests/integration/test_basic.py

@@ -11,7 +11,7 @@ import vcr
 
 def test_nonexistent_directory(tmpdir, httpbin):
     """If we load a cassette in a nonexistent directory, it can save ok"""
-    # Check to make sure directory doesnt exist
+    # Check to make sure directory doesn't exist
     assert not os.path.exists(str(tmpdir.join("nonexistent")))
 
     # Run VCR to create dir and cassette file

tests/integration/test_boto.py

@@ -2,11 +2,13 @@ import pytest
 
 boto = pytest.importorskip("boto")
 
+from configparser import DuplicateSectionError  # NOQA
+
 import boto  # NOQA
 import boto.iam  # NOQA
 from boto.s3.connection import S3Connection  # NOQA
 from boto.s3.key import Key  # NOQA
-from configparser import DuplicateSectionError  # NOQA
+
 import vcr  # NOQA
 
 
@@ -15,6 +17,7 @@ def test_boto_stubs(tmpdir):
         # Perform the imports within the patched context so that
         # CertValidatingHTTPSConnection refers to the patched version.
         from boto.https_connection import CertValidatingHTTPSConnection
+
         from vcr.stubs.boto_stubs import VCRCertValidatingHTTPSConnection
 
         # Prove that the class was patched by the stub and that we can instantiate it.

tests/integration/test_boto3.py

@@ -1,10 +1,12 @@
-import pytest
 import os
 
+import pytest
+
 boto3 = pytest.importorskip("boto3")
 
 import boto3  # NOQA
 import botocore  # NOQA
+
 import vcr  # NOQA
 
 try:
@@ -55,24 +57,6 @@ def get_user(iam_client):
     return _get_user
 
 
-@boto3_skip_vendored_requests
-def test_boto_vendored_stubs(tmpdir):
-    with vcr.use_cassette(str(tmpdir.join("boto3-stubs.yml"))):
-        # Perform the imports within the patched context so that
-        # HTTPConnection, VerifiedHTTPSConnection refers to the patched version.
-        from botocore.vendored.requests.packages.urllib3.connectionpool import (
-            HTTPConnection,
-            VerifiedHTTPSConnection,
-        )
-        from vcr.stubs.boto3_stubs import VCRRequestsHTTPConnection, VCRRequestsHTTPSConnection
-
-        # Prove that the class was patched by the stub and that we can instantiate it.
-        assert issubclass(HTTPConnection, VCRRequestsHTTPConnection)
-        assert issubclass(VerifiedHTTPSConnection, VCRRequestsHTTPSConnection)
-        HTTPConnection("hostname.does.not.matter")
-        VerifiedHTTPSConnection("hostname.does.not.matter")
-
-
 @pytest.mark.skipif(
     os.environ.get("TRAVIS_PULL_REQUEST") != "false",
     reason="Encrypted Environment Variables from Travis Repository Settings"
@@ -80,7 +64,6 @@ def test_boto_vendored_stubs(tmpdir):
     "https://docs.travis-ci.com/user/pull-requests/#pull-requests-and-security-restrictions",
 )
 def test_boto_medium_difficulty(tmpdir, get_user):
-
     with vcr.use_cassette(str(tmpdir.join("boto3-medium.yml"))):
         response = get_user()
         assert response["User"]["UserName"] == IAM_USER_NAME

tests/integration/test_config.py

@@ -1,16 +1,18 @@
-import os
 import json
-import pytest
-import vcr
+import os
 from urllib.request import urlopen
 
+import pytest
 
-def test_set_serializer_default_config(tmpdir, httpbin):
+import vcr
+
+
+def test_set_serializer_default_config(tmpdir, mockbin_request_url):
     my_vcr = vcr.VCR(serializer="json")
 
     with my_vcr.use_cassette(str(tmpdir.join("test.json"))):
         assert my_vcr.serializer == "json"
-        urlopen(httpbin.url + "/get")
+        urlopen(mockbin_request_url)
 
     with open(str(tmpdir.join("test.json"))) as f:
         file_content = f.read()
@@ -18,35 +20,35 @@ def test_set_serializer_default_config(tmpdir, httpbin):
         assert json.loads(file_content)
 
 
-def test_default_set_cassette_library_dir(tmpdir, httpbin):
+def test_default_set_cassette_library_dir(tmpdir, mockbin_request_url):
     my_vcr = vcr.VCR(cassette_library_dir=str(tmpdir.join("subdir")))
 
     with my_vcr.use_cassette("test.json"):
-        urlopen(httpbin.url + "/get")
+        urlopen(mockbin_request_url)
 
     assert os.path.exists(str(tmpdir.join("subdir").join("test.json")))
 
 
-def test_override_set_cassette_library_dir(tmpdir, httpbin):
+def test_override_set_cassette_library_dir(tmpdir, mockbin_request_url):
     my_vcr = vcr.VCR(cassette_library_dir=str(tmpdir.join("subdir")))
 
     cld = str(tmpdir.join("subdir2"))
 
     with my_vcr.use_cassette("test.json", cassette_library_dir=cld):
-        urlopen(httpbin.url + "/get")
+        urlopen(mockbin_request_url)
 
     assert os.path.exists(str(tmpdir.join("subdir2").join("test.json")))
     assert not os.path.exists(str(tmpdir.join("subdir").join("test.json")))
 
 
-def test_override_match_on(tmpdir, httpbin):
+def test_override_match_on(tmpdir, mockbin_request_url):
     my_vcr = vcr.VCR(match_on=["method"])
 
     with my_vcr.use_cassette(str(tmpdir.join("test.json"))):
-        urlopen(httpbin.url)
+        urlopen(mockbin_request_url)
 
     with my_vcr.use_cassette(str(tmpdir.join("test.json"))) as cass:
-        urlopen(httpbin.url + "/get")
+        urlopen(mockbin_request_url)
 
     assert len(cass) == 1
     assert cass.play_count == 1
@@ -58,3 +60,23 @@ def test_missing_matcher():
     with pytest.raises(KeyError):
         with my_vcr.use_cassette("test.yaml", match_on=["notawesome"]):
             pass
+
+
+def test_dont_record_on_exception(tmpdir, mockbin_request_url):
+    my_vcr = vcr.VCR(record_on_exception=False)
+
+    @my_vcr.use_cassette(str(tmpdir.join("dontsave.yml")))
+    def some_test():
+        assert b"Not in content" in urlopen(mockbin_request_url)
+
+    with pytest.raises(AssertionError):
+        some_test()
+
+    assert not os.path.exists(str(tmpdir.join("dontsave.yml")))
+
+    # Make sure context decorator has the same behavior
+    with pytest.raises(AssertionError):
+        with my_vcr.use_cassette(str(tmpdir.join("dontsave2.yml"))):
+            assert b"Not in content" in urlopen(mockbin_request_url).read()
+
+    assert not os.path.exists(str(tmpdir.join("dontsave2.yml")))

tests/integration/test_disksaver.py

@@ -10,19 +10,19 @@ from urllib.request import urlopen
 import vcr
 
 
-def test_disk_saver_nowrite(tmpdir, httpbin):
+def test_disk_saver_nowrite(tmpdir, mockbin_request_url):
     """
     Ensure that when you close a cassette without changing it it doesn't
     rewrite the file
     """
     fname = str(tmpdir.join("synopsis.yaml"))
     with vcr.use_cassette(fname) as cass:
-        urlopen(httpbin.url).read()
+        urlopen(mockbin_request_url).read()
         assert cass.play_count == 0
     last_mod = os.path.getmtime(fname)
 
     with vcr.use_cassette(fname) as cass:
-        urlopen(httpbin.url).read()
+        urlopen(mockbin_request_url).read()
         assert cass.play_count == 1
         assert cass.dirty is False
     last_mod2 = os.path.getmtime(fname)
@@ -30,14 +30,14 @@ def test_disk_saver_nowrite(tmpdir, httpbin):
     assert last_mod == last_mod2
 
 
-def test_disk_saver_write(tmpdir, httpbin):
+def test_disk_saver_write(tmpdir, mockbin_request_url):
     """
     Ensure that when you close a cassette after changing it it does
     rewrite the file
     """
     fname = str(tmpdir.join("synopsis.yaml"))
     with vcr.use_cassette(fname) as cass:
-        urlopen(httpbin.url).read()
+        urlopen(mockbin_request_url).read()
         assert cass.play_count == 0
     last_mod = os.path.getmtime(fname)
 
@@ -46,8 +46,8 @@ def test_disk_saver_write(tmpdir, httpbin):
     time.sleep(1)
 
     with vcr.use_cassette(fname, record_mode=vcr.mode.ANY) as cass:
-        urlopen(httpbin.url).read()
-        urlopen(httpbin.url + "/get").read()
+        urlopen(mockbin_request_url).read()
+        urlopen(mockbin_request_url + "/get").read()
         assert cass.play_count == 1
         assert cass.dirty
     last_mod2 = os.path.getmtime(fname)

tests/integration/test_filter.py

@@ -1,12 +1,14 @@
 import base64
-import pytest
-from urllib.request import urlopen, Request
-from urllib.parse import urlencode
-from urllib.error import HTTPError
-import vcr
 import json
+from urllib.error import HTTPError
+from urllib.parse import urlencode
+from urllib.request import Request, urlopen
+
+import pytest
 from assertions import assert_cassette_has_one_response, assert_is_json
 
+import vcr
+
 
 def _request_with_auth(url, username, password):
     request = Request(url)
@@ -106,6 +108,18 @@ def test_decompress_gzip(tmpdir, httpbin):
     assert_is_json(decoded_response)
 
 
+def test_decomptess_empty_body(tmpdir, httpbin):
+    url = httpbin.url + "/gzip"
+    request = Request(url, headers={"Accept-Encoding": ["gzip, deflate"]}, method="HEAD")
+    cass_file = str(tmpdir.join("gzip_empty_response.yaml"))
+    with vcr.use_cassette(cass_file, decode_compressed_response=True):
+        response = urlopen(request).read()
+    with vcr.use_cassette(cass_file) as cass:
+        decoded_response = urlopen(request).read()
+        assert_cassette_has_one_response(cass)
+    assert decoded_response == response
+
+
 def test_decompress_deflate(tmpdir, httpbin):
     url = httpbin.url + "/deflate"
     request = Request(url, headers={"Accept-Encoding": ["gzip, deflate"]})

tests/integration/test_httplib2.py

@@ -1,16 +1,13 @@
 # -*- coding: utf-8 -*-
 """Integration tests with httplib2"""
-
-import sys
-
 from urllib.parse import urlencode
+
 import pytest
 import pytest_httpbin.certs
+from assertions import assert_cassette_has_one_response
 
 import vcr
 
-from assertions import assert_cassette_has_one_response
-
 httplib2 = pytest.importorskip("httplib2")
 
 
@@ -20,8 +17,6 @@ def http():
     with the certificate replaced by the httpbin one.
     """
     kwargs = {"ca_certs": pytest_httpbin.certs.where()}
-    if sys.version_info[:2] in [(2, 7), (3, 7)]:
-        kwargs["disable_ssl_certificate_validation"] = True
     return httplib2.Http(**kwargs)
 
 
@@ -61,13 +56,14 @@ def test_response_headers(tmpdir, httpbin_both):
         assert set(headers) == set(resp.items())
 
 
-def test_effective_url(tmpdir, httpbin_both):
+def test_effective_url(tmpdir):
     """Ensure that the effective_url is captured"""
-    url = httpbin_both.url + "/redirect-to?url=/html"
+    url = "http://mockbin.org/redirect/301"
+
     with vcr.use_cassette(str(tmpdir.join("headers.yaml"))):
         resp, _ = http().request(url)
         effective_url = resp["content-location"]
-        assert effective_url == httpbin_both + "/html"
+        assert effective_url == "http://mockbin.org/redirect/301/0"
 
     with vcr.use_cassette(str(tmpdir.join("headers.yaml"))):
         resp, _ = http().request(url)

tests/integration/test_httpx.py

@@ -1,50 +1,85 @@
-import pytest
-import contextlib
 import os
 
+import pytest
+
 asyncio = pytest.importorskip("asyncio")
 httpx = pytest.importorskip("httpx")
 
 import vcr  # noqa: E402
+from vcr.stubs.httpx_stubs import HTTPX_REDIRECT_PARAM  # noqa: E402
 
 
 class BaseDoRequest:
     _client_class = None
 
     def __init__(self, *args, **kwargs):
-        self._client = self._client_class(*args, **kwargs)
+        self._client_args = args
+        self._client_kwargs = kwargs
+
+    def _make_client(self):
+        return self._client_class(*self._client_args, **self._client_kwargs)
 
 
 class DoSyncRequest(BaseDoRequest):
     _client_class = httpx.Client
 
+    def __enter__(self):
+        return self
+
+    def __exit__(self, *args):
+        pass
+
+    @property
+    def client(self):
+        try:
+            return self._client
+        except AttributeError:
+            self._client = self._make_client()
+            return self._client
+
     def __call__(self, *args, **kwargs):
-        return self._client.request(*args, timeout=60, **kwargs)
+        return self.client.request(*args, timeout=60, **kwargs)
 
 
 class DoAsyncRequest(BaseDoRequest):
     _client_class = httpx.AsyncClient
 
-    @staticmethod
-    def run_in_loop(coroutine):
-        with contextlib.closing(asyncio.new_event_loop()) as loop:
-            asyncio.set_event_loop(loop)
-            task = loop.create_task(coroutine)
-            return loop.run_until_complete(task)
+    def __enter__(self):
+        # Need to manage both loop and client, because client's implementation
+        # will fail if the loop is closed before the client's end of life.
+        self._loop = asyncio.new_event_loop()
+        asyncio.set_event_loop(self._loop)
+        self._client = self._make_client()
+        self._loop.run_until_complete(self._client.__aenter__())
+        return self
+
+    def __exit__(self, *args):
+        try:
+            self._loop.run_until_complete(self._client.__aexit__(*args))
+        finally:
+            del self._client
+            self._loop.close()
+            del self._loop
+
+    @property
+    def client(self):
+        try:
+            return self._client
+        except AttributeError:
+            raise ValueError('To access async client, use "with do_request() as client"')
 
     def __call__(self, *args, **kwargs):
-        async def _request():
-            async with self._client as c:
-                return await c.request(*args, **kwargs)
+        if hasattr(self, "_loop"):
+            return self._loop.run_until_complete(self.client.request(*args, **kwargs))
 
-        return DoAsyncRequest.run_in_loop(_request())
+        # Use one-time context and dispose of the loop/client afterwards
+        with self:
+            return self(*args, **kwargs)
 
 
 def pytest_generate_tests(metafunc):
     if "do_request" in metafunc.fixturenames:
         metafunc.parametrize("do_request", [DoAsyncRequest, DoSyncRequest])
-    if "scheme" in metafunc.fixturenames:
-        metafunc.parametrize("scheme", ["http", "https"])
 
 
 @pytest.fixture
@@ -52,8 +87,9 @@ def yml(tmpdir, request):
     return str(tmpdir.join(request.function.__name__ + ".yaml"))
 
 
-def test_status(tmpdir, scheme, do_request):
-    url = scheme + "://mockbin.org/request"
+def test_status(tmpdir, mockbin, do_request):
+    url = mockbin
+
     with vcr.use_cassette(str(tmpdir.join("status.yaml"))):
         response = do_request()("GET", url)
 
@@ -63,8 +99,9 @@ def test_status(tmpdir, scheme, do_request):
         assert cassette.play_count == 1
 
 
-def test_case_insensitive_headers(tmpdir, scheme, do_request):
-    url = scheme + "://mockbin.org/request"
+def test_case_insensitive_headers(tmpdir, mockbin, do_request):
+    url = mockbin
+
     with vcr.use_cassette(str(tmpdir.join("whatever.yaml"))):
         do_request()("GET", url)
 
@@ -75,8 +112,9 @@ def test_case_insensitive_headers(tmpdir, scheme, do_request):
         assert cassette.play_count == 1
 
 
-def test_content(tmpdir, scheme, do_request):
-    url = scheme + "://httpbin.org"
+def test_content(tmpdir, mockbin, do_request):
+    url = mockbin
+
     with vcr.use_cassette(str(tmpdir.join("cointent.yaml"))):
         response = do_request()("GET", url)
 
@@ -86,9 +124,10 @@ def test_content(tmpdir, scheme, do_request):
         assert cassette.play_count == 1
 
 
-def test_json(tmpdir, scheme, do_request):
-    url = scheme + "://httpbin.org/get"
-    headers = {"Content-Type": "application/json"}
+def test_json(tmpdir, mockbin, do_request):
+    url = mockbin + "/request"
+
+    headers = {"content-type": "application/json"}
 
     with vcr.use_cassette(str(tmpdir.join("json.yaml"))):
         response = do_request(headers=headers)("GET", url)
@@ -99,8 +138,8 @@ def test_json(tmpdir, scheme, do_request):
         assert cassette.play_count == 1
 
 
-def test_params_same_url_distinct_params(tmpdir, scheme, do_request):
-    url = scheme + "://httpbin.org/get"
+def test_params_same_url_distinct_params(tmpdir, mockbin, do_request):
+    url = mockbin + "/request"
     headers = {"Content-Type": "application/json"}
     params = {"a": 1, "b": False, "c": "c"}
 
@@ -119,15 +158,17 @@ def test_params_same_url_distinct_params(tmpdir, scheme, do_request):
             do_request()("GET", url, params=params, headers=headers)
 
 
-def test_redirect(tmpdir, do_request, yml):
-    url = "https://mockbin.org/redirect/303/2"
+def test_redirect(mockbin, yml, do_request):
+    url = mockbin + "/redirect/303/2"
 
-    response = do_request()("GET", url)
+    redirect_kwargs = {HTTPX_REDIRECT_PARAM.name: True}
+
+    response = do_request()("GET", url, **redirect_kwargs)
     with vcr.use_cassette(yml):
-        response = do_request()("GET", url)
+        response = do_request()("GET", url, **redirect_kwargs)
 
     with vcr.use_cassette(yml) as cassette:
-        cassette_response = do_request()("GET", url)
+        cassette_response = do_request()("GET", url, **redirect_kwargs)
 
         assert cassette_response.status_code == response.status_code
         assert len(cassette_response.history) == len(response.history)
@@ -143,20 +184,23 @@ def test_redirect(tmpdir, do_request, yml):
     }
 
 
-def test_work_with_gzipped_data(tmpdir, do_request, yml):
+def test_work_with_gzipped_data(mockbin, do_request, yml):
+    url = mockbin + "/gzip?foo=bar"
+    headers = {"accept-encoding": "deflate, gzip"}
+
     with vcr.use_cassette(yml):
-        do_request()("GET", "https://httpbin.org/gzip")
+        do_request(headers=headers)("GET", url)
 
     with vcr.use_cassette(yml) as cassette:
-        cassette_response = do_request()("GET", "https://httpbin.org/gzip")
+        cassette_response = do_request(headers=headers)("GET", url)
 
-        assert "gzip" in cassette_response.json()["headers"]["Accept-Encoding"]
+        assert cassette_response.headers["content-encoding"] == "gzip"
         assert cassette_response.read()
         assert cassette.play_count == 1
 
 
 @pytest.mark.parametrize("url", ["https://github.com/kevin1024/vcrpy/issues/" + str(i) for i in range(3, 6)])
-def test_simple_fetching(tmpdir, do_request, yml, url):
+def test_simple_fetching(do_request, yml, url):
     with vcr.use_cassette(yml):
         do_request()("GET", url)
 
@@ -171,9 +215,9 @@ def test_behind_proxy(do_request):
     yml = (
         os.path.dirname(os.path.realpath(__file__)) + "/cassettes/" + "test_httpx_test_test_behind_proxy.yml"
     )
-    url = "https://httpbin.org/headers"
+    url = "https://mockbin.org/headers"
     proxy = "http://localhost:8080"
-    proxies = {"http": proxy, "https": proxy}
+    proxies = {"http://": proxy, "https://": proxy}
 
     with vcr.use_cassette(yml):
         response = do_request(proxies=proxies, verify=False)("GET", url)
@@ -187,67 +231,74 @@ def test_behind_proxy(do_request):
         assert cassette_response.request.url == response.request.url
 
 
-def test_cookies(tmpdir, scheme, do_request):
+def test_cookies(tmpdir, mockbin, do_request):
     def client_cookies(client):
-        return [c for c in client._client.cookies]
+        return [c for c in client.client.cookies]
 
     def response_cookies(response):
         return [c for c in response.cookies]
 
-    client = do_request()
-    assert client_cookies(client) == []
+    url = mockbin + "/bin/26148652-fe25-4f21-aaf5-689b5b4bf65f"
+    headers = {"cookie": "k1=v1;k2=v2"}
 
-    url = scheme + "://httpbin.org"
-    testfile = str(tmpdir.join("cookies.yml"))
-    with vcr.use_cassette(testfile):
-        r1 = client("GET", url + "/cookies/set?k1=v1&k2=v2")
-        assert response_cookies(r1.history[0]) == ["k1", "k2"]
-        assert response_cookies(r1) == []
+    with do_request(headers=headers) as client:
+        assert client_cookies(client) == []
 
-        r2 = client("GET", url + "/cookies")
-        assert len(r2.json()["cookies"]) == 2
+        redirect_kwargs = {HTTPX_REDIRECT_PARAM.name: True}
 
-        assert client_cookies(client) == ["k1", "k2"]
+        testfile = str(tmpdir.join("cookies.yml"))
+        with vcr.use_cassette(testfile):
+            r1 = client("GET", url, **redirect_kwargs)
 
-    new_client = do_request()
-    assert client_cookies(new_client) == []
+            assert response_cookies(r1) == ["k1", "k2"]
 
-    with vcr.use_cassette(testfile) as cassette:
-        cassette_response = new_client("GET", url + "/cookies/set?k1=v1&k2=v2")
-        assert response_cookies(cassette_response.history[0]) == ["k1", "k2"]
-        assert response_cookies(cassette_response) == []
+            r2 = client("GET", url, **redirect_kwargs)
 
-        assert cassette.play_count == 2
-        assert client_cookies(new_client) == ["k1", "k2"]
+            assert response_cookies(r2) == ["k1", "k2"]
+            assert client_cookies(client) == ["k1", "k2"]
+
+    with do_request(headers=headers) as new_client:
+        assert client_cookies(new_client) == []
+
+        with vcr.use_cassette(testfile) as cassette:
+            cassette_response = new_client("GET", url)
+
+            assert cassette.play_count == 1
+            assert response_cookies(cassette_response) == ["k1", "k2"]
+            assert client_cookies(new_client) == ["k1", "k2"]
 
 
-def test_relative_redirects(tmpdir, scheme, do_request):
-    url = scheme + "://mockbin.com/redirect/301?to=/redirect/301?to=/request"
+def test_relative_redirects(tmpdir, scheme, do_request, mockbin):
+    redirect_kwargs = {HTTPX_REDIRECT_PARAM.name: True}
+
+    url = mockbin + "/redirect/301?to=/redirect/301?to=/request"
     testfile = str(tmpdir.join("relative_redirects.yml"))
     with vcr.use_cassette(testfile):
-        response = do_request()("GET", url)
+        response = do_request()("GET", url, **redirect_kwargs)
         assert len(response.history) == 2, response
         assert response.json()["url"].endswith("request")
 
     with vcr.use_cassette(testfile) as cassette:
-        response = do_request()("GET", url)
+        response = do_request()("GET", url, **redirect_kwargs)
         assert len(response.history) == 2
         assert response.json()["url"].endswith("request")
 
         assert cassette.play_count == 3
 
 
-def test_redirect_wo_allow_redirects(do_request, yml):
-    url = "https://mockbin.org/redirect/308/5"
+def test_redirect_wo_allow_redirects(do_request, mockbin, yml):
+    url = mockbin + "/redirect/308/5"
+
+    redirect_kwargs = {HTTPX_REDIRECT_PARAM.name: False}
 
     with vcr.use_cassette(yml):
-        response = do_request()("GET", url, allow_redirects=False)
+        response = do_request()("GET", url, **redirect_kwargs)
 
         assert str(response.url).endswith("308/5")
         assert response.status_code == 308
 
     with vcr.use_cassette(yml) as cassette:
-        response = do_request()("GET", url, allow_redirects=False)
+        response = do_request()("GET", url, **redirect_kwargs)
 
         assert str(response.url).endswith("308/5")
         assert response.status_code == 308

tests/integration/test_ignore.py

@@ -1,6 +1,7 @@
-from urllib.request import urlopen
 import socket
 from contextlib import contextmanager
+from urllib.request import urlopen
+
 import vcr
 
 

tests/integration/test_matchers.py

@@ -1,7 +1,8 @@
-import vcr
-import pytest
 from urllib.request import urlopen
 
+import pytest
+
+import vcr
 
 DEFAULT_URI = "http://httpbin.org/get?p1=q1&p2=q2"  # base uri for testing
 
@@ -35,7 +36,6 @@ def cassette(tmpdir, httpbin, httpbin_secure):
     ],
 )
 def test_matchers(httpbin, httpbin_secure, cassette, matcher, matching_uri, not_matching_uri):
-
     matching_uri = _replace_httpbin(matching_uri, httpbin, httpbin_secure)
     not_matching_uri = _replace_httpbin(not_matching_uri, httpbin, httpbin_secure)
     default_uri = _replace_httpbin(DEFAULT_URI, httpbin, httpbin_secure)
@@ -75,7 +75,6 @@ def test_method_matcher(cassette, httpbin, httpbin_secure):
     "uri", [DEFAULT_URI, "http://httpbin.org/get?p2=q2&p1=q1", "http://httpbin.org/get?p2=q2&p1=q1"]
 )
 def test_default_matcher_matches(cassette, uri, httpbin, httpbin_secure):
-
     uri = _replace_httpbin(uri, httpbin, httpbin_secure)
 
     with vcr.use_cassette(cassette) as cass:

tests/integration/test_multiple.py

@@ -1,7 +1,9 @@
-import pytest
-import vcr
 from urllib.request import urlopen
 
+import pytest
+
+import vcr
+
 
 def test_making_extra_request_raises_exception(tmpdir, httpbin):
     # make two requests in the first request that are considered

tests/integration/test_proxy.py

@@ -1,15 +1,13 @@
 # -*- coding: utf-8 -*-
 """Test using a proxy."""
 
-# External imports
-import multiprocessing
-import pytest
-
 import http.server
+import multiprocessing
 import socketserver
 from urllib.request import urlopen
 
-# Internal imports
+import pytest
+
 import vcr
 
 # Conditional imports
@@ -39,7 +37,7 @@ class Proxy(http.server.SimpleHTTPRequestHandler):
         self.copyfile(upstream_response, self.wfile)
 
 
-@pytest.yield_fixture(scope="session")
+@pytest.fixture(scope="session")
 def proxy_server():
     httpd = socketserver.ThreadingTCPServer(("", 0), Proxy)
     proxy_process = multiprocessing.Process(target=httpd.serve_forever)

tests/integration/test_record_mode.py

@@ -1,7 +1,9 @@
-import pytest
-import vcr
 from urllib.request import urlopen
 
+import pytest
+
+import vcr
+
 
 def test_once_record_mode(tmpdir, httpbin):
     testfile = str(tmpdir.join("recordmode.yml"))
@@ -57,7 +59,7 @@ def test_new_episodes_record_mode(tmpdir, httpbin):
         assert cass.all_played
 
         # in the "new_episodes" record mode, we can add more requests to
-        # a cassette without repurcussions.
+        # a cassette without repercussions.
         urlopen(httpbin.url + "/get").read()
 
         # one of the responses has been played
@@ -108,7 +110,7 @@ def test_all_record_mode(tmpdir, httpbin):
         urlopen(httpbin.url).read()
 
         # in the "all" record mode, we can add more requests to
-        # a cassette without repurcussions.
+        # a cassette without repercussions.
         urlopen(httpbin.url + "/get").read()
 
         # The cassette was never actually played, even though it existed.

tests/integration/test_register_matcher.py

@@ -1,6 +1,7 @@
-import vcr
 from urllib.request import urlopen
 
+import vcr
+
 
 def true_matcher(r1, r2):
     return True
@@ -10,27 +11,27 @@ def false_matcher(r1, r2):
     return False
 
 
-def test_registered_true_matcher(tmpdir, httpbin):
+def test_registered_true_matcher(tmpdir, mockbin_request_url):
     my_vcr = vcr.VCR()
     my_vcr.register_matcher("true", true_matcher)
     testfile = str(tmpdir.join("test.yml"))
     with my_vcr.use_cassette(testfile, match_on=["true"]):
         # These 2 different urls are stored as the same request
-        urlopen(httpbin.url)
-        urlopen(httpbin.url + "/get")
+        urlopen(mockbin_request_url)
+        urlopen(mockbin_request_url + "/get")
 
     with my_vcr.use_cassette(testfile, match_on=["true"]):
         # I can get the response twice even though I only asked for it once
-        urlopen(httpbin.url + "/get")
-        urlopen(httpbin.url + "/get")
+        urlopen(mockbin_request_url)
+        urlopen(mockbin_request_url)
 
 
-def test_registered_false_matcher(tmpdir, httpbin):
+def test_registered_false_matcher(tmpdir, mockbin_request_url):
     my_vcr = vcr.VCR()
     my_vcr.register_matcher("false", false_matcher)
     testfile = str(tmpdir.join("test.yml"))
     with my_vcr.use_cassette(testfile, match_on=["false"]) as cass:
         # These 2 different urls are stored as different requests
-        urlopen(httpbin.url)
-        urlopen(httpbin.url + "/get")
+        urlopen(mockbin_request_url)
+        urlopen(mockbin_request_url + "/get")
         assert len(cass) == 2

tests/integration/test_register_persister.py

@@ -30,7 +30,7 @@ def test_save_cassette_with_custom_persister(tmpdir, httpbin):
     my_vcr = vcr.VCR()
     my_vcr.register_persister(CustomFilesystemPersister)
 
-    # Check to make sure directory doesnt exist
+    # Check to make sure directory doesn't exist
     assert not os.path.exists(str(tmpdir.join("nonexistent")))
 
     # Run VCR to create dir and cassette file using new save_cassette callback

tests/integration/test_request.py

@@ -1,6 +1,7 @@
-import vcr
 from urllib.request import urlopen
 
+import vcr
+
 
 def test_recorded_request_uri_with_redirected_request(tmpdir, httpbin):
     with vcr.use_cassette(str(tmpdir.join("test.yml"))) as cass:

tests/integration/test_requests.py

@@ -1,11 +1,9 @@
-# -*- coding: utf-8 -*-
 """Test requests' interaction with vcr"""
-import platform
 import pytest
-import sys
-import vcr
 from assertions import assert_cassette_empty, assert_is_json
 
+import vcr
+
 requests = pytest.importorskip("requests")
 from requests.exceptions import ConnectionError  # noqa E402
 
@@ -117,10 +115,6 @@ def test_post_chunked_binary(tmpdir, httpbin):
 
 
 @pytest.mark.skipif("sys.version_info >= (3, 6)", strict=True, raises=ConnectionError)
-@pytest.mark.skipif(
-    (3, 5) < sys.version_info < (3, 6) and platform.python_implementation() == "CPython",
-    reason="Fails on CPython 3.5",
-)
 def test_post_chunked_binary_secure(tmpdir, httpbin_secure):
     """Ensure that we can send chunked binary without breaking while trying to concatenate bytes with str."""
     data1 = iter([b"data", b"to", b"send"])

tests/integration/test_stubs.py

@@ -1,10 +1,11 @@
-import vcr
-import zlib
-import json
 import http.client as httplib
+import json
+import zlib
 
 from assertions import assert_is_json
 
+import vcr
+
 
 def _headers_are_case_insensitive(host, port):
     conn = httplib.HTTPConnection(host, port)
@@ -64,7 +65,7 @@ def test_original_decoded_response_is_not_modified(tmpdir, httpbin):
         inside = conn.getresponse()
 
         # Assert that we do not modify the original response while appending
-        # to the casssette.
+        # to the cassette.
         assert "gzip" == inside.headers["content-encoding"]
 
         # They should effectively be the same response.

tests/integration/test_tornado.py

@@ -4,11 +4,11 @@
 import json
 
 import pytest
+from assertions import assert_cassette_empty, assert_is_json
+
 import vcr
 from vcr.errors import CannotOverwriteExistingCassetteException
 
-from assertions import assert_cassette_empty, assert_is_json
-
 tornado = pytest.importorskip("tornado")
 http = pytest.importorskip("tornado.httpclient")
 
@@ -44,12 +44,6 @@ def post(client, url, data=None, **kwargs):
     return client.fetch(http.HTTPRequest(url, method="POST", **kwargs))
 
 
-@pytest.fixture(params=["https", "http"])
-def scheme(request):
-    """Fixture that returns both http and https."""
-    return request.param
-
-
 @pytest.mark.gen_test
 def test_status_code(get_client, scheme, tmpdir):
     """Ensure that we can read the status code"""

tests/integration/test_urllib2.py

@@ -2,15 +2,15 @@
 """Integration tests with urllib2"""
 
 import ssl
-from urllib.request import urlopen
 from urllib.parse import urlencode
+from urllib.request import urlopen
+
 import pytest_httpbin.certs
+from assertions import assert_cassette_has_one_response
 
 # Internal imports
 import vcr
 
-from assertions import assert_cassette_has_one_response
-
 
 def urlopen_with_cafile(*args, **kwargs):
     context = ssl.create_default_context(cafile=pytest_httpbin.certs.where())
@@ -56,12 +56,13 @@ def test_response_headers(httpbin_both, tmpdir):
         assert sorted(open1) == sorted(open2)
 
 
-def test_effective_url(httpbin_both, tmpdir):
+def test_effective_url(tmpdir):
     """Ensure that the effective_url is captured"""
-    url = httpbin_both.url + "/redirect-to?url=/html"
+    url = "http://mockbin.org/redirect/301"
+
     with vcr.use_cassette(str(tmpdir.join("headers.yaml"))):
         effective_url = urlopen_with_cafile(url).geturl()
-        assert effective_url == httpbin_both.url + "/html"
+        assert effective_url == "http://mockbin.org/redirect/301/0"
 
     with vcr.use_cassette(str(tmpdir.join("headers.yaml"))):
         assert effective_url == urlopen_with_cafile(url).geturl()

tests/integration/test_urllib3.py

@@ -4,9 +4,11 @@
 
 import pytest
 import pytest_httpbin
+from assertions import assert_cassette_empty, assert_is_json
+
 import vcr
 from vcr.patch import force_reset
-from assertions import assert_cassette_empty, assert_is_json
+from vcr.stubs.compat import get_headers
 
 urllib3 = pytest.importorskip("urllib3")
 
@@ -40,7 +42,8 @@ def test_headers(tmpdir, httpbin_both, verify_pool_mgr):
         headers = verify_pool_mgr.request("GET", url).headers
 
     with vcr.use_cassette(str(tmpdir.join("headers.yaml"))):
-        assert headers == verify_pool_mgr.request("GET", url).headers
+        new_headers = verify_pool_mgr.request("GET", url).headers
+        assert sorted(get_headers(headers)) == sorted(get_headers(new_headers))
 
 
 def test_body(tmpdir, httpbin_both, verify_pool_mgr):
@@ -94,9 +97,10 @@ def test_post(tmpdir, httpbin_both, verify_pool_mgr):
     assert req1 == req2
 
 
-def test_redirects(tmpdir, httpbin_both, verify_pool_mgr):
+def test_redirects(tmpdir, verify_pool_mgr):
     """Ensure that we can handle redirects"""
-    url = httpbin_both.url + "/redirect-to?url=bytes/1024"
+    url = "http://mockbin.org/redirect/301"
+
     with vcr.use_cassette(str(tmpdir.join("verify_pool_mgr.yaml"))):
         content = verify_pool_mgr.request("GET", url).data
 
@@ -104,8 +108,9 @@ def test_redirects(tmpdir, httpbin_both, verify_pool_mgr):
         assert content == verify_pool_mgr.request("GET", url).data
         # Ensure that we've now cached *two* responses. One for the redirect
         # and one for the final fetch
-        assert len(cass) == 2
-        assert cass.play_count == 2
+
+    assert len(cass) == 2
+    assert cass.play_count == 2
 
 
 def test_cross_scheme(tmpdir, httpbin, httpbin_secure, verify_pool_mgr):
@@ -142,18 +147,18 @@ def test_https_with_cert_validation_disabled(tmpdir, httpbin_secure, pool_mgr):
 
 
 def test_urllib3_force_reset():
-    cpool = urllib3.connectionpool
-    http_original = cpool.HTTPConnection
-    https_original = cpool.HTTPSConnection
-    verified_https_original = cpool.VerifiedHTTPSConnection
+    conn = urllib3.connection
+    http_original = conn.HTTPConnection
+    https_original = conn.HTTPSConnection
+    verified_https_original = conn.VerifiedHTTPSConnection
     with vcr.use_cassette(path="test"):
-        first_cassette_HTTPConnection = cpool.HTTPConnection
-        first_cassette_HTTPSConnection = cpool.HTTPSConnection
-        first_cassette_VerifiedHTTPSConnection = cpool.VerifiedHTTPSConnection
+        first_cassette_HTTPConnection = conn.HTTPConnection
+        first_cassette_HTTPSConnection = conn.HTTPSConnection
+        first_cassette_VerifiedHTTPSConnection = conn.VerifiedHTTPSConnection
         with force_reset():
-            assert cpool.HTTPConnection is http_original
-            assert cpool.HTTPSConnection is https_original
-            assert cpool.VerifiedHTTPSConnection is verified_https_original
-        assert cpool.HTTPConnection is first_cassette_HTTPConnection
-        assert cpool.HTTPSConnection is first_cassette_HTTPSConnection
-        assert cpool.VerifiedHTTPSConnection is first_cassette_VerifiedHTTPSConnection
+            assert conn.HTTPConnection is http_original
+            assert conn.HTTPSConnection is https_original
+            assert conn.VerifiedHTTPSConnection is verified_https_original
+        assert conn.HTTPConnection is first_cassette_HTTPConnection
+        assert conn.HTTPSConnection is first_cassette_HTTPSConnection
+        assert conn.VerifiedHTTPSConnection is first_cassette_VerifiedHTTPSConnection

tests/integration/test_wild.py

@@ -1,9 +1,10 @@
 import http.client as httplib
 import multiprocessing
-import pytest
 from xmlrpc.client import ServerProxy
 from xmlrpc.server import SimpleXMLRPCServer
 
+import pytest
+
 requests = pytest.importorskip("requests")
 
 import vcr  # NOQA
@@ -63,9 +64,10 @@ def test_cookies(tmpdir, httpbin):
     with vcr.use_cassette(testfile):
         s = requests.Session()
         s.get(httpbin.url + "/cookies/set?k1=v1&k2=v2")
+        assert s.cookies.keys() == ["k1", "k2"]
 
         r2 = s.get(httpbin.url + "/cookies")
-        assert len(r2.json()["cookies"]) == 2
+        assert sorted(r2.json()["cookies"].keys()) == ["k1", "k2"]
 
 
 def test_amazon_doctype(tmpdir):
@@ -83,7 +85,7 @@ def start_rpc_server(q):
     httpd.serve_forever()
 
 
-@pytest.yield_fixture(scope="session")
+@pytest.fixture(scope="session")
 def rpc_server():
     q = multiprocessing.Queue()
     proxy_process = multiprocessing.Process(target=start_rpc_server, args=(q,))

tests/unit/test_cassettes.py

@@ -7,6 +7,7 @@ from unittest import mock
 
 import pytest
 import yaml
+
 from vcr.cassette import Cassette
 from vcr.errors import UnhandledHTTPRequestError
 from vcr.patch import force_reset
@@ -208,7 +209,7 @@ def test_nesting_cassette_context_managers(*args):
         )
         assert_get_response_body_is("first_response")
 
-        # Make sure a second cassette can supercede the first
+        # Make sure a second cassette can supersede the first
         with Cassette.use(path="test") as second_cassette:
             with mock.patch.object(second_cassette, "play_response", return_value=second_response):
                 assert_get_response_body_is("second_response")
@@ -310,16 +311,16 @@ def test_func_path_generator():
 
 
 def test_use_as_decorator_on_coroutine():
-    original_http_connetion = httplib.HTTPConnection
+    original_http_connection = httplib.HTTPConnection
 
     @Cassette.use(inject=True)
     def test_function(cassette):
         assert httplib.HTTPConnection.cassette is cassette
-        assert httplib.HTTPConnection is not original_http_connetion
+        assert httplib.HTTPConnection is not original_http_connection
         value = yield 1
         assert value == 1
         assert httplib.HTTPConnection.cassette is cassette
-        assert httplib.HTTPConnection is not original_http_connetion
+        assert httplib.HTTPConnection is not original_http_connection
         value = yield 2
         assert value == 2
 
@@ -333,15 +334,15 @@ def test_use_as_decorator_on_coroutine():
 
 
 def test_use_as_decorator_on_generator():
-    original_http_connetion = httplib.HTTPConnection
+    original_http_connection = httplib.HTTPConnection
 
     @Cassette.use(inject=True)
     def test_function(cassette):
         assert httplib.HTTPConnection.cassette is cassette
-        assert httplib.HTTPConnection is not original_http_connetion
+        assert httplib.HTTPConnection is not original_http_connection
         yield 1
         assert httplib.HTTPConnection.cassette is cassette
-        assert httplib.HTTPConnection is not original_http_connetion
+        assert httplib.HTTPConnection is not original_http_connection
         yield 2
 
     assert list(test_function()) == [1, 2]

tests/unit/test_filters.py

@@ -1,18 +1,19 @@
-from io import BytesIO
-from vcr.filters import (
-    remove_headers,
-    replace_headers,
-    remove_query_parameters,
-    replace_query_parameters,
-    remove_post_data_parameters,
-    replace_post_data_parameters,
-    decode_response,
-)
-from vcr.request import Request
 import gzip
 import json
-from unittest import mock
 import zlib
+from io import BytesIO
+from unittest import mock
+
+from vcr.filters import (
+    decode_response,
+    remove_headers,
+    remove_post_data_parameters,
+    remove_query_parameters,
+    replace_headers,
+    replace_post_data_parameters,
+    replace_query_parameters,
+)
+from vcr.request import Request
 
 
 def test_replace_headers():
@@ -220,6 +221,49 @@ def test_remove_all_json_post_data_parameters():
     assert request.body == b"{}"
 
 
+def test_replace_dict_post_data_parameters():
+    # This tests all of:
+    #   1. keeping a parameter
+    #   2. removing a parameter
+    #   3. replacing a parameter
+    #   4. replacing a parameter using a callable
+    #   5. removing a parameter using a callable
+    #   6. replacing a parameter that doesn't exist
+    body = {"one": "keep", "two": "lose", "three": "change", "four": "shout", "five": "whisper"}
+    request = Request("POST", "http://google.com", body, {})
+    request.headers["Content-Type"] = "application/x-www-form-urlencoded"
+    replace_post_data_parameters(
+        request,
+        [
+            ("two", None),
+            ("three", "tada"),
+            ("four", lambda key, value, request: value.upper()),
+            ("five", lambda key, value, request: None),
+            ("six", "doesntexist"),
+        ],
+    )
+    expected_data = {"one": "keep", "three": "tada", "four": "SHOUT"}
+    assert request.body == expected_data
+
+
+def test_remove_dict_post_data_parameters():
+    # Test the backward-compatible API wrapper.
+    body = {"id": "secret", "foo": "bar", "baz": "qux"}
+    request = Request("POST", "http://google.com", body, {})
+    request.headers["Content-Type"] = "application/x-www-form-urlencoded"
+    remove_post_data_parameters(request, ["id"])
+    expected_data = {"foo": "bar", "baz": "qux"}
+    assert request.body == expected_data
+
+
+def test_remove_all_dict_post_data_parameters():
+    body = {"id": "secret", "foo": "bar"}
+    request = Request("POST", "http://google.com", body, {})
+    request.headers["Content-Type"] = "application/x-www-form-urlencoded"
+    replace_post_data_parameters(request, [("id", None), ("foo", None)])
+    assert request.body == {}
+
+
 def test_decode_response_uncompressed():
     recorded_response = {
         "status": {"message": "OK", "code": 200},

tests/unit/test_json_serializer.py

@@ -1,6 +1,7 @@
 import pytest
-from vcr.serializers.jsonserializer import serialize
+
 from vcr.request import Request
+from vcr.serializers.jsonserializer import serialize
 
 
 def test_serialize_binary():

tests/unit/test_matchers.py

@@ -3,8 +3,7 @@ from unittest import mock
 
 import pytest
 
-from vcr import matchers
-from vcr import request
+from vcr import matchers, request
 
 # the dict contains requests with corresponding to its key difference
 # with 'base' request.

tests/unit/test_migration.py

@@ -1,6 +1,7 @@
 import filecmp
 import json
 import shutil
+
 import yaml
 
 import vcr.migration
@@ -44,4 +45,4 @@ def test_try_migrate_with_invalid_or_new_cassettes(tmpdir):
     for file_path in files:
         shutil.copy(file_path, cassette)
         assert not vcr.migration.try_migrate(cassette)
-        assert filecmp.cmp(cassette, file_path)  # shold not change file
+        assert filecmp.cmp(cassette, file_path)  # should not change file

tests/unit/test_request.py

@@ -1,6 +1,6 @@
 import pytest
 
-from vcr.request import Request, HeadersDict
+from vcr.request import HeadersDict, Request
 
 
 @pytest.mark.parametrize(
@@ -60,7 +60,6 @@ def test_uri(method, uri):
 
 
 def test_HeadersDict():
-
     # Simple test of CaseInsensitiveDict
     h = HeadersDict()
     assert h == {}

tests/unit/test_serialize.py

@@ -5,7 +5,7 @@ import pytest
 
 from vcr.request import Request
 from vcr.serialize import deserialize, serialize
-from vcr.serializers import yamlserializer, jsonserializer, compat
+from vcr.serializers import compat, jsonserializer, yamlserializer
 
 
 def test_deserialize_old_yaml_cassette():

tests/unit/test_stubs.py

@@ -1,12 +1,12 @@
 from unittest import mock
 
 from vcr import mode
-from vcr.stubs import VCRHTTPSConnection
 from vcr.cassette import Cassette
+from vcr.stubs import VCRHTTPSConnection
 
 
 class TestVCRConnection:
-    def test_setting_of_attributes_get_propogated_to_real_connection(self):
+    def test_setting_of_attributes_get_propagated_to_real_connection(self):
         vcr_connection = VCRHTTPSConnection("www.examplehost.com")
         vcr_connection.ssl_version = "example_ssl_version"
         assert vcr_connection.real_connection.ssl_version == "example_ssl_version"

tests/unit/test_vcr.py

@@ -1,13 +1,14 @@
-from unittest import mock
+import http.client as httplib
 import os
+from pathlib import Path
+from unittest import mock
 
 import pytest
-import http.client as httplib
 
 from vcr import VCR, mode, use_cassette
+from vcr.patch import _HTTPConnection, force_reset
 from vcr.request import Request
 from vcr.stubs import VCRHTTPSConnection
-from vcr.patch import _HTTPConnection, force_reset
 
 
 def test_vcr_use_cassette():
@@ -31,7 +32,7 @@ def test_vcr_use_cassette():
         function()
         assert mock_cassette_load.call_args[1]["record_mode"] == test_vcr.record_mode
 
-        # Ensure that explicitly provided arguments still supercede
+        # Ensure that explicitly provided arguments still supersede
         # those on the vcr.
         new_record_mode = mock.Mock()
 
@@ -40,7 +41,7 @@ def test_vcr_use_cassette():
 
 
 def test_vcr_before_record_request_params():
-    base_path = "http://httpbin.org/"
+    base_path = "http://whatever.test/"
 
     def before_record_cb(request):
         if request.path != "/get":
@@ -95,7 +96,6 @@ def test_vcr_before_record_response_iterable():
 
     # Prevent actually saving the cassette
     with mock.patch("vcr.cassette.FilesystemPersister.save_cassette"):
-
         # Baseline: non-iterable before_record_response should work
         mock_filter = mock.Mock()
         vcr = VCR(before_record_response=mock_filter)
@@ -119,7 +119,6 @@ def test_before_record_response_as_filter():
 
     # Prevent actually saving the cassette
     with mock.patch("vcr.cassette.FilesystemPersister.save_cassette"):
-
         filter_all = mock.Mock(return_value=None)
         vcr = VCR(before_record_response=filter_all)
         with vcr.use_cassette("test") as cassette:
@@ -133,7 +132,6 @@ def test_vcr_path_transformer():
 
     # Prevent actually saving the cassette
     with mock.patch("vcr.cassette.FilesystemPersister.save_cassette"):
-
         # Baseline: path should be unchanged
         vcr = VCR()
         with vcr.use_cassette("test") as cassette:
@@ -226,7 +224,7 @@ def test_with_current_defaults():
 
 
 def test_cassette_library_dir_with_decoration_and_no_explicit_path():
-    library_dir = "/libary_dir"
+    library_dir = "/library_dir"
     vcr = VCR(inject_cassette=True, cassette_library_dir=library_dir)
 
     @vcr.use_cassette()
@@ -237,7 +235,7 @@ def test_cassette_library_dir_with_decoration_and_no_explicit_path():
 
 
 def test_cassette_library_dir_with_decoration_and_explicit_path():
-    library_dir = "/libary_dir"
+    library_dir = "/library_dir"
     vcr = VCR(inject_cassette=True, cassette_library_dir=library_dir)
 
     @vcr.use_cassette(path="custom_name")
@@ -248,7 +246,7 @@ def test_cassette_library_dir_with_decoration_and_explicit_path():
 
 
 def test_cassette_library_dir_with_decoration_and_super_explicit_path():
-    library_dir = "/libary_dir"
+    library_dir = "/library_dir"
     vcr = VCR(inject_cassette=True, cassette_library_dir=library_dir)
 
     @vcr.use_cassette(path=os.path.join(library_dir, "custom_name"))
@@ -259,7 +257,7 @@ def test_cassette_library_dir_with_decoration_and_super_explicit_path():
 
 
 def test_cassette_library_dir_with_path_transformer():
-    library_dir = "/libary_dir"
+    library_dir = "/library_dir"
     vcr = VCR(
         inject_cassette=True, cassette_library_dir=library_dir, path_transformer=lambda path: path + ".json"
     )
@@ -360,3 +358,11 @@ def test_dynamically_added(self):
 
 TestVCRClass.test_dynamically_added = test_dynamically_added
 del test_dynamically_added
+
+
+def test_path_class_as_cassette():
+    path = Path(__file__).parent.parent.joinpath(
+        "integration/cassettes/test_httpx_test_test_behind_proxy.yml"
+    )
+    with use_cassette(path):
+        pass

tests/unit/test_vcr_import.py

@@ -2,7 +2,6 @@ import sys
 
 
 def test_vcr_import_deprecation(recwarn):
-
     if "vcr" in sys.modules:
         # Remove imported module entry if already loaded in another test
         del sys.modules["vcr"]

tox.ini

@@ -3,12 +3,21 @@ skip_missing_interpreters=true
 envlist =
   cov-clean,
   lint,
-  {py35,py36,py37,py38}-{requests,httplib2,urllib3,tornado4,boto3,aiohttp},
-  {py36,py37,py38}-{httpx}
-  {pypy3}-{requests,httplib2,urllib3,tornado4,boto3},
+  {py37,py38,py39,py310,py311}-{requests-urllib3-1,requests-urllib3-2,httplib2,urllib3-1,urllib3-2,tornado4,boto3,aiohttp,httpx},
+  {pypy3}-{requests-urllib3-1,requests-urllib3-2,httplib2,urllib3-1,urllib3-2,tornado4,boto3},
+  {py310}-httpx019,
   cov-report
 
 
+[gh-actions]
+python =
+    3.7: py37
+    3.8: py38
+    3.9: py39
+    3.10: py310, lint
+    3.11: py311
+    pypy-3: pypy3
+
 # Coverage environment tasks: cov-clean and cov-report
 # https://pytest-cov.readthedocs.io/en/latest/tox.html
 [testenv:cov-clean]
@@ -28,12 +37,16 @@ skipsdist = True
 commands =
     black --version
     black --check --diff .
+    isort --version
+    isort . --check --diff
     flake8 --version
-    flake8 --exclude=./docs/conf.py,./.tox/
+    flake8 --exclude=./docs/conf.py,./.tox/,./venv/
     pyflakes ./docs/conf.py
 deps =
   flake8
   black
+  isort
+basepython = python3.10
 
 [testenv:docs]
 # Running sphinx from inside the "docs" directory
@@ -62,30 +75,33 @@ basepython = python3.7
 usedevelop=true
 commands =
     ./runtests.sh --cov=./vcr --cov-branch --cov-report=xml --cov-append {posargs}
+allowlist_externals =
+    ./runtests.sh
 deps =
-    Flask
+    Werkzeug==2.0.3
     pytest
-    pytest-httpbin
+    pytest-httpbin>=1.0.1
     pytest-cov
     PyYAML
     ipaddress
     requests: requests>=2.22.0
     httplib2: httplib2
-    urllib3: urllib3
-    {py35,py36}-tornado4: tornado>=4,<5
-    {py35,py36}-tornado4: pytest-tornado
-    {py35,py36}-tornado4: pycurl
+    urllib3-1: urllib3<2
+    urllib3-2: urllib3<3
     boto3: boto3
-    boto3: urllib3
     aiohttp: aiohttp
     aiohttp: pytest-asyncio
     aiohttp: pytest-aiohttp
     httpx: httpx
-    {py36,py37,py38}-{httpx}: httpx
-    {py36,py37,py38}-{httpx}: pytest-asyncio
+    {py37,py38,py39,py310}-{httpx}: httpx
+    {py37,py38,py39,py310}-{httpx}: pytest-asyncio
+    httpx: httpx>0.19
+    # httpx==0.19 is the latest version that supports allow_redirects, newer versions use follow_redirects
+    httpx019: httpx==0.19
+    {py37,py38,py39,py310}-{httpx}: pytest-asyncio
 depends =
-  lint,{py35,py36,py37,py38,pypy3}-{requests,httplib2,urllib3,tornado4,boto3},{py35,py36,py37,py38}-{aiohttp},{py36,py37,py38}-{httpx}: cov-clean
-  cov-report: lint,{py35,py36,py37,py38,pypy3}-{requests,httplib2,urllib3,tornado4,boto3},{py35,py36,py37,py38}-{aiohttp}
+  lint,{py37,py38,py39,py310,py311,pypy3}-{requests-urllib3-1,requests-urllib3-2,httplib2,urllib3-1,urllib3-2,tornado4,boto3},{py37,py38,py39,py310,py311}-{aiohttp},{py37,py38,py39,py310,py311}-{httpx}: cov-clean
+  cov-report: lint,{py37,py38,py39,py310,py311,pypy3}-{requests-urllib3-1,requests-urllib3-2,httplib2,urllib3-1,urllib3-2,tornado4,boto3},{py37,py38,py39,py310,py311}-{aiohttp}
 passenv =
     AWS_ACCESS_KEY_ID
     AWS_DEFAULT_REGION

vcr/__init__.py

@@ -1,9 +1,10 @@
 import logging
-from .config import VCR
 from logging import NullHandler
+
+from .config import VCR
 from .record_mode import RecordMode as mode  # noqa import is not used in this file
 
-__version__ = "4.1.1"
+__version__ = "4.2.1"
 
 logging.getLogger(__name__).addHandler(NullHandler())
 

vcr/cassette.py

@@ -1,20 +1,20 @@
 import collections
 import contextlib
 import copy
-import sys
 import inspect
 import logging
+import sys
 
 import wrapt
 
-from .errors import UnhandledHTTPRequestError
-from .matchers import requests_match, uri, method, get_matchers_results
-from .patch import CassettePatcherBuilder
-from .serializers import yamlserializer
-from .persisters.filesystem import FilesystemPersister
-from .util import partition_dict
 from ._handle_coroutine import handle_coroutine
+from .errors import UnhandledHTTPRequestError
+from .matchers import get_matchers_results, method, requests_match, uri
+from .patch import CassettePatcherBuilder
+from .persisters.filesystem import FilesystemPersister
 from .record_mode import RecordMode
+from .serializers import yamlserializer
+from .util import partition_dict
 
 try:
     from asyncio import iscoroutinefunction
@@ -45,7 +45,11 @@ class CassetteContextDecorator:
     this class as a context manager in ``__exit__``.
     """
 
-    _non_cassette_arguments = ("path_transformer", "func_path_generator")
+    _non_cassette_arguments = (
+        "path_transformer",
+        "func_path_generator",
+        "record_on_exception",
+    )
 
     @classmethod
     def from_args(cls, cassette_class, **kwargs):
@@ -55,6 +59,7 @@ class CassetteContextDecorator:
         self.cls = cls
         self._args_getter = args_getter
         self.__finish = None
+        self.__cassette = None
 
     def _patch_generator(self, cassette):
         with contextlib.ExitStack() as exit_stack:
@@ -64,9 +69,6 @@ class CassetteContextDecorator:
             log.debug(log_format.format(action="Entering", path=cassette._path))
             yield cassette
             log.debug(log_format.format(action="Exiting", path=cassette._path))
-            # TODO(@IvanMalison): Hmmm. it kind of feels like this should be
-            # somewhere else.
-            cassette._save()
 
     def __enter__(self):
         # This assertion is here to prevent the dangerous behavior
@@ -84,10 +86,22 @@ class CassetteContextDecorator:
         if other_kwargs.get("path_transformer"):
             transformer = other_kwargs["path_transformer"]
             cassette_kwargs["path"] = transformer(cassette_kwargs["path"])
-        self.__finish = self._patch_generator(self.cls.load(**cassette_kwargs))
+        self.__cassette = self.cls.load(**cassette_kwargs)
+        self.__finish = self._patch_generator(self.__cassette)
         return next(self.__finish)
 
-    def __exit__(self, *args):
+    def __exit__(self, *exc_info):
+        exception_was_raised = any(exc_info)
+        record_on_exception = self._args_getter().get("record_on_exception", True)
+        if record_on_exception or not exception_was_raised:
+            self.__cassette._save()
+            self.__cassette = None
+        # Fellow programmer, don't remove this `next`, if `self.__finish` is
+        # not consumed the unpatcher functions accumulated in the `exit_stack`
+        # object created in `_patch_generator` will not be called until
+        # `exit_stack` is not garbage collected.
+        # This works in CPython but not in Pypy, where the unpatchers will not
+        # be called until much later.
         next(self.__finish, None)
         self.__finish = None
 

vcr/config.py

@@ -1,20 +1,19 @@
 import copy
-
-from collections import abc as collections_abc
 import functools
 import inspect
 import os
 import types
+from collections import abc as collections_abc
+from pathlib import Path
 
 import six
 
+from . import filters, matchers
 from .cassette import Cassette
-from .serializers import yamlserializer, jsonserializer
 from .persisters.filesystem import FilesystemPersister
-from .util import compose, auto_decorate
 from .record_mode import RecordMode
-from . import matchers
-from . import filters
+from .serializers import jsonserializer, yamlserializer
+from .util import auto_decorate, compose
 
 
 class VCR:
@@ -50,6 +49,7 @@ class VCR:
         cassette_library_dir=None,
         func_path_generator=None,
         decode_compressed_response=False,
+        record_on_exception=True,
     ):
         self.serializer = serializer
         self.match_on = match_on
@@ -81,6 +81,7 @@ class VCR:
         self.path_transformer = path_transformer
         self.func_path_generator = func_path_generator
         self.decode_compressed_response = decode_compressed_response
+        self.record_on_exception = record_on_exception
         self._custom_patches = tuple(custom_patches)
 
     def _get_serializer(self, serializer_name):
@@ -100,7 +101,7 @@ class VCR:
         return matchers
 
     def use_cassette(self, path=None, **kwargs):
-        if path is not None and not isinstance(path, str):
+        if path is not None and not isinstance(path, (str, Path)):
             function = path
             # Assume this is an attempt to decorate a function
             return self._use_cassette(**kwargs)(function)
@@ -124,6 +125,7 @@ class VCR:
         func_path_generator = kwargs.get("func_path_generator", self.func_path_generator)
         cassette_library_dir = kwargs.get("cassette_library_dir", self.cassette_library_dir)
         additional_matchers = kwargs.get("additional_matchers", ())
+        record_on_exception = kwargs.get("record_on_exception", self.record_on_exception)
 
         if cassette_library_dir:
 
@@ -150,6 +152,7 @@ class VCR:
             "path_transformer": path_transformer,
             "func_path_generator": func_path_generator,
             "allow_playback_repeats": kwargs.get("allow_playback_repeats", False),
+            "record_on_exception": record_on_exception,
         }
         path = kwargs.get("path")
         if path:

vcr/filters.py

@@ -1,8 +1,8 @@
-from io import BytesIO
-from urllib.parse import urlparse, urlencode, urlunparse
 import copy
 import json
 import zlib
+from io import BytesIO
+from urllib.parse import urlencode, urlparse, urlunparse
 
 from .util import CaseInsensitiveDict
 
@@ -84,7 +84,17 @@ def replace_post_data_parameters(request, replacements):
 
     replacements = dict(replacements)
     if request.method == "POST" and not isinstance(request.body, BytesIO):
-        if request.headers.get("Content-Type") == "application/json":
+        if isinstance(request.body, dict):
+            new_body = request.body.copy()
+            for k, rv in replacements.items():
+                if k in new_body:
+                    ov = new_body.pop(k)
+                    if callable(rv):
+                        rv = rv(key=k, value=ov, request=request)
+                    if rv is not None:
+                        new_body[k] = rv
+            request.body = new_body
+        elif request.headers.get("Content-Type") == "application/json":
             json_data = json.loads(request.body.decode("utf-8"))
             for k, rv in replacements.items():
                 if k in json_data:
@@ -140,6 +150,8 @@ def decode_response(response):
         """Returns decompressed body according to encoding using zlib.
         to (de-)compress gzip format, use wbits = zlib.MAX_WBITS | 16
         """
+        if not body:
+            return ""
         if encoding == "gzip":
             return zlib.decompress(body, zlib.MAX_WBITS | 16)
         else:  # encoding == 'deflate'

vcr/matchers.py

@@ -1,9 +1,9 @@
 import json
+import logging
 import urllib
 import xmlrpc.client
-from .util import read_body
-import logging
 
+from .util import read_body
 
 log = logging.getLogger(__name__)
 

vcr/migration.py

@@ -17,11 +17,12 @@ import os
 import shutil
 import sys
 import tempfile
+
 import yaml
 
-from .serializers import yamlserializer, jsonserializer
-from .serialize import serialize
 from . import request
+from .serialize import serialize
+from .serializers import jsonserializer, yamlserializer
 from .stubs.compat import get_httpmessage
 
 # Use the libYAML versions if possible

vcr/patch.py

@@ -1,13 +1,12 @@
 """Utilities for patching in cassettes"""
 import contextlib
 import functools
+import http.client as httplib
 import itertools
+import logging
 from unittest import mock
 
 from .stubs import VCRHTTPConnection, VCRHTTPSConnection
-import http.client as httplib
-
-import logging
 
 log = logging.getLogger(__name__)
 # Save some of the original types for the purposes of unpatching
@@ -16,42 +15,47 @@ _HTTPSConnection = httplib.HTTPSConnection
 
 # Try to save the original types for boto3
 try:
-    from botocore.awsrequest import AWSHTTPSConnection, AWSHTTPConnection
-except ImportError:
+    from botocore.awsrequest import AWSHTTPConnection, AWSHTTPSConnection
+except ImportError as e:
     try:
-        import botocore.vendored.requests.packages.urllib3.connectionpool as cpool
+        import botocore.vendored.requests  # noqa: F401
     except ImportError:  # pragma: no cover
         pass
     else:
-        _Boto3VerifiedHTTPSConnection = cpool.VerifiedHTTPSConnection
-        _cpoolBoto3HTTPConnection = cpool.HTTPConnection
-        _cpoolBoto3HTTPSConnection = cpool.HTTPSConnection
+        raise RuntimeError(
+            "vcrpy >=4.2.2 and botocore <1.11.0 are not compatible"
+            "; please upgrade botocore (or downgrade vcrpy)"
+        ) from e
 else:
     _Boto3VerifiedHTTPSConnection = AWSHTTPSConnection
     _cpoolBoto3HTTPConnection = AWSHTTPConnection
     _cpoolBoto3HTTPSConnection = AWSHTTPSConnection
 
 cpool = None
+conn = None
 # Try to save the original types for urllib3
 try:
+    import urllib3.connection as conn
     import urllib3.connectionpool as cpool
 except ImportError:  # pragma: no cover
     pass
 else:
-    _VerifiedHTTPSConnection = cpool.VerifiedHTTPSConnection
-    _cpoolHTTPConnection = cpool.HTTPConnection
-    _cpoolHTTPSConnection = cpool.HTTPSConnection
+    _VerifiedHTTPSConnection = conn.VerifiedHTTPSConnection
+    _connHTTPConnection = conn.HTTPConnection
+    _connHTTPSConnection = conn.HTTPSConnection
 
 # Try to save the original types for requests
 try:
-    if not cpool:
-        import requests.packages.urllib3.connectionpool as cpool
+    import requests
 except ImportError:  # pragma: no cover
     pass
 else:
-    _VerifiedHTTPSConnection = cpool.VerifiedHTTPSConnection
-    _cpoolHTTPConnection = cpool.HTTPConnection
-    _cpoolHTTPSConnection = cpool.HTTPSConnection
+    if requests.__build__ < 0x021602:
+        raise RuntimeError(
+            "vcrpy >=4.2.2 and requests <2.16.2 are not compatible"
+            "; please upgrade requests (or downgrade vcrpy)"
+        )
+
 
 # Try to save the original types for httplib2
 try:
@@ -196,24 +200,15 @@ class CassettePatcherBuilder:
             from .stubs import requests_stubs
         except ImportError:  # pragma: no cover
             return ()
-        return self._urllib3_patchers(cpool, requests_stubs)
+        return self._urllib3_patchers(cpool, conn, requests_stubs)
 
     @_build_patchers_from_mock_triples_decorator
     def _boto3(self):
-
         try:
             # botocore using awsrequest
             import botocore.awsrequest as cpool
         except ImportError:  # pragma: no cover
-            try:
-                # botocore using vendored requests
-                import botocore.vendored.requests.packages.urllib3.connectionpool as cpool
-            except ImportError:  # pragma: no cover
-                pass
-            else:
-                from .stubs import boto3_stubs
-
-                yield self._urllib3_patchers(cpool, boto3_stubs)
+            pass
         else:
             from .stubs import boto3_stubs
 
@@ -255,12 +250,13 @@ class CassettePatcherBuilder:
 
     def _urllib3(self):
         try:
+            import urllib3.connection as conn
             import urllib3.connectionpool as cpool
         except ImportError:  # pragma: no cover
             return ()
         from .stubs import urllib3_stubs
 
-        return self._urllib3_patchers(cpool, urllib3_stubs)
+        return self._urllib3_patchers(cpool, conn, urllib3_stubs)
 
     @_build_patchers_from_mock_triples_decorator
     def _httplib2(self):
@@ -269,8 +265,7 @@ class CassettePatcherBuilder:
         except ImportError:  # pragma: no cover
             pass
         else:
-            from .stubs.httplib2_stubs import VCRHTTPConnectionWithTimeout
-            from .stubs.httplib2_stubs import VCRHTTPSConnectionWithTimeout
+            from .stubs.httplib2_stubs import VCRHTTPConnectionWithTimeout, VCRHTTPSConnectionWithTimeout
 
             yield cpool, "HTTPConnectionWithTimeout", VCRHTTPConnectionWithTimeout
             yield cpool, "HTTPSConnectionWithTimeout", VCRHTTPSConnectionWithTimeout
@@ -338,7 +333,7 @@ class CassettePatcherBuilder:
             new_sync_client_send = sync_vcr_send(self._cassette, _HttpxSyncClient_send)
             yield httpx.Client, "send", new_sync_client_send
 
-    def _urllib3_patchers(self, cpool, stubs):
+    def _urllib3_patchers(self, cpool, conn, stubs):
         http_connection_remover = ConnectionRemover(
             self._get_cassette_subclass(stubs.VCRRequestsHTTPConnection)
         )
@@ -346,9 +341,9 @@ class CassettePatcherBuilder:
             self._get_cassette_subclass(stubs.VCRRequestsHTTPSConnection)
         )
         mock_triples = (
-            (cpool, "VerifiedHTTPSConnection", stubs.VCRRequestsHTTPSConnection),
-            (cpool, "HTTPConnection", stubs.VCRRequestsHTTPConnection),
-            (cpool, "HTTPSConnection", stubs.VCRRequestsHTTPSConnection),
+            (conn, "VerifiedHTTPSConnection", stubs.VCRRequestsHTTPSConnection),
+            (conn, "HTTPConnection", stubs.VCRRequestsHTTPConnection),
+            (conn, "HTTPSConnection", stubs.VCRRequestsHTTPSConnection),
             (cpool, "is_connection_dropped", mock.Mock(return_value=False)),  # Needed on Windows only
             (cpool.HTTPConnectionPool, "ConnectionCls", stubs.VCRRequestsHTTPConnection),
             (cpool.HTTPSConnectionPool, "ConnectionCls", stubs.VCRRequestsHTTPSConnection),
@@ -418,69 +413,23 @@ def reset_patchers():
     yield mock.patch.object(httplib, "HTTPSConnection", _HTTPSConnection)
 
     try:
-        import requests
-
-        if requests.__build__ < 0x021603:
-            # Avoid double unmock if requests 2.16.3
-            # First, this is pointless, requests.packages.urllib3 *IS* urllib3 (see packages.py)
-            # Second, this is unmocking twice the same classes with different namespaces
-            # and is creating weird issues and bugs:
-            # > AssertionError: assert <class 'urllib3.connection.HTTPConnection'>
-            # >   is <class 'requests.packages.urllib3.connection.HTTPConnection'>
-            # This assert should work!!!
-            # Note that this also means that now, requests.packages is never imported
-            # if requests 2.16.3 or greater is used with VCRPy.
-            import requests.packages.urllib3.connectionpool as cpool
-        else:
-            raise ImportError("Skip requests not vendored anymore")
-    except ImportError:  # pragma: no cover
-        pass
-    else:
-        # unpatch requests v1.x
-        yield mock.patch.object(cpool, "VerifiedHTTPSConnection", _VerifiedHTTPSConnection)
-        yield mock.patch.object(cpool, "HTTPConnection", _cpoolHTTPConnection)
-        # unpatch requests v2.x
-        if hasattr(cpool.HTTPConnectionPool, "ConnectionCls"):
-            yield mock.patch.object(cpool.HTTPConnectionPool, "ConnectionCls", _cpoolHTTPConnection)
-            yield mock.patch.object(cpool.HTTPSConnectionPool, "ConnectionCls", _cpoolHTTPSConnection)
-
-        if hasattr(cpool, "HTTPSConnection"):
-            yield mock.patch.object(cpool, "HTTPSConnection", _cpoolHTTPSConnection)
-
-    try:
+        import urllib3.connection as conn
         import urllib3.connectionpool as cpool
     except ImportError:  # pragma: no cover
         pass
     else:
-        yield mock.patch.object(cpool, "VerifiedHTTPSConnection", _VerifiedHTTPSConnection)
-        yield mock.patch.object(cpool, "HTTPConnection", _cpoolHTTPConnection)
-        yield mock.patch.object(cpool, "HTTPSConnection", _cpoolHTTPSConnection)
+        yield mock.patch.object(conn, "VerifiedHTTPSConnection", _VerifiedHTTPSConnection)
+        yield mock.patch.object(conn, "HTTPConnection", _connHTTPConnection)
+        yield mock.patch.object(conn, "HTTPSConnection", _connHTTPSConnection)
         if hasattr(cpool.HTTPConnectionPool, "ConnectionCls"):
-            yield mock.patch.object(cpool.HTTPConnectionPool, "ConnectionCls", _cpoolHTTPConnection)
-            yield mock.patch.object(cpool.HTTPSConnectionPool, "ConnectionCls", _cpoolHTTPSConnection)
+            yield mock.patch.object(cpool.HTTPConnectionPool, "ConnectionCls", _connHTTPConnection)
+            yield mock.patch.object(cpool.HTTPSConnectionPool, "ConnectionCls", _connHTTPSConnection)
 
     try:
         # unpatch botocore with awsrequest
         import botocore.awsrequest as cpool
     except ImportError:  # pragma: no cover
-        try:
-            # unpatch botocore with vendored requests
-            import botocore.vendored.requests.packages.urllib3.connectionpool as cpool
-        except ImportError:  # pragma: no cover
-            pass
-        else:
-            # unpatch requests v1.x
-            yield mock.patch.object(cpool, "VerifiedHTTPSConnection", _Boto3VerifiedHTTPSConnection)
-            yield mock.patch.object(cpool, "HTTPConnection", _cpoolBoto3HTTPConnection)
-            # unpatch requests v2.x
-            if hasattr(cpool.HTTPConnectionPool, "ConnectionCls"):
-                yield mock.patch.object(cpool.HTTPConnectionPool, "ConnectionCls", _cpoolBoto3HTTPConnection)
-                yield mock.patch.object(
-                    cpool.HTTPSConnectionPool, "ConnectionCls", _cpoolBoto3HTTPSConnection
-                )
-
-            if hasattr(cpool, "HTTPSConnection"):
-                yield mock.patch.object(cpool, "HTTPSConnection", _cpoolBoto3HTTPSConnection)
+        pass
     else:
         if hasattr(cpool.AWSHTTPConnectionPool, "ConnectionCls"):
             yield mock.patch.object(cpool.AWSHTTPConnectionPool, "ConnectionCls", _cpoolBoto3HTTPConnection)

vcr/persisters/filesystem.py

@@ -1,25 +1,32 @@
 # .. _persister_example:
 
-import os
-from ..serialize import serialize, deserialize
+from pathlib import Path
+
+from ..serialize import deserialize, serialize
 
 
 class FilesystemPersister:
     @classmethod
     def load_cassette(cls, cassette_path, serializer):
-        try:
-            with open(cassette_path) as f:
-                cassette_content = f.read()
-        except OSError:
+        cassette_path = Path(cassette_path)  # if cassette path is already Path this is no operation
+        if not cassette_path.is_file():
             raise ValueError("Cassette not found.")
-        cassette = deserialize(cassette_content, serializer)
-        return cassette
+        try:
+            with cassette_path.open() as f:
+                data = f.read()
+        except UnicodeEncodeError as err:
+            raise ValueError("Can't read Cassette, Encoding is broken") from err
+
+        return deserialize(data, serializer)
 
     @staticmethod
     def save_cassette(cassette_path, cassette_dict, serializer):
         data = serialize(cassette_dict, serializer)
-        dirname, filename = os.path.split(cassette_path)
-        if dirname and not os.path.exists(dirname):
-            os.makedirs(dirname)
-        with open(cassette_path, "w") as f:
+        cassette_path = Path(cassette_path)  # if cassette path is already Path this is no operation
+
+        cassette_folder = cassette_path.parent
+        if not cassette_folder.exists():
+            cassette_folder.mkdir(parents=True)
+
+        with cassette_path.open("w") as f:
             f.write(data)

vcr/record_mode.py

@@ -3,7 +3,7 @@ from enum import Enum
 
 class RecordMode(str, Enum):
     """
-    Configues when VCR will record to the cassette.
+    Configures when VCR will record to the cassette.
 
     Can be declared by either using the enumerated value (`vcr.mode.ONCE`)
     or by simply using the defined string (`once`).

vcr/request.py

@@ -1,8 +1,9 @@
+import logging
 import warnings
 from io import BytesIO
-from urllib.parse import urlparse, parse_qsl
+from urllib.parse import parse_qsl, urlparse
+
 from .util import CaseInsensitiveDict
-import logging
 
 log = logging.getLogger(__name__)
 

vcr/serialize.py

@@ -1,7 +1,8 @@
-from vcr.serializers import compat
-from vcr.request import Request
 import yaml
 
+from vcr.request import Request
+from vcr.serializers import compat
+
 # version 1 cassettes started with VCR 1.0.x.
 # Before 1.0.x, there was no versioning.
 CASSETTE_FORMAT_VERSION = 1

vcr/serializers/yamlserializer.py

@@ -2,9 +2,10 @@ import yaml
 
 # Use the libYAML versions if possible
 try:
-    from yaml import CLoader as Loader, CDumper as Dumper
+    from yaml import CDumper as Dumper
+    from yaml import CLoader as Loader
 except ImportError:
-    from yaml import Loader, Dumper
+    from yaml import Dumper, Loader
 
 
 def deserialize(cassette_string):

vcr/stubs/__init__.py

@@ -1,14 +1,12 @@
 """Stubs for patching HTTP and HTTPS requests"""
 
 import logging
-from http.client import (
-    HTTPConnection,
-    HTTPSConnection,
-    HTTPResponse,
-)
+from http.client import HTTPConnection, HTTPResponse, HTTPSConnection
 from io import BytesIO
-from vcr.request import Request
+
 from vcr.errors import CannotOverwriteExistingCassetteException
+from vcr.request import Request
+
 from . import compat
 
 log = logging.getLogger(__name__)
@@ -49,8 +47,9 @@ def parse_headers(header_list):
 
 
 def serialize_headers(response):
+    headers = response.headers if response.msg is None else response.msg
     out = {}
-    for key, values in compat.get_headers(response.msg):
+    for key, values in compat.get_headers(headers):
         out.setdefault(key, [])
         out[key].extend(values)
     return out
@@ -69,6 +68,7 @@ class VCRHTTPResponse(HTTPResponse):
         self.version = None
         self._content = BytesIO(self.recorded_response["body"]["string"])
         self._closed = False
+        self._original_response = self  # for requests.session.Session cookie extraction
 
         headers = self.recorded_response["headers"]
         # Since we are loading a response that has already been serialized, our
@@ -87,7 +87,7 @@ class VCRHTTPResponse(HTTPResponse):
     def closed(self):
         # in python3, I can't change the value of self.closed.  So I'
         # twiddling self._closed and using this property to shadow the real
-        # self.closed from the superclas
+        # self.closed from the superclass
         return self._closed
 
     def read(self, *args, **kwargs):
@@ -145,6 +145,28 @@ class VCRHTTPResponse(HTTPResponse):
     def readable(self):
         return self._content.readable()
 
+    @property
+    def length_remaining(self):
+        return self._content.getbuffer().nbytes - self._content.tell()
+
+    def get_redirect_location(self):
+        """
+        Returns (a) redirect location string if we got a redirect
+        status code and valid location, (b) None if redirect status and
+        no location, (c) False if not a redirect status code.
+        See https://urllib3.readthedocs.io/en/stable/reference/urllib3.response.html .
+        """
+        if not (300 <= self.status <= 399):
+            return False
+        return self.getheader("Location")
+
+    @property
+    def data(self):
+        return self._content.getbuffer().tobytes()
+
+    def drain_conn(self):
+        pass
+
 
 class VCRConnection:
     # A reference to the cassette that's currently being patched in
@@ -250,12 +272,13 @@ class VCRConnection:
 
             # get the response
             response = self.real_connection.getresponse()
+            response_data = response.data if hasattr(response, "data") else response.read()
 
             # put the response into the cassette
             response = {
                 "status": {"code": response.status, "message": response.reason},
                 "headers": serialize_headers(response),
-                "body": {"string": response.read()},
+                "body": {"string": response_data},
             }
             self.cassette.append(self._vcr_request, response)
         return VCRHTTPResponse(response)
@@ -314,7 +337,7 @@ class VCRConnection:
     def __setattr__(self, name, value):
         """
         We need to define this because any attributes that are set on the
-        VCRConnection need to be propogated to the real connection.
+        VCRConnection need to be propagated to the real connection.
 
         For example, urllib3 will set certain attributes on the connection,
         such as 'ssl_version'. These attributes need to get set on the real

vcr/stubs/aiohttp_stubs.py

@@ -1,14 +1,14 @@
 """Stubs for aiohttp HTTP clients"""
 import asyncio
 import functools
-import logging
 import json
-
-from aiohttp import ClientConnectionError, ClientResponse, RequestInfo, streams
-from aiohttp import hdrs, CookieJar
+import logging
 from http.cookies import CookieError, Morsel, SimpleCookie
+from typing import Mapping, Union
+
+from aiohttp import ClientConnectionError, ClientResponse, CookieJar, RequestInfo, hdrs, streams
 from aiohttp.helpers import strip_auth_from_url
-from multidict import CIMultiDict, CIMultiDictProxy
+from multidict import CIMultiDict, CIMultiDictProxy, MultiDict
 from yarl import URL
 
 from vcr.errors import CannotOverwriteExistingCassetteException
@@ -116,14 +116,15 @@ def _deserialize_headers(headers):
     return CIMultiDictProxy(deserialized_headers)
 
 
-def play_responses(cassette, vcr_request):
+def play_responses(cassette, vcr_request, kwargs):
     history = []
+    allow_redirects = kwargs.get("allow_redirects", True)
     vcr_response = cassette.play_response(vcr_request)
     response = build_response(vcr_request, vcr_response, history)
 
     # If we're following redirects, continue playing until we reach
     # our final destination.
-    while 300 <= response.status <= 399:
+    while allow_redirects and 300 <= response.status <= 399:
         if "location" not in response.headers:
             break
 
@@ -175,14 +176,14 @@ async def record_responses(cassette, vcr_request, response):
     to the final destination.
     """
 
-    for past_response in response.history:
+    for i, past_response in enumerate(response.history):
         aiohttp_request = past_response.request_info
 
-        # No data because it's following a redirect.
         past_request = Request(
             aiohttp_request.method,
             str(aiohttp_request.url),
-            None,
+            # Record body of first request, rest are following a redirect.
+            None if i else vcr_request.body,
             _serialize_headers(aiohttp_request.headers),
         )
         await record_response(cassette, past_request, past_response)
@@ -228,6 +229,16 @@ def _build_cookie_header(session, cookies, cookie_header, url):
     return c.output(header="", sep=";").strip()
 
 
+def _build_url_with_params(url_str: str, params: Mapping[str, Union[str, int, float]]) -> URL:
+    # This code is basically a copy&paste of aiohttp.
+    # https://github.com/aio-libs/aiohttp/blob/master/aiohttp/client_reqrep.py#L225
+    url = URL(url_str)
+    q = MultiDict(url.query)
+    url2 = url.with_query(params)
+    q.extend(url2.query)
+    return url.with_query(q)
+
+
 def vcr_request(cassette, real_request):
     @functools.wraps(real_request)
     async def new_request(self, method, url, **kwargs):
@@ -241,12 +252,7 @@ def vcr_request(cassette, real_request):
         if auth is not None:
             headers["AUTHORIZATION"] = auth.encode()
 
-        request_url = URL(url)
-        if params:
-            for k, v in params.items():
-                params[k] = str(v)
-            request_url = URL(url).with_query(params)
-
+        request_url = URL(url) if not params else _build_url_with_params(url, params)
         c_header = headers.pop(hdrs.COOKIE, None)
         cookie_header = _build_cookie_header(self, cookies, c_header, request_url)
         if cookie_header:
@@ -256,7 +262,7 @@ def vcr_request(cassette, real_request):
 
         if cassette.can_play_response_for(vcr_request):
             log.info("Playing response for {} from cassette".format(vcr_request))
-            response = play_responses(cassette, vcr_request)
+            response = play_responses(cassette, vcr_request, kwargs)
             for redirect in response.history:
                 self._cookie_jar.update_cookies(redirect.cookies, redirect.url)
             self._cookie_jar.update_cookies(response.cookies, response.url)

vcr/stubs/boto3_stubs.py

@@ -1,17 +1,6 @@
 """Stubs for boto3"""
-try:
-    # boto using awsrequest
-    from botocore.awsrequest import AWSHTTPConnection as HTTPConnection
-    from botocore.awsrequest import AWSHTTPSConnection as VerifiedHTTPSConnection
-
-except ImportError:  # pragma: nocover
-    # boto using vendored requests
-    # urllib3 defines its own HTTPConnection classes, which boto3 goes ahead and assumes
-    # you're using.  It includes some polyfills for newer features missing in older pythons.
-    try:
-        from urllib3.connectionpool import HTTPConnection, VerifiedHTTPSConnection
-    except ImportError:  # pragma: nocover
-        from requests.packages.urllib3.connectionpool import HTTPConnection, VerifiedHTTPSConnection
+from botocore.awsrequest import AWSHTTPConnection as HTTPConnection
+from botocore.awsrequest import AWSHTTPSConnection as VerifiedHTTPSConnection
 
 from ..stubs import VCRHTTPConnection, VCRHTTPSConnection
 

vcr/stubs/boto_stubs.py

@@ -1,6 +1,7 @@
 """Stubs for boto"""
 
 from boto.https_connection import CertValidatingHTTPSConnection
+
 from ..stubs import VCRHTTPSConnection
 
 

vcr/stubs/compat.py

@@ -1,6 +1,5 @@
-from io import BytesIO
 import http.client
-
+from io import BytesIO
 
 """
 The python3 http.client api moved some stuff around, so this is an abstraction
@@ -13,7 +12,7 @@ def get_header(message, name):
 
 
 def get_header_items(message):
-    for (key, values) in get_headers(message):
+    for key, values in get_headers(message):
         for value in values:
             yield key, value
 

vcr/stubs/httplib2_stubs.py

@@ -1,6 +1,7 @@
 """Stubs for httplib2"""
 
 from httplib2 import HTTPConnectionWithTimeout, HTTPSConnectionWithTimeout
+
 from ..stubs import VCRHTTPConnection, VCRHTTPSConnection
 
 
@@ -27,7 +28,6 @@ class VCRHTTPSConnectionWithTimeout(VCRHTTPSConnection, HTTPSConnectionWithTimeo
     _baseclass = HTTPSConnectionWithTimeout
 
     def __init__(self, *args, **kwargs):
-
         # Delete the keyword arguments that HTTPSConnection would not recognize
         safe_keys = {
             "host",

vcr/stubs/httpx_stubs.py

@@ -1,35 +1,44 @@
 import functools
+import inspect
 import logging
-from unittest.mock import patch, MagicMock
+from unittest.mock import MagicMock, patch
 
 import httpx
-from vcr.request import Request as VcrRequest
+
 from vcr.errors import CannotOverwriteExistingCassetteException
+from vcr.request import Request as VcrRequest
+
+_httpx_signature = inspect.signature(httpx.Client.request)
+
+try:
+    HTTPX_REDIRECT_PARAM = _httpx_signature.parameters["follow_redirects"]
+except KeyError:
+    HTTPX_REDIRECT_PARAM = _httpx_signature.parameters["allow_redirects"]
 
 
 _logger = logging.getLogger(__name__)
 
 
-def _transform_headers(httpx_reponse):
+def _transform_headers(httpx_response):
     """
     Some headers can appear multiple times, like "Set-Cookie".
     Therefore transform to every header key to list of values.
     """
 
     out = {}
-    for key, var in httpx_reponse.headers.raw:
+    for key, var in httpx_response.headers.raw:
         decoded_key = key.decode("utf-8")
         out.setdefault(decoded_key, [])
         out[decoded_key].append(var.decode("utf-8"))
     return out
 
 
-def _to_serialized_response(httpx_reponse):
+def _to_serialized_response(httpx_response):
     return {
-        "status_code": httpx_reponse.status_code,
-        "http_version": httpx_reponse.http_version,
-        "headers": _transform_headers(httpx_reponse),
-        "content": httpx_reponse.content.decode("utf-8", "ignore"),
+        "status_code": httpx_response.status_code,
+        "http_version": httpx_response.http_version,
+        "headers": _transform_headers(httpx_response),
+        "content": httpx_response.content.decode("utf-8", "ignore"),
     }
 
 
@@ -98,7 +107,11 @@ def _record_responses(cassette, vcr_request, real_response):
 
 def _play_responses(cassette, request, vcr_request, client, kwargs):
     history = []
-    allow_redirects = kwargs.get("allow_redirects", True)
+
+    allow_redirects = kwargs.get(
+        HTTPX_REDIRECT_PARAM.name,
+        HTTPX_REDIRECT_PARAM.default,
+    )
     vcr_response = cassette.play_response(vcr_request)
     response = _from_serialized_response(request, vcr_response)
 

vcr/stubs/requests_stubs.py

@@ -1,9 +1,6 @@
 """Stubs for requests"""
 
-try:
-    from urllib3.connectionpool import HTTPConnection, VerifiedHTTPSConnection
-except ImportError:
-    from requests.packages.urllib3.connectionpool import HTTPConnection, VerifiedHTTPSConnection
+from urllib3.connection import HTTPConnection, VerifiedHTTPSConnection
 
 from ..stubs import VCRHTTPConnection, VCRHTTPSConnection
 

vcr/stubs/urllib3_stubs.py

@@ -1,6 +1,7 @@
 """Stubs for urllib3"""
 
-from urllib3.connectionpool import HTTPConnection, VerifiedHTTPSConnection
+from urllib3.connection import HTTPConnection, VerifiedHTTPSConnection
+
 from ..stubs import VCRHTTPConnection, VCRHTTPSConnection
 
 # urllib3 defines its own HTTPConnection classes. It includes some polyfills